detox が依存性の解決に失敗してたまにコケる話

書こう、書こうとおもってさぼってた話を書きます。

複数の Python バージョンをサポートするライブラリを作っていると、tox があるとテストがとても楽になります。
ただし、Python のバージョンによって依存するライブラリが変化する場合は、
setup.py と tox.ini の両方にその記述を書く必要があります。

例えば、python2.6 では unittest2 を使い、python2.7 以降では unittest2 を使わない場合は次のようになります。

# setup.py

# -*- coding: utf-8 -*-
test_requires = ['mock']
if sys.version_info < (2, 7):
    test_requires.append('unittest2')

setup(
    name='example',
    extras_require=dict(
        testing=test_requires,
    ),
)

; tox.ini

[tox]
envlist=py26,py27,py32,py33

[testenv]
deps=
    mock

[testenv:py26]
deps=
    mock
    unittest2

python2.6 用の定義を setup.py と tox.ini の両方に書かなくてはならないのでちょっと厄介ですね。

作っているものによっては、この依存パッケージの定義が複雑怪奇になってしまうことがあり、
書き換えるのがめんどうになります。
たとえば、拙作の diff-highlight はとても読みづらいことになっています。

tox で extra_requires を参照してみた

両方に定義を書くのがイヤだったので、tox で extra_require の定義を参照するようにしてみました。

[tox]
envlist=py26,py27,py32,py33

[testenv]
deps=
    mock
commands=
    pip install -e .[testing]

pip の -e オプションで extra_require を指定することができるので、
このやり方であれば tox.ini に同じことを書かずに済むようになります。

実際に tox を実行して試してみると、正しく動きます。

detox に罠があった

うまくいきました、めでたしめでたし、と終わりにしたかったのですがそうも行きませんでした。

detox コマンドで tox を並列実行するようにすると、このやり方は簡単に崩壊します。
なんどか detox を呼ぶと、数回に一回、実行が失敗します。

$ detox
GLOB sdist-make: /Users/tkomiya/work/experimental/tox-test/setup.py
py33 inst-nodeps: /Users/tkomiya/work/experimental/tox-test/.tox/dist/example-0.0.0.zip
py27 inst-nodeps: /Users/tkomiya/work/experimental/tox-test/.tox/dist/example-0.0.0.zip
py32 inst-nodeps: /Users/tkomiya/work/experimental/tox-test/.tox/dist/example-0.0.0.zip
py26 inst-nodeps: /Users/tkomiya/work/experimental/tox-test/.tox/dist/example-0.0.0.zip
py27 runtests: PYTHONHASHSEED='248044164'
py27 runtests: commands[0] | pip install -e .[testing]
py26 runtests: PYTHONHASHSEED='248044164'
py26 runtests: commands[0] | pip install -e .[testing]
py32 runtests: PYTHONHASHSEED='248044164'
py32 runtests: commands[0] | pip install -e .[testing]
py33 runtests: PYTHONHASHSEED='248044164'
py33 runtests: commands[0] | pip install -e .[testing]
ERROR: invocation failed, logfile: /Users/tkomiya/work/experimental/tox-test/.tox/py32/log/py32-20.log
ERROR: actionid=py32
msg=runtests
cmdargs=[local('/Users/tkomiya/work/experimental/tox-test/.tox/py32/bin/pip'), 'install', '-e', '.[testing]']
env={'DESTINATION': '/var/folders/2s/79pf79d94r9gxhzl4kpztrr00000gp/T/iTerm 1.0.0.20140112 Update', 'PYENV_DIR': '/Users/tkomiya/work/experimental/tox-test', 'LOGNAME': 'tkomiya', 'USER': 'tkomiya', 'PATH': '/Users/tkomiya/work/experimental/tox-test/.tox/py32/bin:/Users/tkomiya/work/experimental/tox-test/.tox/py27/bin:/Users/tkomiya/work/experimental/tox-test/.tox/py33/bin:/Users/tkomiya/.pyenv/versions/2.7.6/bin:/Users/tkomiya/.pyenv/libexec:/Users/tkomiya/.pyenv/plugins/python-build/bin:/Users/tkomiya/.pyenv/plugins/python-virtualenv/bin:/Users/tkomiya/.plenv/shims:/Users/tkomiya/.plenv/bin:/Users/tkomiya/.pyenv/shims:/Users/tkomiya/.pyenv/bin:/Users/tkomiya/.rbenv/shims:/Users/tkomiya/.rbenv/bin:/Users/tkomiya/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin/X11:/usr/games', 'HOME': '/Users/tkomiya', 'TERM_PROGRAM': 'iTerm.app', 'LANG': 'ja_JP.UTF-8', 'TERM': 'xterm-256color', 'SHELL': '/bin/zsh', 'COLORFGBG': '7;0', 'PYENV_SHELL': 'zsh', 'SHLVL': '1', 'SECURITYSESSIONID': '186a4', 'ITERM_SESSION_ID': 'w0t3p0', 'RBENV_SHELL': 'zsh', 'PYENV_VERSION': '2.7.6:2.5.6:2.6.9:3.2.5:3.3.5:3.4.0', 'PYTHONHASHSEED': '248044164', 'EDITOR': 'vi', 'GPG_AGENT_INFO': '/Users/tkomiya/.gnupg/S.gpg-agent:278:1', 'RPROMPT': '%F{green}%~%f', 'PROMPT': '%B%(?..[%?] )%b%n@%U%m%u> ', 'SSH_AUTH_SOCK': '/tmp/launch-EfapyJ/Listeners', 'Apple_PubSub_Socket_Render': '/tmp/launch-viag1O/Render', 'ITERM_PROFILE': 'Default', 'LC_ALL': 'ja_JP.UTF-8', 'TMPDIR': '/var/folders/2s/79pf79d94r9gxhzl4kpztrr00000gp/T/', 'PYENV_HOOK_PATH': ':/Users/tkomiya/.pyenv/pyenv.d:/usr/local/etc/pyenv.d:/etc/pyenv.d:/usr/lib/pyenv/hooks', 'PYENV_ROOT': '/Users/tkomiya/.pyenv', '__CF_USER_TEXT_ENCODING': '0x1F6:1:14', 'PWD': '/Users/tkomiya/work/experimental/tox-test', '__CHECKFIX1436934': '1', 'COMMAND_MODE': 'unix2003'}
Obtaining file:///Users/tkomiya/work/experimental/tox-test
  Running setup.py (path:/Users/tkomiya/work/experimental/tox-test/setup.py) egg_info for package from file:///Users/tkomiya/work/experimental/tox-test

  Installing extra requirements: 'testing'
Requirement already satisfied (use --upgrade to upgrade): mock in ./.tox/py32/lib/python3.2/site-packages (from example==0.0.0)
Downloading/unpacking unittest2 (from example==0.0.0)
  Running setup.py (path:/Users/tkomiya/work/experimental/tox-test/.tox/py32/build/unittest2/setup.py) egg_info for package unittest2
    Traceback (most recent call last):
      File "<string>", line 17, in <module>
      File "/Users/tkomiya/work/experimental/tox-test/.tox/py32/build/unittest2/setup.py", line 12, in <module>
        from unittest2 import __version__ as VERSION
      File "unittest2/__init__.py", line 40, in <module>
        from unittest2.collector import collector
      File "unittest2/collector.py", line 3, in <module>
        from unittest2.loader import defaultTestLoader
      File "unittest2/loader.py", line 92
        except Exception, e:
                        ^
    SyntaxError: invalid syntax
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 17, in <module>

  File "/Users/tkomiya/work/experimental/tox-test/.tox/py32/build/unittest2/setup.py", line 12, in <module>

    from unittest2 import __version__ as VERSION

  File "unittest2/__init__.py", line 40, in <module>

    from unittest2.collector import collector

  File "unittest2/collector.py", line 3, in <module>

    from unittest2.loader import defaultTestLoader

  File "unittest2/loader.py", line 92

    except Exception, e:

                    ^

SyntaxError: invalid syntax

----------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in /Users/tkomiya/work/experimental/tox-test/.tox/py32/build/unittest2
Storing debug log for failure in /Users/tkomiya/.pip/pip.log

ERROR: InvocationError: /Users/tkomiya/work/experimental/tox-test/.tox/py32/bin/pip install -e .[testing] (see /Users/tkomiya/work/experimental/tox-test/.tox/py32/log/py32-20.log)
ERROR: invocation failed, logfile: /Users/tkomiya/work/experimental/tox-test/.tox/py33/log/py33-20.log
ERROR: actionid=py33
msg=runtests
cmdargs=[local('/Users/tkomiya/work/experimental/tox-test/.tox/py33/bin/pip'), 'install', '-e', '.[testing]']
env={'DESTINATION': '/var/folders/2s/79pf79d94r9gxhzl4kpztrr00000gp/T/iTerm 1.0.0.20140112 Update', 'PYENV_DIR': '/Users/tkomiya/work/experimental/tox-test', 'LOGNAME': 'tkomiya', 'USER': 'tkomiya', 'PATH': '/Users/tkomiya/work/experimental/tox-test/.tox/py33/bin:/Users/tkomiya/.pyenv/versions/2.7.6/bin:/Users/tkomiya/.pyenv/libexec:/Users/tkomiya/.pyenv/plugins/python-build/bin:/Users/tkomiya/.pyenv/plugins/python-virtualenv/bin:/Users/tkomiya/.plenv/shims:/Users/tkomiya/.plenv/bin:/Users/tkomiya/.pyenv/shims:/Users/tkomiya/.pyenv/bin:/Users/tkomiya/.rbenv/shims:/Users/tkomiya/.rbenv/bin:/Users/tkomiya/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin/X11:/usr/games', 'HOME': '/Users/tkomiya', 'TERM_PROGRAM': 'iTerm.app', 'LANG': 'ja_JP.UTF-8', 'TERM': 'xterm-256color', 'SHELL': '/bin/zsh', 'COLORFGBG': '7;0', 'PYENV_SHELL': 'zsh', 'SHLVL': '1', 'SECURITYSESSIONID': '186a4', 'ITERM_SESSION_ID': 'w0t3p0', 'RBENV_SHELL': 'zsh', 'PYENV_VERSION': '2.7.6:2.5.6:2.6.9:3.2.5:3.3.5:3.4.0', 'PYTHONHASHSEED': '248044164', 'EDITOR': 'vi', 'GPG_AGENT_INFO': '/Users/tkomiya/.gnupg/S.gpg-agent:278:1', 'RPROMPT': '%F{green}%~%f', 'PROMPT': '%B%(?..[%?] )%b%n@%U%m%u> ', 'SSH_AUTH_SOCK': '/tmp/launch-EfapyJ/Listeners', 'Apple_PubSub_Socket_Render': '/tmp/launch-viag1O/Render', 'ITERM_PROFILE': 'Default', 'LC_ALL': 'ja_JP.UTF-8', 'TMPDIR': '/var/folders/2s/79pf79d94r9gxhzl4kpztrr00000gp/T/', 'PYENV_HOOK_PATH': ':/Users/tkomiya/.pyenv/pyenv.d:/usr/local/etc/pyenv.d:/etc/pyenv.d:/usr/lib/pyenv/hooks', 'PYENV_ROOT': '/Users/tkomiya/.pyenv', '__CF_USER_TEXT_ENCODING': '0x1F6:1:14', 'PWD': '/Users/tkomiya/work/experimental/tox-test', '__CHECKFIX1436934': '1', 'COMMAND_MODE': 'unix2003'}
Obtaining file:///Users/tkomiya/work/experimental/tox-test
  Running setup.py (path:/Users/tkomiya/work/experimental/tox-test/setup.py) egg_info for package from file:///Users/tkomiya/work/experimental/tox-test

  Installing extra requirements: 'testing'
Requirement already satisfied (use --upgrade to upgrade): mock in ./.tox/py33/lib/python3.3/site-packages (from example==0.0.0)
Downloading/unpacking unittest2 (from example==0.0.0)
  Running setup.py (path:/Users/tkomiya/work/experimental/tox-test/.tox/py33/build/unittest2/setup.py) egg_info for package unittest2
    Traceback (most recent call last):
      File "<string>", line 17, in <module>
      File "/Users/tkomiya/work/experimental/tox-test/.tox/py33/build/unittest2/setup.py", line 12, in <module>
        from unittest2 import __version__ as VERSION
      File "./unittest2/__init__.py", line 40, in <module>
        from unittest2.collector import collector
      File "./unittest2/collector.py", line 3, in <module>
        from unittest2.loader import defaultTestLoader
      File "./unittest2/loader.py", line 92
        except Exception, e:
                        ^
    SyntaxError: invalid syntax
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 17, in <module>

  File "/Users/tkomiya/work/experimental/tox-test/.tox/py33/build/unittest2/setup.py", line 12, in <module>

    from unittest2 import __version__ as VERSION

  File "./unittest2/__init__.py", line 40, in <module>

    from unittest2.collector import collector

  File "./unittest2/collector.py", line 3, in <module>

    from unittest2.loader import defaultTestLoader

  File "./unittest2/loader.py", line 92

    except Exception, e:

                    ^

SyntaxError: invalid syntax

----------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in /Users/tkomiya/work/experimental/tox-test/.tox/py33/build/unittest2
Storing debug log for failure in /Users/tkomiya/.pip/pip.log

ERROR: InvocationError: /Users/tkomiya/work/experimental/tox-test/.tox/py33/bin/pip install -e .[testing] (see /Users/tkomiya/work/experimental/tox-test/.tox/py33/log/py33-20.log)
______________________________________________________________ summary _______________________________________________________________
  py26: commands succeeded
  py27: commands succeeded
ERROR:   py32: commands failed
ERROR:   py33: commands failed

ログを読むと、本来 python3.x では依存していないはずのパッケージ(unittest2)を python3.x にインストールしようとしているために失敗しています。

調べていくと、detox によって "pip install -e .[testing]" が並列実行される際に、
*.egg-info/requires.txt が上書きしあって、実行するタイミングや順序によって、
本来「python2.6 で適用されるべき依存関係」が「python3.x に適用されてしまう」という現象が起きていることが分かりました。

まとめ

ダブっててつらいけど setup.cfg と tox.ini に依存関係を書くことにしました。