py3 venv creation fails when a virtualenv is active

[rpkilby]

If a user attempts to create a python 3 "venv" while working under an active "virtualenv", the resulting virtual environment will not be properly created.

As you can see below, the venv and virtualenv environments are both created with python 3's "venv" module. However, the latter was created under an active outer virtualenv, and is missing a number of binaries as well as the contents of its site-packages directory. python -m pip does function though.

Setup:

[vagrant@vagrant-arch ~]$ python3.6 -m venv venv
[vagrant@vagrant-arch ~]$ virtualenv -p python3.6 outer
[vagrant@vagrant-arch ~]$ source outer/bin/activate
(outer) [vagrant@vagrant-arch ~]$ python -m venv virtualenv

bin contents:

[vagrant@vagrant-arch ~]$ ls venv/bin/
activate  activate.csh  activate.fish  easy_install  easy_install-3.6  pip  pip3  pip3.6  python  python3  python3.6

[vagrant@vagrant-arch ~]$ ls virtualenv/bin/
activate  activate.csh  activate.fish  python  python3

site-packages contents:

[vagrant@vagrant-arch ~]$ ls venv/lib/python3.6/site-packages/
__pycache__  easy_install.py  pip  pip-9.0.1.dist-info  pkg_resources  setuptools  setuptools-28.8.0.dist-info

[vagrant@vagrant-arch ~]$ ls virtualenv/lib/python3.6/site-packages/
[vagrant@vagrant-arch ~]$ 

[rpkilby]

It turns out that python -m pip "works", however it installs packages into the original virtualenv.

(outer) [vagrant@vagrant-arch ~]$ deactivate 
[vagrant@vagrant-arch ~]$ source virtualenv/bin/activate
(virtualenv) [vagrant@vagrant-arch ~]$ python -m pip install tox

Is the active venv, should contain tox

(virtualenv) [vagrant@vagrant-arch ~]$ ls virtualenv/lib/python3.6/site-packages/
(virtualenv) [vagrant@vagrant-arch ~]$ ls virtualenv/bin/
activate  activate.csh  activate.fish  python  python3

But tox is installed to to the original virtualenv

(virtualenv) [vagrant@vagrant-arch ~]$ ls outer/lib/python3.6/site-packages/
__pycache__      pip                  pkg_resources  pluggy-0.5.2.dist-info  py-1.4.34.dist-info  setuptools-36.6.0.dist-info  six.py  tox-2.9.1.dist-info          virtualenv.py       wheel
easy_install.py  pip-9.0.1.dist-info  pluggy         py                      setuptools           six-1.11.0.dist-info         tox     virtualenv-15.1.0.dist-info  virtualenv_support  wheel-0.30.0.dist-info
(virtualenv) [vagrant@vagrant-arch ~]$ ls outer/bin 
activate  activate.csh  activate.fish  activate_this.py  easy_install  easy_install-3.6  pip  pip3  pip3.6  python  python-config  python3  python3.6  tox  tox-quickstart  virtualenv  wheel

I've tried creating a similar venv with the --copies option, but it had no effect.

[asottile]

Hitting this here as well: pre-commit/pre-commit#755

I suspect a fix is needed from the venv side to do the .real_prefix dance. I'm going to investigate that approach as a workaround for pre-commit's sake

[asottile]

This appears to be the bpo issue: https://bugs.python.org/issue30811

[rpkilby]

I suspect a fix is needed from the venv side to do the .real_prefix dance.

It's not clear to me who should be responsible for the fix. Should venv detect that it's being executed from within a virtualenv, or should virtualenv create an environment that's compatible with venv? On the one hand, virtualenv predates venv. On the other, venv is part of the standard library.

Regardless, here's the workaround that I currently have in tox-venv:

https://github.com/tox-dev/tox-venv/blob/58401663fda66dfba4f344553525c73d57432d5e/src/tox_venv/hooks.py#L10-L49

The executable path detection could be improved, as it currently just looks for python3. eg, you may want python3.6, but python3 points to python3.4.

[asottile]

I opted for a similar workaround: https://github.com/pre-commit/pre-commit/blob/805a2921ad0d34698433972c6fcb1a6dca47191d/pre_commit/languages/python_venv.py#L13-L39

(main difference is pre-commit already does some normalization so I use bin/$(basename exe) instead of just bin/python3)

[gaborbernat]

Eh, yeah overall I feel like we should be venv compatible.