{"action":"created","issue":{"url":"https://api.github.com/repos/pypa/virtualenv/issues/697","labels_url":"https://api.github.com/repos/pypa/virtualenv/issues/697/labels{/name}","comments_url":"https://api.github.com/repos/pypa/virtualenv/issues/697/comments","events_url":"https://api.github.com/repos/pypa/virtualenv/issues/697/events","html_url":"https://github.com/pypa/virtualenv/pull/697","id":53358848,"number":697,"title":"Rewrite virtualenv","user":{"login":"dstufft","id":145979,"avatar_url":"https://avatars.githubusercontent.com/u/145979?v=3","gravatar_id":"","url":"https://api.github.com/users/dstufft","html_url":"https://github.com/dstufft","followers_url":"https://api.github.com/users/dstufft/followers","following_url":"https://api.github.com/users/dstufft/following{/other_user}","gists_url":"https://api.github.com/users/dstufft/gists{/gist_id}","starred_url":"https://api.github.com/users/dstufft/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dstufft/subscriptions","organizations_url":"https://api.github.com/users/dstufft/orgs","repos_url":"https://api.github.com/users/dstufft/repos","events_url":"https://api.github.com/users/dstufft/events{/privacy}","received_events_url":"https://api.github.com/users/dstufft/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":44,"created_at":"2015-01-05T02:34:39Z","updated_at":"2015-03-04T20:04:56Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/pypa/virtualenv/pulls/697","html_url":"https://github.com/pypa/virtualenv/pull/697","diff_url":"https://github.com/pypa/virtualenv/pull/697.diff","patch_url":"https://github.com/pypa/virtualenv/pull/697.patch"},"body":"*This Pull Request was previously #691 however I've moved the rewrite branch to pypa/virtualenv to make it more official.*\r\n\r\nThis is a major change to the virtualenv code base, and as thus I want to get some sign off and opinions from the other @pypa/virtualenv-developers. I don't consider this idea a forgone conclusion without input from the other core developers.\r\n\r\nThis rewrites the entirety of virtualenv, this is a \"dangerous\" thing to do which risks breaking things that have been working and generally complete rewrites are considered a bad idea. However while attempting to fix errors like #694, #671, #666, #650, #672, #687, #674, #648, etc I ran into situation where I believe it's not feasible to fix all of the issues with virtualenv given the current methods it employs. In particular I believe that the way ``-p`` is implemented makes it extremely fragile if not impossible to work accurately in all situations.\r\n\r\nIn particular I believe there are a number of issues, most of which stem from architectural issues with virtualenv itself. The (incomplete) list of issues are:\r\n\r\n* The ``-p`` functionality to restart the script, which is likely installed into one Python's ``site-packages`` directory, seems to be fundamentally flawed in a way where it's not reasonable to fix it in all situations.\r\n* The previous code relied on a number of hacks in order to implement isolation. This comes from a time when Python had no first class support for isolation and the only way to do it was to implement hacks. This also forces us to maintain a different set of hacks for each target interpreter and sometimes different hacks for different interpreters for different platforms. However since Python 3.3 Python itself has a virtualenv like module called venv. While I do not believe that venv is (or should) be a complete replacement for virtualenv, I do believe that the underlying isolation mechanism is likely to be far more robust given that it comes from the interpreter itself. This also pushes the maintenance burden of implementing the isolation to the people best able to control what the interpreter does.\r\n* The code is extremely hard to read, it is not well factored and over time it's only grown more unwieldy and more gnarly.\r\n* Over time the code has grown more and more hacks to solve specific problems. A lot of these hacks are not documented or commented at all and it's extremely difficult to determine which hacks apply any longer or which are no longer required.\r\n* There is almost a non existent test suite. I have wanted to attempt to write a test suite previously however the way the code is structured it makes it extremely difficult to actually write tests.\r\n* The ``virtualenv.py`` module attempts to run as a single file, however it can't actually do that because it requires the ``virtualenv_support`` directory to exist alongside it. I believe that it's saner and easier to have a single directory ``virtualenv/`` directory which contains everything that needs to exist for an install to work. This also removes giant base64 encoded blobs from needing to be maintained inside of the ``virtualenv.py`` file.\r\n* The current means of isolation relies on installing a custom patched ``site.py`` that we maintain inside of virtualenv. However this ``site.py`` misses features from later versions of Python which means that there are subtle differences in how Python behaves from within a virtual environment and without.\r\n\r\nIn contrast the rewrite is structured in a different way.\r\n\r\nIn order to still support the ``-p`` option without needing to implement the hackish \"restart in target interpreter\" logic, the rewrite instead (for the \"legacy\" isolation method) inspects the target environment for any values it needs. It does this by running a subprocess which will shove all of the required values into a dictionary, and then dump that as json. When the virtualenv script pulls these values out it then loads the json and then has all of the values it needs in order to construct the virtual environment without needing to execute itself with the target interpreter.\r\n\r\nGiven that newer versions of Python has built in isolation the rewrite wants to take advantage of that. It will detect if the target Python has the ``venv`` module, and if they do it will execute a short script (via ``-c``) that will do the actual creation of the virtual environment (without any scripts or tools installed). This will also allow alternative implementations (like PyPy) to implement the isolation themselves so that they can ensure that isolation works correctly in their interpreter without needing to maintain a set of instructions inside of virtualenv for handling their interpreter.\r\n\r\nThe rewrite also attempts to start fresh with none of the additional hacks. This will mean that we'll likely be broken in certain situations and we'll need to determine how the old virtualenv solved that problem and reapply any additional hacks. However this time we can hopefully ensure that we document/comment what those hacks are and why we're doing it. We'll also ideally write a good set of tests for those hacks so that if we ever remove them in the future we'll have a test suite that will tell us if we broke anything by removing them. Another benefit is that since the hacks are (almost?) always related to the legacy isolation mechanism they can all be quarantined to the ``virtualenv.builders.legacy.LegacyBuilder`` class instead of leaking all over the code base.\r\n\r\nThe new code base is also implemented in a much more modular way. Instead of a ball of spaghetti where the implementation of isolation is spread all over the place, the new code uses \"Builder\" classes which handle creating the actual environments. In addition (thanks to @ionelmc) it utilizes a \"flavor\" system which abstracts away the differences between Windows and Posix in the builder code.\r\n\r\nIn addition to the better (imo) layout of the code base, the new code base is also going to mandate that there is 100% unit test coverage at all times. This will help ensure that we continue to work through refactoring and the like in the future.\r\n\r\nNo matter which isolation method is in use, virtualenv itself will install it's own scripts and install it's own tools (pip and setuptools). This is so we can have consistent scripts and tool versions no matter which isolation mechanism is in play. It also makes it easier to get newer versions of the tooling and scripts without requiring people to update their entire Python interpreter. In other words if the venv isolation mechanism is available, we will *only* use it to create the isolated environment and nothing else.\r\n\r\nThe high level overview of what the major differences in capabilities are:\r\n* ``--relocatable`` no longer is supported as it never worked very well and I don't believe it's possible to actually make it work in all cases. It appears the major use case for this is to avoid re-compiling software, for which I believe Wheels are a better solution.\r\n* The virtual environment always contains copies instead of attempting to use symlink. The new ``site.py`` file in the rewrite allows us to copy a lot less files over than previously required. Almost the entire size of an empty virtual environment in the new rewrite is the executable files, which are required to be copied no matter what.\r\n* Support a clean and simple API for using virtualenv instead of the hackish bootstrap script method that the previous versions used to use.\r\n* The deprecated ``--distribute`` option is gone.\r\n* The ``--no-pip`` and ``--no-setuptools`` options now operate independently of each other. In addition there are now ``--setuptools`` and ``--pip`` options which will negate the ``--no-*`` options if they were given previously.\r\n* Instead of ``sys.real_prefix`` we have ``sys.base_prefix`` and ``sys.base_exec_prefix`` in order to match what the venv module does.\r\n* The new code base is tested on Windows using Appveyor.\r\n* The new code drops the ``virtualenv-X.Y`` entry points which didn't work correctly with Wheel anyways and are unneeded since it doesn't really matter what version of Python virtualenv is installed with.\r\n* The new code only uses setuptools in its ``setup.py`` and will always use the script wrappers in that situation.\r\n\r\nThat major things left to do are:\r\n* [ ] Write all of the unit tests (mostly done).\r\n* [ ] Write all of the functional tests.\r\n* [ ] Ensure/fix the legacy isolation mechanism on PyPy (Jython and IronPython too maybe?).\r\n* [ ] Fix virtualenv-in-virtualenv (including for environments created pre-rewrite) so that it correctly uses the \"real\" Python.\r\n* [ ] Update the documentation to match the state of the repository now.\r\n\r\nThere is an open question about how best to handle the transition. If we just merge this and release it then we risk regressing behavior for a lot of people since this is essentially brand new untested code. In the previous Pull Request @ncoghlan suggested releasing it on PyPI as something like virtualenv-rewrite for a period of time and asking people to switch to using that instead. I think that's a good idea and that's how I would plan the migration to go I think.\r\n\r\nSummary of outstanding issues that get fixed/closed:\r\n\r\n* Closes #694 - Cannot create virtualenv after upgrading OS X 10.6.8 Python to 2.7.\r\n* Closes #692 - virtualenv-2.7 no longer exists.\r\n* Closes #671 - virtualenv -p python3.4 fails with ImportError: No module named 'copy_reg'\r\n* Closes #666 - virtualenv 1.11.6 not working with pypy 2.4 and pypy3 2.4\r\n* Closes #659 - Installing a virtualenv doesn't link pyc files.\r\n* Closes #656 - bootstrap -> No module named pip\r\n* Closes #654 - --always-copy flag doesn't copy\r\n* Closes #640 - The executable python does not exist\r\n* Closes #626 - tests not included in tarballs on pypi\r\n* Closes #625 - ImportError for Python3\r\n* Closes #622 - 1.11.6 creates right shebang but wrong file name\r\n* Closes #617 - Skip \"fix_local_scheme\" for \"virtualenv .\"? (symlinks all files)\r\n* Closes #615 - Problem with symlinked paths\r\n* Closes #605 - option always-copy doesn't work with OOTB CentOS 6.5\r\n* Closes #587 - Python2.7 site.py Not Properly Symlinked\r\n* Closes #586 - virtualenv wheel's entry-point script is version specific to the python its built with\r\n* Closes #576 - ImportError: No module named optparse.\r\n* Closes #571 - site.getsitepackages() doesn't exist in virtualenv\r\n* Closes #565 - Breakage in --always-copy - fails with: ImportError: No module named time\r\n* Closes #560 - The tests/ directory is not included in the tarball.\r\n* Closes #558 - Make --relocatable even more relocatable\r\n* Closes #557 - --system-site-packages fails if PIP_REQUIRE_VIRTUALENV=true\r\n* Closes #555 - DeprecationWarning\r\n* Closes #530 - test_always_copy_option() fails\r\n* Closes #500 - add in a test framework similar to pip\r\n* Closes #464 - Why I read « Helper script to rebuild virtualenv.py from virtualenv_support » in comment but the code don't use virtualenv_support folder content ?\r\n* Closes #434 - Missing site.get*() methods on 3.2 and later\r\n* Closes #414 - Review the use of \"embedded binary blobs\" for support scripts\r\n* Closes #405 - need test to confirm rebuild-script.py has been run\r\n* Closes #368 - Consolidate the test suite\r\n* Closes #355 - site.py not compatible with python 2.7\r\n* Closes #340 - Generated bootstrap script's install_pip isn't specific enough\r\n* Closes #319 - AssertionError while creating virtualenv on windows 7\r\n* Closes #286 - Add code comment re: base64-encoded code in virtualenv.py\r\n* Closes #204 - --relocatable doesn't work with PyPy\r\n* Closes #172 - virtualenv's main should take arguments\r\n* Closes #125 - Improved support for --exec-prefix\r\n* Closes #90 - --relocatable is not reliable\r\n* Closes #11 - --relocatable does not point activate script to the correct path\r\n* Closes #2 - --clear doesn't clear out bin/\r\n\r\nSummary of outstanding pull requests that get fixed/closed:\r\n\r\n* Closes #686 - Document when --always-copy was introduced (in PR #409)\r\n* Closes #672 - Avoid sys.path pollution when switching interpreters\r\n* Closes #663 - Autodetect if symlinks are supported on target file system\r\n* Closes #657 - Add a note, that bootstrap script can't install pip.\r\n* Closes #647 - Fix formating of activate_this.py\r\n* Closes #638 - Split out this setup.py fix from https://github.com/pypa/virtualenv/pull/635\r\n* Closes #636 - Fix to better support python installs where prefix != exec_prefix.\r\n* Closes #634 - A collection of fixes for when prefix != exec_prefix.\r\n* Closes #597 - [WIP] Do some testing\r\n* Closes #561 - Add the tests/ directory to the tarball.\r\n* Closes #547 - Use decorator to classmethod.\r\n* Closes #544 - Source distributions should include tests\r\n* Closes #508 - Add getsitepackages() to site module (Issue #355)\r\n* Closes #485 - Simplify setting up symlinks/copying files\r\n* Closes #479 - Making activate.bat relocatable.\r\n* Closes #473 - Relative local symlinks\r\n* Closes #389 - Made make_relative_path handle symlinks correctly.\r\n* Closes #236 - relocatable activate for bash, dash, zsh, ksh, fish, Windows\r\n\r\nI am pretty sure that this fixes more issues and pull requests than I've listed here. However I don't feel like going through every single issue and determining which ones are still valid when they would require me to actually attempt to reproduce the issues instead of simply looking at the issue and determining if the new code can handle that situation.\r\n"},"comment":{"url":"https://api.github.com/repos/pypa/virtualenv/issues/comments/77235173","html_url":"https://github.com/pypa/virtualenv/pull/697#issuecomment-77235173","issue_url":"https://api.github.com/repos/pypa/virtualenv/issues/697","id":77235173,"user":{"login":"dstufft","id":145979,"avatar_url":"https://avatars.githubusercontent.com/u/145979?v=3","gravatar_id":"","url":"https://api.github.com/users/dstufft","html_url":"https://github.com/dstufft","followers_url":"https://api.github.com/users/dstufft/followers","following_url":"https://api.github.com/users/dstufft/following{/other_user}","gists_url":"https://api.github.com/users/dstufft/gists{/gist_id}","starred_url":"https://api.github.com/users/dstufft/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dstufft/subscriptions","organizations_url":"https://api.github.com/users/dstufft/orgs","repos_url":"https://api.github.com/users/dstufft/repos","events_url":"https://api.github.com/users/dstufft/events{/privacy}","received_events_url":"https://api.github.com/users/dstufft/received_events","type":"User","site_admin":false},"created_at":"2015-03-04T20:04:55Z","updated_at":"2015-03-04T20:04:55Z","body":"The most important part of ``venv`` isn't the actual code behind ``venv``, it's the changes to interpreter start up so that it looks for a ``pyvenv.cfg`` file and localizes the paths it uses based on that."}}