{"action":"created","issue":{"url":"https://api.github.com/repos/galaxyproject/galaxy/issues/428","labels_url":"https://api.github.com/repos/galaxyproject/galaxy/issues/428/labels{/name}","comments_url":"https://api.github.com/repos/galaxyproject/galaxy/issues/428/comments","events_url":"https://api.github.com/repos/galaxyproject/galaxy/issues/428/events","html_url":"https://github.com/galaxyproject/galaxy/pull/428","id":93096775,"number":428,"title":"Replace eggs with wheels","user":{"login":"natefoo","id":286969,"avatar_url":"https://avatars.githubusercontent.com/u/286969?v=3","gravatar_id":"","url":"https://api.github.com/users/natefoo","html_url":"https://github.com/natefoo","followers_url":"https://api.github.com/users/natefoo/followers","following_url":"https://api.github.com/users/natefoo/following{/other_user}","gists_url":"https://api.github.com/users/natefoo/gists{/gist_id}","starred_url":"https://api.github.com/users/natefoo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/natefoo/subscriptions","organizations_url":"https://api.github.com/users/natefoo/orgs","repos_url":"https://api.github.com/users/natefoo/repos","events_url":"https://api.github.com/users/natefoo/events{/privacy}","received_events_url":"https://api.github.com/users/natefoo/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/galaxyproject/galaxy/labels/kind/enhancement","name":"kind/enhancement","color":"84b6eb"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":45,"created_at":"2015-07-05T09:47:33Z","updated_at":"2015-10-22T18:59:12Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/galaxyproject/galaxy/pulls/428","html_url":"https://github.com/galaxyproject/galaxy/pull/428","diff_url":"https://github.com/galaxyproject/galaxy/pull/428.diff","patch_url":"https://github.com/galaxyproject/galaxy/pull/428.patch"},"body":"# Smash Eggs\r\n\r\nMore importantly, replace all of our custom egg installation code with a very slightly modified version of pip. The modification is necessary if/until [wheel is updated to include support for Linux wheels](https://bitbucket.org/pypa/wheel/pull-requests/55/soabi-support-for-python-2x-and-pypy).\r\n\r\n## Other changes include\r\n\r\n* Removal of the awful `lib/pkg_resources.py`\r\n* Relocated the bit of eggs code that checked the Galaxy config to determine what dependencies are needed to a wheels module.\r\n* [galaxyproject/docker-build](https://github.com/galaxyproject/docker-build/) has been updated with scripts and images to support building wheels for distribution on 32 and 64 bit Linux.\r\n\r\n## On virtualenvs\r\n\r\n`scripts/common_startup.sh` will create a virtualenv at `$galaxy_root/.venv` if one is not already active and wheels will be installed to here. Otherwise, Galaxy will use whatever virtualenv is currently active (in `$VIRTUAL_ENV`).\r\n\r\nWhatever virtualenv Galaxy is started under will be passed to jobs if a virtualenv is not already active in the job environment (and it exists at the same path on the compute resource). You can force jobs to use a different virtualenv using the `<env>` tag in a job destination, e.g.:\r\n\r\n```xml\r\n<destination id=\"foo\">\r\n    <env file=\"/path/to/venv/bin/activate\"/>\r\n</destination>\r\n```\r\n\r\nGalaxy needs to become installable in the traditional `python setup.py install` manner which will change some things about how virtualenvs are handled in the future (tl;dr, we don't need to \"handle\" them all unless you are running directly from source), but that is the next project. This was all recently done for Pulsar so Galaxy can steal from there. =)\r\n\r\n## TODO\r\n\r\n- [x] Replace distribute in bx-python with setuptools.\r\n- [x] Figure out how to build OS X wheels for distribution.\r\n- [x] Remove all eggs imports and eggs require()s\r\n- [x] ~~Get rid of the SVGFig dependency~~\r\n- [x] Do not attempt to install all dependencies every time (determine what's missing and attempt to install those).\r\n- [x] Fail to start if a virtualenv does not exist and cannot be created.\r\n- [x] Figure out how to support modified packages (Whoosh, sqlalchemy-migrate). `pip install git+https://` works, but then it's a little trickier to correctly answer \"Is sqlalchemy-migrate installed, and is it the right version?\"\r\n- [x] Build [and host](https://wheels.galaxyproject.org/) wheels for everything that does not already have them (and possibly even the stuff already in PyPI). We should provide wheels for anything that does not provide them, possibly even for pure-python modules since it'll still be faster to install a wheel even though there is no C compilation involved.\r\n- [x] Figure out how to build psycopg2 ~~and MySQL_Python~~ (anyone using MySQL can just `pip install MySQL-python` (which `common-startup.sh` will in fact attempt to do)\r\n- [x] The pysam wheel is broken: `ImportError: /home/nate/galaxy/galaxy/.venv/local/lib/python2.7/site-packages/pysam/calignmentfile.so: undefined symbol: hts_idx_load_literal`\r\n- [x] The bx wheel is broken (possibly due to being built w/o numpy installed?):  `ImportError: No module named bigbed_file`\r\n- [x] Test drmaa and pbs runners\r\n- [x] Create an `sdist` for everything on wheels.galaxyproject.org\r\n- [x] Test and document creating (or sharing) a venv w/ necessary wheels when running on the cluster\r\n- [x] Build remaining OS X wheels\r\n- [x] When using `run.sh`, ignore whatever virtualenv is active and use `.venv`. If you want to use something else, either start without `run.sh` or use `run.sh --skip-venv`. In the future, we won't mess with venvs at all since Galaxy should install in the standard Pythonic way.\r\n- [x] Merge updated SOABI code from pypa/pip#3075 into natefoo/pip linux-wheels branch\r\n\r\n## Questions\r\n\r\n* ~~Should we pin exact versions of dependencies? I have a strong preference to say no, and as a concession say we should pin to \"less than the next major version\" e.g. `SQLAlchemy<1.1`. Although this does present one small problem - we will have to ensure that we always have wheels built for every version published on PyPI (we can certainly automate this) or else pip will bypass the wheels and install from source, unless we use `pip install --only-binary`. However, if we use `--only-binary`, then we cannot trivially fall back to compiling from source on platforms that we do not have wheels for.~~ I was won over by the argument that we should pin exact versions (\"exact set of versioned dependencies for an exact version of Galaxy), but that we will:\r\n * Fetch new dependency versions as they are released\r\n * Build wheels for them\r\n * Test Galaxy against these new wheel versions\r\n * If the tests pass, automatically issue a PR against Galaxy to bump the version\r\n* ~~Should the existence of `$galaxy_root/.venv` take precedence over `$VIRTUAL_ENV` (since I keep installing all of Galaxy's wheels in my tox venv =P)?~~use `--skip-venv` if you don't want `.venv` to take precedence."},"comment":{"url":"https://api.github.com/repos/galaxyproject/galaxy/issues/comments/150323050","html_url":"https://github.com/galaxyproject/galaxy/pull/428#issuecomment-150323050","issue_url":"https://api.github.com/repos/galaxyproject/galaxy/issues/428","id":150323050,"user":{"login":"natefoo","id":286969,"avatar_url":"https://avatars.githubusercontent.com/u/286969?v=3","gravatar_id":"","url":"https://api.github.com/users/natefoo","html_url":"https://github.com/natefoo","followers_url":"https://api.github.com/users/natefoo/followers","following_url":"https://api.github.com/users/natefoo/following{/other_user}","gists_url":"https://api.github.com/users/natefoo/gists{/gist_id}","starred_url":"https://api.github.com/users/natefoo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/natefoo/subscriptions","organizations_url":"https://api.github.com/users/natefoo/orgs","repos_url":"https://api.github.com/users/natefoo/repos","events_url":"https://api.github.com/users/natefoo/events{/privacy}","received_events_url":"https://api.github.com/users/natefoo/received_events","type":"User","site_admin":false},"created_at":"2015-10-22T18:59:12Z","updated_at":"2015-10-22T18:59:12Z","body":"@erasche Well why'd you type `wheels.galxay` instead of `wheels.galaxy`... oh... nevermind. (fixed my comment above where I typo'd it). Also, `--skip-venv` support coming shortly.\r\n\r\n@dannon What if you start with `PYTHONPATH` unset?\r\n\r\nIf it works, that's gonna be a fun one to deal with. Perhaps we should `unset PYTHONPATH` in the event that `--skip-venv` is not used."}}