{"action":"created","issue":{"url":"https://api.github.com/repos/pyinstaller/pyinstaller/issues/1231","labels_url":"https://api.github.com/repos/pyinstaller/pyinstaller/issues/1231/labels{/name}","comments_url":"https://api.github.com/repos/pyinstaller/pyinstaller/issues/1231/comments","events_url":"https://api.github.com/repos/pyinstaller/pyinstaller/issues/1231/events","html_url":"https://github.com/pyinstaller/pyinstaller/pull/1231","id":66820257,"number":1231,"title":"\"six\" Support for Python 3","user":{"login":"leycec","id":217028,"avatar_url":"https://avatars.githubusercontent.com/u/217028?v=3","gravatar_id":"","url":"https://api.github.com/users/leycec","html_url":"https://github.com/leycec","followers_url":"https://api.github.com/users/leycec/followers","following_url":"https://api.github.com/users/leycec/following{/other_user}","gists_url":"https://api.github.com/users/leycec/gists{/gist_id}","starred_url":"https://api.github.com/users/leycec/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/leycec/subscriptions","organizations_url":"https://api.github.com/users/leycec/orgs","repos_url":"https://api.github.com/users/leycec/repos","events_url":"https://api.github.com/users/leycec/events{/privacy}","received_events_url":"https://api.github.com/users/leycec/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/pyinstaller/pyinstaller/labels/@medium","name":"@medium","color":"FFD07D"},{"url":"https://api.github.com/repos/pyinstaller/pyinstaller/labels/PY3","name":"PY3","color":"fbca04"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":10,"created_at":"2015-04-07T08:19:20Z","updated_at":"2015-06-03T06:25:08Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/pyinstaller/pyinstaller/pulls/1231","html_url":"https://github.com/pyinstaller/pyinstaller/pull/1231","diff_url":"https://github.com/pyinstaller/pyinstaller/pull/1231.diff","patch_url":"https://github.com/pyinstaller/pyinstaller/pull/1231.patch"},"body":"## Synopsis\r\n\r\nThis pull request adds support for [six](https://pythonhosted.org/six), a popular Python 2 and 3 compatibility layer, to PyInstaller's Python 3 branch.\r\n\r\n## Motivation\r\n\r\nIf SWIG is [Pretty Big Stuff in the Python community](https://github.com/pyinstaller/pyinstaller/pull/1226), six is Even Bigger Stuff. While there appears to be no official list, a few prominent six-based projects include: Django, OpenStack, Matplotlib >= 1.4.0, and python-modernize (and thus all projects using python-modernize).\r\n\r\n## The Problem\r\n\r\nsix allows modules whose names changed between Python 2 and 3 to be imported in a version-agnostic manner. To do so, six provides a submodule `six.moves` whose attributes are safely importable modules. For example, six allows `tkinter`'s drag-and-drop module (named `Tkdnd` under Python 2 and `tkinter.dnd` under Python 3) to be imported under both Python 2 and 3 like so:\r\n\r\n    from six.moves import tkinter_dnd as Tkdnd\r\n\r\nWhile PyInstaller *does* successfully freeze scripts containing such imports, the resulting executables fail to run. Why? Because PyInstaller fails to realize that the `six.moves.tkinter_dnd` attribute is actually an alias for the `tkinter.dnd` module (under Python 3) and thus fails to:\r\n\r\n* Add the `tkinter` package to the PYZ archive at freeze time.\r\n* Run `tkinter` import hooks at freeze time.\r\n* Run `tkinter` runtime hooks at bootloader time.\r\n\r\nSee #1230 for details.\r\n\r\n## How Does six Do It?\r\n\r\nBefore we get to the solution proposed by this pull request, let's discuss how six internally implements import aliases. As expected, six molests Python's runtime import system by:\r\n\r\n* Dynamically transforming itself from a module into a package at runtime.\r\n* Dynamically defining a fake `six.moves` module at runtime.\r\n* Dynamically defining a PEP 302-compliant importer for this module at runtime, converting imports of attributes from this module into standard imports specific to the current Python version.\r\n\r\nPyInstaller doesn't support *any* of this. For one thing, PyInstaller only supports modules that physically exist on disk; it currently ignores modules dynamically declared in memory. For another, PyInstaller has no support for PEP 302-compliant importers; it currently ignores `sys.meta_path` entirely.\r\n\r\nIn other words, the solution proposed by this pull request is non-trivial. It involves changes to numerous low-level modules at both freeze and bootloader time. To sweeten the deal, these changes have been implemented in a generic fashion independent of six. Once committed, they should make it simpler to add support for other compatibility layers (e.g., [python-future](http://python-future.org), [python-modernize](https://github.com/mitsuhiko/python-modernize) and PEP 302-compliant importers.\r\n\r\nThis request does *not* add support for such importers. While that should eventually be done, doing so here would have been too crazy. (Indeed, it may not even be feasible for PyInstaller to support such importers. Discussion for another time.)\r\n\r\n## The Solution\r\n\r\n`py2exe` has had six support since [last October](http://sourceforge.net/p/py2exe/mailman/message/32941826/). If `py2exe` did it, so can we. However, PyInstaller is a *much* deeper beast than `py2exe` – usually because it has to be, occasionally because it just is. What worked for them won't work for us. Moreover, the `py2exe` solution could be regarded as something of a hack. (Sorry, guys.) It sort of works, but it's hardly elegant and probably fails to scale to more complex PEP 302-compliant importers.\r\n\r\nSo we'd like to do this right. Let's get to it.\r\n\r\n### Problem #1: Import Hooks\r\n\r\nTo support `six.moves`-style imports, we need to import `six.moves` in a subprocess, convert the dynamically declared `six.moves` data structure into a generic data structure evaluatable in the current process, and then add the module aliases implied by that structure to the current module graph. In other words, we need a `six.moves` hook.\r\n\r\nThere's just one problem: import hooks are run *after* PyInstaller has already parsed apart the import statements triggering those hooks and added appropriate module nodes to the graph. That doesn't work in this case. To understand why, consider what PyInstaller does on parsing the import statement `from six.moves import tkinter_dnd as Tkdnd`.\r\n\r\nFirst, it successfully imports the top-level `six` module from a script `six.py`. Then, it attempts to import the `six.moves` submodule by searching for a script `moves.py` in the package directory `six/`. That obviously fails: there *is* no directory `six/` and hence no script `moves.py`. The `six.moves` submodule exists only in memory as a side effect of importing `six`, which PyInstaller of course *never* does. So, PyInstaller adds `six.moves` to the graph as a MissingModule and the attribute `tkinter_dnd` as an entry in the `_namespace` attribute of that module. And that's all.\r\n\r\nBy the time a hypothetical import hook `hooks/hook-six.moves.py` would be run, all `six.moves` import statements in the user's script would have long since been added to the graph as attributes of a MissingModule and thus effectively ignored.\r\n\r\nSo import hooks don't help us. **We need a fundamentally new type of hook.** We need a hook run immediately *before* PyInstaller has parsed apart the import statement triggering that hook and added the appropriate module node to the graph. That hook can then preemptively intervene in the import process by modifying the graph in such a way as to mimic custom behaviour – like, say, a PEP 302-compliant importer.\r\n\r\n### Solution #1: Graph Hooks\r\n\r\nWe call this new type of hook the **graph hook**. Graph hooks reside in the `PyInstaller/hooks/graph/` subdirectory. Like import hooks, graph hook basenames follow the format `hook-{module_name}.py`. While import hooks may optionally define a `hook()` function accepting a `FakeModule` object, graph hooks *must* define a `hook()` function accepting a `PyiModuleGraph` object.\r\n\r\nGraph hooks are handled by the `PyiModuleGraph` class, which also currently handles runtime hooks. On instantiation, `PyiModuleGraph` caches a dictionary mapping module names to graph hooks for efficient lookup. On `import_module()` method calls, `PyiModuleGraph` runs the graph hook for the passed module name if any and then removes that hook from that dictionary, guaranteenig graph hooks to be run exactly once.\r\n\r\nGraph hooks are *not* PEP 302-compliant importers. They're just hooks run at a slightly different time than standard import hooks and passed a slightly different object. That's it. No fancy import logic and no modification of existing behaviour.\r\n\r\nThis request adds *no* support for user-defined custom graph hooks. (It's certainly doable, but *not* required for six support.)\r\n\r\n### Problem #2: Aliased Modules\r\n\r\nPyInstaller has support for adding module aliases to the graph – but *only* at graph instantiation time. The `six.moves` graph hook needs to add such aliases *after* the graph has already been instantiated.\r\n\r\n### Solution #2: alias_module\r\n\r\nWe define a new high-level `ModuleGraph.alias_module()` method, aliasing the name of a physically importable module (e.g., `tkinter.dnd`) to the name of an alias for that module (e.g., `six.moves.tkinter_dnd`).\r\n\r\nSimple.\r\n\r\n### Problem #3: Runtime Modules\r\n\r\nPyInstaller currently has no support for adding modules dynamically defined at runtime (e.g., `six.moves`) to the graph. This prevents a dependency (directed edge) from being added between the parent module defining that module and that module, preventing the parent module from being bundled into the PYZ archive, resulting in an `ImportError` exception at runtime.\r\n\r\nWhich is bad.\r\n\r\n### Solution #3: RuntimeModule and add_module\r\n\r\nWe declare a new type of graph node representing dynamically defined modules in the module graph: `RuntimeModule`. Since runtime modules do *not* physically exist, these nodes *cannot* be added to the graph via typical import methods (e.g., `ModuleGraph.import_hook()`, `ModuleGraph.import_module()`).\r\n\r\nInstead, we define a new high-level `ModuleGraph.add_module()` method adding the passed graph node to the graph *without* importing that node's module file. This method is intended to be used only with `RuntimeModule` nodes, but should technically work for any node type.\r\n\r\n### Problem #4: ObjectGraph.get_edges() Bug\r\n\r\nThe `ObjectGraph.get_edges()` method contains a subtle bug with respect to `MissingModule` and `AliasNode` nodes, resulting in the following exception:\r\n\r\n    59521 INFO: Looking for dynamic libraries\r\n    ldd: warning: you do not have execution permission for `/usr/lib/gcc/x86_64-pc-linux-gnu/4.8.4/libgcc_s.so.1'\r\n    63940 INFO: Looking for eggs - TODO\r\n    64009 INFO: Multiarch directory not detected.\r\n    64057 INFO: Using Python library /usr/lib/libpython3.3.so.1.0\r\n    64058 INFO: Adding Python library to binary dependencies\r\n    Traceback (most recent call last):\r\n      File \"/usr/lib/python-exec/python3.3/pyinstaller\", line 9, in <module>\r\n        load_entry_point('PyInstaller===3.0dev-be553f0-mod', 'console_scripts', 'pyinstaller')()\r\n      File \"/usr/lib64/python3.3/site-packages/PyInstaller/main.py\", line 88, in run\r\n        run_build(opts, spec_file, pyi_config)\r\n      File \"/usr/lib64/python3.3/site-packages/PyInstaller/main.py\", line 46, in run_build\r\n        PyInstaller.build.main(pyi_config, spec_file, **opts.__dict__)\r\n      File \"/usr/lib64/python3.3/site-packages/PyInstaller/build.py\", line 1903, in main\r\n        build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))\r\n      File \"/usr/lib64/python3.3/site-packages/PyInstaller/build.py\", line 1850, in build\r\n        exec(compile(open(spec).read(), spec, 'exec'))\r\n      File \"freeze/betse.dir.spec\", line 14, in <module>\r\n        exec(core_spec.read())\r\n      File \"<string>\", line 100, in <module>\r\n      File \"/usr/lib64/python3.3/site-packages/PyInstaller/build.py\", line 355, in __init__\r\n        self.__postinit__()\r\n      File \"/usr/lib64/python3.3/site-packages/PyInstaller/build.py\", line 215, in __postinit__\r\n        self.assemble()\r\n      File \"/usr/lib64/python3.3/site-packages/PyInstaller/build.py\", line 756, in assemble\r\n        importer_names = self.graph.importer_names(n)\r\n      File \"/usr/lib64/python3.3/site-packages/PyInstaller/depend/analysis.py\", line 303, in importer_names\r\n        return [importer.identifier for importer in iter_inc]\r\n      File \"/usr/lib64/python3.3/site-packages/PyInstaller/depend/analysis.py\", line 303, in <listcomp>\r\n        return [importer.identifier for importer in iter_inc]\r\n    AttributeError: 'NoneType' object has no attribute 'identifier'\r\n\r\nIf that looks inscrutable to you too, you're not alone. What's going on here?  Let's find out.\r\n\r\nObjectGraph.get_edges() returns a 2-tuple whose first item is a generator yielding the nodes connected to outgoing edges and whose second item is a generator yielding the nodes connected to incoming edges for the passed node. There's one small problem: if any such node is a `MissingModule`, that node's entry in the internal edge data structure will be **the current graph** rather than a valid node. This results in the generator yielding the value `None` for that node. However, the `PyiModuleGraph.importer_names()` method assumes these generators to yield only valid nodes. The exception follows.\r\n\r\nWhy does this relate to the current pull request? Because six aliases `winreg` to `_winreg` under Python 3, resulting in an edge being added from `six.moves` to `winreg`. On OS X and Linux, `winreg` doesn't exist and is thus added to the graph as a `MissingModule`. The rest is tiresome history.\r\n\r\n### Solution #4: Fix the ObjectGraph.get_edges() Bug\r\n\r\nThis one's dead simple. We fix the ObjectGraph.get_edges() method as follows:\r\n\r\n* If the currently iterated node's entry in the internal edge data structure is the current graph (rather than a valid node), then:\r\n** Skip that node.\r\n\r\nThat's it. Next!\r\n\r\n### Problem #5: Module Aliases Are Unmarshallable\r\n\r\nWe stated above that \"PyInstaller has support for adding module aliases to the graph.\" Technically, that's true: PyInstaller *does* support that. It just doesn't actually work.\r\n\r\nThis probably explains why the `lib.modulegraph.find_modules` module – a nearly 400-line file – is no longer used at all (apart from a single use of the `find_needed_modules()` function elsewhere). `find_modules` is mostly concerned with establishing module aliases. But module aliases don't actually work. So that module was entirely disabled but *not* removed. (It's like my vestigial third eye.)\r\n\r\nBefore we get to why module aliases are currently broken, let's describe how module aliases are actually added to the graph. The `ModuleGraph.alias_module()` method basically reduces to the following line:\r\n\r\n    self.lazynodes[trg_module_name] = Alias(src_module_name)\r\n\r\nThe `Alias` class subclasses the `str` class... and that's it. It's only used for typing: specifically, to tell the `ModuleGraph.findNode()` method that the module with the passed name should be created as a new `AliasNode`. To do that, that method contains the following line:\r\n\r\n    other = self._safe_import_hook(deps, None, None).pop()\r\n\r\nThe `deps` object passed here is the `Alias` object created above. Ultimately, this disastrous logic results in the TOC (of files to be frozen) containing an `Alias` rather than `str` object. Is that a problem? **Yes.** Python's marshalling facilities are *not* smart. (Some might say \"stupid.\") The `marshal.dump()` function requires the passed object and all objects transitively contained by that object to be standard types. `Alias` is *not* a standard type. `str` is.\r\n\r\nOn receiving a TOC object containing an `Alias` object, `marshal.dump()` raises the following unreadable exception:\r\n\r\n    5963 INFO: Looking for dynamic libraries\r\n    7021 INFO: Looking for eggs - TODO\r\n    7074 INFO: Multiarch directory not detected.\r\n    7098 INFO: Using Python library /usr/lib/libpython3.3.so.1.0\r\n    7098 INFO: Adding Python library to binary dependencies\r\n    7171 INFO: Warnings written to /home/leycec/tmp/testem/build/saxo/warnsaxo.txt\r\n    7173 DEBUG: compiled /usr/lib64/python3.3/struct.py\r\n    7174 INFO: checking PYZ\r\n    7174 INFO: rebuilding out00-PYZ.toc because out00-PYZ.pyz is missing\r\n    7174 INFO: building PYZ (ZlibArchive) out00-PYZ.toc\r\n    Traceback (most recent call last):\r\n      File \"/usr/lib/python-exec/python3.3/pyinstaller\", line 9, in <module>\r\n        load_entry_point('PyInstaller===3.0dev-be553f0-mod', 'console_scripts', 'pyinstaller')()\r\n      File \"/usr/lib64/python3.3/site-packages/PyInstaller/main.py\", line 88, in run\r\n        run_build(opts, spec_file, pyi_config)\r\n      File \"/usr/lib64/python3.3/site-packages/PyInstaller/main.py\", line 46, in run_build\r\n        PyInstaller.build.main(pyi_config, spec_file, **opts.__dict__)\r\n      File \"/usr/lib64/python3.3/site-packages/PyInstaller/build.py\", line 1903, in main\r\n        build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))\r\n      File \"/usr/lib64/python3.3/site-packages/PyInstaller/build.py\", line 1850, in build\r\n        exec(compile(open(spec).read(), spec, 'exec'))\r\n      File \"/home/leycec/tmp/testem/saxo.spec\", line 7, in <module>\r\n        pyz = PYZ(a.pure)\r\n      File \"/usr/lib64/python3.3/site-packages/PyInstaller/build.py\", line 829, in __init__\r\n        self.__postinit__()\r\n      File \"/usr/lib64/python3.3/site-packages/PyInstaller/build.py\", line 215, in __postinit__\r\n        self.assemble()\r\n      File \"/usr/lib64/python3.3/site-packages/PyInstaller/build.py\", line 851, in assemble\r\n        pyz.build(self.name, toc)\r\n      File \"/usr/lib64/python3.3/site-packages/PyInstaller/loader/pyi_archive.py\", line 226, in build\r\n        self._finalize()\r\n      File \"/usr/lib64/python3.3/site-packages/PyInstaller/loader/pyi_archive.py\", line 212, in _finalize\r\n        self.save_toc(toc_pos)\r\n      File \"/usr/lib64/python3.3/site-packages/PyInstaller/loader/pyi_archive.py\", line 260, in save_toc\r\n        marshal.dump(self.toc, self.lib)\r\n    ValueError: unmarshallable object\r\n\r\nGood luck debugging *that*. To simplify debugging of similar issues in the future, `Archive.save_toc()` has been patched to find and print the responsible nested TOC object on catching such exceptions. For example, the Python script containing only `from six.moves import tkinter` results in this output:\r\n\r\n    Module name \"tkinter\" (<class 'modulegraph.modulegraph.Alias'>) unmarshallable. \r\n\r\nAs expected, the non-standard `Alias` type kills TOC marshalling.\r\n\r\n### Solution #5: Make Module Aliases Marshallable\r\n\r\nWe correct this by coercing non-`str` to `str` types in both the `Archive.add()` and `ZlibArchive.add()` methods.\r\n\r\n**And we're done.** Believe it.\r\n\r\n## Other Changes\r\n\r\nYou know the deal by now. Sphinx-compatible docstrings (in standard numpydoc format) have been added to critical classes, functions, and methods.\r\n\r\n## Summary\r\n\r\nThis pull request fixes #1230. It's good. It's hard. You know you want it."},"comment":{"url":"https://api.github.com/repos/pyinstaller/pyinstaller/issues/comments/108208502","html_url":"https://github.com/pyinstaller/pyinstaller/pull/1231#issuecomment-108208502","issue_url":"https://api.github.com/repos/pyinstaller/pyinstaller/issues/1231","id":108208502,"user":{"login":"leycec","id":217028,"avatar_url":"https://avatars.githubusercontent.com/u/217028?v=3","gravatar_id":"","url":"https://api.github.com/users/leycec","html_url":"https://github.com/leycec","followers_url":"https://api.github.com/users/leycec/followers","following_url":"https://api.github.com/users/leycec/following{/other_user}","gists_url":"https://api.github.com/users/leycec/gists{/gist_id}","starred_url":"https://api.github.com/users/leycec/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/leycec/subscriptions","organizations_url":"https://api.github.com/users/leycec/orgs","repos_url":"https://api.github.com/users/leycec/repos","events_url":"https://api.github.com/users/leycec/events{/privacy}","received_events_url":"https://api.github.com/users/leycec/received_events","type":"User","site_admin":false},"created_at":"2015-06-03T06:25:08Z","updated_at":"2015-06-03T06:25:08Z","body":"**Rebased**, with a gracious nod to all of the badass work you've done.\r\n\r\nUnfortunately, I can't test this. \"It's game over, man!\" PyInstaller now yields the following fatal exception on startup:\r\n\r\n    Traceback (most recent call last):\r\n      File \"/usr/lib/python-exec/python3.3/pyinstaller\", line 9, in <module>\r\n        load_entry_point('PyInstaller===3.0dev-be553f0-mod', 'console_scripts', 'pyinstaller')()\r\n      File \"/usr/lib64/python3.3/site-packages/pkg_resources/__init__.py\", line 519, in load_entry_point\r\n        return get_distribution(dist).load_entry_point(group, name)\r\n      File \"/usr/lib64/python3.3/site-packages/pkg_resources/__init__.py\", line 2630, in load_entry_point\r\n        return ep.load()\r\n      File \"/usr/lib64/python3.3/site-packages/pkg_resources/__init__.py\", line 2310, in load\r\n        return self.resolve()\r\n      File \"/usr/lib64/python3.3/site-packages/pkg_resources/__init__.py\", line 2316, in resolve\r\n        module = __import__(self.module_name, fromlist=['__name__'], level=0)\r\n      File \"/usr/lib64/python3.3/site-packages/PyInstaller/main.py\", line 21, in <module>\r\n        import PyInstaller.build\r\n      File \"/usr/lib64/python3.3/site-packages/PyInstaller/build.py\", line 24, in <module>\r\n        from PyInstaller.depend.analysis import PyiModuleGraph, TOC, FakeModule\r\n      File \"/usr/lib64/python3.3/site-packages/PyInstaller/depend/analysis.py\", line 41, in <module>\r\n        from modulegraph.modulegraph import ModuleGraph\r\n      File \"/home/leycec/py/modulegraph/modulegraph/__init__.py\", line 2, in <module>\r\n        __version__ = pkg_resources.require('modulegraph')[0].version\r\n      File \"/usr/lib64/python3.3/site-packages/pkg_resources/__init__.py\", line 918, in require\r\n        needed = self.resolve(parse_requirements(requirements))\r\n      File \"/usr/lib64/python3.3/site-packages/pkg_resources/__init__.py\", line 805, in resolve\r\n        raise DistributionNotFound(req)\r\n    pkg_resources.DistributionNotFound: modulegraph\r\n\r\nAdmittedly, this is sorta my fault. To simplify testing of my `modulegraph` pull request, I installed `modulegraph` in a non-standard manner. That's my problem, and I should *definitely* fix it.\r\n\r\nBut the `PyInstaller.depend.analysis` module now imports the top-level and hence official version of `modulegraph` rather than our `PyInstaller.lib.modulegraph` \"branch.\" (Wink-wink, nudge-nudge.) That doesn't seem quite right. That said, I haven't plumbed through our `__init__.py` scripts in a while. We're probably still mucking around with `sys.path` somewhere, in which case importing `modulegraph` *should* import `PyInstaller.lib.modulegraph` instead.\r\n\r\n**Anyway.** I clearly have no idea what I'm talking about. It's late. I'm hungry. I'm tired. Thus ends this pull request: not with a bang, but a baleful whimper."}}