{"action":"created","comment":{"url":"https://api.github.com/repos/PyConPL/Book/pulls/comments/38243291","id":38243291,"diff_hunk":"@@ -0,0 +1,297 @@\n+# Import Deep Dive Detours - Petr Viktorin\n+\n+My talk, Import Deep Dive, gave you a simplified overview of Python's import\n+machinery, enough to understand what's going on and where to look for more\n+details.\n+But, there are still some topics that I haven't covered in the 30 minutes.\n+This article dives into a few more tidbits and special cases related to\n+importing: namespace packages, real-world examples of project that use custom\n+importers, Loader-assisted resource loading, the rules for C extension modules,\n+and more information on the `__main__` module.\n+\n+\n+I assume you've seen the talk. If not, watch it! It should be put online\n+after the conference. (Or you can look for the EuroPython version.)\n+\n+Here's the code I talked about. Hopefully I've explained it enough:\n+\n+![CODE SLIDE HERE](code)\n+\n+You also get the package/submodule advice:\n+\n+* `__init__` should:\n+    * import from submodules\n+    * set `__all__`\n+    * nothing else\n+* submodules should:\n+    * not import directly from `__init__`\n+    * not have internal import cycles\n+\n+Now, if you're in for a dive into a few more tidbits, read on!\n+\n+\n+## Namespace packages and plugin systems\n+\n+Probably the most important omission in my talk is namespace packages.\n+\n+When `PathFinder.find_spec` loops through the path entries, it looks for\n+either a file (which would become a module), or a directory with an\n+`__init__` file (which would become a package).\n+\n+What I didn't tell you is that it actually looks for *all* directories,\n+even if they don't contain `__init__`. Any `__init__`-less directories\n+are remembered for later.\n+\n+If the finder finishes going through `path` without finding a suitable module,\n+it makes a *namespace package*: an empty module whose `__path__` contains all\n+the directories remembered.\n+\n+But why do that?\n+\n+Namespace packages are usually used to implement plugin systems.\n+Suppose there is an extensible sound editor, where individual filters\n+(like \"echo\", \"reverb\" and \"speedup\") are implemented as Python modules,\n+and can be installed individually.\n+\n+\"Individually\" means \"to different locations\".\n+Your `sys.path` will usually include several kinds of locations. On a Unixy\n+machine, the path will typically have:\n+\n+* Directories from `$PYTHONPATH`, and the current directory\n+* The Python standard library (e.g. `/usr/lib/python3.4`)\n+* User-specific locations (such as `~/.local/lib/python3.4/site-packages`)\n+* A place for packages installed system-wide (like `/usr/lib/python3.4/site-packages`)\n+\n+Suppose you install the editor and a built-in \"echo\" filter system-wide,\n+a private \"reverb\" filter to your home directory, and you have \"speedup\"\n+(which you're currently developing) in the current directory (on `$PYTHONPATH`).\n+The paths to these modules would be:\n+\n+* /usr/lib/python3.4/site-packages/sound_editor_filters/echo.py\n+* ~/.local/lib/python3.4/site-packages/sound_editor_filters/reverb.py\n+* ./sound_editor_filters/speedup.py\n+\n+And that repeated `sound_editor_filters` directory becomes a namespace package:\n+\n+    >>> import sound_editor_filters\n+    >>> sound_editor_filters.__path__\n+    _NamespacePath([\n+        './sound_editor_filters',\n+        '~/.local/lib/python3.4/site-packages/sound_editor_filters',\n+        '/usr/lib/python3.4/site-packages/sound_editor_filters'])\n+\n+Importing `sound_editor_filters.echo` will get you the correct module,\n+because submodules are searched for in the `__path__` of their parent.\n+\n+Remember that adding an `__init__.py` to *any* of the `sound_editor_filters`\n+directories will cancel the magic. Any directory with `__init__` is a regular\n+package, its `__init__`-less siblings are ignored.\n+\n+Long before namespace packages were added to core Python, `setuptools` contained\n+[an utility](https://pythonhosted.org/setuptools/setuptools.html#namespace-packages)\n+to declare namespaces manually and explicitly.\n+Of course, this still works – and if you need to support Python 2,\n+it's the way to go.\n+\n+\n+## Customization examples\n+\n+Another topic that didn't make it to my talk is customization:\n+how to extend importlib to do new, exciting things?\n+\n+If you're interested in this, I'll just direct you to\n+[importlib documentation](https://docs.python.org/3/library/importlib.html).\n+Now that you know the terminology and and the \"normal\" way of how things are\n+done, those docs should make much more sense.\n+\n+Instead, I'll mention some real-world projects that use custom import hooks,\n+which you can use as inspiration about what's possible.\n+\n+Hy (hy.readthedocs.org) allows you to import modules written in a dialect of\n+Lisp. Or as its docs put it, it's \"lisp-stick on a Python\".\n+It uses a custom finder that looks for `*.hy` files instead of `*.py` ones,\n+and a corresponding loader that parses the Lisp code, runs it, and populates\n+a Python module object with the resulting functions and objects.\n+\n+This is a good approach for any project that wants to embed a custom language\n+in the Python runtime.\n+\n+The Macropy project uses a similar technique to add syntactic macros to\n+Python code. Conceptually, it's the same as Hy, except the language is a bit\n+more similar to Python.\n+\n+Perhaps most widely used of import customizations is Cython's `pyximport`,\n+which finds a `.pyx` file, compiles it into a C extension, and then imports it.\n+\n+If you just want to load Python modules that are stored in a non-standard\n+location, like a database, Git repository, or just a string,\n+look near the end of David Beazley's PyCon US 2015 tutorial for his\n+Redis importer.\n+\n+\n+## Loading Module Resources\n+\n+Every module uses a `Loader` object when it is first initialized, but the\n+usefulness of loaders doesn't end there.\n+\n+Loaders of most types of modules implement the `ResourceLoader` ABC, a protocol\n+that allows loading data files stored alongside a module.\n+For example, to open a data file stored \"next to\" the current module on the\n+filesystem, a common approach is to do:\n+\n+    >>> import os\n+    >>> filename = os.path.join(os.path.dirname(__file__), \"file.dat\")\n+    >>> with open(filename, 'rb') as f:\n+    ...     data = f.read()\n+\n+However, this will only work when the module and data is loaded from regular\n+files. If they were, for example, distributed in a zip file, the `open`","path":"2015/presentations/import_deep_dive/text.md","position":148,"original_position":148,"commit_id":"dd5105d3642c25d38d83b5b5ebf2bfd38c471273","original_commit_id":"dd5105d3642c25d38d83b5b5ebf2bfd38c471273","user":{"login":"soltysh","id":576341,"avatar_url":"https://avatars.githubusercontent.com/u/576341?v=3","gravatar_id":"","url":"https://api.github.com/users/soltysh","html_url":"https://github.com/soltysh","followers_url":"https://api.github.com/users/soltysh/followers","following_url":"https://api.github.com/users/soltysh/following{/other_user}","gists_url":"https://api.github.com/users/soltysh/gists{/gist_id}","starred_url":"https://api.github.com/users/soltysh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/soltysh/subscriptions","organizations_url":"https://api.github.com/users/soltysh/orgs","repos_url":"https://api.github.com/users/soltysh/repos","events_url":"https://api.github.com/users/soltysh/events{/privacy}","received_events_url":"https://api.github.com/users/soltysh/received_events","type":"User","site_admin":false},"body":"s/files/file","created_at":"2015-08-28T20:54:43Z","updated_at":"2015-08-28T20:54:43Z","html_url":"https://github.com/PyConPL/Book/pull/21#discussion_r38243291","pull_request_url":"https://api.github.com/repos/PyConPL/Book/pulls/21","_links":{"self":{"href":"https://api.github.com/repos/PyConPL/Book/pulls/comments/38243291"},"html":{"href":"https://github.com/PyConPL/Book/pull/21#discussion_r38243291"},"pull_request":{"href":"https://api.github.com/repos/PyConPL/Book/pulls/21"}}},"pull_request":{"url":"https://api.github.com/repos/PyConPL/Book/pulls/21","id":42525258,"html_url":"https://github.com/PyConPL/Book/pull/21","diff_url":"https://github.com/PyConPL/Book/pull/21.diff","patch_url":"https://github.com/PyConPL/Book/pull/21.patch","issue_url":"https://api.github.com/repos/PyConPL/Book/issues/21","number":21,"state":"open","locked":false,"title":"Article with additional info for \"Import Deep Dive\".","user":{"login":"encukou","id":302922,"avatar_url":"https://avatars.githubusercontent.com/u/302922?v=3","gravatar_id":"","url":"https://api.github.com/users/encukou","html_url":"https://github.com/encukou","followers_url":"https://api.github.com/users/encukou/followers","following_url":"https://api.github.com/users/encukou/following{/other_user}","gists_url":"https://api.github.com/users/encukou/gists{/gist_id}","starred_url":"https://api.github.com/users/encukou/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/encukou/subscriptions","organizations_url":"https://api.github.com/users/encukou/orgs","repos_url":"https://api.github.com/users/encukou/repos","events_url":"https://api.github.com/users/encukou/events{/privacy}","received_events_url":"https://api.github.com/users/encukou/received_events","type":"User","site_admin":false},"body":"Here is an article on importing, with lots of information beyond what I can fit in 30min.\r\n\r\nI will be on an offline vacation until August 31, which is the article submission deadline. So hopefully I didn't make any mistakes here – I won't be able to correct them :(\r\n\r\nAs far as I know, Markdown doesn't support arbitrary indents (outside of\r\nmonospaced code), so I included the take-away slide with all the simplified pseudocode \r\nas a PDF document. Hopefully it will be possible to insert it.","created_at":"2015-08-15T20:24:24Z","updated_at":"2015-08-28T20:54:43Z","closed_at":null,"merged_at":null,"merge_commit_sha":"6559377ad52f2d6ebe05b4876e6e3d1226ae7950","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/PyConPL/Book/pulls/21/commits","review_comments_url":"https://api.github.com/repos/PyConPL/Book/pulls/21/comments","review_comment_url":"https://api.github.com/repos/PyConPL/Book/pulls/comments{/number}","comments_url":"https://api.github.com/repos/PyConPL/Book/issues/21/comments","statuses_url":"https://api.github.com/repos/PyConPL/Book/statuses/dd5105d3642c25d38d83b5b5ebf2bfd38c471273","head":{"label":"encukou:import-deep-dive","ref":"import-deep-dive","sha":"dd5105d3642c25d38d83b5b5ebf2bfd38c471273","user":{"login":"encukou","id":302922,"avatar_url":"https://avatars.githubusercontent.com/u/302922?v=3","gravatar_id":"","url":"https://api.github.com/users/encukou","html_url":"https://github.com/encukou","followers_url":"https://api.github.com/users/encukou/followers","following_url":"https://api.github.com/users/encukou/following{/other_user}","gists_url":"https://api.github.com/users/encukou/gists{/gist_id}","starred_url":"https://api.github.com/users/encukou/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/encukou/subscriptions","organizations_url":"https://api.github.com/users/encukou/orgs","repos_url":"https://api.github.com/users/encukou/repos","events_url":"https://api.github.com/users/encukou/events{/privacy}","received_events_url":"https://api.github.com/users/encukou/received_events","type":"User","site_admin":false},"repo":{"id":40782593,"name":"Book","full_name":"encukou/Book","owner":{"login":"encukou","id":302922,"avatar_url":"https://avatars.githubusercontent.com/u/302922?v=3","gravatar_id":"","url":"https://api.github.com/users/encukou","html_url":"https://github.com/encukou","followers_url":"https://api.github.com/users/encukou/followers","following_url":"https://api.github.com/users/encukou/following{/other_user}","gists_url":"https://api.github.com/users/encukou/gists{/gist_id}","starred_url":"https://api.github.com/users/encukou/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/encukou/subscriptions","organizations_url":"https://api.github.com/users/encukou/orgs","repos_url":"https://api.github.com/users/encukou/repos","events_url":"https://api.github.com/users/encukou/events{/privacy}","received_events_url":"https://api.github.com/users/encukou/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/encukou/Book","description":"This is a book which will be created by PyCon PL Organizers, based on PyCon PL Speaker's talks for PyCon PL Attendees.","fork":true,"url":"https://api.github.com/repos/encukou/Book","forks_url":"https://api.github.com/repos/encukou/Book/forks","keys_url":"https://api.github.com/repos/encukou/Book/keys{/key_id}","collaborators_url":"https://api.github.com/repos/encukou/Book/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/encukou/Book/teams","hooks_url":"https://api.github.com/repos/encukou/Book/hooks","issue_events_url":"https://api.github.com/repos/encukou/Book/issues/events{/number}","events_url":"https://api.github.com/repos/encukou/Book/events","assignees_url":"https://api.github.com/repos/encukou/Book/assignees{/user}","branches_url":"https://api.github.com/repos/encukou/Book/branches{/branch}","tags_url":"https://api.github.com/repos/encukou/Book/tags","blobs_url":"https://api.github.com/repos/encukou/Book/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/encukou/Book/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/encukou/Book/git/refs{/sha}","trees_url":"https://api.github.com/repos/encukou/Book/git/trees{/sha}","statuses_url":"https://api.github.com/repos/encukou/Book/statuses/{sha}","languages_url":"https://api.github.com/repos/encukou/Book/languages","stargazers_url":"https://api.github.com/repos/encukou/Book/stargazers","contributors_url":"https://api.github.com/repos/encukou/Book/contributors","subscribers_url":"https://api.github.com/repos/encukou/Book/subscribers","subscription_url":"https://api.github.com/repos/encukou/Book/subscription","commits_url":"https://api.github.com/repos/encukou/Book/commits{/sha}","git_commits_url":"https://api.github.com/repos/encukou/Book/git/commits{/sha}","comments_url":"https://api.github.com/repos/encukou/Book/comments{/number}","issue_comment_url":"https://api.github.com/repos/encukou/Book/issues/comments{/number}","contents_url":"https://api.github.com/repos/encukou/Book/contents/{+path}","compare_url":"https://api.github.com/repos/encukou/Book/compare/{base}...{head}","merges_url":"https://api.github.com/repos/encukou/Book/merges","archive_url":"https://api.github.com/repos/encukou/Book/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/encukou/Book/downloads","issues_url":"https://api.github.com/repos/encukou/Book/issues{/number}","pulls_url":"https://api.github.com/repos/encukou/Book/pulls{/number}","milestones_url":"https://api.github.com/repos/encukou/Book/milestones{/number}","notifications_url":"https://api.github.com/repos/encukou/Book/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/encukou/Book/labels{/name}","releases_url":"https://api.github.com/repos/encukou/Book/releases{/id}","created_at":"2015-08-15T20:13:59Z","updated_at":"2015-08-15T20:13:59Z","pushed_at":"2015-08-15T20:16:27Z","git_url":"git://github.com/encukou/Book.git","ssh_url":"git@github.com:encukou/Book.git","clone_url":"https://github.com/encukou/Book.git","svn_url":"https://github.com/encukou/Book","homepage":"http://pl.pycon.org/","size":4770,"stargazers_count":0,"watchers_count":0,"language":"TeX","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"PyConPL:master","ref":"master","sha":"4711793b4d0ad4e65ad1c7fc6278fe5761c94759","user":{"login":"PyConPL","id":5347274,"avatar_url":"https://avatars.githubusercontent.com/u/5347274?v=3","gravatar_id":"","url":"https://api.github.com/users/PyConPL","html_url":"https://github.com/PyConPL","followers_url":"https://api.github.com/users/PyConPL/followers","following_url":"https://api.github.com/users/PyConPL/following{/other_user}","gists_url":"https://api.github.com/users/PyConPL/gists{/gist_id}","starred_url":"https://api.github.com/users/PyConPL/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PyConPL/subscriptions","organizations_url":"https://api.github.com/users/PyConPL/orgs","repos_url":"https://api.github.com/users/PyConPL/repos","events_url":"https://api.github.com/users/PyConPL/events{/privacy}","received_events_url":"https://api.github.com/users/PyConPL/received_events","type":"Organization","site_admin":false},"repo":{"id":22459875,"name":"Book","full_name":"PyConPL/Book","owner":{"login":"PyConPL","id":5347274,"avatar_url":"https://avatars.githubusercontent.com/u/5347274?v=3","gravatar_id":"","url":"https://api.github.com/users/PyConPL","html_url":"https://github.com/PyConPL","followers_url":"https://api.github.com/users/PyConPL/followers","following_url":"https://api.github.com/users/PyConPL/following{/other_user}","gists_url":"https://api.github.com/users/PyConPL/gists{/gist_id}","starred_url":"https://api.github.com/users/PyConPL/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PyConPL/subscriptions","organizations_url":"https://api.github.com/users/PyConPL/orgs","repos_url":"https://api.github.com/users/PyConPL/repos","events_url":"https://api.github.com/users/PyConPL/events{/privacy}","received_events_url":"https://api.github.com/users/PyConPL/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/PyConPL/Book","description":"This is a book which will be created by PyCon PL Organizers, based on PyCon PL Speaker's talks for PyCon PL Attendees.","fork":false,"url":"https://api.github.com/repos/PyConPL/Book","forks_url":"https://api.github.com/repos/PyConPL/Book/forks","keys_url":"https://api.github.com/repos/PyConPL/Book/keys{/key_id}","collaborators_url":"https://api.github.com/repos/PyConPL/Book/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/PyConPL/Book/teams","hooks_url":"https://api.github.com/repos/PyConPL/Book/hooks","issue_events_url":"https://api.github.com/repos/PyConPL/Book/issues/events{/number}","events_url":"https://api.github.com/repos/PyConPL/Book/events","assignees_url":"https://api.github.com/repos/PyConPL/Book/assignees{/user}","branches_url":"https://api.github.com/repos/PyConPL/Book/branches{/branch}","tags_url":"https://api.github.com/repos/PyConPL/Book/tags","blobs_url":"https://api.github.com/repos/PyConPL/Book/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/PyConPL/Book/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/PyConPL/Book/git/refs{/sha}","trees_url":"https://api.github.com/repos/PyConPL/Book/git/trees{/sha}","statuses_url":"https://api.github.com/repos/PyConPL/Book/statuses/{sha}","languages_url":"https://api.github.com/repos/PyConPL/Book/languages","stargazers_url":"https://api.github.com/repos/PyConPL/Book/stargazers","contributors_url":"https://api.github.com/repos/PyConPL/Book/contributors","subscribers_url":"https://api.github.com/repos/PyConPL/Book/subscribers","subscription_url":"https://api.github.com/repos/PyConPL/Book/subscription","commits_url":"https://api.github.com/repos/PyConPL/Book/commits{/sha}","git_commits_url":"https://api.github.com/repos/PyConPL/Book/git/commits{/sha}","comments_url":"https://api.github.com/repos/PyConPL/Book/comments{/number}","issue_comment_url":"https://api.github.com/repos/PyConPL/Book/issues/comments{/number}","contents_url":"https://api.github.com/repos/PyConPL/Book/contents/{+path}","compare_url":"https://api.github.com/repos/PyConPL/Book/compare/{base}...{head}","merges_url":"https://api.github.com/repos/PyConPL/Book/merges","archive_url":"https://api.github.com/repos/PyConPL/Book/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/PyConPL/Book/downloads","issues_url":"https://api.github.com/repos/PyConPL/Book/issues{/number}","pulls_url":"https://api.github.com/repos/PyConPL/Book/pulls{/number}","milestones_url":"https://api.github.com/repos/PyConPL/Book/milestones{/number}","notifications_url":"https://api.github.com/repos/PyConPL/Book/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/PyConPL/Book/labels{/name}","releases_url":"https://api.github.com/repos/PyConPL/Book/releases{/id}","created_at":"2014-07-31T07:04:58Z","updated_at":"2015-08-25T13:07:24Z","pushed_at":"2015-08-27T12:08:41Z","git_url":"git://github.com/PyConPL/Book.git","ssh_url":"git@github.com:PyConPL/Book.git","clone_url":"https://github.com/PyConPL/Book.git","svn_url":"https://github.com/PyConPL/Book","homepage":"http://pl.pycon.org/","size":5375,"stargazers_count":9,"watchers_count":9,"language":"TeX","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":25,"mirror_url":null,"open_issues_count":4,"forks":25,"open_issues":4,"watchers":9,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/PyConPL/Book/pulls/21"},"html":{"href":"https://github.com/PyConPL/Book/pull/21"},"issue":{"href":"https://api.github.com/repos/PyConPL/Book/issues/21"},"comments":{"href":"https://api.github.com/repos/PyConPL/Book/issues/21/comments"},"review_comments":{"href":"https://api.github.com/repos/PyConPL/Book/pulls/21/comments"},"review_comment":{"href":"https://api.github.com/repos/PyConPL/Book/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/PyConPL/Book/pulls/21/commits"},"statuses":{"href":"https://api.github.com/repos/PyConPL/Book/statuses/dd5105d3642c25d38d83b5b5ebf2bfd38c471273"}}}}