How do you organise a python project that contains multiple packages so that each file in a package can still be run individually?
<p><strong>TL;DR</strong></p>

<p>Here's an example repository that is set up as described in the first diagram (below): <a href="https://github.com/Poddster/package_problems" rel="noreferrer">https://github.com/Poddster/package_problems</a></p>

<p>If you could please make it look like the second diagram in terms of project organisation and can still run the following commands, then you've answered the question:</p>

<pre><code>$ git clone https://github.com/Poddster/package_problems.git
$ cd package_problems
&lt;do your magic here&gt;

$ nosetests

$ ./my_tool/my_tool.py
$ ./my_tool/t.py
$ ./my_tool/d.py

 (or for the above commands, $ cd ./my_tool/ &amp;&amp; ./my_tool.py is also acceptable)
</code></pre>

<p>Alternatively: Give me a different project structure that allows me to group together related files ('package'), run all of the files individually, import the files into other files in the same package, and import the packages/files into other package's files.</p>

<hr>

<h1>Current situation</h1>

<p>I have a bunch of python files. Most of them are useful when callable from the command line i.e. they all use argparse and <code>if __name__ == "__main__"</code> to do useful things.</p>

<p>Currently I have this directory structure, and everything is working fine:</p>

<pre><code>.
├── config.txt
├── docs/
│   ├── ...
├── my_tool.py
├── a.py
├── b.py
├── c.py
├── d.py
├── e.py
├── README.md
├── tests
│   ├── __init__.py
│   ├── a.py
│   ├── b.py
│   ├── c.py
│   ├── d.py
│   └── e.py
└── resources
    ├── ...
</code></pre>

<p>Some of the scripts <code>import</code> things from other scripts to do their work. But no script is merely a library, they are all invokable. e.g. I could invoke <code>./my_tool.py</code>, <code>./a.by</code>, <code>./b.py</code>, <code>./c.py</code> etc and they would do useful things for the user. </p>

<p>"my_tool.py" is the main script that leverages all of the other scripts.</p>

<h1>What I want to happen</h1>

<p>However I want to change the way the project is organised. The project itself represents an entire program useable by the user, and will be distributed as such, but I know that parts of it will be useful in different projects later so I want to try and encapsulate the current files into a package. In the immediate future I will also add other packages to this same project. </p>

<p>To facilitate this I've decided to re-organise the project to something like the following:</p>

<pre><code>.
├── config.txt
├── docs/
│   ├── ...
├── my_tool
│   ├── __init__.py
│   ├── my_tool.py
│   ├── a.py
│   ├── b.py
│   ├── c.py
│   ├── d.py
│   ├── e.py
│   └── tests
│       ├── __init__.py
│       ├── a.py
│       ├── b.py
│       ├── c.py
│       ├── d.py
│       └── e.py
├── package2
│   ├── __init__.py
│   ├── my_second_package.py
|   ├── ...
├── README.md
└── resources
    ├── ...
</code></pre>

<p>However, I can't figure out an project organisation that satisfies the following criteria:</p>

<ol>
<li>All of the scripts are invokable on the command line (either as <code>my_tool\a.py</code> or <code>cd my_tool &amp;&amp; a.py</code>)</li>
<li>The tests actually run :) </li>
<li>Files in package2 can do <code>import my_tool</code></li>
</ol>

<p>The main problem is with the import statements used by the packages and the tests.</p>

<p>Currently, all of the packages, including the tests, simply do <code>import &lt;module&gt;</code> and it's resolved correctly.  But when jiggering things around it doesn't work.</p>

<p>Note that supporting py2.7 is a requirement so all of the files have <code>from __future__ import absolute_import, ...</code> at the top.</p>

<h1>What I've tried, and the disastrous results</h1>

<h2>1</h2>

<p>If I move the files around as shown above, but leave all of the import statements as they currently are:</p>

<ol>
<li><code>$ ./my_tool/*.py</code> works and they all run properly</li>
<li><code>$ nosetests</code> run from the top directory doesn't work. The tests fail to import the packages scripts.</li>
<li>pycharm highlights import statements in red when editing those files :(</li>
</ol>

<h2>2</h2>

<p>If I then change the test scripts to do:</p>

<pre><code>from my_tool import x
</code></pre>

<ol>
<li><code>$ ./my_tool/*.py</code> still works and they all run properly</li>
<li><code>$ nosetests</code> run from the top directory doesn't work. Then tests can import the correct scripts, but the imports in the scripts themselves fail when the test scripts import them. </li>
<li>pycharm highlights import statements in red in the main scripts still :(</li>
</ol>

<h2>3</h2>

<p>If I keep the same structure and change <em>everything</em> to be <code>from my_tool import</code> then:</p>

<ol>
<li><code>$ ./my_tool/*.py</code> results in <code>ImportError</code>s </li>
<li><code>$ nosetests</code> runs everything ok. </li>
<li>pycharm doesn't complain about anything </li>
</ol>

<p>e.g. of 1.:</p>

<pre><code>Traceback (most recent call last):
  File "./my_tool/a.py", line 34, in &lt;module&gt;
    from my_tool import b
ImportError: cannot import name b
</code></pre>

<h2>4</h2>

<p>I also tried <code>from . import x</code> but that just ends up with <code>ValueError: Attempted relative import in non-package</code> for the direct running of scripts.</p>

<h1>Looking at some other SO answers:</h1>

<p>I can't just use <code>python -m pkg.tests.core_test</code> as</p>

<p>a) I don't have <strong>main</strong>.py. I guess I could have one?<br>
b) I want to be able to run all of the scripts, not just main?</p>

<p>I've tried:</p>

<pre><code>if __name__ == '__main__' and __package__ is None:
    from os import sys, path
    sys.path.append(path.dirname(path.dirname(path.abspath(__file__))))
</code></pre>

<p>but it didn't help. </p>

<p>I also tried:</p>

<pre><code>__package__ = "my_tool"
from . import b
</code></pre>

<p>But received:</p>

<pre><code>SystemError: Parent module 'loading_tool' not loaded, cannot perform relative import
</code></pre>

<p>adding <code>import my_tool</code> before <code>from . import b</code> just ends up back with <code>ImportError: cannot import name b</code></p>

<h2>Fix?</h2>

<p>What's the correct set of magical incantations and directory layout to make all of this work?</p>