How to distribute files in a Python sdist that are not VCS tracked?
<p>I would like to find the proper way to include files in a python sdist that are not tracked by git.</p>

<h1>Context</h1>

<p>The <code>.mo</code> files from my project are not tracked by <code>git</code> (like some other <code>.txt</code> files that need to be created at install time).</p>

<p>I have written a small function in <code>setup.py</code> to create them at install time, that I call in <code>setup()</code>:</p>

<pre><code>setup(
    .
    .
    .
    data_files=create_extra_files(),
    include_package_data=True,
    .
    .
    .
)
</code></pre>

<p>Note that they should belong to <code>data_dir</code> because <a href="https://docs.python.org/3.4/distutils/setupscript.html#installing-additional-files" rel="noreferrer">the documentation says</a>:</p>

<blockquote>
  <p>The data_files option can be used to specify additional files needed
  by the module distribution: configuration files, <strong>message catalogs</strong>,
  data files, anything which doesn’t fit in the previous categories.</p>
</blockquote>

<p>So, this works well with <code>python3 setup.py install</code> (and <code>bdist</code> too). The <code>.mo</code> files are generated and stored at the right place.</p>

<p>But if I want it to work with <code>sdist</code>, then I must include them in <code>MANIFEST.in</code> (e.g. <code>recursive-include mathmaker *.mo</code>). <a href="https://docs.python.org/3.4/distutils/setupscript.html#installing-additional-files" rel="noreferrer">Documentation says indeed</a>:</p>

<blockquote>
  <p>Changed in version 3.1:  All the files that match data_files will be added to the MANIFEST file
  <strong>if no template is provided</strong>. See <a href="https://docs.python.org/3.4/distutils/sourcedist.html#manifest" rel="noreferrer">Specifying the files to distribute</a>.</p>
</blockquote>

<p>(The link doesn't help much).</p>

<p>I am reluctant to include <code>*.mo</code> files in <code>MANIFEST.in</code> as they are not tracked by git. And <a href="https://pypi.python.org/pypi/check-manifest" rel="noreferrer">check-manifest</a> doesn't like this kind of situation, it complains about the fact that <code>lists of files in version control and sdist do not match!</code></p>

<p>So, is there a way to fix this ugly situation?</p>

<h1>Steps to reproduce the situation</h1>

<h2>Environment and project</h2>

<p>To avoid polluting your environment, create and activate a dedicated virtual environment (python3.4+) in the directory of your choice:</p>



<pre class="lang-none prettyprint-override"><code>$ pyvenv-3.4 v0
$ source v0/bin/activate
(v0) $
</code></pre>

<p>Reproduce following tree in a <code>project0</code> directory:</p>

<pre class="lang-none prettyprint-override"><code>.
├── .gitignore
├── MANIFEST.in
├── README.rst
├── setup.py
└── project0
    ├── __init__.py
    ├── main.py
    └── data
        └── dummy_versioned.po
</code></pre>

<p>Where <code>README.rst</code>, <code>__init__.py</code> and <code>dummy_versioned.po</code> are empty. </p>

<p>Content of the other files:</p>

<ul>
<li><p><code>.gitignore</code>:</p>

<pre class="lang-none prettyprint-override"><code>build/
dist/
*.egg-info
project0/data/*.txt
*~
</code></pre></li>
<li><p><code>MANIFEST.in</code>:</p>

<pre class="lang-none prettyprint-override"><code>recursive-include project0 *.po
recursive-include project0 *.txt
</code></pre></li>
<li><p><code>main.py</code>:</p>

<pre class="lang-py prettyprint-override"><code>#!/usr/bin/env python3
# -*- coding: utf-8 -*-


def entry_point():
    with open('project0/data/a_file.txt', mode='rt') as f:
        print(f.read())
</code></pre></li>
<li><p><code>setup.py</code>:</p>

<pre class="lang-py prettyprint-override"><code>#!/usr/bin/env python3
# -*- coding: utf-8 -*-

import platform
from setuptools import setup, find_packages


def create_files():
    txt_file_path = 'project0/data/a_file.txt'
    with open(txt_file_path, mode='w+') as f:
        f.write("Some dummy platform information: " + platform.platform())
    return [('project0/data', [txt_file_path])]


setup(
    name='project0',
    version='0.0.1',
    author='J. Doe',
    author_email='j.doe@someprovider.com',
    url='http://myproject.url',
    packages=find_packages(),
    data_files=create_files(),
    include_package_data=True,
    entry_points={
        'console_scripts': ['myscript0 = project0.main:entry_point'],
    }
)
</code></pre></li>
</ul>

<p>Start a local <code>git</code> repo:</p>

<pre class="lang-none prettyprint-override"><code>(v0) $ git init
(v0) $ git add .
</code></pre>

<p>Install <code>check-manifest</code>:</p>

<pre class="lang-none prettyprint-override"><code>(v0) $ pip3 install check-manifest
</code></pre>

<h2>Install and test</h2>

<p><code>install</code> works:</p>

<pre class="lang-none prettyprint-override"><code>(v0) $ python3 setup.py install
.
.
.
copying project0/data/a_file.txt -&gt; build/lib/project0/data
.
.
.
Finished processing dependencies for project0==0.0.1
(v0) $ myscript0 
Some dummy platform information: Linux-3.16.0-29-generic-x86_64-with-Ubuntu-14.04-trusty
</code></pre>

<p>If you <code>rm project0/data/a_file.txt</code>, then <code>myscript0</code> doesn't work anymore, but reinstall it and it works again, as expected.</p>

<p>Building the sdist also includes <code>a_file.txt</code>:</p>

<pre class="lang-none prettyprint-override"><code>(v0) $ python3 setup.py sdist
.
.
.
hard linking project0/data/a_file.txt -&gt; project0-0.0.1/project0/data
.
.
.
</code></pre>

<p>Note that to have this file included in the sdist, it looks necessary (as explained in the "context" part below) to have <code>recursive-include project0 *.txt</code> in <code>MANIFEST.in</code>. Would you remove this line, <code>python3 setup.py sdist</code> woudln't mention <code>a_file.txt</code> any more (do not forget to remove any previous <code>build/</code> or <code>dist/</code> directories to observe this).</p>

<h2>Conclusion</h2>

<p>So, everything works as it is, but there is this discrepancy: <code>a_file.txt</code> is not tracked by <code>git</code>, but is included in <code>MANIFEST.in</code>.</p>

<p><code>check-manifest</code> tells clearly:</p>

<pre class="lang-none prettyprint-override"><code>lists of files in version control and sdist do not match!
missing from VCS:
  project0/data/a_file.txt
</code></pre>

<p>So, is there a proper way to handle this situation?</p>