Using a shared npm node_modules/ for multiple workspaces on the same server
<p>Our Jenkins/CI server runs hundreds of builds a day for our node/js project, and I would like to be able to run each build in a completely clean workspace. However, the <code>npm install</code> step can take >10 minutes, which is just far too slow. Instead, as our npm dependencies only change for a small number of builds (approx 1% of all builds), I'd like to only run <code>npm install</code> once every time <code>npm-shrinkwrap.json</code> changes (check md5sum at every build). If the shrinkwrap file hasn't changed, use a cached <code>node_modules/</code> directory.</p>

<p>This plan works well enough if I copy the cached <code>node_modules/</code>, however even that operation can take up to a minute. To further optimize our build times, I would like to be able to symlink to the cached <code>node_modules/</code>, which should drastically improve our overall build performance.</p>

<pre><code>ln -s /path/to/cache/ /path/to/workspace/node_modules
</code></pre>

<p>However, simply symlinking to the path of the cache doesn't work in a case where a dependency exists at multiple levels of the dependency tree. As an example, our top level project depends on both <code>gulp</code> and <code>gulp-util</code>. The top level dependency also depends on <code>gulp-util</code>. After an <code>npm install</code>, <code>gulp-util</code> will be install in the top level <code>node_modules/</code> but not in <code>node_modules/gulp/node_modules</code>.</p>

<p>If the dependencies exist in the local workspace (i.e. a real directory <code>/path/to/workspace/node_modules/</code>), then any instance of <code>require('gulp-util')</code> within <code>node_modules/gulp</code> will (I think) recurse up the dependency tree until it finds an appropriate <code>gulp-util</code> module. That is, it starts off looking in <code>/path/to/workspace/node_modules/gulp/node_modules/gulp-util</code>, doesn't find anything, then looks in <code>/path/to/workspace/node_modules/gulp-util</code>, finds an appropriate module, imports it and moves on.</p>

<p>However, when this is a symlink I'll get an error like:</p>

<pre><code>module.js:339
    throw err;
    ^

Error: Cannot find module 'gulp-util'
    at Function.Module._resolveFilename (module.js:337:15)
    at Function.Module._load (module.js:287:25)
    at Module.require (module.js:366:17)
    at require (module.js:385:17)
    at Object.&lt;anonymous&gt; (/path/to/cache/gulp/index.js:4:15)
    at Module._compile (module.js:435:26)
    at Object.Module._extensions..js (module.js:442:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:311:12)
    at Module.require (module.js:366:17)
    at require (module.js:385:17)
</code></pre>

<p>I assume that this tries to do the same as the other version, but I can't see why it fails to find <code>gulp-util</code>. Whether it looks in <code>/path/to/workspace/node_modules/gulp-util</code> or <code>/path/to/cache/gulp-util</code>, it should find the module and be able to import it.</p>

<p>I've tried resolving this by manually installing the module <code>gulp/node_modules/gulp-util</code>, but I encounter dozens of such errors and dealing with this manually on a build server is unfeasible. Writing some code to search for dependencies of this type and installing them is possible, but it feels like the wrong thing to do.</p>

<p>npm must have some way of supporting such a workflow, right? Am I missing something obvious? Have I glossed over something in the documentation?</p>