{"action":"created","issue":{"url":"https://api.github.com/repos/timoxley/linklocal/issues/14","labels_url":"https://api.github.com/repos/timoxley/linklocal/issues/14/labels{/name}","comments_url":"https://api.github.com/repos/timoxley/linklocal/issues/14/comments","events_url":"https://api.github.com/repos/timoxley/linklocal/issues/14/events","html_url":"https://github.com/timoxley/linklocal/issues/14","id":53221277,"number":14,"title":"strategies for installing dependencies of local dependencies","user":{"login":"timoxley","id":43438,"avatar_url":"https://avatars.githubusercontent.com/u/43438?v=3","gravatar_id":"","url":"https://api.github.com/users/timoxley","html_url":"https://github.com/timoxley","followers_url":"https://api.github.com/users/timoxley/followers","following_url":"https://api.github.com/users/timoxley/following{/other_user}","gists_url":"https://api.github.com/users/timoxley/gists{/gist_id}","starred_url":"https://api.github.com/users/timoxley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/timoxley/subscriptions","organizations_url":"https://api.github.com/users/timoxley/orgs","repos_url":"https://api.github.com/users/timoxley/repos","events_url":"https://api.github.com/users/timoxley/events{/privacy}","received_events_url":"https://api.github.com/users/timoxley/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2015-01-01T14:57:32Z","updated_at":"2015-11-23T13:44:00Z","closed_at":null,"body":"## Scenario\r\n* `parent-pkg` has local dependency: `file: ./local-pkg`.\r\n* `npm install` in `parent-pkg`.\r\n* `./local-pkg` is copied and installed into `parent_package/node_modules, along with the dependencies of `local-pkg`.\r\n* post-install, run `linklocal`.\r\n* `node_modules/local-pkg` is now replaced with a relative symlink to `../local-pkg`. Dependencies of `local-pkg` are not installed.\r\n* Use `linklocal list` to produce a list of all local dependencies, which we then pass to something like `xargs`, `find -exec` or [bulk](https://github.com/timoxley/bulk) to run `npm install --production` for us: `linklocal list | bulk -c \"npm install --production\"`.\r\n* Local dependencies have now got their dependencies installed.\r\n\r\n## Issues\r\n1. Dependencies of local dependencies were installed twice. First when the local dependencies were installed with the regular `npm install`, second during the bulk `npm install` in the local dependencies. This can lead to considerable increases in install time if you depend on anything that requires building.\r\n2. No deduplication occurs between local dependencies. If two local packages depend on the same thing, it will be installed twice.\r\n\r\n## Some Improvements\r\n\r\n### Don't hit remote registry\r\nDisable accessing the remote registry with `npm install --cache-min=Infinity` for the bulk install. You've already just installed everything you need so there's a good chance everything is already in the cache. This prevents overhead of hitting npm servers over and over for every local dependency. Doesn't solve rebuilding over and over again though.\r\n\r\n## Don't install local dependencies then symlink them away\r\nTo avoid the initial install of everything before symlinking it all away, I've been doing `linklocal` in the preinstall script. Of course this requires linklocal and bulk to be installed before starting, so my preinstall (Makefile) looks something like:\r\n\r\n```\r\npreinstall:\r\n    npm install linklocal bulk # this unfortunately ignores the version specified in package.json\r\n    linklocal link -r\r\n    linklocal list -r --unique | bulk -c \"npm install --cache-min=Infinity\"\r\n```\r\n\r\nbut what's really killing install time is rebuilding dependencies over and over due to lack of deduplication.\r\n\r\ncc @hughsk @yoshuawuyts any ideas on what we can do to improve this?"},"comment":{"url":"https://api.github.com/repos/timoxley/linklocal/issues/comments/158936114","html_url":"https://github.com/timoxley/linklocal/issues/14#issuecomment-158936114","issue_url":"https://api.github.com/repos/timoxley/linklocal/issues/14","id":158936114,"user":{"login":"jbach","id":109169,"avatar_url":"https://avatars.githubusercontent.com/u/109169?v=3","gravatar_id":"","url":"https://api.github.com/users/jbach","html_url":"https://github.com/jbach","followers_url":"https://api.github.com/users/jbach/followers","following_url":"https://api.github.com/users/jbach/following{/other_user}","gists_url":"https://api.github.com/users/jbach/gists{/gist_id}","starred_url":"https://api.github.com/users/jbach/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jbach/subscriptions","organizations_url":"https://api.github.com/users/jbach/orgs","repos_url":"https://api.github.com/users/jbach/repos","events_url":"https://api.github.com/users/jbach/events{/privacy}","received_events_url":"https://api.github.com/users/jbach/received_events","type":"User","site_admin":false},"created_at":"2015-11-23T13:44:00Z","updated_at":"2015-11-23T13:44:00Z","body":"Old issue, but still: I think the \"stealing\" approach is the way to go. Considering npm3’s behaviour, `node_modules/local-pkg/node_modules` should only include module versions that are specific to `local-pkg`, thus not in the flat hierarch. Copying the `node_modules/local-pkg/node_modules` folder to `./local-pkg/node_modules` before symlinking works, at least in npm3. I’m currently doing exactly that by script but would love to replace it with `linklocal`."}}