{"action":"created","issue":{"url":"https://api.github.com/repos/npm/npm/issues/7153","labels_url":"https://api.github.com/repos/npm/npm/issues/7153/labels{/name}","comments_url":"https://api.github.com/repos/npm/npm/issues/7153/comments","events_url":"https://api.github.com/repos/npm/npm/issues/7153/events","html_url":"https://github.com/npm/npm/issues/7153","id":54500089,"number":7153,"title":"Revamp npm search / local index caching etc.","user":{"login":"silkentrance","id":6068824,"avatar_url":"https://avatars.githubusercontent.com/u/6068824?v=3","gravatar_id":"","url":"https://api.github.com/users/silkentrance","html_url":"https://github.com/silkentrance","followers_url":"https://api.github.com/users/silkentrance/followers","following_url":"https://api.github.com/users/silkentrance/following{/other_user}","gists_url":"https://api.github.com/users/silkentrance/gists{/gist_id}","starred_url":"https://api.github.com/users/silkentrance/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/silkentrance/subscriptions","organizations_url":"https://api.github.com/users/silkentrance/orgs","repos_url":"https://api.github.com/users/silkentrance/repos","events_url":"https://api.github.com/users/silkentrance/events{/privacy}","received_events_url":"https://api.github.com/users/silkentrance/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/npm/npm/labels/feature-request","name":"feature-request","color":"ACE1AF"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":16,"created_at":"2015-01-15T20:31:01Z","updated_at":"2015-01-16T02:19:00Z","closed_at":"2015-01-16T00:56:40Z","body":"Since npm search is rather memory hungry on the client as it first downloads the whole package index into memory and from there instantiates the JSON object, I think that there is a need for redoing both the client side and the server side handling of the package index.\r\n\r\nI guess that on the server side there is a single couchdb view that will generate the package index. This needs to be changed. Each package index needs to be associated with a timestamp. So whenever one publishes or otherwise causes an update to the index, a new index will be written with a different timestamp.\r\n\r\nOr something similar to that.\r\n\r\nNext, on the client side, whenever building an index for the first time, the index downloaded from the server will have a timestamp and of course the actual data. This index will be streamed directly to the filesystem instead of streaming it into memory. In case of the download being aborted, the system can then call upon the service, requesting the parts of the index that have not been downloaded yet for the timestamp that the index was registered with the service and which is also known to the client.\r\n\r\nThat timestamp could also be the filename of the file to which the index (update) is being streamed to, e.g. ~/.npm/update/1234567890.part etc.\r\n\r\nThe client on the other hand will store the index data in a local sqlite3 database by reading the cached JSON index file using a streaming/event based parser to cut down on overall memory usage.\r\n\r\nThe rationale for using sqlite3 instead of couchdb would be that it can be embedded more easily and is readily available for a multitude of platforms. And, looking at `npm-search`, it uses a Google inspired key-value store, which may or may not be as well supported as sqlite3.\r\n\r\nThe sql schema for that database would be so that it allows npm to quickly search through that index and also update that index incrementally with new data received from the service, e.g.\r\n\r\n```\r\ntable system\r\n   last_update int\r\n\r\ntable package\r\n   name text\r\n\r\n   unique name\r\n\r\ntable version\r\n   package text\r\n   version text\r\n   summary text\r\n   description text\r\n\r\n   unique package, version\r\n\r\nand so on\r\n```\r\n\r\nFeel free to comment on this in order to make npm a better experience for everyone."},"comment":{"url":"https://api.github.com/repos/npm/npm/issues/comments/70198877","html_url":"https://github.com/npm/npm/issues/7153#issuecomment-70198877","issue_url":"https://api.github.com/repos/npm/npm/issues/7153","id":70198877,"user":{"login":"othiym23","id":418097,"avatar_url":"https://avatars.githubusercontent.com/u/418097?v=3","gravatar_id":"","url":"https://api.github.com/users/othiym23","html_url":"https://github.com/othiym23","followers_url":"https://api.github.com/users/othiym23/followers","following_url":"https://api.github.com/users/othiym23/following{/other_user}","gists_url":"https://api.github.com/users/othiym23/gists{/gist_id}","starred_url":"https://api.github.com/users/othiym23/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/othiym23/subscriptions","organizations_url":"https://api.github.com/users/othiym23/orgs","repos_url":"https://api.github.com/users/othiym23/repos","events_url":"https://api.github.com/users/othiym23/events{/privacy}","received_events_url":"https://api.github.com/users/othiym23/received_events","type":"User","site_admin":false},"created_at":"2015-01-16T02:19:00Z","updated_at":"2015-01-16T02:19:00Z","body":"> I have discovered that hashing names and having 100 different data files is actually **worse** performance than just reading the whole damn JSON file into memory.  So I have to revert `pure-fts` one step.\r\n\r\nRemember that one of the sad-clown.gif aspects of the current search story is that it's pretty much impossible to use `npm search` in memory-constrained environments like the Raspberry Pi or mobile. I think some query time loss is acceptable if it keeps the runtime memory consumption of the search process under, say, 128MiB.\r\n\r\n> I also want to do some thinking about `rsync`-able compression, whether that's done by hand (I've done that before, it's not fun, but do-able) or by a pure-js rsync-friendly algorithm (not `zip` or `gz` unfortunately; maybe `xz`?)\r\n\r\nThe last time I played around with these things, I found [`xdelta`](http://en.wikipedia.org/wiki/Xdelta) pretty easy to implement. Granted, I was doing it in OCaml, but still."}}