{"action":"created","issue":{"url":"https://api.github.com/repos/npm/npm/issues/6016","labels_url":"https://api.github.com/repos/npm/npm/issues/6016/labels{/name}","comments_url":"https://api.github.com/repos/npm/npm/issues/6016/comments","events_url":"https://api.github.com/repos/npm/npm/issues/6016/events","html_url":"https://github.com/npm/npm/issues/6016","id":40898893,"number":6016,"title":"npm.commands.search fails and dumps 100+ Mb to output log","user":{"login":"outbounder","id":125437,"avatar_url":"https://avatars.githubusercontent.com/u/125437?v=3","gravatar_id":"","url":"https://api.github.com/users/outbounder","html_url":"https://github.com/outbounder","followers_url":"https://api.github.com/users/outbounder/followers","following_url":"https://api.github.com/users/outbounder/following{/other_user}","gists_url":"https://api.github.com/users/outbounder/gists{/gist_id}","starred_url":"https://api.github.com/users/outbounder/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/outbounder/subscriptions","organizations_url":"https://api.github.com/users/outbounder/orgs","repos_url":"https://api.github.com/users/outbounder/repos","events_url":"https://api.github.com/users/outbounder/events{/privacy}","received_events_url":"https://api.github.com/users/outbounder/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/npm/npm/labels/bug","name":"bug","color":"C40233"},{"url":"https://api.github.com/repos/npm/npm/labels/next-minor","name":"next-minor","color":"FF00FF"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":36,"created_at":"2014-08-22T11:04:08Z","updated_at":"2015-05-23T17:36:24Z","closed_at":null,"body":"Hi npm team,\r\n\r\nI've been playing with the programmatic api of npm, namely `npm.commands.search`.\r\n\r\nYesterday I've left a script based on that api command to run for some time and observed interesting behavior which I though might help:\r\n\r\n1. here is the [source code](https://github.com/node-organic/node-organic.com/blob/ecf584c0c66a4c33c77a74de9f5dadeec906b398/plasma/npm-organic-modules.js) which produces the odd behavior/results\r\n2. I've inspected the produced output log and it contains the following content:\r\n\r\n```\r\nError: Unexpected end of input\r\n...\r\n... a very very long json dump 138+Mb ...\r\n...\r\nsm.git\"},\"author\":{\"name\":\"Carsten Nielsen\",\"email\":\" <-- email intentionally removed -->\"},\"bugs\":{\"url\":\"https://github.com/heycarsten/ember-fsm/issues\"},\"license\":\"MIT\",\"readmeFilename\":\"README.md\",\"time\":{\"modified\":\"2014-06-02T19:10:37.452Z\"},\"versions\":{\"0.3.0\":\"latest\"}}\r\n        at Object.parse (native)\r\n        at RegClient.<anonymous> (/home/node/node-organic.com/node_modules/npm/node_modules/npm-registry-client/lib/request.js:266:23)\r\n        at Request._callback (/home/node/node-organic.com/node_modules/npm/node_modules/npm-registry-client/lib/request.js:238:65)\r\n        at Request.self.callback (/home/node/node-organic.com/node_modules/npm/node_modules/request/request.js:123:22)\r\n        at Request.emit (events.js:98:17)\r\n        at Request.<anonymous> (/home/node/node-organic.com/node_modules/npm/node_modules/request/request.js:893:14)\r\n        at Request.emit (events.js:117:20)\r\n        at IncomingMessage.<anonymous> (/home/node/node-organic.com/node_modules/npm/node_modules/request/request.js:844:12)\r\n        at IncomingMessage.emit (events.js:117:20)\r\n        at _stream_readable.js:938:16\r\n```\r\n\r\nHere are couple of observational notes of mine in addition :\r\n\r\n* the same source code runs on my local workstation without any errors. I guess that is because of cache which seems not present on the remote server :? \r\n* the source code atm is modified to run `npm.commands.search` only once and the issue is found everytime. However yesterday it was working just fine within reasonable comamnd exeuction time of 8-10s and refresh interval of 5 mins. Today I've disabled the refresh interval as to prevent any further complications in regards to debugging it :)\r\n\r\n\r\nHaving the above said do you think that using `npm.commands.search` to acquire metadata for specific `set` of npm published modules is good idea?\r\n\r\nI've introduced myself today to gulpjs plugins website implementation also but sadly I think that running java and elasticsearch for just querying 45 modules in total for the moment seems a little bit too heavy :?"},"comment":{"url":"https://api.github.com/repos/npm/npm/issues/comments/104928908","html_url":"https://github.com/npm/npm/issues/6016#issuecomment-104928908","issue_url":"https://api.github.com/repos/npm/npm/issues/6016","id":104928908,"user":{"login":"maxpolk","id":389424,"avatar_url":"https://avatars.githubusercontent.com/u/389424?v=3","gravatar_id":"","url":"https://api.github.com/users/maxpolk","html_url":"https://github.com/maxpolk","followers_url":"https://api.github.com/users/maxpolk/followers","following_url":"https://api.github.com/users/maxpolk/following{/other_user}","gists_url":"https://api.github.com/users/maxpolk/gists{/gist_id}","starred_url":"https://api.github.com/users/maxpolk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/maxpolk/subscriptions","organizations_url":"https://api.github.com/users/maxpolk/orgs","repos_url":"https://api.github.com/users/maxpolk/repos","events_url":"https://api.github.com/users/maxpolk/events{/privacy}","received_events_url":"https://api.github.com/users/maxpolk/received_events","type":"User","site_admin":false},"created_at":"2015-05-23T17:36:24Z","updated_at":"2015-05-23T17:36:24Z","body":"On a 512MB mem, 20GB SSD drive VPS, new Ubuntu install of \"nodejs\" and \"npm\", first use of \"npm search backbone\" failed until I jacked up the swap file size:\r\n(1) command took 20 minutes; (2) about 1000 lines of output; (3) peak swap usage 1.81GB; (4) network bandwidth reasonably small; (5) high CPU usage for the full 20 minutes; (6) high disk I/O averaged 3 to 10 MB/sec continuously probably thrashing; (7) the swapfile expanded to permit \"npm search\" to complete now takes up 1/7 of the whole drive.\r\nThe space and time complexity is enmeshed in finite-space systems when the algorithm exceeds available memory, because swapping begins to occur, which means slow disk I/O, even on the SSD drive."}}