{"action":"created","issue":{"url":"https://api.github.com/repos/meteor/meteor/issues/2668","labels_url":"https://api.github.com/repos/meteor/meteor/issues/2668/labels{/name}","comments_url":"https://api.github.com/repos/meteor/meteor/issues/2668/comments","events_url":"https://api.github.com/repos/meteor/meteor/issues/2668/events","html_url":"https://github.com/meteor/meteor/issues/2668","id":43555138,"number":2668,"title":"Large number of operations hangs server","user":{"login":"davidworkman9","id":1152186,"avatar_url":"https://avatars.githubusercontent.com/u/1152186?v=3","gravatar_id":"","url":"https://api.github.com/users/davidworkman9","html_url":"https://github.com/davidworkman9","followers_url":"https://api.github.com/users/davidworkman9/followers","following_url":"https://api.github.com/users/davidworkman9/following{/other_user}","gists_url":"https://api.github.com/users/davidworkman9/gists{/gist_id}","starred_url":"https://api.github.com/users/davidworkman9/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidworkman9/subscriptions","organizations_url":"https://api.github.com/users/davidworkman9/orgs","repos_url":"https://api.github.com/users/davidworkman9/repos","events_url":"https://api.github.com/users/davidworkman9/events{/privacy}","received_events_url":"https://api.github.com/users/davidworkman9/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/meteor/meteor/labels/Project:Livequery","name":"Project:Livequery","color":"006b75"}],"state":"closed","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/meteor/meteor/milestones/4","html_url":"https://github.com/meteor/meteor/milestones/Winter%202015%20Mongo%20Maintenance","labels_url":"https://api.github.com/repos/meteor/meteor/milestones/4/labels","id":944811,"number":4,"title":"Winter 2015 Mongo Maintenance","description":"","creator":{"login":"glasser","id":16724,"avatar_url":"https://avatars.githubusercontent.com/u/16724?v=3","gravatar_id":"","url":"https://api.github.com/users/glasser","html_url":"https://github.com/glasser","followers_url":"https://api.github.com/users/glasser/followers","following_url":"https://api.github.com/users/glasser/following{/other_user}","gists_url":"https://api.github.com/users/glasser/gists{/gist_id}","starred_url":"https://api.github.com/users/glasser/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/glasser/subscriptions","organizations_url":"https://api.github.com/users/glasser/orgs","repos_url":"https://api.github.com/users/glasser/repos","events_url":"https://api.github.com/users/glasser/events{/privacy}","received_events_url":"https://api.github.com/users/glasser/received_events","type":"User","site_admin":false},"open_issues":6,"closed_issues":19,"state":"open","created_at":"2015-01-21T06:32:09Z","updated_at":"2015-03-06T04:38:52Z","due_on":null,"closed_at":null},"comments":25,"created_at":"2014-09-22T21:22:26Z","updated_at":"2015-03-11T21:14:55Z","closed_at":"2015-02-10T20:17:01Z","body":"Originally discussed [here](https://groups.google.com/forum/#!topic/meteor-talk/Y547Hh2z39Y).\r\n\r\nAs our application data and usage has begun to scale we've run into an issue where a large number of MongoDB operations performed at one time will lock our server entirely for extended periods.  The application becomes non-responsive to all users. What seems to be happening is a few thousand operations go into the oplog and Meteor struggles with the workload to determine which of those operations need to be passed down to the client.  This could become a deal breaker for using Meteor with a large class of really useful applications unless addressed.\r\n\r\nFor our app this is unavoidable; no schema redesign will save us. There are tasks that need to affect huge numbers of MongoDB documents. Today it's 5,000 and up to a dozen users, but in the coming months it could easily be 100,000 documents and hundreds of users.  Given what we are seeing with today's performance we know we are going to be in trouble.\r\n\r\nOur solution for the immediate future is to turn oplog tailing off and revert to the original polling approach. With oplog tailing off I can do thousands of operations (directly to Mongo, not through a Meteor.Collection) and the server doesn't DDOS's itself. But we lose the benefit of the oplog tailor.\r\n\r\nWe wonder if the following (or something else that accomplishes the same goal) could be built into Meteor core somehow:\r\n* Provide the ability to turn oplog tailing off\r\n* Then we could execute our big query\r\n* Turn oplog tailing back on, which would invoke a poll and diff of all the subscriptions to get the missed changes"},"comment":{"url":"https://api.github.com/repos/meteor/meteor/issues/comments/78373403","html_url":"https://github.com/meteor/meteor/issues/2668#issuecomment-78373403","issue_url":"https://api.github.com/repos/meteor/meteor/issues/2668","id":78373403,"user":{"login":"glasser","id":16724,"avatar_url":"https://avatars.githubusercontent.com/u/16724?v=3","gravatar_id":"","url":"https://api.github.com/users/glasser","html_url":"https://github.com/glasser","followers_url":"https://api.github.com/users/glasser/followers","following_url":"https://api.github.com/users/glasser/following{/other_user}","gists_url":"https://api.github.com/users/glasser/gists{/gist_id}","starred_url":"https://api.github.com/users/glasser/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/glasser/subscriptions","organizations_url":"https://api.github.com/users/glasser/orgs","repos_url":"https://api.github.com/users/glasser/repos","events_url":"https://api.github.com/users/glasser/events{/privacy}","received_events_url":"https://api.github.com/users/glasser/received_events","type":"User","site_admin":false},"created_at":"2015-03-11T21:14:55Z","updated_at":"2015-03-11T21:14:55Z","body":"@mizzao I'd be happy to hear if 1.0.4 helps your apps!"}}