{"action":"created","issue":{"url":"https://api.github.com/repos/colinmollenhour/Cm_Cache_Backend_Mongo/issues/3","labels_url":"https://api.github.com/repos/colinmollenhour/Cm_Cache_Backend_Mongo/issues/3/labels{/name}","comments_url":"https://api.github.com/repos/colinmollenhour/Cm_Cache_Backend_Mongo/issues/3/comments","events_url":"https://api.github.com/repos/colinmollenhour/Cm_Cache_Backend_Mongo/issues/3/events","html_url":"https://github.com/colinmollenhour/Cm_Cache_Backend_Mongo/issues/3","id":58981993,"number":3,"title":"ensure_indexes seems to make things slower","user":{"login":"kschroeder","id":237332,"avatar_url":"https://avatars.githubusercontent.com/u/237332?v=3","gravatar_id":"","url":"https://api.github.com/users/kschroeder","html_url":"https://github.com/kschroeder","followers_url":"https://api.github.com/users/kschroeder/followers","following_url":"https://api.github.com/users/kschroeder/following{/other_user}","gists_url":"https://api.github.com/users/kschroeder/gists{/gist_id}","starred_url":"https://api.github.com/users/kschroeder/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kschroeder/subscriptions","organizations_url":"https://api.github.com/users/kschroeder/orgs","repos_url":"https://api.github.com/users/kschroeder/repos","events_url":"https://api.github.com/users/kschroeder/events{/privacy}","received_events_url":"https://api.github.com/users/kschroeder/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-02-25T22:40:37Z","updated_at":"2015-02-26T23:14:29Z","closed_at":null,"body":"One of the tests I've done on this shows that the default setting for ensure_indexes makes general load significantly slower during certain remove() operations.  Following is a chart from a single-threaded load test that simulates reads and writes at about a 10:1 ratio.\r\n![Mongo with ensure_index turned on](http://content.screencast.com/users/KevinSchroeder/folders/Jing/media/85a526d4-af55-4006-a85b-ea2e5b704833/2015-02-25_1635.png)\r\n\r\nIf I turn ensure_index off the response times are much improved. (note the Y axis)\r\n\r\n![Mongo with ensure_index turned off](http://content.screencast.com/users/KevinSchroeder/folders/Jing/media/8fa5b63d-cfad-4fdb-89e8-5bcd102fdce8/2015-02-25_1628.png)\r\n\r\nObviously this is a simple configuration item to note but it would probably make sense to make the least prohibitive option the default."},"comment":{"url":"https://api.github.com/repos/colinmollenhour/Cm_Cache_Backend_Mongo/issues/comments/76296302","html_url":"https://github.com/colinmollenhour/Cm_Cache_Backend_Mongo/issues/3#issuecomment-76296302","issue_url":"https://api.github.com/repos/colinmollenhour/Cm_Cache_Backend_Mongo/issues/3","id":76296302,"user":{"login":"colinmollenhour","id":38738,"avatar_url":"https://avatars.githubusercontent.com/u/38738?v=3","gravatar_id":"","url":"https://api.github.com/users/colinmollenhour","html_url":"https://github.com/colinmollenhour","followers_url":"https://api.github.com/users/colinmollenhour/followers","following_url":"https://api.github.com/users/colinmollenhour/following{/other_user}","gists_url":"https://api.github.com/users/colinmollenhour/gists{/gist_id}","starred_url":"https://api.github.com/users/colinmollenhour/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/colinmollenhour/subscriptions","organizations_url":"https://api.github.com/users/colinmollenhour/orgs","repos_url":"https://api.github.com/users/colinmollenhour/repos","events_url":"https://api.github.com/users/colinmollenhour/events{/privacy}","received_events_url":"https://api.github.com/users/colinmollenhour/received_events","type":"User","site_admin":false},"created_at":"2015-02-26T23:14:29Z","updated_at":"2015-02-26T23:14:29Z","body":"Yes, the id already has an index so lookups are not affected. However, the index on the expiration is required for proper automatic expiration. The index on tags could be omitted, but I think as the dataset grows it would get to the point that this index is more useful. I would venture to guess that this tipping point is probably when the full dataset will no longer fit in memory.\r\n\r\nThanks for sharing your results. I'm interested in more details. Perhaps it would be beneficial to split \"ensure_index\" into \"ensure_tags_index\" and \"ensure_expiration_index\"?\r\n\r\nNow I'm thinking the three spikes in your graph must correspond to when the Mongo automatic expiration thread was running. That's pretty terrible performance... I think an external script doing expiration is probably not going to be any faster so I'm guessing the tags index is the performance killer since it is the \"largest\"."}}