{"action":"created","issue":{"url":"https://api.github.com/repos/cloudfoundry-incubator/cf-abacus/issues/69","labels_url":"https://api.github.com/repos/cloudfoundry-incubator/cf-abacus/issues/69/labels{/name}","comments_url":"https://api.github.com/repos/cloudfoundry-incubator/cf-abacus/issues/69/comments","events_url":"https://api.github.com/repos/cloudfoundry-incubator/cf-abacus/issues/69/events","html_url":"https://github.com/cloudfoundry-incubator/cf-abacus/issues/69","id":110159812,"number":69,"title":"Unusually large number of partitions ","user":{"login":"rajkiranrbala","id":493560,"avatar_url":"https://avatars.githubusercontent.com/u/493560?v=3","gravatar_id":"","url":"https://api.github.com/users/rajkiranrbala","html_url":"https://github.com/rajkiranrbala","followers_url":"https://api.github.com/users/rajkiranrbala/followers","following_url":"https://api.github.com/users/rajkiranrbala/following{/other_user}","gists_url":"https://api.github.com/users/rajkiranrbala/gists{/gist_id}","starred_url":"https://api.github.com/users/rajkiranrbala/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rajkiranrbala/subscriptions","organizations_url":"https://api.github.com/users/rajkiranrbala/orgs","repos_url":"https://api.github.com/users/rajkiranrbala/repos","events_url":"https://api.github.com/users/rajkiranrbala/events{/privacy}","received_events_url":"https://api.github.com/users/rajkiranrbala/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/cloudfoundry-incubator/cf-abacus/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-10-07T06:24:07Z","updated_at":"2015-10-07T20:01:58Z","closed_at":null,"body":"I was working on some APIs to query data from accumulated-usage-log database. Queries that took seconds to execute took unusually large amount of time after merging with the latest source. Also the logs showed that many requests failed owing to connection time out. \r\n\r\nFrom the logs I could see unusually large number of partition maps being created. I modified the dbserver to see the number of requests that are being sent to the db while running the demo. The [latest commit](https://github.com/cloudfoundry-incubator/cf-abacus/commit/174a62d5b8a3c0e5d886e732a26bf4db2154334b)  had 773 requests to the db, while an [earlier commit](https://github.com/cloudfoundry-incubator/cf-abacus/commit/be64bc50408e523671a24670d4d16fdadfd0fb7c) had only 38 requests \r\n\r\nHere is the code that I added to the dbserver to count the requests\r\n\r\n```javascript\r\nconst debug2 = require('abacus-debug')('request-counter');\r\nlet i = 0;\r\napp.use(function(req,res,next) {\r\n        i++;\r\n        debug2('[%s] [%s] %s', i, req.method, req.url);\r\n        next();\r\n});\r\n```\r\n\r\nI was not able to attach the log files here and hence I uploaded it to dropbox. You can find the [Logs Here](https://www.dropbox.com/sh/qo1e1ypwgslua0r/AAAnArjPNnroDDLcp_vM2UFJa?dl=0)"},"comment":{"url":"https://api.github.com/repos/cloudfoundry-incubator/cf-abacus/issues/comments/146312051","html_url":"https://github.com/cloudfoundry-incubator/cf-abacus/issues/69#issuecomment-146312051","issue_url":"https://api.github.com/repos/cloudfoundry-incubator/cf-abacus/issues/69","id":146312051,"user":{"login":"BetaFood","id":10606422,"avatar_url":"https://avatars.githubusercontent.com/u/10606422?v=3","gravatar_id":"","url":"https://api.github.com/users/BetaFood","html_url":"https://github.com/BetaFood","followers_url":"https://api.github.com/users/BetaFood/followers","following_url":"https://api.github.com/users/BetaFood/following{/other_user}","gists_url":"https://api.github.com/users/BetaFood/gists{/gist_id}","starred_url":"https://api.github.com/users/BetaFood/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/BetaFood/subscriptions","organizations_url":"https://api.github.com/users/BetaFood/orgs","repos_url":"https://api.github.com/users/BetaFood/repos","events_url":"https://api.github.com/users/BetaFood/events{/privacy}","received_events_url":"https://api.github.com/users/BetaFood/received_events","type":"User","site_admin":false},"created_at":"2015-10-07T20:01:58Z","updated_at":"2015-10-07T20:01:58Z","body":"One of the issues here lies in the partitioning logic. Specifically when determining the periods from the bounds in a range Operation.\r\n```javascript\r\nconst periods = range(bounds[0], bounds[1] + 1, bounds[1] < bounds[0] ?\r\n        -1 : 1);\r\n```\r\nSince this logic does a pure +1 or -1 as a step regardless of the period function passed in, db names of 201499, 201498 will happening. It would good to make this logic extendible since limiting it would limit all use of partition to a monthly date range.\r\n\r\nThis range is used in dbpartitions() inside of dbclient and subsequently pool(). pool() eventually calls the the default dbcons() function(in this case, it's using the default)\r\n```javascript\r\nconst dbcons = (uri, opt, cb) => cb(undefined,\r\n  /:/.test(uri) ? new PouchDB(uri, defaults(opt || {}, {\r\n    ajax: {\r\n      rejectUnauthorized: false\r\n    }\r\n  })) : new PouchDB(uri, defaults(opt || {}, {\r\n    db: memdown\r\n  })));\r\n```\r\nCalling PouchDb creates the database in this case. So querying the range of [201401, 201501] will create about 100 databases before it tries to do the query.\r\n\r\nSince reporting is doing a descending range query of a limit of 1, it should ideally stop once it finds its first record(or if it finds nothing). rangeOp has such logic for doing that\r\n```javascript\r\n if(opt.limit && accum.length === opt.limit)\r\n          return rcb(undefined, accum);\r\n```\r\nHowever, as shown above. The list of dbhandles is already created, so this line only stops subsequently queries into each database.\r\n\r\n\r\nThere's a couple of things here to consider:\r\n* The range creation based upon the bounds from the period\r\n  * Currently, the range creation is using a simple step of +1 or -1. This results in creating dbhandles for databases with invalid months.\r\n  * This functionality would need to be extendible to prevent any limitations in the partitioning module\r\n* Databases are being created on an allDoc requests.\r\n  * Databases should probably not be created on a retrieval operation since they would have no data to return in that case\r\n* The query itself\r\n  * The query does a range operation from the month of the time passed to the beginning of the year before the time passed in. Since the time windows will be keep track in a single document up to the previous year, the query starts with the beginning of the last year\r\n    * 1444246062000 would do a query from 201401 to 201510\r\n  * We might have to understand how far we look in terms of a report. If by chance a report for an organization is asked for 2015-10, would it make sense to return 2014-04 assuming that's the latest usage of that org?\r\n\r\nI'm thinking that the second point would solve this current issue, but that we would need to also solve the first issue(since a bulkDocs request could create many databases with invalid months).\r\n\r\nThoughts?"}}