{"action":"created","issue":{"url":"https://api.github.com/repos/18F/midas/issues/813","labels_url":"https://api.github.com/repos/18F/midas/issues/813/labels{/name}","comments_url":"https://api.github.com/repos/18F/midas/issues/813/comments","events_url":"https://api.github.com/repos/18F/midas/issues/813/events","html_url":"https://github.com/18F/midas/issues/813","id":81581226,"number":813,"title":"Performance on /tasks is slow","user":{"login":"dhcole","id":170641,"avatar_url":"https://avatars.githubusercontent.com/u/170641?v=3","gravatar_id":"","url":"https://api.github.com/users/dhcole","html_url":"https://github.com/dhcole","followers_url":"https://api.github.com/users/dhcole/followers","following_url":"https://api.github.com/users/dhcole/following{/other_user}","gists_url":"https://api.github.com/users/dhcole/gists{/gist_id}","starred_url":"https://api.github.com/users/dhcole/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dhcole/subscriptions","organizations_url":"https://api.github.com/users/dhcole/orgs","repos_url":"https://api.github.com/users/dhcole/repos","events_url":"https://api.github.com/users/dhcole/events{/privacy}","received_events_url":"https://api.github.com/users/dhcole/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/18F/midas/labels/in%20progress","name":"in progress","color":"ededed"},{"url":"https://api.github.com/repos/18F/midas/labels/Open%20Opps","name":"Open Opps","color":"4488aa"}],"state":"open","locked":false,"assignee":{"login":"dhcole","id":170641,"avatar_url":"https://avatars.githubusercontent.com/u/170641?v=3","gravatar_id":"","url":"https://api.github.com/users/dhcole","html_url":"https://github.com/dhcole","followers_url":"https://api.github.com/users/dhcole/followers","following_url":"https://api.github.com/users/dhcole/following{/other_user}","gists_url":"https://api.github.com/users/dhcole/gists{/gist_id}","starred_url":"https://api.github.com/users/dhcole/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dhcole/subscriptions","organizations_url":"https://api.github.com/users/dhcole/orgs","repos_url":"https://api.github.com/users/dhcole/repos","events_url":"https://api.github.com/users/dhcole/events{/privacy}","received_events_url":"https://api.github.com/users/dhcole/received_events","type":"User","site_admin":false},"milestone":null,"comments":3,"created_at":"2015-05-27T19:05:57Z","updated_at":"2015-06-03T16:55:54Z","closed_at":null,"body":"Just a general capture of thought about performance on the tasks page.\r\n\r\nPage loads on https://midas.18f.us/tasks are taking about 4-5 seconds. This is down by about 50% due to the #656 Tag Associations refactor, but still too high.\r\n\r\n![screen shot 2015-05-27 at 2 53 56 pm](https://cloud.githubusercontent.com/assets/170641/7844723/5e4a13e2-0480-11e5-9ec2-47423627ad96.png)\r\n\r\nThe delay on this page is mostly due to the `/api/tasks` endpoint. The manual association between tagEntities and other models was inefficient. \r\n\r\nHere are additional sources of slowness:\r\n\r\n- `/api/file/get/*` used to require querying file blobs from the database. In dev, this is moved to disk or S3, but it's not rolled out to production yet. This accounts for 17% of request time: \r\n![screen shot 2015-05-27 at 2 55 52 pm](https://cloud.githubusercontent.com/assets/170641/7844777/eb8b89d4-0480-11e5-9325-577dd74e86ea.png)\r\n- Sessions are stored in the main postgres DB (That's what all the `head` requests in the previous chart show). We can move sessions to a separate database to help with this. Sails / Express recommend redis, which should be available in AWS through the Elasticache service\r\n- The `/api/task` endpoint is currently returning more data then necessary. We could limit these requests with pagination. There is an infinite scroll implementation of pagination available in the front-end code, however, we need to confirm that this works well with the search and filters on the `/tasks` page\r\n- There is a `like` query on the `/api/task` endpoint that seems to cause the bulk of slowness now. We could track down where this query is coming from and figure out how to optimize it"},"comment":{"url":"https://api.github.com/repos/18F/midas/issues/comments/108522834","html_url":"https://github.com/18F/midas/issues/813#issuecomment-108522834","issue_url":"https://api.github.com/repos/18F/midas/issues/813","id":108522834,"user":{"login":"dhcole","id":170641,"avatar_url":"https://avatars.githubusercontent.com/u/170641?v=3","gravatar_id":"","url":"https://api.github.com/users/dhcole","html_url":"https://github.com/dhcole","followers_url":"https://api.github.com/users/dhcole/followers","following_url":"https://api.github.com/users/dhcole/following{/other_user}","gists_url":"https://api.github.com/users/dhcole/gists{/gist_id}","starred_url":"https://api.github.com/users/dhcole/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dhcole/subscriptions","organizations_url":"https://api.github.com/users/dhcole/orgs","repos_url":"https://api.github.com/users/dhcole/repos","events_url":"https://api.github.com/users/dhcole/events{/privacy}","received_events_url":"https://api.github.com/users/dhcole/received_events","type":"User","site_admin":false},"created_at":"2015-06-03T16:55:54Z","updated_at":"2015-06-03T16:55:54Z","body":"We're currently running an `db.m1.small\t` instance, which costs $0.120 / hour. It's also two generations old, so newer generations have more resources at lower cost.\r\n\r\nI recommend we upgrade to a `db.t2.small` instance at $0.072. It has comparable resources, with slightly more memory.\r\n\r\nThen we can test to see how well it performance. If it's still not desirable, we can either scale up to a `db.t2.medium` or `db.m3.medium`, or we can switch from magnetic storage to SSD.\r\n\r\nSince this may cause a few minutes of downtime, I'm going to start testing different instances on the staging database. \r\n\r\nFirst, I'll update staging to match the production data so we can benchmark effectively and make sure this is really an instance size issue. Currently, with outdated data, we see this on staging:\r\n\r\n```\r\ndavidcole@dc:~/dev/midas$ time curl https://openopps-staging.18f.us/api/task > /dev/null\r\n  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\r\n                                 Dload  Upload   Total   Spent    Left  Speed\r\n100 73957  100 73957    0     0  33203      0  0:00:02  0:00:02 --:--:-- 33209\r\n\r\nreal\t0m2.237s\r\nuser\t0m0.032s\r\nsys\t0m0.012s\r\n```"}}