{"action":"created","issue":{"url":"https://api.github.com/repos/wmde/phragile/issues/154","labels_url":"https://api.github.com/repos/wmde/phragile/issues/154/labels{/name}","comments_url":"https://api.github.com/repos/wmde/phragile/issues/154/comments","events_url":"https://api.github.com/repos/wmde/phragile/issues/154/events","html_url":"https://github.com/wmde/phragile/pull/154","id":104024647,"number":154,"title":"Fix sprint overviews for large projects","user":{"login":"jakobw","id":453024,"avatar_url":"https://avatars.githubusercontent.com/u/453024?v=3","gravatar_id":"","url":"https://api.github.com/users/jakobw","html_url":"https://github.com/jakobw","followers_url":"https://api.github.com/users/jakobw/followers","following_url":"https://api.github.com/users/jakobw/following{/other_user}","gists_url":"https://api.github.com/users/jakobw/gists{/gist_id}","starred_url":"https://api.github.com/users/jakobw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jakobw/subscriptions","organizations_url":"https://api.github.com/users/jakobw/orgs","repos_url":"https://api.github.com/users/jakobw/repos","events_url":"https://api.github.com/users/jakobw/events{/privacy}","received_events_url":"https://api.github.com/users/jakobw/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2015-08-31T10:17:31Z","updated_at":"2015-09-16T10:46:42Z","closed_at":"2015-09-16T10:45:01Z","pull_request":{"url":"https://api.github.com/repos/wmde/phragile/pulls/154","html_url":"https://github.com/wmde/phragile/pull/154","diff_url":"https://github.com/wmde/phragile/pull/154.diff","patch_url":"https://github.com/wmde/phragile/pull/154.patch"},"body":"Task on Phab: https://phabricator.wikimedia.org/T110330\r\n\r\nThere were two different problems with large projects.\r\n\r\n1. **Timeouts on task transaction requests**\r\nIt is not possible to load all task transactions of large projects at once through Conduit (Phabricator's API). This was solved by fetching transactions in batches of 200. More details: https://phabricator.wikimedia.org/T109036\r\n\r\n2. **Memory issues when processing/storing the transactions**\r\nLoading all transactions is solved by loading them in batches, however, processing all the transactions consumes a lot of memory. This can be improved by filtering out irrelevant transactions. For snapshots it'll store all transactions regarding workboard and status changes so that switching between workboard and status mode still works. Unfortunately this still exceeded the memory limit for sprint overviews but ~~can be~~ **that has been** fixed by also removing status transactions in workboard mode and workboard transactions in status mode.\r\n\r\nI deployed the changes to a test instance and you can check it out here: http://phragile-test.wmflabs.org/sprints/483\r\n\r\n~~Snapshots are still somewhat broken for large projects unfortunately. For some reason they can be created but not read.~~\r\nSnapshots are fixed in #155 "},"comment":{"url":"https://api.github.com/repos/wmde/phragile/issues/comments/140703171","html_url":"https://github.com/wmde/phragile/pull/154#issuecomment-140703171","issue_url":"https://api.github.com/repos/wmde/phragile/issues/154","id":140703171,"user":{"login":"Benestar","id":2998254,"avatar_url":"https://avatars.githubusercontent.com/u/2998254?v=3","gravatar_id":"","url":"https://api.github.com/users/Benestar","html_url":"https://github.com/Benestar","followers_url":"https://api.github.com/users/Benestar/followers","following_url":"https://api.github.com/users/Benestar/following{/other_user}","gists_url":"https://api.github.com/users/Benestar/gists{/gist_id}","starred_url":"https://api.github.com/users/Benestar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Benestar/subscriptions","organizations_url":"https://api.github.com/users/Benestar/orgs","repos_url":"https://api.github.com/users/Benestar/repos","events_url":"https://api.github.com/users/Benestar/events{/privacy}","received_events_url":"https://api.github.com/users/Benestar/received_events","type":"User","site_admin":false},"created_at":"2015-09-16T10:46:42Z","updated_at":"2015-09-16T10:46:42Z","body":"I just noticed that http://phragile-test.wmflabs.org/sprints/782 looks very strange. But at least it loads something!"}}