{"action":"created","issue":{"url":"https://api.github.com/repos/mozilla/kuma/issues/3386","labels_url":"https://api.github.com/repos/mozilla/kuma/issues/3386/labels{/name}","comments_url":"https://api.github.com/repos/mozilla/kuma/issues/3386/comments","events_url":"https://api.github.com/repos/mozilla/kuma/issues/3386/events","html_url":"https://github.com/mozilla/kuma/pull/3386","id":99042574,"number":3386,"title":"Bug 1190171 - Revision content denormalization and other cleanups of the compare revision view.","user":{"login":"jezdez","id":1610,"avatar_url":"https://avatars.githubusercontent.com/u/1610?v=3","gravatar_id":"","url":"https://api.github.com/users/jezdez","html_url":"https://github.com/jezdez","followers_url":"https://api.github.com/users/jezdez/followers","following_url":"https://api.github.com/users/jezdez/following{/other_user}","gists_url":"https://api.github.com/users/jezdez/gists{/gist_id}","starred_url":"https://api.github.com/users/jezdez/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jezdez/subscriptions","organizations_url":"https://api.github.com/users/jezdez/orgs","repos_url":"https://api.github.com/users/jezdez/repos","events_url":"https://api.github.com/users/jezdez/events{/privacy}","received_events_url":"https://api.github.com/users/jezdez/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/mozilla/kuma/labels/not%20ready","name":"not ready","color":"f7c6c7"}],"state":"open","locked":false,"assignee":{"login":"jezdez","id":1610,"avatar_url":"https://avatars.githubusercontent.com/u/1610?v=3","gravatar_id":"","url":"https://api.github.com/users/jezdez","html_url":"https://github.com/jezdez","followers_url":"https://api.github.com/users/jezdez/followers","following_url":"https://api.github.com/users/jezdez/following{/other_user}","gists_url":"https://api.github.com/users/jezdez/gists{/gist_id}","starred_url":"https://api.github.com/users/jezdez/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jezdez/subscriptions","organizations_url":"https://api.github.com/users/jezdez/orgs","repos_url":"https://api.github.com/users/jezdez/repos","events_url":"https://api.github.com/users/jezdez/events{/privacy}","received_events_url":"https://api.github.com/users/jezdez/received_events","type":"User","site_admin":false},"milestone":null,"comments":5,"created_at":"2015-08-04T19:22:32Z","updated_at":"2015-08-07T14:58:10Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/mozilla/kuma/pulls/3386","html_url":"https://github.com/mozilla/kuma/pull/3386","diff_url":"https://github.com/mozilla/kuma/pull/3386.diff","patch_url":"https://github.com/mozilla/kuma/pull/3386.patch"},"body":"- Adds ratelimiting to the compare revision view.\n- Refactors the signal handling to use new patterns and less confusing code.\n- Prefetch revision tags and document for fewer queries.\n- Denormalize the revision content in a separate field that is filled on save and uses tidy to do the CPU-intense work once instead of on every request."},"comment":{"url":"https://api.github.com/repos/mozilla/kuma/issues/comments/128723611","html_url":"https://github.com/mozilla/kuma/pull/3386#issuecomment-128723611","issue_url":"https://api.github.com/repos/mozilla/kuma/issues/3386","id":128723611,"user":{"login":"jezdez","id":1610,"avatar_url":"https://avatars.githubusercontent.com/u/1610?v=3","gravatar_id":"","url":"https://api.github.com/users/jezdez","html_url":"https://github.com/jezdez","followers_url":"https://api.github.com/users/jezdez/followers","following_url":"https://api.github.com/users/jezdez/following{/other_user}","gists_url":"https://api.github.com/users/jezdez/gists{/gist_id}","starred_url":"https://api.github.com/users/jezdez/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jezdez/subscriptions","organizations_url":"https://api.github.com/users/jezdez/orgs","repos_url":"https://api.github.com/users/jezdez/repos","events_url":"https://api.github.com/users/jezdez/events{/privacy}","received_events_url":"https://api.github.com/users/jezdez/received_events","type":"User","site_admin":false},"created_at":"2015-08-07T14:58:10Z","updated_at":"2015-08-07T14:58:10Z","body":"I actually decided to increase the rate limit of the tidy_revision_content Celery task to `120/s`. The reasoning is simple: we have 550k revisions right now, with a rate limit of `15/m` we'd wait 12 days for even running the tasks. On the other hand, MySQL is currently able to handle `UPDATE` queries very fast and hasn't hit performance ceilings (not even close) with that when I looked at the last week's stats. We also are able to have the Celery nodes run through the CPU intense tidying easier given we can spread it to their collective CPUs, in other words, they aren't fully utilized right now as well.\r\n\r\nThat said I agree that there is a big risk with scheduling Celery tasks on every call to `get_tidied_content` since if assuming we create a big stack of tasks by normal site browsing it's guarenteed that we'll create duplicates of the tasks. That's why I've added a simple cache based check to the `get_tidied_content` function that prevents duplicate scheduling. It's set to timeout after 3 days, after which a call to `get_tidied_content` would again schedule a task. The calculation for that timeout is: `(546235 / 120) / 60 / 24 = 3.1 days` -- the number of days it takes in theory to run through all revisions."}}