{"action":"created","issue":{"url":"https://api.github.com/repos/HabitRPG/habitrpg/issues/1057","labels_url":"https://api.github.com/repos/HabitRPG/habitrpg/issues/1057/labels{/name}","comments_url":"https://api.github.com/repos/HabitRPG/habitrpg/issues/1057/comments","events_url":"https://api.github.com/repos/HabitRPG/habitrpg/issues/1057/events","html_url":"https://github.com/HabitRPG/habitrpg/issues/1057","id":14799894,"number":1057,"title":"Dailies still checked from yesterday (cron did not run) [$271]","user":{"login":"impr","id":4540635,"avatar_url":"https://avatars.githubusercontent.com/u/4540635?v=3","gravatar_id":"","url":"https://api.github.com/users/impr","html_url":"https://github.com/impr","followers_url":"https://api.github.com/users/impr/followers","following_url":"https://api.github.com/users/impr/following{/other_user}","gists_url":"https://api.github.com/users/impr/gists{/gist_id}","starred_url":"https://api.github.com/users/impr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/impr/subscriptions","organizations_url":"https://api.github.com/users/impr/orgs","repos_url":"https://api.github.com/users/impr/repos","events_url":"https://api.github.com/users/impr/events{/privacy}","received_events_url":"https://api.github.com/users/impr/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/HabitRPG/habitrpg/labels/bounty","name":"bounty","color":"129e5e"},{"url":"https://api.github.com/repos/HabitRPG/habitrpg/labels/bug","name":"bug","color":"e102d8"},{"url":"https://api.github.com/repos/HabitRPG/habitrpg/labels/cron","name":"cron","color":"bfd4f2"},{"url":"https://api.github.com/repos/HabitRPG/habitrpg/labels/hammer","name":"hammer","color":"e11d21"},{"url":"https://api.github.com/repos/HabitRPG/habitrpg/labels/Memorable","name":"Memorable","color":"207de5"},{"url":"https://api.github.com/repos/HabitRPG/habitrpg/labels/priority+-+critical","name":"priority - critical","color":"e10c02"},{"url":"https://api.github.com/repos/HabitRPG/habitrpg/labels/tag2-bug","name":"tag2-bug","color":"5319e7"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/HabitRPG/habitrpg/milestones/3","labels_url":"https://api.github.com/repos/HabitRPG/habitrpg/milestones/3/labels","id":556556,"number":3,"title":"Site Stability","description":"Users have been experiencing many issues over the last week. Fear not, this isn't the *Derby Days* all over again. Those times were due to one giant bad architecture, this time it's a bunch of smaller issues here and there, connecting to cause a chain of bad events. Twas an inevitability of outgrowing our training wheels, and now it's time to build for scalability & fix those issues. Things will improve over coming weeks, and we're focusing on stability before introducing new big features.\r\n\r\n## TL;DR:\r\n\r\nCause:\r\n\r\n* Database slow for array updates\r\n* Mishandling of errors\r\n\r\nResult:\r\n\r\n* 503 Errors showing for many people\r\n* Duplicate operations (duplicate tasks being created, users dying unexpectedly, etc)\r\n* Actions not \"sticking\"\r\n* Cron running at incorrect times or more than once\r\n\r\nWorkarounds:\r\n\r\n* Click slower for now. VersionError is caused by [clicking things off too fast](https://github.com/HabitRPG/habitrpg/commit/e36d3264bed5a015888f834c967d92f953365e51#commitcomment-5312406). We're [working on a fix](https://github.com/HabitRPG/habitrpg/issues/2301)\r\n* If you see an error, refresh before proceeding\r\n\r\n## Full Explanation\r\n\r\n#### Slow Server & Database\r\nAfter installing [New Relic](http://newrelic.com/), we were able to pinpoint bottlenecks in our application. One major bottleneck was MongoDB throughput & availability. We were previously using a 5GB single shared DB instance (MongoHQ). This had low throughput and high locking / page-fault. The solution was to upgrade to a dedicated DB cluster (Replica Set) on [MongoLab](https://mongolab.com/home), which improved throughput and availability. The server is still slow at the `post /user/batch-update` route ([screenshot](https://www.evernote.com/shard/s17/sh/c14505df-7ea5-475f-aa35-6d0aa2130624/aea6bb8a261b2aac94c4b48863fee7a0)) which needs tuning. We made one major server perf improvement by enabling [cluster](http://nodejs.org/api/cluster.html), which now allows us to use more resource per Heroku dyno than before (utilizing each dyno's 4 cpus, where previously we only used one). These two updates were a *major* improvement, but we're not yet off the hook.\r\n\r\nWhat we *think* happens when the backend is slow is the following. `batch-update` will send a queue of operations to perform. eg, `[score, updateTask, sortTasks, cron, etc]`. It sends it against one instance (there are **12** instances: `3 dynos * 4 cluster servers`). If that instance takes too long (either due to lag, or due to an instance crash which I'll explain later), then the operation says \"screw it\", and tries against the next instance. Both go through. Or in some cases, the \"screw it\" happens many times and *many* go through. This is why you sometimes see a ton of duplicate, strange-text tasks; or you die suddenly; or you bought something, and now you're down way too much GP. This is our *theory*, we haven't been able to verify it completely, but we've been digging for the last week and we're leaning pretty heavily towards this.\r\n\r\n#### Bad Error Handling\r\n\r\nAs explained above, we have server 12 instances. If someone triggers an error (the site has bugs, we all know) then what happens is this. That server instance becomes \"unstable\", and will return the same error that the user experienced, for every next request to that instance. Eg, User1 flagged the error & sees \"Application Error\" or \"503\" or whatever. Users 2,3,4,5 connect to that instance in their next HTTP request - they *also* see the error. Additionally, as per our theory above, whatever operation they'd sent up gets queued against the next instance. Both go through, duplicate operations. The solution here is:\r\n\r\n* Fix bugs so the errors don't happen in the first place. This is an on-going process and will never really be \"complete\". Software has bugs, but we can squash the common big ones.\r\n* When an instance becomes unstable, kill it immediately and don't let others connect to it. This is currently in progress, we've pegged it [for the most part](https://github.com/HabitRPG/habitrpg/commit/4f02ccb3b0779c330e326f72726ba0e9e935d354), but it's still having some issues that we're tweaking.\r\n\r\n#### Sudden Death\r\nSee [this comment](https://github.com/HabitRPG/habitrpg/issues/2601#issuecomment-34134544) if you've experienced sudden, unexpected death.","creator":{"login":"colegleason","id":232327,"avatar_url":"https://avatars.githubusercontent.com/u/232327?v=3","gravatar_id":"","url":"https://api.github.com/users/colegleason","html_url":"https://github.com/colegleason","followers_url":"https://api.github.com/users/colegleason/followers","following_url":"https://api.github.com/users/colegleason/following{/other_user}","gists_url":"https://api.github.com/users/colegleason/gists{/gist_id}","starred_url":"https://api.github.com/users/colegleason/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/colegleason/subscriptions","organizations_url":"https://api.github.com/users/colegleason/orgs","repos_url":"https://api.github.com/users/colegleason/repos","events_url":"https://api.github.com/users/colegleason/events{/privacy}","received_events_url":"https://api.github.com/users/colegleason/received_events","type":"User","site_admin":false},"open_issues":7,"closed_issues":13,"state":"open","created_at":"2014-02-02T23:50:51Z","updated_at":"2014-09-20T12:44:59Z","due_on":"2014-03-02T08:00:00Z","closed_at":null},"comments":347,"created_at":"2013-05-27T14:16:48Z","updated_at":"2015-01-13T11:43:03Z","closed_at":null,"body":"\r\nWhen I logged in this morning my dailies from yesterday were still showing as checked. I tried reopening habitrpg, I tried logging out and logging back in, and I tried waiting (it's been an hour now). No luck, my dailies still show as checked even though I haven't gotten out of bed yet today...\r\n\r\nI do have two dailies which I was not supposed to do yesterday (MWF dailies) and they are now showing as tickable so my account clearly knows that it is Monday. Maybe this is just the site's way of telling me those are the only dailies I should do today and I should take the day off from everything else... \r\n\r\n----\r\n\r\n_edit by admin:_\r\n\r\n## What to do if your Dailies are still ticked from yesterday:\r\n\r\nGo to the [Tavern](https://habitrpg.com/#/options/groups/tavern) to see if you are Resting in the Inn. If the left-hand column has a large green button that says **\"Check Out of the Inn\"**, click that, and your Dailies will reset tomorrow ([more information about Resting in the Inn](http://habitrpg.wikia.com/wiki/Tavern)). If you didn't see that button, then either wait until tomorrow because your Dailies will almost certainly reset correctly tonight, or:\r\n\r\n1. Record your current values for HP (health), XP (experience), GP (gold), and MP (mana).\r\n1. Untick all your Dailies.\r\n1. Go to [Settings -> Site](https://habitrpg.com/#/options/settings/settings) and use **Fix Character Values** to put your values back to the correct settings.\r\n1. For each Daily, edit it and go into the Advanced Settings to increase the streak count by one.\r\n\r\n<!---\r\n@huboard:{\"order\":127.82386779785156}\r\n-->\r\n\r\n\r\n<bountysource-plugin>\r\n\r\n---\r\nThere is a **[<total-bounty>$271</total-bounty> open bounty](https://www.bountysource.com/issues/868018-dailies-still-checked-from-yesterday?utm_campaign=plugin&utm_content=tracker%2F68393&utm_medium=issues&utm_source=github)** on this issue. Add to the bounty at [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F68393&utm_medium=issues&utm_source=github).\r\n</bountysource-plugin>"},"comment":{"url":"https://api.github.com/repos/HabitRPG/habitrpg/issues/comments/69732549","html_url":"https://github.com/HabitRPG/habitrpg/issues/1057#issuecomment-69732549","issue_url":"https://api.github.com/repos/HabitRPG/habitrpg/issues/1057","id":69732549,"user":{"login":"flutterbye","id":10045485,"avatar_url":"https://avatars.githubusercontent.com/u/10045485?v=3","gravatar_id":"","url":"https://api.github.com/users/flutterbye","html_url":"https://github.com/flutterbye","followers_url":"https://api.github.com/users/flutterbye/followers","following_url":"https://api.github.com/users/flutterbye/following{/other_user}","gists_url":"https://api.github.com/users/flutterbye/gists{/gist_id}","starred_url":"https://api.github.com/users/flutterbye/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/flutterbye/subscriptions","organizations_url":"https://api.github.com/users/flutterbye/orgs","repos_url":"https://api.github.com/users/flutterbye/repos","events_url":"https://api.github.com/users/flutterbye/events{/privacy}","received_events_url":"https://api.github.com/users/flutterbye/received_events","type":"User","site_admin":false},"created_at":"2015-01-13T11:43:03Z","updated_at":"2015-01-13T11:43:03Z","body":"yes I'm in a boss battle, here is the message from this morning:\r\nflutterbye attacks The Laundromancer for 2.7 damage, The Laundromancer\r\nattacks party for 6.7 damage\r\n\r\nthe dailies i ticked off in the afternoon (3 of them) reset, but all of the\r\nones i ticked off in the evening (i think it was around 10.30pm) did not\r\nreset.  looking at the damage dealt i suspect a dark red to-do did not deal\r\ndamage and hoping it will register tonight.\r\n\r\nOn 13 January 2015 at 11:35, Alice Harris <notifications@github.com> wrote:\r\n\r\n> @flutterbye <https://github.com/flutterbye> This morning, did ALL of your\r\n> completed Dailies remain ticked, or did only some of them stay ticked?\r\n>\r\n> Are you in a quest at the moment? If so, does your party chat contain\r\n> messages about your quest progress from yesterday/this morning? Please\r\n> click \"fetch recent messages\" in your party chat before you look for quest\r\n> progress messages.\r\n>\r\n> —\r\n> Reply to this email directly or view it on GitHub\r\n> <https://github.com/HabitRPG/habitrpg/issues/1057#issuecomment-69731772>.\r\n>"}}