{"action":"created","issue":{"url":"https://api.github.com/repos/Metaswitch/sprout/issues/1010","labels_url":"https://api.github.com/repos/Metaswitch/sprout/issues/1010/labels{/name}","comments_url":"https://api.github.com/repos/Metaswitch/sprout/issues/1010/comments","events_url":"https://api.github.com/repos/Metaswitch/sprout/issues/1010/events","html_url":"https://github.com/Metaswitch/sprout/issues/1010","id":69145050,"number":1010,"title":"Deregistration leaves subscriber registered","user":{"login":"dtheobald","id":6430927,"avatar_url":"https://avatars.githubusercontent.com/u/6430927?v=3","gravatar_id":"","url":"https://api.github.com/users/dtheobald","html_url":"https://github.com/dtheobald","followers_url":"https://api.github.com/users/dtheobald/followers","following_url":"https://api.github.com/users/dtheobald/following{/other_user}","gists_url":"https://api.github.com/users/dtheobald/gists{/gist_id}","starred_url":"https://api.github.com/users/dtheobald/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dtheobald/subscriptions","organizations_url":"https://api.github.com/users/dtheobald/orgs","repos_url":"https://api.github.com/users/dtheobald/repos","events_url":"https://api.github.com/users/dtheobald/events{/privacy}","received_events_url":"https://api.github.com/users/dtheobald/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/Metaswitch/sprout/labels/high-priority","name":"high-priority","color":"e11d21"}],"state":"open","locked":false,"assignee":{"login":"dtheobald","id":6430927,"avatar_url":"https://avatars.githubusercontent.com/u/6430927?v=3","gravatar_id":"","url":"https://api.github.com/users/dtheobald","html_url":"https://github.com/dtheobald","followers_url":"https://api.github.com/users/dtheobald/followers","following_url":"https://api.github.com/users/dtheobald/following{/other_user}","gists_url":"https://api.github.com/users/dtheobald/gists{/gist_id}","starred_url":"https://api.github.com/users/dtheobald/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dtheobald/subscriptions","organizations_url":"https://api.github.com/users/dtheobald/orgs","repos_url":"https://api.github.com/users/dtheobald/repos","events_url":"https://api.github.com/users/dtheobald/events{/privacy}","received_events_url":"https://api.github.com/users/dtheobald/received_events","type":"User","site_admin":false},"milestone":null,"comments":16,"created_at":"2015-04-17T14:03:42Z","updated_at":"2015-04-22T15:47:05Z","closed_at":null,"body":"While performing some non-clearwater tests, I've run into a flaky situation where I register a subscriber and subsequently de-register that subscriber, yet it remains registered in both homestead and the HSS. I say \"flaky\" because it's not consistently reproducible and it seems that the longer the clearwater system is up, the more likely this problem is to occur.\r\n\r\nDiags have been captured."},"comment":{"url":"https://api.github.com/repos/Metaswitch/sprout/issues/comments/95243110","html_url":"https://github.com/Metaswitch/sprout/issues/1010#issuecomment-95243110","issue_url":"https://api.github.com/repos/Metaswitch/sprout/issues/1010","id":95243110,"user":{"login":"mirw","id":2963879,"avatar_url":"https://avatars.githubusercontent.com/u/2963879?v=3","gravatar_id":"","url":"https://api.github.com/users/mirw","html_url":"https://github.com/mirw","followers_url":"https://api.github.com/users/mirw/followers","following_url":"https://api.github.com/users/mirw/following{/other_user}","gists_url":"https://api.github.com/users/mirw/gists{/gist_id}","starred_url":"https://api.github.com/users/mirw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mirw/subscriptions","organizations_url":"https://api.github.com/users/mirw/orgs","repos_url":"https://api.github.com/users/mirw/repos","events_url":"https://api.github.com/users/mirw/events{/privacy}","received_events_url":"https://api.github.com/users/mirw/received_events","type":"User","site_admin":false},"created_at":"2015-04-22T15:47:05Z","updated_at":"2015-04-22T15:47:05Z","body":"The reason that the memcached write fails with contention is that the connection to memcached instance 1 was failed when we tried to read, so we read from memcached instance 2 instead.  By the time we tried to write back, memcached instance 1 was alive again, and so we tried to CAS to that.  Memcached instance 1 had a different CAS value (as CAS values are not transferrable between servers) so this write failed.  I think the immediate fix is to do the usual retry behavior on contention, but we should probably also look at improving handling of this specific memcached failure mode."}}