{"action":"created","issue":{"url":"https://api.github.com/repos/dotnet/orleans/issues/1209","labels_url":"https://api.github.com/repos/dotnet/orleans/issues/1209/labels{/name}","comments_url":"https://api.github.com/repos/dotnet/orleans/issues/1209/comments","events_url":"https://api.github.com/repos/dotnet/orleans/issues/1209/events","html_url":"https://github.com/dotnet/orleans/issues/1209","id":124087315,"number":1209,"title":"Forwarding failed exceptions","user":{"login":"k1918u42","id":10634378,"avatar_url":"https://avatars.githubusercontent.com/u/10634378?v=3","gravatar_id":"","url":"https://api.github.com/users/k1918u42","html_url":"https://github.com/k1918u42","followers_url":"https://api.github.com/users/k1918u42/followers","following_url":"https://api.github.com/users/k1918u42/following{/other_user}","gists_url":"https://api.github.com/users/k1918u42/gists{/gist_id}","starred_url":"https://api.github.com/users/k1918u42/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/k1918u42/subscriptions","organizations_url":"https://api.github.com/users/k1918u42/orgs","repos_url":"https://api.github.com/users/k1918u42/repos","events_url":"https://api.github.com/users/k1918u42/events{/privacy}","received_events_url":"https://api.github.com/users/k1918u42/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-12-28T16:23:07Z","updated_at":"2015-12-28T19:08:31Z","closed_at":null,"body":"We have about 12 Orleans silos (azure worker roles) supporting around 100 web roles.  Every couple weeks some of the Orleans silos start throwing forwarding failed exceptions:\r\n\r\n    Forwarding failed: tried to forward message Request S[ip]:11111:188604562*cli/5bebe9bf@7d8cfb97->S[ip]:11111:186066337*grn/7A74AB6F/45a25a8d@12e5569b #116808238[ForwardCount=2]: [GrainType + Method] for 2 times after Non-existent activation to invalid activation. Rejecting now.\r\n\r\nThrough analyzing the ip addresses in the exception and correlating to the SiloInstances table we're able to determine which roles are generating the error.  Restarting the role fixes the issue temporarily (usually for a week or two).\r\n\r\nSearching the existing issues I see this error can occur when the Global Grain GC limit is less than 1 minute (issue #271).  We haven't set this value though and the default limit of 2 hours should apply.\r\n\r\nHave you encountered this issue before and if so what was the resolution?"},"comment":{"url":"https://api.github.com/repos/dotnet/orleans/issues/comments/167630207","html_url":"https://github.com/dotnet/orleans/issues/1209#issuecomment-167630207","issue_url":"https://api.github.com/repos/dotnet/orleans/issues/1209","id":167630207,"user":{"login":"k1918u42","id":10634378,"avatar_url":"https://avatars.githubusercontent.com/u/10634378?v=3","gravatar_id":"","url":"https://api.github.com/users/k1918u42","html_url":"https://github.com/k1918u42","followers_url":"https://api.github.com/users/k1918u42/followers","following_url":"https://api.github.com/users/k1918u42/following{/other_user}","gists_url":"https://api.github.com/users/k1918u42/gists{/gist_id}","starred_url":"https://api.github.com/users/k1918u42/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/k1918u42/subscriptions","organizations_url":"https://api.github.com/users/k1918u42/orgs","repos_url":"https://api.github.com/users/k1918u42/repos","events_url":"https://api.github.com/users/k1918u42/events{/privacy}","received_events_url":"https://api.github.com/users/k1918u42/received_events","type":"User","site_admin":false},"created_at":"2015-12-28T19:08:31Z","updated_at":"2015-12-28T19:08:31Z","body":"We're running 1.0.5.  If #955 is in 1.1.0 we'll try upgrading and see if that fixes."}}