{"action":"closed","issue":{"url":"https://api.github.com/repos/dotnet/orleans/issues/935","labels_url":"https://api.github.com/repos/dotnet/orleans/issues/935/labels{/name}","comments_url":"https://api.github.com/repos/dotnet/orleans/issues/935/comments","events_url":"https://api.github.com/repos/dotnet/orleans/issues/935/events","html_url":"https://github.com/dotnet/orleans/issues/935","id":113014669,"number":935,"title":"Dead silo","user":{"login":"Merroy","id":4007564,"avatar_url":"https://avatars.githubusercontent.com/u/4007564?v=3","gravatar_id":"","url":"https://api.github.com/users/Merroy","html_url":"https://github.com/Merroy","followers_url":"https://api.github.com/users/Merroy/followers","following_url":"https://api.github.com/users/Merroy/following{/other_user}","gists_url":"https://api.github.com/users/Merroy/gists{/gist_id}","starred_url":"https://api.github.com/users/Merroy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Merroy/subscriptions","organizations_url":"https://api.github.com/users/Merroy/orgs","repos_url":"https://api.github.com/users/Merroy/repos","events_url":"https://api.github.com/users/Merroy/events{/privacy}","received_events_url":"https://api.github.com/users/Merroy/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/dotnet/orleans/labels/question","name":"question","color":"cc317c"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":17,"created_at":"2015-10-23T13:02:54Z","updated_at":"2015-10-24T13:20:24Z","closed_at":"2015-10-24T13:20:24Z","body":"I have a cluster with 3 silo, after a few days of uninterrupted work, dead one silo\r\n```\r\n[2015-10-22 21:25:24.290 GMT    10\tWARNING\t100618\tMembershipOracle\t192.168.3.24:11111]\tI should be Dead according to membership table (in CleanupTableEntries): entry = SiloAddress=S192.168.3.24:11111:183104555 InstanceName=kontent3 Status=Dead.\t\r\n[2015-10-22 21:25:24.290 GMT    10\tERROR  \t100627\tMembershipOracle\t192.168.3.24:11111]\t!!!!!!!!!! I have been told I am dead, so this silo will commit suicide! I should be Dead according to membership table (in CleanupTableEntries): entry = SiloAddress=S192.168.3.24:11111:183104555 InstanceName=kontent3 Status=Dead.\t\r\n[2015-10-22 21:25:24.446 GMT    10\tERROR  \t100627\tMembershipOracle\t192.168.3.24:11111]\t!!!!!!!!!! INTERNAL FAILURE! About to crash! Fail message is: I have been told I am dead, so this silo will commit suicide! I should be Dead according to membership table (in CleanupTableEntries): entry = SiloAddress=S192.168.3.24:11111:183104555 InstanceName=kontent3 Status=Dead.\r\n   в System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)\r\n   в System.Environment.get_StackTrace()\r\n   в Orleans.Runtime.TraceLogger.Fail(ErrorCode errorCode, String message)\r\n   в Orleans.Runtime.MembershipService.MembershipOracle.KillMyselfLocally(String reason)\r\n   в Orleans.Runtime.MembershipService.MembershipOracle.<CleanupTableEntries>d__65.MoveNext()\r\n   в System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)\r\n   в Orleans.Runtime.MembershipService.MembershipOracle.CleanupTableEntries(MembershipTableData table)\r\n   в Orleans.Runtime.MembershipService.MembershipOracle.<ProcessTableUpdate>d__55.MoveNext()\r\n   в System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)\r\n   в Orleans.Runtime.MembershipService.MembershipOracle.ProcessTableUpdate(MembershipTableData table, String caller, Boolean logAtInfoLevel)\r\n   в Orleans.Runtime.MembershipService.MembershipOracle.<<OnGetTableUpdateTimer>b__7d>d__7e.MoveNext()\r\n   в System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)\r\n   в Orleans.Runtime.MembershipService.MembershipOracle.<OnGetTableUpdateTimer>b__7d(Task`1 task)\r\n   в System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()\r\n   в System.Threading.Tasks.Task.Execute()\r\n   в System.Threading.Tasks.Task.ExecutionContextCallback(Object obj)\r\n   в System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)\r\n   в System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)\r\n   в System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)\r\n   в System.Threading.Tasks.Task.ExecuteEntry(Boolean bPreventDoubleExecution)\r\n   в System.Threading.Tasks.TaskScheduler.TryExecuteTask(Task task)\r\n   в Orleans.Runtime.Scheduler.ActivationTaskScheduler.RunTask(Task task)\r\n   в Orleans.Runtime.Scheduler.WorkItemGroup.Execute()\r\n   в Orleans.Runtime.Scheduler.WorkerPoolThread.Run()\r\n   в Orleans.Runtime.AsynchAgent.AgentThreadProc(Object obj)\r\n   в System.Threading.ThreadHelper.ThreadStart_Context(Object state)\r\n   в System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)\r\n   в System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)\r\n   в System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)\r\n   в System.Threading.ThreadHelper.ThreadStart(Object obj)\t\r\n[2015-10-22 21:25:36.661 GMT    18\tINFO   \t100328\tRuntime.GrainDirectory.AdaptiveDirectoryCacheMaintainer`1\t192.168.3.24:11111]\tStopping AsyncAgent Runtime.GrainDirectory.AdaptiveDirectoryCacheMaintainer`1 that runs on managed thread 18\t\r\n[2015-10-22 21:25:36.661 GMT    47\tWARNING\t100150\tWatchdog\t192.168.3.24:11111]\t.NET Runtime Platform stalled for 00:00:11.8564560 - possibly GC? We are now using total of 37MB memory. gc=31249, 4032, 602\t\r\n[2015-10-22 21:25:36.677 GMT    10\tERROR  \t100001\tMembershipOracle\t192.168.3.24:11111]\t!!!!!!!!!! INTERNAL FAILURE! Application mini-dump written to file ServerSiloHost-MiniDump-2015-10-22-21-25-24-493Z.dmp\t\r\n[2015-10-22 21:25:36.677 GMT    10\tERROR  \t100002\tMembershipOracle\t192.168.3.24:11111]\t!!!!!!!!!! INTERNAL FAILURE! Process crashing!\t\r\n```\r\nWhat could be the problem?"}}