{"action":"created","issue":{"url":"https://api.github.com/repos/rabbitmq/rabbitmq-server/issues/298","labels_url":"https://api.github.com/repos/rabbitmq/rabbitmq-server/issues/298/labels{/name}","comments_url":"https://api.github.com/repos/rabbitmq/rabbitmq-server/issues/298/comments","events_url":"https://api.github.com/repos/rabbitmq/rabbitmq-server/issues/298/events","html_url":"https://github.com/rabbitmq/rabbitmq-server/issues/298","id":104957271,"number":298,"title":"Crash when trying to reload rabbitmq-management plugin after removing a node","user":{"login":"evanccnyc","id":2079454,"avatar_url":"https://avatars.githubusercontent.com/u/2079454?v=3","gravatar_id":"","url":"https://api.github.com/users/evanccnyc","html_url":"https://github.com/evanccnyc","followers_url":"https://api.github.com/users/evanccnyc/followers","following_url":"https://api.github.com/users/evanccnyc/following{/other_user}","gists_url":"https://api.github.com/users/evanccnyc/gists{/gist_id}","starred_url":"https://api.github.com/users/evanccnyc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/evanccnyc/subscriptions","organizations_url":"https://api.github.com/users/evanccnyc/orgs","repos_url":"https://api.github.com/users/evanccnyc/repos","events_url":"https://api.github.com/users/evanccnyc/events{/privacy}","received_events_url":"https://api.github.com/users/evanccnyc/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":{"login":"michaelklishin","id":1090,"avatar_url":"https://avatars.githubusercontent.com/u/1090?v=3","gravatar_id":"","url":"https://api.github.com/users/michaelklishin","html_url":"https://github.com/michaelklishin","followers_url":"https://api.github.com/users/michaelklishin/followers","following_url":"https://api.github.com/users/michaelklishin/following{/other_user}","gists_url":"https://api.github.com/users/michaelklishin/gists{/gist_id}","starred_url":"https://api.github.com/users/michaelklishin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/michaelklishin/subscriptions","organizations_url":"https://api.github.com/users/michaelklishin/orgs","repos_url":"https://api.github.com/users/michaelklishin/repos","events_url":"https://api.github.com/users/michaelklishin/events{/privacy}","received_events_url":"https://api.github.com/users/michaelklishin/received_events","type":"User","site_admin":false},"milestone":null,"comments":3,"created_at":"2015-09-04T19:39:29Z","updated_at":"2015-09-05T14:57:00Z","closed_at":"2015-09-04T22:11:20Z","body":"Hi there,\r\n\r\nI ran into a rabbitmq crash after the following scenario:\r\n\r\n\r\nRabbitMQ 3.5.3\r\nUbuntu 14.04\r\n3 servers, 1c,1d,1e\r\n\r\nThere was a network blip and 1e briefly connected/reconnected, at which point the management GUI showed it not connected to the cluster.  We drained the queues and then disconnected the 1e node. At which point we lost all information on the management GUI on the *remaining two servers*.  The interface showed lots of ????'s and error messages like \"Statistics database could not be contacted.\".\r\n\r\nAt which point we removed the management plugin from one of the servers (1c) via the command line and tried to readd it back with:  \"rabbitmq-plugins enable rabbitmq_management\"\r\n\r\n\r\nAt which point 1c crashed with the following debug message:\r\n\r\n```\r\n=CRASH REPORT==== 4-Sep-2015::14:23:27 ===\r\n  crasher:\r\n    initial call: gen:init_it/6\r\n    pid: <0.12775.6568>\r\n    registered_name: []\r\n    exception exit: {function_clause,\r\n                        [{rabbit_channel,handle_info,2,[]},\r\n                         {gen_server2,handle_msg,2,[]},\r\n                         {rabbit_mgmt_wm_overview,module_info,1,[]}]}\r\n      in function  gen_server2:terminate/3\r\n      in call from rabbit_mgmt_wm_overview:module_info/1\r\n    ancestors: [<0.28642.6557>,<0.3525.6568>,<0.12250.6567>,<0.4055.6567>,\r\n                  rabbit_tcp_client_sup,rabbit_sup,<0.4863.0>]\r\n    messages: []\r\n    links: [<0.28642.6557>]\r\n    dictionary: [{{queue_stats,{resource,<<\"/\">>,queue,<<\"djjob.quick\">>}},\r\n                   [{deliver,1},{redeliver,1}]},\r\n                  {credit_blocked,[]},\r\n                  {delegate,delegate_3},\r\n                  {msg_size_for_gc,2077},\r\n                  {process_name,\r\n                      {rabbit_channel,\r\n                          {<<\"10.178.189.57:59650 -> 10.0.11.174:5672\">>,1}}}]\r\n    trap_exit: true\r\n    status: running\r\n    heap_size: 4185\r\n    stack_size: 27\r\n    reductions: 8423\r\n  neighbours:\r\n\r\n=SUPERVISOR REPORT==== 4-Sep-2015::14:23:56 ===\r\n     Supervisor: {<0.29726.6380>,rabbit_channel_sup}\r\n     Context:    child_terminated\r\n     Reason:     {function_clause,\r\n                     [{rabbit_channel,handle_info,2,[]},\r\n                      {gen_server2,handle_msg,2,[]},\r\n                      {rabbit_mgmt_wm_overview,module_info,1,[]}]}\r\n     Offender:   [{pid,<0.22944.6380>},\r\n                  {name,channel},\r\n                  {mfargs,\r\n                      {rabbit_channel,start_link,\r\n                          [1,<0.10639.6372>,<0.8494.6369>,<0.10639.6372>,\r\n                           <<\"10.140.163.98:52524 -> 10.0.11.174:5672\">>,\r\n                           rabbit_framing_amqp_0_9_1,\r\n                           {user,<<\"uberxxx\">>,[],\r\n                               [{rabbit_auth_backend_internal,none}]},\r\n                           <<\"/\">>,\r\n                           [{<<\"publisher_confirms\">>,bool,true},\r\n                            {<<\"consumer_cancel_notify\">>,bool,true},\r\n                            {<<\"exchange_exchange_bindings\">>,bool,true},\r\n                            {<<\"basic.nack\">>,bool,true},\r\n                            {<<\"connection.blocked\">>,bool,true},\r\n                            {<<\"authentication_failure_close\">>,bool,true}],\r\n                           <0.20398.6380>,<0.3201.6381>]}},\r\n                  {restart_type,intrinsic},\r\n                  {shutdown,4294967295},\r\n                  {child_type,worker}]\r\n\r\n\r\n=SUPERVISOR REPORT==== 4-Sep-2015::14:23:56 ===\r\n     Supervisor: {<0.29726.6380>,rabbit_channel_sup}\r\n     Context:    shutdown\r\n     Reason:     reached_max_restart_intensity\r\n     Offender:   [{pid,<0.22944.6380>},\r\n                  {name,channel},\r\n                  {mfargs,\r\n                      {rabbit_channel,start_link,\r\n                          [1,<0.10639.6372>,<0.8494.6369>,<0.10639.6372>,\r\n                           <<\"10.140.163.98:52524 -> 10.0.11.174:5672\">>,\r\n                           rabbit_framing_amqp_0_9_1,\r\n                           {user,<<\"uberxxx\">>,[],\r\n                               [{rabbit_auth_backend_internal,none}]},\r\n                           <<\"/\">>,\r\n                           [{<<\"publisher_confirms\">>,bool,true},\r\n                            {<<\"consumer_cancel_notify\">>,bool,true},\r\n                            {<<\"exchange_exchange_bindings\">>,bool,true},\r\n                            {<<\"basic.nack\">>,bool,true},\r\n                            {<<\"connection.blocked\">>,bool,true},\r\n                            {<<\"authentication_failure_close\">>,bool,true}],\r\n                           <0.20398.6380>,<0.3201.6381>]}},\r\n                  {restart_type,intrinsic},\r\n                  {shutdown,4294967295},\r\n                  {child_type,worker}]\r\n\r\n\r\n=CRASH REPORT==== 4-Sep-2015::14:23:56 ===\r\n  crasher:\r\n    initial call: gen:init_it/6\r\n    pid: <0.7322.5965>\r\n    registered_name: []\r\n    exception exit: {function_clause,\r\n                        [{rabbit_channel,handle_info,2,[]},\r\n                         {gen_server2,handle_msg,2,[]},\r\n                         {rabbit_mgmt_wm_overview,module_info,1,[]}]}\r\n      in function  gen_server2:terminate/3\r\n      in call from rabbit_mgmt_wm_overview:module_info/1\r\n    ancestors: [<0.4950.5965>,<0.24355.5952>,<0.7810.5965>,<0.8862.5947>,\r\n                  rabbit_tcp_client_sup,rabbit_sup,<0.4863.0>]\r\n    messages: []\r\n    links: [<0.4950.5965>]\r\n    dictionary: [{delegate,delegate_9},\r\n                  {guid_secure,\r\n                      {{11,'rabbit@rabbitmq-us-east-1c',#Ref<0.0.3333.114301>},\r\n                       0}},\r\n                  {msg_size_for_gc,695},\r\n                  {{credit_from,<9876.5461.0>},196},\r\n                  {process_name,\r\n                      {rabbit_channel,\r\n                          {<<\"10.178.189.57:42651 -> 10.0.11.174:5672\">>,1}}},\r\n                  {{credit_from,<0.5465.0>},196},\r\n                  {{credit_to,<0.19404.5943>},48},\r\n                  {{xtype_to_module,direct},rabbit_exchange_type_direct},\r\n                  {credit_blocked,[]},\r\n                  {guid,{{829018877,3024599086,2624743100,3587034967},1}},\r\n                  {{exchange_stats,{resource,<<\"/\">>,exchange,<<\"cronx\">>}},\r\n                   [{publish,2}]}]\r\n    trap_exit: true\r\n    status: running\r\n    heap_size: 987\r\n    stack_size: 27\r\n    reductions: 5867\r\n  neighbours:\r\n\r\n=SUPERVISOR REPORT==== 4-Sep-2015::14:23:56 ===\r\n     Supervisor: {<0.4950.5965>,rabbit_channel_sup}\r\n     Context:    child_terminated\r\n     Reason:     {function_clause,\r\n                     [{rabbit_channel,handle_info,2,[]},\r\n                      {gen_server2,handle_msg,2,[]},\r\n                      {rabbit_mgmt_wm_overview,module_info,1,[]}]}\r\n     Offender:   [{pid,<0.7322.5965>},\r\n                  {name,channel},\r\n                  {mfargs,\r\n                      {rabbit_channel,start_link,\r\n                          [1,<0.19404.5943>,<0.27371.5962>,<0.19404.5943>,\r\n                           <<\"10.178.189.57:42651 -> 10.0.11.174:5672\">>,\r\n                           rabbit_framing_amqp_0_9_1,\r\n                           {user,<<\"cronx\">>,[],\r\n                               [{rabbit_auth_backend_internal,none}]},\r\n                           <<\"/\">>,[],<0.5646.5965>,<0.7247.5965>]}},\r\n                  {restart_type,intrinsic},\r\n                  {shutdown,4294967295},\r\n                  {child_type,worker}]\r\n\r\n\r\n=SUPERVISOR REPORT==== 4-Sep-2015::14:23:56 ===\r\n     Supervisor: {<0.4950.5965>,rabbit_channel_sup}\r\n     Context:    shutdown\r\n     Reason:     reached_max_restart_intensity\r\n     Offender:   [{pid,<0.7322.5965>},\r\n                  {name,channel},\r\n                  {mfargs,\r\n                      {rabbit_channel,start_link,\r\n                          [1,<0.19404.5943>,<0.27371.5962>,<0.19404.5943>,\r\n                           <<\"10.178.189.57:42651 -> 10.0.11.174:5672\">>,\r\n                           rabbit_framing_amqp_0_9_1,\r\n                           {user,<<\"cronx\">>,[],\r\n                               [{rabbit_auth_backend_internal,none}]},\r\n                           <<\"/\">>,[],<0.5646.5965>,<0.7247.5965>]}},\r\n                  {restart_type,intrinsic},\r\n                  {shutdown,4294967295},\r\n                  {child_type,worker}]\r\n```"},"comment":{"url":"https://api.github.com/repos/rabbitmq/rabbitmq-server/issues/comments/137962871","html_url":"https://github.com/rabbitmq/rabbitmq-server/issues/298#issuecomment-137962871","issue_url":"https://api.github.com/repos/rabbitmq/rabbitmq-server/issues/298","id":137962871,"user":{"login":"michaelklishin","id":1090,"avatar_url":"https://avatars.githubusercontent.com/u/1090?v=3","gravatar_id":"","url":"https://api.github.com/users/michaelklishin","html_url":"https://github.com/michaelklishin","followers_url":"https://api.github.com/users/michaelklishin/followers","following_url":"https://api.github.com/users/michaelklishin/following{/other_user}","gists_url":"https://api.github.com/users/michaelklishin/gists{/gist_id}","starred_url":"https://api.github.com/users/michaelklishin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/michaelklishin/subscriptions","organizations_url":"https://api.github.com/users/michaelklishin/orgs","repos_url":"https://api.github.com/users/michaelklishin/repos","events_url":"https://api.github.com/users/michaelklishin/events{/privacy}","received_events_url":"https://api.github.com/users/michaelklishin/received_events","type":"User","site_admin":false},"created_at":"2015-09-05T14:57:00Z","updated_at":"2015-09-05T14:57:00Z","body":"1. https://groups.google.com/d/msg/rabbitmq-users/P5g8CRI1Owg/T01ob_Bn9oAJ\r\n2. I don't see any evidence of a \"plugin crash\". The errors you see in the log are from channel processes that couldn't emit stats because they couldn't contact stats DB. Eventually their supervisor was restarted.\r\nEven though there is a \"crash\" in the name, these are non-fatal termination and only affect a single unfortunate connection."}}