{"action":"closed","issue":{"url":"https://api.github.com/repos/rabbitmq/rabbitmq-lvc-plugin/issues/2","labels_url":"https://api.github.com/repos/rabbitmq/rabbitmq-lvc-plugin/issues/2/labels{/name}","comments_url":"https://api.github.com/repos/rabbitmq/rabbitmq-lvc-plugin/issues/2/comments","events_url":"https://api.github.com/repos/rabbitmq/rabbitmq-lvc-plugin/issues/2/events","html_url":"https://github.com/rabbitmq/rabbitmq-lvc-plugin/issues/2","id":95391820,"number":2,"title":"Issue with LVC exchange - Mnesia is not replicating the contents for a LVC table","user":{"login":"Ashish123gs","id":13363432,"avatar_url":"https://avatars.githubusercontent.com/u/13363432?v=3","gravatar_id":"","url":"https://api.github.com/users/Ashish123gs","html_url":"https://github.com/Ashish123gs","followers_url":"https://api.github.com/users/Ashish123gs/followers","following_url":"https://api.github.com/users/Ashish123gs/following{/other_user}","gists_url":"https://api.github.com/users/Ashish123gs/gists{/gist_id}","starred_url":"https://api.github.com/users/Ashish123gs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Ashish123gs/subscriptions","organizations_url":"https://api.github.com/users/Ashish123gs/orgs","repos_url":"https://api.github.com/users/Ashish123gs/repos","events_url":"https://api.github.com/users/Ashish123gs/events{/privacy}","received_events_url":"https://api.github.com/users/Ashish123gs/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":{"login":"videlalvaro","id":30834,"avatar_url":"https://avatars.githubusercontent.com/u/30834?v=3","gravatar_id":"","url":"https://api.github.com/users/videlalvaro","html_url":"https://github.com/videlalvaro","followers_url":"https://api.github.com/users/videlalvaro/followers","following_url":"https://api.github.com/users/videlalvaro/following{/other_user}","gists_url":"https://api.github.com/users/videlalvaro/gists{/gist_id}","starred_url":"https://api.github.com/users/videlalvaro/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/videlalvaro/subscriptions","organizations_url":"https://api.github.com/users/videlalvaro/orgs","repos_url":"https://api.github.com/users/videlalvaro/repos","events_url":"https://api.github.com/users/videlalvaro/events{/privacy}","received_events_url":"https://api.github.com/users/videlalvaro/received_events","type":"User","site_admin":false},"milestone":null,"comments":0,"created_at":"2015-07-16T09:31:48Z","updated_at":"2015-07-16T13:07:04Z","closed_at":"2015-07-16T13:07:04Z","body":"Please file an issue on the LVC repo and we'll try to port the fixes to this exchange\r\n\r\nOn Thu, Jul 16, 2015 at 11:08 AM Ojha, Ashish <Ashish.Ojha@gs.com> wrote:\r\nSome of the tests I did when we had this issue .\r\n \r\n1.       Declaring a new LVC exchange is not an issue .\r\n2.       Deleting a LVC exchange is a problem  , it gives below error when I try deleting a LVC exchange .\r\n \r\nGot response code 500 with body {\"error\":\"Internal Server Error\",\"reason\":\"{exit,\\n {{{error,{no_exists,lvc}},\\n [{rabbit_misc,execute_mnesia_transaction,1,\\n [{file,\\\"src/rabbit_misc.erl\\\"},{line,540}]},\\n {rabbit_misc,execute_mnesia_tx_with_tail,1,\\n [{file,\\\"src/rabbit_misc.erl\\\"},{line,562}]},\\n {rabbit_channel,handle_method,3,\\n [{file,\\\"src/rabbit_channel.erl\\\"},{line,1093}]},\\n {rabbit_channel,handle_cast,2,\\n [{file,\\\"src/rabbit_channel.erl\\\"},{line,326}]},\\n {gen_server2,handle_msg,2,[{file,\\\"src/gen_server2.erl\\\"},{line,1034}]},\\n {proc_lib,init_p_do_apply,3,[{file,\\\"proc_lib.erl\\\"},{line,239}]}]},\\n {gen_server,call,\\n [<0.23460.100>,\\n {call,{'exchange.delete',0,<<\\\"ddd\\\">>,false,false},none,<0.7589.100>},\\n infinity]}},\\n [{gen_server,call,3,[{file,\\\"gen_server.erl\\\"},{line,188}]},\\n {rabbit_mgmt_util,'-amqp_request/5-fun-0-',4,\\n [{file,\\\"rabbitmq-management/src/rabbit_mgmt_util.erl\\\"},{line,416}]},\\n {rabbit_mgmt_util,with_channel,5,\\n [{file,\\\"rabbitmq-management/src/rabbit_mgmt_util.erl\\\"},{line,435}]},\\n {webmachine_resource,resource_call,3,\\n [{file,\\n \\\"webmachine-wrapper/webmachine-git/src/webmachine_resource.erl\\\"},\\n {line,186}]},\\n {webmachine_resource,do,3,\\n [{file,\\n \\\"webmachine-wrapper/webmachine-git/src/webmachine_resource.erl\\\"},\\n {line,142}]},\\n {webmachine_decision_core,resource_call,1,\\n [{file,\\n \\\"webmachine-wrapper/webmachine-git/src/webmachine_decision_core.erl\\\"},\\n {line,48}]},\\n {webmachine_decision_core,decision,1,\\n [{file,\\n \\\"webmachine-wrapper/webmachine-git/src/webmachine_decision_core.erl\\\"},\\n {line,440}]},\\n {webmachine_decision_core,handle_request,2,\\n [{file,\\n \\\"webmachine-wrapper/webmachine-git/src/webmachine_decision_core.erl\\\"},\\n {line,33}]}]}\\n\"}\r\n \r\n3.       During this issue , publisher code always fails while consumers still can consumer from the queue which is bind to the LVC exchange .\r\n4.       In 3 node cluster if we shut down one node , then publisher fails to publish to LVC exchange . Consumers still works fine .\r\n \r\n \r\nIs it possible to fix the LVC exchange for the bug you have mentioned ?\r\nor is LVC exchange no more supported and users should move off LVC and use Recent History Exchange ?\r\n \r\nWe have LVC exchange deployed in production so wondering if the LVC can be fixed . Moving to Recent History Exchange will need time/effort/testing ..\r\n \r\nFrom: Alvaro Videla [mailto:videlalvaro@gmail.com] \r\nSent: Thursday, July 16, 2015 1:56 PM\r\n\r\nTo: Ojha, Ashish [Tech]\r\nCc: Michael Klishin; rabbitmq-users@googlegroups.com\r\nSubject: Re: [rabbitmq-users] Issue while publishing to LVC exchange\r\n \r\nAh OK, sorry, I thought it was a different error this time.\r\n \r\nSo if while the cluster was let's call it \"in error state\", if you declared a new LVC exchange, with a different name from the one you were using, you still get those errors?\r\n \r\nIt looks like Mnesia is not replicating the contents for that particular LVC table. \r\n \r\nLooking at the code it seems LVC doesn't add table copies:\r\n \r\nhttps://github.com/rabbitmq/rabbitmq-lvc-plugin/blob/master/src/rabbit_lvc_plugin.erl#L18\r\n \r\nWhich is a bug that we fixed for the recent history exchange plugin: https://github.com/videlalvaro/rabbitmq-recent-history-exchange/blob/master/src/rabbit_exchange_type_recent_history.erl#L119 \r\n \r\nThis was the related issue https://github.com/videlalvaro/rabbitmq-recent-history-exchange/issues/9\r\n \r\nSo I think using the LVC plugin you could run on the same issue as well. \r\n \r\nIf you use the recent-history-exchange with a \"x-recent-history-length\" value of 1, then you should get the same behaviour as with the LVC plugin.\r\n \r\n \r\n \r\nOn Thu, Jul 16, 2015 at 10:15 AM, Ojha, Ashish <Ashish.Ojha@gs.com> wrote:\r\nsure , I pasted the error in my first email .\r\nadding the error again .\r\n \r\n \r\nclient logs ;\r\n \r\n[20150710 10:32:39.842-S-OmaServerMessage-Thread-3] Unexpected exception in OmaServerMessage.process(): com.rabbitmq.client.AlreadyClosedException: connection is already closed due to connection error; protocol method: #method<connection.close>(reply-code=541, reply-text=INTERNAL_ERROR, class-id=0, method-id=0)\r\n\r\ncom.rabbitmq.client.AlreadyClosedException: connection is already closed due to connection error; protocol method: #method<connection.close>(reply-code=541, reply-text=INTERNAL_ERROR, class-id=0, method-id=0)\r\n        at com.rabbitmq.client.impl.AMQChannel.ensureIsOpen(AMQChannel.java:195)\r\n        at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:296)\r\n        at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:648)\r\n        at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:631)\r\n        at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:622)\r\n        at com.rabbitmq.client.impl.recovery.AutorecoveringChannel.basicPublish(AutorecoveringChannel.java:160)\r\nBroker logs :\r\n \r\n=ERROR REPORT==== 10-Jul-2015::09:43:46 ===\r\n** Generic server <0.9385.62> terminating\r\n** Last message in was {'$gen_cast',\r\n                           {method,\r\n                               {'queue.bind',0,\r\n                                   <<\"NONPROD.JETSTREAM.QA.JetStreamOma.jtdb\">>,\r\n                                   <<\"NONPROD.JETSTREAM.QA.JSQA.Lvc\">>,\r\n                                   <<\"JetStreamOma.jtdb\">>,false,[]},\r\n                               none,noflow}}\r\n** When Server state == {ch,running,rabbit_framing_amqp_0_9_1,1,<0.9381.62>,\r\n                            <0.9387.62>,<0.9381.62>,\r\n                            <<\"10.152.132.150:55490 -> 10.152.155.201:25009\">>,\r\n                            {lstate,<0.9386.62>,false},\r\n                                                                                    \r\n                            <<\"10.152.132.150:55490 -> 10.152.155.201:25009\">>,\r\n                            {lstate,<0.9386.62>,false},\r\n                            none,1,\r\n                            {[],[]},\r\n                            {user,<<\"nonprod.jetstream-oma\">>,[],\r\n                                [{rabbit_auth_backend_internal,none}]},\r\n                            <<\"/\">>,\r\n                            <<\"NONPROD.JETSTREAM.QA.JetStreamOma.jtdb\">>,\r\n                            {dict,0,16,16,8,80,48,\r\n                                {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],\r\n                                 []},\r\n                                {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],\r\n                                  [],[]}}},\r\n                            {state,\r\n                                {dict,0,16,16,8,80,48,\r\n                                    {[],[],[],[],[],[],[],[],[],[],[],[],[],\r\n                                     [],[],[]},\r\n                                    {{[],[],[],[],[],[],[],[],[],[],[],[],[],\r\n                                      [],[],[]}}},\r\n                                erlang},\r\n                            {dict,0,16,16,8,80,48,\r\n                                {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],\r\n                                 []},\r\n                                {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],\r\n                                  [],[]}}},\r\n                            {dict,0,16,16,8,80,48,\r\n                                {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],\r\n                                 []},\r\n                                {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],\r\n                                  [],[]}}},\r\n                            {set,0,16,16,8,80,48,\r\n                                {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],\r\n                                 []},\r\n                                {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],\r\n                                  [],[]}}},\r\n                            <0.9380.62>,\r\n                            {state,fine,5000,#Ref<0.0.66.224148>},\r\n                            false,1,\r\n                            {{0,nil},{0,nil}},\r\n                            [],\r\n                            {{0,nil},{0,nil}},\r\n                            [{<<\"exchange_exchange_bindings\">>,bool,true},\r\n                             {<<\"authentication_failure_close\">>,bool,true},\r\n                                                                                       \r\n                            [{<<\"exchange_exchange_bindings\">>,bool,true},\r\n                             {<<\"authentication_failure_close\">>,bool,true},\r\n                             {<<\"consumer_cancel_notify\">>,bool,true},\r\n                             {<<\"basic.nack\">>,bool,true},\r\n                             {<<\"publisher_confirms\">>,bool,true},\r\n                             {<<\"connection.blocked\">>,bool,true}],\r\n                            none,5,none}\r\n** Reason for termination ==\r\n** {{aborted,\r\n        {no_exists,\r\n            [lvc,\r\n             {cachekey,\r\n                 {resource,<<\"/\">>,exchange,\r\n                     <<\"NONPROD.JETSTREAM.QA.JSQA.Lvc\">>},\r\n                 <<\"JetStreamOma.jtdb\">>}]}},\r\n    [{mnesia,abort,1,[{file,\"mnesia.erl\"},{line,309}]},\r\n     {rabbit_exchange_type_lvc,add_binding,3,\r\n         [{file,\"src/rabbit_exchange_type_lvc.erl\"},{line,66}]},\r\n     {rabbit_binding,x_callback,4,\r\n         [{file,\"src/rabbit_binding.erl\"},{line,561}]},\r\n     {rabbit_binding,'-add/3-fun-0-',3,\r\n         [{file,\"src/rabbit_binding.erl\"},{line,193}]},\r\n     {rabbit_channel,binding_action,9,\r\n         [{file,\"src/rabbit_channel.erl\"},{line,1471}]},\r\n     {rabbit_channel,handle_cast,2,\r\n         [{file,\"src/rabbit_channel.erl\"},{line,326}]},\r\n     {gen_server2,handle_msg,2,[{file,\"src/gen_server2.erl\"},{line,1034}]},\r\n     {proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,239}]}]}\r\n \r\n=ERROR REPORT==== 10-Jul-2015::09:43:46 ===\r\nError on AMQP connection <0.9381.62> (10.152.132.150:55490 -> 10.152.155.201:25009, vhost: '/', user: 'nonprod.jetstream-oma', state: running), channel 1:\r\n{{aborted,\r\n     {no_exists,\r\n         [lvc,\r\n          {cachekey,\r\n              {resource,<<\"/\">>,exchange,<<\"NONPROD.JETSTREAM.QA.JSQA.Lvc\">>},\r\n              <<\"JetStreamOma.jtdb\">>}]}},\r\n[{mnesia,abort,1,[{file,\"mnesia.erl\"},{line,309}]},\r\n  {rabbit_exchange_type_lvc,add_binding,3,\r\n      [{file,\"src/rabbit_exchange_type_lvc.erl\"},{line,66}]},\r\n {rabbit_binding,x_callback,4,[{file,\"src/rabbit_binding.erl\"},{line,561}]},\r\n  {rabbit_binding,'-add/3-fun-0-',3,\r\n      [{file,\"src/rabbit_binding.erl\"},{line,193}]},\r\n                                                                                           \r\n  {rabbit_binding,'-add/3-fun-0-',3,\r\n      [{file,\"src/rabbit_binding.erl\"},{line,193}]},\r\n  {rabbit_channel,binding_action,9,\r\n      [{file,\"src/rabbit_channel.erl\"},{line,1471}]},\r\n  {rabbit_channel,handle_cast,2,[{file,\"src/rabbit_channel.erl\"},{line,326}]},\r\n  {gen_server2,handle_msg,2,[{file,\"src/gen_server2.erl\"},{line,1034}]},\r\n  {proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,239}]}]}\r\n \r\n=WARNING REPORT==== 10-Jul-2015::09:43:46 ===\r\nNon-AMQP exit reason '{{aborted,\r\n                        {no_exists,\r\n                         [lvc,\r\n                          {cachekey,\r\n                           {resource,<<\"/\">>,exchange,\r\n                            <<\"NONPROD.JETSTREAM.QA.JSQA.Lvc\">>},\r\n                           <<\"JetStreamOma.jtdb\">>}]}},\r\n                       [{mnesia,abort,1,[{file,\"mnesia.erl\"},{line,309}]},\r\n                        {rabbit_exchange_type_lvc,add_binding,3,\r\n                         [{file,\"src/rabbit_exchange_type_lvc.erl\"},\r\n                          {line,66}]},\r\n                        {rabbit_binding,x_callback,4,\r\n                         [{file,\"src/rabbit_binding.erl\"},{line,561}]},\r\n                        {rabbit_binding,'-add/3-fun-0-',3,\r\n                         [{file,\"src/rabbit_binding.erl\"},{line,193}]},\r\n                        {rabbit_channel,binding_action,9,\r\n                         [{file,\"src/rabbit_channel.erl\"},{line,1471}]},\r\n                        {rabbit_channel,handle_cast,2,\r\n                         [{file,\"src/rabbit_channel.erl\"},{line,326}]},\r\n                        {gen_server2,handle_msg,2,\r\n                         [{file,\"src/gen_server2.erl\"},{line,1034}]},\r\n                        {proc_lib,init_p_do_apply,3,\r\n                         [{file,\"proc_lib.erl\"},{line,239}]}]}'\r\n \r\n \r\n \r\nFrom: Alvaro Videla [mailto:videlalvaro@gmail.com] \r\nSent: Thursday, July 16, 2015 1:44 PM\r\nTo: Ojha, Ashish [Tech]\r\nCc: Michael Klishin; rabbitmq-users@googlegroups.com\r\n\r\nSubject: Re: [rabbitmq-users] Issue while publishing to LVC exchange\r\n \r\nIt would be nice to get what errors were produced in the failing scenario to see if it's something we can fix on our side.\r\n \r\nOn Thu, Jul 16, 2015 at 10:09 AM, Ojha, Ashish <Ashish.Ojha@gs.com> wrote:\r\nThanks Alvaro , really appreciate your help on this one .\r\n \r\nso my assumption and resolution looks fine to you if we hit this issue again ? \r\n \r\nFrom: Alvaro Videla [mailto:videlalvaro@gmail.com] \r\nSent: Thursday, July 16, 2015 1:36 PM\r\nTo: Ojha, Ashish [Tech]; Michael Klishin\r\nCc: rabbitmq-users@googlegroups.com\r\nSubject: Re: [rabbitmq-users] Issue while publishing to LVC exchange\r\n \r\nThe LVC exchange (and the recent history exchange as well) keeps a small table in Mnesia where they store the last N messages.\r\n \r\nOn Thu, Jul 16, 2015 at 9:05 AM Ojha, Ashish <Ashish.Ojha@gs.com> wrote:\r\nHi Michael ,\r\n \r\nWe had this issue with LVC exchange and we figured out a way to fix it . Is it possible if we can get some understanding of the issue please or the behavior of the LVC exchange \r\n \r\nScenario :\r\n \r\n3 Node cluster ( A,B,C ) with Mirroring enabled for all the queues .\r\nOn Sunday , Node B and Node C were rebuilt from scratch ( mnesia was recreated ) meaning they went down and joined the cluster as virgin nodes .\r\nNode A was untouched .\r\nAfter Node B and Node C were back up , it clustered with Node A and everything worked fine .\r\nLater we realized that LVC exchange is not working fine .\r\nAny connection made to LVC exchange was closed down by the broker ( when the publisher invoked basicPublish() method )\r\nAll other exchanges were working fine ( topic , direct exchange )\r\n \r\n \r\nQuestion :\r\n \r\n- Does LVC exchange maintains any kind of state between the nodes in the cluster ?\r\n- In our scenario , only Node A was 'unchanged' , so is it possible that only Node A knew about LVC exchange state and when Node B and Node C joined the cluster there was a conflict between the LVC exchange state in the mnesia LVC table  ?\r\n \r\nHow we resolved the issue \r\n \r\n- disable LVC plugin .\r\n-  shutdown all the nodes and start them in order ( last disc to go down was the first to come back up )\r\n- enable the LVC plugin .\r\n \r\nwe tested LVC exchange after this resolution and everything was working fine .\r\n \r\n \r\n-----Original Message-----\r\nFrom: Michael Klishin [mailto:mklishin@pivotal.io] \r\nSent: Friday, July 10, 2015 5:33 PM\r\nTo: Ojha, Ashish [Tech]\r\nCc: rabbitmq-users@googlegroups.com\r\nSubject: RE: [rabbitmq-users] Issue while publishing to LVC exchange\r\n \r\n On 10 July 2015 at 15:02:08, Ojha, Ashish (ashish.ojha@gs.com) wrote:\r\n> plugin version is also 3.5.0 and we get it from github \r\n> https://github.com/rabbitmq/rabbitmq-lvc-plugin\r\n> and build it from source\r\n \r\nPlease use RabbitMQ 3.5.3 and the pre-compiled plugin version from our community plugins page.\r\n--\r\nMK  \r\n \r\nStaff Software Engineer, Pivotal/RabbitMQ  \r\n \r\n \r\n \r\n-- \r\nYou received this message because you are subscribed to the Google Groups \"rabbitmq-users\" group.\r\nTo unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.\r\nTo post to this group, send email to rabbitmq-users@googlegroups.com.\r\nFor more options, visit https://groups.google.com/d/optout.\r\n \r\n \r\n"}}