{"action":"created","issue":{"url":"https://api.github.com/repos/basho/machi/issues/37","labels_url":"https://api.github.com/repos/basho/machi/issues/37/labels{/name}","comments_url":"https://api.github.com/repos/basho/machi/issues/37/comments","events_url":"https://api.github.com/repos/basho/machi/issues/37/events","html_url":"https://github.com/basho/machi/issues/37","id":114259150,"number":37,"title":"client error when chain UPI=[] (unavailable), also perhaps file descriptor/other leak?","user":{"login":"slfritchie","id":75320,"avatar_url":"https://avatars.githubusercontent.com/u/75320?v=3","gravatar_id":"","url":"https://api.github.com/users/slfritchie","html_url":"https://github.com/slfritchie","followers_url":"https://api.github.com/users/slfritchie/followers","following_url":"https://api.github.com/users/slfritchie/following{/other_user}","gists_url":"https://api.github.com/users/slfritchie/gists{/gist_id}","starred_url":"https://api.github.com/users/slfritchie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/slfritchie/subscriptions","organizations_url":"https://api.github.com/users/slfritchie/orgs","repos_url":"https://api.github.com/users/slfritchie/repos","events_url":"https://api.github.com/users/slfritchie/events{/privacy}","received_events_url":"https://api.github.com/users/slfritchie/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-10-30T12:19:55Z","updated_at":"2015-10-30T13:03:42Z","closed_at":null,"body":"When I was at dinner, running `env EQC_TIMEOUT=3600 rebar skip_deps=true -v eunit suites=machi_ap_repair_eqc tests=prop_repair_par_test_` on the commit b5005c35263e79389c42b1808fce1171b44f4fb3 (branch `ss-repair-with-partition-simulator`) with the https://gist.github.com/slfritchie/12e40859a08d5e4a678a patch applied, I saw this when I returned.\r\n\r\nAt a minimum, the machi_cr_client should do something less silly when the chain is not available.\r\n\r\nAlso, there may be a resource leak (e.g. file descriptor) that caused this test to fail after about 52 iterations (estimate, based on test output).\r\n\r\n    .  Got stable chain: [{a,{1113,[a,b,c],[],[]}},{b,{1113,[a,b,c],[],[]}},{c,{1113,[a,b,c],[],[]}}]\r\n    ==== Start post operations, stabilize and confirm results\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.31552.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.31538.2>\r\n      NOT YET stable chain: [{a,{1117,[c],[a,b],[]}},{b,{1117,[c],[a,b],[]}},{c,{1117,[c],[a,b],[]}}]\r\n      NOT YET stable chain: [{a,{1117,[c],[a,b],[]}},{b,{1117,[c],[a,b],[]}},{c,{1117,[c],[a,b],[]}}]\r\n      NOT YET stable chain: [{a,{1117,[c],[a,b],[]}},{b,{1117,[c],[a,b],[]}},{c,{1117,[c],[a,b],[]}}]\r\n    MissingFileSummary [{<<\"pre^3ef5a24e-3833-4ce3-9b32-e24aaf968795^1\">>,\r\n                         {1094,[a,b]}},\r\n                        {<<\"pre^69061121-5f50-4610-ac0e-4d9df67e249d^2\">>,\r\n                         {1044,[]}},\r\n                        {<<\"pre^b6be4085-b1a4-44b7-93a2-7f907acf9c95^1\">>,\r\n                         {1084,[b,c]}}]\r\n    Make repair directives: ... done\r\n    Out-of-sync data for FLU a: 0.1 MBytes\r\n    Out-of-sync data for FLU b: 0.1 MBytes\r\n    Out-of-sync data for FLU c: 0.1 MBytes\r\n    Execute repair directives: .. done\r\n      Got stable chain: [{a,{1119,[c,a,b],[],[]}},{b,{1119,[c,a,b],[],[]}},{c,{1119,[c,a,b],[],[]}}]\r\n      Written=10, DATALOSS=0, Acceptable=5\r\n      Failed=0, Critical=0\r\n    \r\n    .  Got stable chain: [{a,{1113,[a,b,c],[],[]}},{b,{1113,[a,b,c],[],[]}},{c,{1113,[a,b,c],[],[]}}]\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32479.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32479.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32479.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32479.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32479.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32479.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32479.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32479.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32479.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32479.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32455.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32455.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32455.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32455.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32455.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32455.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32455.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32455.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32479.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32455.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32455.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32483.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32483.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32483.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32483.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32483.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32483.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32483.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32455.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32479.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32483.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32483.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32483.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32455.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32483.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32463.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32463.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32463.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32463.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32463.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32463.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32463.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32463.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32463.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32479.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32463.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32483.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32463.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32455.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32447.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32447.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32447.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32447.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32447.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32463.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32447.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32447.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32447.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32483.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32447.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32487.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32487.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32487.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32487.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32487.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32487.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32487.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32487.2>\r\n    ==== Start post operations, stabilize and confirm results\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32463.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32463.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32463.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32463.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32463.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32463.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32463.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32463.2>\r\n      NOT YET stable chain: [{a,{1126,[c],[b,a],[]}},{b,{1126,[c],[b,a],[]}},{c,{1126,[c],[b,a],[]}}]\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32463.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32463.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32463.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32463.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32463.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32463.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32463.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32455.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32463.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32463.2>\r\n      NOT YET stable chain: [{a,{1126,[c],[b,a],[]}},{b,{1126,[c],[b,a],[]}},{c,{1126,[c],[b,a],[]}}]\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32463.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32483.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32463.2>\r\n      NOT YET stable chain: [{a,{1126,[c],[b,a],[]}},{b,{1126,[c],[b,a],[]}},{c,{1126,[c],[b,a],[]}}]\r\n      NOT YET stable chain: [{a,{1126,[c],[b,a],[]}},{b,{1126,[c],[b,a],[]}},{c,{1126,[c],[b,a],[]}}]\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32463.2>\r\n      NOT YET stable chain: [{a,{1126,[c],[b,a],[]}},{b,{1126,[c],[b,a],[]}},{c,{1126,[c],[b,a],[]}}]\r\n      NOT YET stable chain: [{a,{1126,[c],[b,a],[]}},{b,{1126,[c],[b,a],[]}},{c,{1126,[c],[b,a],[]}}]\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32463.2>\r\n    MissingFileSummary [{<<\"pre^58d8ff5c-2371-4e5e-85f8-4fca1f44eedf^1\">>,\r\n                         {1034,[]}},\r\n                        {<<\"pre^8e67a7c2-570d-41e6-bf48-9405bfd777c2^1\">>,\r\n                         {1054,[c]}},\r\n                        {<<\"pre^979fad11-8e0e-4674-af5f-58aa3e5be461^2\">>,\r\n                         {1034,[a]}},\r\n                        {<<\"pre^e7670123-4c9e-41a3-8c20-93518dca6e01^1\">>,\r\n                         {1064,[a,b]}}]\r\n    Make repair directives: .... done\r\n    Out-of-sync data for FLU a: 0.1 MBytes\r\n    Out-of-sync data for FLU b: 0.1 MBytes\r\n    Out-of-sync data for FLU c: 0.1 MBytes\r\n    Execute repair directives: ... done\r\n      Got stable chain: [{a,{1128,[c,b,a],[],[]}},{b,{1128,[c,b,a],[],[]}},{c,{1128,[c,b,a],[],[]}}]\r\n      Written=8, DATALOSS=0, Acceptable=6\r\n      Failed=0, Critical=0\r\n    \r\n    .TODO: Using ?WORST_PROJ, chain is not available <0.32463.2>\r\n    TODO: Using ?WORST_PROJ, chain is not available <0.32483.2>\r\n    *failed*\r\n    in function eqc:quickcheck/1 (../src/eqc.erl, line 1270)\r\n    in call from machi_ap_repair_eqc:'-prop_repair_par_test_/0-fun-1-'/3 (test/machi_ap_repair_eqc.erl, line 91)\r\n    **exit:{{badmatch,[]},\r\n     [{machi_cr_client,do_append_head2,7,\r\n                       [{file,\"src/machi_cr_client.erl\"},{line,330}]},\r\n      {gen_server,try_handle_call,4,[{file,\"gen_server.erl\"},{line,607}]},\r\n      {gen_server,handle_msg,5,[{file,\"gen_server.erl\"},{line,639}]},\r\n      {proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,237}]}]}\r\n      output:<<\"Starting Quviq QuickCheck version 1.35.2\r\n       (compiled at {{2015,6,23},{16,47,22}})\r\n    Licence for Basho reserved until {{2015,10,31},{2,57,8}}\r\n    \">>\r\n"},"comment":{"url":"https://api.github.com/repos/basho/machi/issues/comments/152517984","html_url":"https://github.com/basho/machi/issues/37#issuecomment-152517984","issue_url":"https://api.github.com/repos/basho/machi/issues/37","id":152517984,"user":{"login":"slfritchie","id":75320,"avatar_url":"https://avatars.githubusercontent.com/u/75320?v=3","gravatar_id":"","url":"https://api.github.com/users/slfritchie","html_url":"https://github.com/slfritchie","followers_url":"https://api.github.com/users/slfritchie/followers","following_url":"https://api.github.com/users/slfritchie/following{/other_user}","gists_url":"https://api.github.com/users/slfritchie/gists{/gist_id}","starred_url":"https://api.github.com/users/slfritchie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/slfritchie/subscriptions","organizations_url":"https://api.github.com/users/slfritchie/orgs","repos_url":"https://api.github.com/users/slfritchie/repos","events_url":"https://api.github.com/users/slfritchie/events{/privacy}","received_events_url":"https://api.github.com/users/slfritchie/received_events","type":"User","site_admin":false},"created_at":"2015-10-30T13:03:42Z","updated_at":"2015-10-30T13:03:42Z","body":"More info, when running with the SASL app running & logging errors.\r\n\r\n    =CRASH REPORT==== 30-Oct-2015::21:41:22 ===\r\n      crasher:\r\n        initial call: machi_cr_client:init/1\r\n        pid: <0.1933.4>\r\n        registered_name: []\r\n        exception exit: {{badmatch,[]},\r\n                         [{machi_cr_client,do_append_head2,7,\r\n                              [{file,\"src/machi_cr_client.erl\"},{line,330}]},\r\n                          {gen_server,try_handle_call,4,\r\n                              [{file,\"gen_server.erl\"},{line,607}]},\r\n                          {gen_server,handle_msg,5,\r\n                              [{file,\"gen_server.erl\"},{line,639}]},\r\n                          {proc_lib,init_p_do_apply,3,\r\n                              [{file,\"proc_lib.erl\"},{line,237}]}]}\r\n          in function  gen_server:terminate/7 (gen_server.erl, line 804)\r\n    [....]\r\n        neighbour: [{pid,<0.182.0>},  %%% SLF note that this is a really \"old\" PID compared to the crasher\r\n                      {registered_name,[]},\r\n                      {initial_call,{erlang,apply,2}},\r\n                      {current_function,{gen,do_call,4}},\r\n                      {ancestors,[]},\r\n                      {messages,\r\n                       [{#Ref<0.0.0.8917>,{error,partition}},\r\n    [... a few hundred messages more ...]\r\n                      {links,\r\n                       [<0.31733.1>,<0.15066.2>,<0.22243.3>,<0.28288.3>,\r\n                        <0.31857.3>,<0.4687.4>,<0.4779.4>,<0.4867.4>,<0.4907.4>,\r\n    [... very roughly, about 100,000 other process links ...]\r\n                      {dictionary,\r\n                       [{{memo,eqc_statem,non_interfering,\r\n                          {machi_ap_repair_eqc,\r\n                           {state,4,\r\n                            {1445,935441,287549},\r\n                            false,\r\n                            [a,b,c,d],\r\n                            [{a,a_chmgr},{b,b_chmgr},{c,c_chmgr},{d,d_chmgr}],\r\n                            [{a,<0.8424.1>},\r\n                             {b,<0.8425.1>},\r\n                             {c,<0.8427.1>},\r\n                             {d,<0.8429.1>}],\r\n                            [{a,<0.8430.1>},\r\n                             {b,<0.8437.1>},\r\n                             {c,<0.8444.1>},\r\n                             {d,<0.8451.1>},\r\n                             {a,<0.8458.1>},\r\n                             {b,<0.8465.1>},\r\n                             {c,<0.8472.1>},\r\n                             {d,<0.8479.1>},\r\n                             {a,<0.8486.1>},\r\n                             {b,<0.8493.1>},\r\n                             {c,<0.8500.1>},\r\n                             {d,<0.8507.1>}]},\r\n    [... are really super enormous process dictionary continues ...]\r\n"}}