{"action":"created","issue":{"url":"https://api.github.com/repos/Metaswitch/calico/issues/645","labels_url":"https://api.github.com/repos/Metaswitch/calico/issues/645/labels{/name}","comments_url":"https://api.github.com/repos/Metaswitch/calico/issues/645/comments","events_url":"https://api.github.com/repos/Metaswitch/calico/issues/645/events","html_url":"https://github.com/Metaswitch/calico/issues/645","id":86966941,"number":645,"title":"Empty SG chain after power cycle ","user":{"login":"DaveLangridge","id":9798178,"avatar_url":"https://avatars.githubusercontent.com/u/9798178?v=3","gravatar_id":"","url":"https://api.github.com/users/DaveLangridge","html_url":"https://github.com/DaveLangridge","followers_url":"https://api.github.com/users/DaveLangridge/followers","following_url":"https://api.github.com/users/DaveLangridge/following{/other_user}","gists_url":"https://api.github.com/users/DaveLangridge/gists{/gist_id}","starred_url":"https://api.github.com/users/DaveLangridge/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DaveLangridge/subscriptions","organizations_url":"https://api.github.com/users/DaveLangridge/orgs","repos_url":"https://api.github.com/users/DaveLangridge/repos","events_url":"https://api.github.com/users/DaveLangridge/events{/privacy}","received_events_url":"https://api.github.com/users/DaveLangridge/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/Metaswitch/calico/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/Metaswitch/calico/labels/felix","name":"felix","color":"5319e7"}],"state":"open","locked":false,"assignee":{"login":"matthewdupre","id":9053570,"avatar_url":"https://avatars.githubusercontent.com/u/9053570?v=3","gravatar_id":"","url":"https://api.github.com/users/matthewdupre","html_url":"https://github.com/matthewdupre","followers_url":"https://api.github.com/users/matthewdupre/followers","following_url":"https://api.github.com/users/matthewdupre/following{/other_user}","gists_url":"https://api.github.com/users/matthewdupre/gists{/gist_id}","starred_url":"https://api.github.com/users/matthewdupre/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/matthewdupre/subscriptions","organizations_url":"https://api.github.com/users/matthewdupre/orgs","repos_url":"https://api.github.com/users/matthewdupre/repos","events_url":"https://api.github.com/users/matthewdupre/events{/privacy}","received_events_url":"https://api.github.com/users/matthewdupre/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/Metaswitch/calico/milestones/3","html_url":"https://github.com/Metaswitch/calico/milestones/1.0","labels_url":"https://api.github.com/repos/Metaswitch/calico/milestones/3/labels","id":1074686,"number":3,"title":"1.0","description":"","creator":{"login":"Lukasa","id":1382556,"avatar_url":"https://avatars.githubusercontent.com/u/1382556?v=3","gravatar_id":"","url":"https://api.github.com/users/Lukasa","html_url":"https://github.com/Lukasa","followers_url":"https://api.github.com/users/Lukasa/followers","following_url":"https://api.github.com/users/Lukasa/following{/other_user}","gists_url":"https://api.github.com/users/Lukasa/gists{/gist_id}","starred_url":"https://api.github.com/users/Lukasa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Lukasa/subscriptions","organizations_url":"https://api.github.com/users/Lukasa/orgs","repos_url":"https://api.github.com/users/Lukasa/repos","events_url":"https://api.github.com/users/Lukasa/events{/privacy}","received_events_url":"https://api.github.com/users/Lukasa/received_events","type":"User","site_admin":false},"open_issues":9,"closed_issues":32,"state":"open","created_at":"2015-04-20T07:33:48Z","updated_at":"2015-06-29T14:04:12Z","due_on":null,"closed_at":null},"comments":19,"created_at":"2015-06-10T13:15:10Z","updated_at":"2015-06-30T09:54:57Z","closed_at":null,"body":"The test creates some VMs, does some pings, then power cycles one of the compute nodes. Then it does some more ping tests. The test fails more times than it passes (2:1 at a guess). A similar test which reboots the host passes most/all of the time.\r\n\r\nWhen it fails, the test script can’t ssh into a VM on the host that’s been cycled. Looking at the iptables, the structure seems to be present, but most of the SG stuff is missing:\r\n\r\nChain felix-from-cd250fed-c7 (1 references)\r\npkts bytes target     prot opt in     out     source               destination\r\n    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate INVALID\r\n    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED\r\n    2   630 RETURN     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp spt:68 dpt:67\r\n   11   924 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            MARK and 0x0\r\n   11   924 felix-p-_a0438770e0e8efc-o  all  --  *      *       10.28.1.171          0.0.0.0/0            MAC FA:16:3E:20:5A:19\r\n    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match ! 0x1/0x1 /* No mark means profile accepted packet */\r\n    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* Default DROP if no match (endpoint cd250fed-c73e-41b9-8dd2-e7f01512cbe7): */\r\n\r\nChain felix-p-_a0438770e0e8efc-i (1 references)\r\npkts bytes target     prot opt in     out     source               destination\r\n   59  4584 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* WARNING Missing chain DROP: */\r\n\r\nChain felix-p-_a0438770e0e8efc-o (1 references)\r\npkts bytes target     prot opt in     out     source               destination\r\n   11   924 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* WARNING Missing chain DROP: */\r\n\r\nChain felix-to-cd250fed-c7 (1 references)\r\npkts bytes target     prot opt in     out     source               destination\r\n    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate INVALID\r\n    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED\r\n   59  4584 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            MARK and 0x0\r\n   59  4584 felix-p-_a0438770e0e8efc-i  all  --  *      *       0.0.0.0/0            0.0.0.0/0\r\n    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match ! 0x1/0x1 /* No mark means profile accepted packet */\r\n    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* Endpoint cd250fed-c73e-41b9-8dd2-e7f01512cbe7: */\r\n\r\nWhich should be:\r\n\r\nChain felix-from-cd250fed-c7 (1 references)\r\npkts bytes target     prot opt in     out     source               destination\r\n    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate INVALID\r\n    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED\r\n    0     0 RETURN     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp spt:68 dpt:67\r\n    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            MARK and 0x0\r\n    0     0 felix-p-_a0438770e0e8efc-o  all  --  *      *       10.28.1.171          0.0.0.0/0            MAC FA:16:3E:20:5A:19\r\n    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match ! 0x1/0x1 /* No mark means profile accepted packet */\r\n    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* Default DROP if no match (endpoint cd250fed-c73e-41b9-8dd2-e7f01512cbe7): */\r\n\r\nChain felix-p-_a0438770e0e8efc-i (1 references)\r\npkts bytes target     prot opt in     out     source               destination\r\n    0     0 RETURN     icmp --  *      *       0.0.0.0/0            0.0.0.0/0            match-set felix-v4-_a0438770e0e8efc src\r\n    0     0 RETURN     tcp  --  *      *       172.18.203.223       0.0.0.0/0            multiport dports 22\r\n    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* Mark as not matched */ MARK set 0x1\r\n\r\nChain felix-p-_a0438770e0e8efc-o (1 references)\r\npkts bytes target     prot opt in     out     source               destination\r\n    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0\r\n    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* Mark as not matched */ MARK set 0x1\r\n\r\nChain felix-to-cd250fed-c7 (1 references)\r\npkts bytes target     prot opt in     out     source               destination\r\n    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate INVALID\r\n    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED\r\n    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            MARK and 0x0\r\n    0     0 felix-p-_a0438770e0e8efc-i  all  --  *      *       0.0.0.0/0            0.0.0.0/0\r\n    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match ! 0x1/0x1 /* No mark means profile accepted packet */\r\n    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* Endpoint cd250fed-c73e-41b9-8dd2-e7f01512cbe7: */\r\n\r\nThe power cycle occurred at ~10:56. There are some logs like\r\n\r\n2015-06-10 10:56:01,232 [INFO][1768/18] calico.felix.profilerules 203: Can't program rules 97dd63d7-4853-4aaa-970f-b1f46d09a670 yet, waiting on ipsets\r\n\r\nBut later it seems to work:\r\n\r\n2015-06-10 10:56:01,294 [INFO][1768/5] calico.felix.fiptables 189: Iptables update: {'felix-p-_a0438770e0e8efc-i': ['--append felix-p-_a0438770e0e8efc-i --protocol tcp --source fec0::250:56ff:fe92:e7/128 --match multiport --destination-ports 22 --jump RETURN', '--append felix-p-_a0438770e0e8efc-i --match comment --comment \"Mark as not matched\" --jump MARK --set-mark 1'], 'felix-p-_a0438770e0e8efc-o': ['--append felix-p-_a0438770e0e8efc-o --destination ::/0 --jump RETURN', '--append felix-p-_a0438770e0e8efc-o --match comment --comment \"Mark as not matched\" --jump MARK --set-mark 1']}\r\n2015-06-10 10:56:01,294 [INFO][1768/5] calico.felix.fiptables 190: Iptables deps: {}\r\n2015-06-10 10:56:01,296 [INFO][1768/17] calico.felix.endpoint 393: Endpoint EndpointId<calico-st03/openstack/b10597c5-3b94-4343-a70a-194bf4d4b51f/cd250fed-c73e-41b9-8dd2-e7f01512cbe7> acquired all required profile references\r\n2015-06-10 10:56:01,302 [INFO][1768/4] calico.felix.fiptables 336: IptablesUpdater<queue_len=1,live=True,msg=None> Successfully processed iptables updates.\r\n2015-06-10 10:56:01,303 [INFO][1768/4] calico.felix.fiptables 189: Iptables update: {'felix-p-_a0438770e0e8efc-i': ['--append felix-p-_a0438770e0e8efc-i --protocol icmp --match set --match-set felix-v4-_a0438770e0e8efc src --jump RETURN', '--append felix-p-_a0438770e0e8efc-i --protocol tcp --source 172.18.203.223/32 --match multiport --destination-ports 22 --jump RETURN', '--append felix-p-_a0438770e0e8efc-i --match comment --comment \"Mark as not matched\" --jump MARK --set-mark 1'], 'felix-p-_a0438770e0e8efc-o': ['--append felix-p-_a0438770e0e8efc-o --destination 0.0.0.0/0 --jump RETURN', '--append felix-p-_a0438770e0e8efc-o --match comment --comment \"Mark as not matched\" --jump MARK --set-mark 1']}\r\n2015-06-10 10:56:01,303 [INFO][1768/4] calico.felix.fiptables 190: Iptables deps: {}\r\n2015-06-10 10:56:01,304 [INFO][1768/15] calico.felix.endpoint 393: Endpoint EndpointId<calico-st03/openstack/b10597c5-3b94-4343-a70a-194bf4d4b51f/cd250fed-c73e-41b9-8dd2-e7f01512cbe7> acquired all required profile references\r\n2015-06-10 10:56:01,324 [INFO][1768/5] calico.felix.fiptables 336: IptablesUpdater<queue_len=0,live=True,msg=None> Successfully processed iptables updates.\r\n2015-06-10 10:56:01,339 [INFO][1768/4] calico.felix.fiptables 336: IptablesUpdater<queue_len=0,live=True,msg=None> Successfully processed iptables updates.\r\n\r\n\r\nBut later (~12:10) , I restarted calico-felix and my rules came back. The ‘Iptables update’ logs look similar to the above.\r\n\r\nFull diags here: https://transfer.sh/vNLJM/controller-diags.tar.xz\r\n"},"comment":{"url":"https://api.github.com/repos/Metaswitch/calico/issues/comments/117103673","html_url":"https://github.com/Metaswitch/calico/issues/645#issuecomment-117103673","issue_url":"https://api.github.com/repos/Metaswitch/calico/issues/645","id":117103673,"user":{"login":"Lukasa","id":1382556,"avatar_url":"https://avatars.githubusercontent.com/u/1382556?v=3","gravatar_id":"","url":"https://api.github.com/users/Lukasa","html_url":"https://github.com/Lukasa","followers_url":"https://api.github.com/users/Lukasa/followers","following_url":"https://api.github.com/users/Lukasa/following{/other_user}","gists_url":"https://api.github.com/users/Lukasa/gists{/gist_id}","starred_url":"https://api.github.com/users/Lukasa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Lukasa/subscriptions","organizations_url":"https://api.github.com/users/Lukasa/orgs","repos_url":"https://api.github.com/users/Lukasa/repos","events_url":"https://api.github.com/users/Lukasa/events{/privacy}","received_events_url":"https://api.github.com/users/Lukasa/received_events","type":"User","site_admin":false},"created_at":"2015-06-30T09:54:57Z","updated_at":"2015-06-30T09:54:57Z","body":"nova-api-metadata is the culprit there. ;)"}}