{"action":"closed","issue":{"url":"https://api.github.com/repos/projectcalico/calico/issues/768","labels_url":"https://api.github.com/repos/projectcalico/calico/issues/768/labels{/name}","comments_url":"https://api.github.com/repos/projectcalico/calico/issues/768/comments","events_url":"https://api.github.com/repos/projectcalico/calico/issues/768/events","html_url":"https://github.com/projectcalico/calico/issues/768","id":101632430,"number":768,"title":"Felix dies if ipset gets too big","user":{"login":"fasaxc","id":469264,"avatar_url":"https://avatars.githubusercontent.com/u/469264?v=3","gravatar_id":"","url":"https://api.github.com/users/fasaxc","html_url":"https://github.com/fasaxc","followers_url":"https://api.github.com/users/fasaxc/followers","following_url":"https://api.github.com/users/fasaxc/following{/other_user}","gists_url":"https://api.github.com/users/fasaxc/gists{/gist_id}","starred_url":"https://api.github.com/users/fasaxc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fasaxc/subscriptions","organizations_url":"https://api.github.com/users/fasaxc/orgs","repos_url":"https://api.github.com/users/fasaxc/repos","events_url":"https://api.github.com/users/fasaxc/events{/privacy}","received_events_url":"https://api.github.com/users/fasaxc/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/projectcalico/calico/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/projectcalico/calico/labels/felix","name":"felix","color":"5319e7"},{"url":"https://api.github.com/repos/projectcalico/calico/labels/priority:high","name":"priority:high","color":"f7c6c7"}],"state":"closed","locked":false,"assignee":{"login":"fasaxc","id":469264,"avatar_url":"https://avatars.githubusercontent.com/u/469264?v=3","gravatar_id":"","url":"https://api.github.com/users/fasaxc","html_url":"https://github.com/fasaxc","followers_url":"https://api.github.com/users/fasaxc/followers","following_url":"https://api.github.com/users/fasaxc/following{/other_user}","gists_url":"https://api.github.com/users/fasaxc/gists{/gist_id}","starred_url":"https://api.github.com/users/fasaxc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fasaxc/subscriptions","organizations_url":"https://api.github.com/users/fasaxc/orgs","repos_url":"https://api.github.com/users/fasaxc/repos","events_url":"https://api.github.com/users/fasaxc/events{/privacy}","received_events_url":"https://api.github.com/users/fasaxc/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/projectcalico/calico/milestones/18","html_url":"https://github.com/projectcalico/calico/milestones/1.2.0","labels_url":"https://api.github.com/repos/projectcalico/calico/milestones/18/labels","id":1297741,"number":18,"title":"1.2.0","description":"","creator":{"login":"fasaxc","id":469264,"avatar_url":"https://avatars.githubusercontent.com/u/469264?v=3","gravatar_id":"","url":"https://api.github.com/users/fasaxc","html_url":"https://github.com/fasaxc","followers_url":"https://api.github.com/users/fasaxc/followers","following_url":"https://api.github.com/users/fasaxc/following{/other_user}","gists_url":"https://api.github.com/users/fasaxc/gists{/gist_id}","starred_url":"https://api.github.com/users/fasaxc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fasaxc/subscriptions","organizations_url":"https://api.github.com/users/fasaxc/orgs","repos_url":"https://api.github.com/users/fasaxc/repos","events_url":"https://api.github.com/users/fasaxc/events{/privacy}","received_events_url":"https://api.github.com/users/fasaxc/received_events","type":"User","site_admin":false},"open_issues":4,"closed_issues":22,"state":"open","created_at":"2015-09-09T13:55:31Z","updated_at":"2015-10-21T14:54:20Z","due_on":null,"closed_at":null},"comments":2,"created_at":"2015-08-18T11:15:30Z","updated_at":"2015-10-21T14:54:20Z","closed_at":"2015-10-21T14:54:20Z","body":"An ipset can hold many entries (maybe ~65k, by default) but we don't put a limit on how many entries we'll try to put in one.  If the the ipset restore call fails due to the number of entries being exceeded then felix dies with \r\n\r\n```\r\n<snip>\r\nadd felix-tmp-v4-prof-000 1.1.68.108\r\nadd felix-tmp-v4-prof-000 1.1.68.107\r\nadd felix-tmp-v4-prof-000 1.1.68.106\r\nadd felix-tmp-v4-prof-000 1.1.68.105\r\nadd felix-tmp-v4-prof-000 1.1.68.104\r\nadd felix-tmp-v4-prof-000 1.1.68.103\r\nadd felix-tmp-v4-prof-000 1.1.68.102\r\nadd felix-tmp-v4-prof-000 1.0.50.108\r\nadd felix-tmp-v4-prof-000 1.1.68.100\r\nswap felix-v4-prof-000 felix-tmp-v4-prof-000\r\ndestroy felix-tmp-v4-prof-000\r\nCOMMIT\r\n\r\n\r\nTrackedAsyncResult ('ipsets.py:100:_update_active_ipset', 'IpsetManager(IPv4) (processing 6a605efe775a)', 'TagIpset(prof-000)', 'replace_members') was leaked with exception FailedSystemCall('ipset', 'restore').  Dying.\r\n```\r\n\r\nI'm not sure if the limit can be raised (so that we can dynamically choose to increase it once we're getting close) or if it's a hard limit (in which case we should isolate the failure to the single ipset and log an error)."}}