{"action":"created","issue":{"url":"https://api.github.com/repos/Metaswitch/calico/issues/534","labels_url":"https://api.github.com/repos/Metaswitch/calico/issues/534/labels{/name}","comments_url":"https://api.github.com/repos/Metaswitch/calico/issues/534/comments","events_url":"https://api.github.com/repos/Metaswitch/calico/issues/534/events","html_url":"https://github.com/Metaswitch/calico/issues/534","id":76070054,"number":534,"title":"Felix runs out of file descriptors at around 160 instances on one host","user":{"login":"matthewdupre","id":9053570,"avatar_url":"https://avatars.githubusercontent.com/u/9053570?v=3","gravatar_id":"","url":"https://api.github.com/users/matthewdupre","html_url":"https://github.com/matthewdupre","followers_url":"https://api.github.com/users/matthewdupre/followers","following_url":"https://api.github.com/users/matthewdupre/following{/other_user}","gists_url":"https://api.github.com/users/matthewdupre/gists{/gist_id}","starred_url":"https://api.github.com/users/matthewdupre/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/matthewdupre/subscriptions","organizations_url":"https://api.github.com/users/matthewdupre/orgs","repos_url":"https://api.github.com/users/matthewdupre/repos","events_url":"https://api.github.com/users/matthewdupre/events{/privacy}","received_events_url":"https://api.github.com/users/matthewdupre/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/Metaswitch/calico/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/Metaswitch/calico/labels/felix","name":"felix","color":"5319e7"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/Metaswitch/calico/milestones/3","html_url":"https://github.com/Metaswitch/calico/milestones/1.0","labels_url":"https://api.github.com/repos/Metaswitch/calico/milestones/3/labels","id":1074686,"number":3,"title":"1.0","description":"","creator":{"login":"Lukasa","id":1382556,"avatar_url":"https://avatars.githubusercontent.com/u/1382556?v=3","gravatar_id":"","url":"https://api.github.com/users/Lukasa","html_url":"https://github.com/Lukasa","followers_url":"https://api.github.com/users/Lukasa/followers","following_url":"https://api.github.com/users/Lukasa/following{/other_user}","gists_url":"https://api.github.com/users/Lukasa/gists{/gist_id}","starred_url":"https://api.github.com/users/Lukasa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Lukasa/subscriptions","organizations_url":"https://api.github.com/users/Lukasa/orgs","repos_url":"https://api.github.com/users/Lukasa/repos","events_url":"https://api.github.com/users/Lukasa/events{/privacy}","received_events_url":"https://api.github.com/users/Lukasa/received_events","type":"User","site_admin":false},"open_issues":20,"closed_issues":2,"state":"open","created_at":"2015-04-20T07:33:48Z","updated_at":"2015-05-13T23:59:11Z","due_on":null,"closed_at":null},"comments":3,"created_at":"2015-05-13T17:56:34Z","updated_at":"2015-05-14T11:41:52Z","closed_at":null,"body":"I'm running with the default ulimit of 1024, and after creating 160 VMs it stopped setting up new VMs and cyclically restarted.  We should consider reducing the number of file handles used and (if we can't remove the relation to number of instances) document that the limit may need increasing.\r\n```\r\n2015-05-13 17:47:25,134 [ERROR][12169/352] calico.felix.actor 227: Exception processing 597b3eb38cf7 (on_endpoint_update)\r\nTraceback (most recent call last):\r\n  File \"/usr/lib/python2.7/dist-packages/calico/felix/actor.py\", line 225, in _step\r\n    result = msg.method()\r\n  File \"/usr/lib/python2.7/dist-packages/calico/felix/endpoint.py\", line 235, in on_endpoint_update\r\n    self._configure_interface()\r\n  File \"/usr/lib/python2.7/dist-packages/calico/felix/endpoint.py\", line 360, in _configure_interface\r\n    self.endpoint[\"mac\"])\r\n  File \"/usr/lib/python2.7/dist-packages/calico/felix/devices.py\", line 182, in set_routes\r\n  File \"/usr/lib/python2.7/dist-packages/calico/felix/devices.py\", line 62, in list_interface_ips\r\n  File \"/usr/lib/python2.7/dist-packages/calico/felix/futils.py\", line 100, in check_call\r\n  File \"/usr/lib/python2.7/dist-packages/gevent/subprocess.py\", line 238, in __init__\r\n  File \"/usr/lib/python2.7/dist-packages/gevent/subprocess.py\", line 638, in _execute_child\r\n  File \"/usr/lib/python2.7/dist-packages/gevent/subprocess.py\", line 594, in pipe_cloexec\r\nOSError: [Errno 24] Too many open files\r\n2015-05-13 17:47:25,140 [CRITICAL][12169/352] calico.felix.actor 511: TrackedAsyncResult ('endpoint.py:82:_on_object_started', 'EndpointManager(IPv6) (processing 27dd27bbd479)', u'LocalEndpoint(e3257adf-2ba3-4011-a339-2ceb2d0886c4(IPv6))', 'on_endpoint_update') was leaked with exception OSError(24, 'Too many open files').  Dying.\r\n```"},"comment":{"url":"https://api.github.com/repos/Metaswitch/calico/issues/comments/102009955","html_url":"https://github.com/Metaswitch/calico/issues/534#issuecomment-102009955","issue_url":"https://api.github.com/repos/Metaswitch/calico/issues/534","id":102009955,"user":{"login":"matthewdupre","id":9053570,"avatar_url":"https://avatars.githubusercontent.com/u/9053570?v=3","gravatar_id":"","url":"https://api.github.com/users/matthewdupre","html_url":"https://github.com/matthewdupre","followers_url":"https://api.github.com/users/matthewdupre/followers","following_url":"https://api.github.com/users/matthewdupre/following{/other_user}","gists_url":"https://api.github.com/users/matthewdupre/gists{/gist_id}","starred_url":"https://api.github.com/users/matthewdupre/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/matthewdupre/subscriptions","organizations_url":"https://api.github.com/users/matthewdupre/orgs","repos_url":"https://api.github.com/users/matthewdupre/repos","events_url":"https://api.github.com/users/matthewdupre/events{/privacy}","received_events_url":"https://api.github.com/users/matthewdupre/received_events","type":"User","site_admin":false},"created_at":"2015-05-14T11:41:52Z","updated_at":"2015-05-14T11:41:52Z","body":"I can further add that increasing the limit (put a `limit nofile 10000 10000` stanza into /etc/init/calico-felix.conf) does increase the number of VMs Felix can handle."}}