{"action":"created","issue":{"url":"https://api.github.com/repos/openaustralia/morph/issues/521","labels_url":"https://api.github.com/repos/openaustralia/morph/issues/521/labels{/name}","comments_url":"https://api.github.com/repos/openaustralia/morph/issues/521/comments","events_url":"https://api.github.com/repos/openaustralia/morph/issues/521/events","html_url":"https://github.com/openaustralia/morph/issues/521","id":65736488,"number":521,"title":"mitmproxy too many open files and list index out of range","user":{"login":"mlandauer","id":2221,"avatar_url":"https://avatars.githubusercontent.com/u/2221?v=3","gravatar_id":"","url":"https://api.github.com/users/mlandauer","html_url":"https://github.com/mlandauer","followers_url":"https://api.github.com/users/mlandauer/followers","following_url":"https://api.github.com/users/mlandauer/following{/other_user}","gists_url":"https://api.github.com/users/mlandauer/gists{/gist_id}","starred_url":"https://api.github.com/users/mlandauer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mlandauer/subscriptions","organizations_url":"https://api.github.com/users/mlandauer/orgs","repos_url":"https://api.github.com/users/mlandauer/repos","events_url":"https://api.github.com/users/mlandauer/events{/privacy}","received_events_url":"https://api.github.com/users/mlandauer/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/openaustralia/morph/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/openaustralia/morph/labels/ready","name":"ready","color":"00c5fe"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-04-01T17:25:03Z","updated_at":"2015-04-03T00:57:38Z","closed_at":null,"body":"Recent tail of `/var/log/morph/mitmproxy-1.log`\r\n\r\n```\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nException in thread SlaveThread (0.0.0.0:8080):\r\nTraceback (most recent call last):\r\n  File \"/usr/lib/python2.7/threading.py\", line 551, in __bootstrap_inner\r\n  File \"/usr/local/lib/python2.7/dist-packages/libmproxy/controller.py\", line 77, in run\r\n  File \"/usr/local/lib/python2.7/dist-packages/netlib/tcp.py\", line 479, in serve_forever\r\n  File \"/usr/lib/python2.7/socket.py\", line 202, in accept\r\nerror: [Errno 24] Too many open files\r\n\r\nException in thread ScriptThread:\r\nTraceback (most recent call last):\r\n  File \"/usr/lib/python2.7/threading.py\", line 551, in __bootstrap_inner\r\n    self.run()\r\n  File \"/usr/lib/python2.7/threading.py\", line 504, in run\r\n    self.__target(*self.__args, **self.__kwargs)\r\n  File \"/usr/local/lib/python2.7/dist-packages/libmproxy/script.py\", line 141, in run\r\n    fn(*args, **kwargs)\r\n  File \"mitmproxy/log_to_morph.py\", line 15, in response\r\n    'host': flow.request.headers[\"Host\"][0],\r\nIndexError: list index out of range\r\n\r\nException in thread SlaveThread (0.0.0.0:8080):\r\nTraceback (most recent call last):\r\n  File \"/usr/lib/python2.7/threading.py\", line 551, in __bootstrap_inner\r\n    self.run()\r\n  File \"/usr/local/lib/python2.7/dist-packages/libmproxy/controller.py\", line 77, in run\r\n  File \"/usr/local/lib/python2.7/dist-packages/netlib/tcp.py\", line 479, in serve_forever\r\n  File \"/usr/lib/python2.7/socket.py\", line 202, in accept\r\nerror: [Errno 24] Too many open files\r\n\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nException in thread ScriptThread:\r\nTraceback (most recent call last):\r\n  File \"/usr/lib/python2.7/threading.py\", line 551, in __bootstrap_inner\r\n    self.run()\r\n  File \"/usr/lib/python2.7/threading.py\", line 504, in run\r\n    self.__target(*self.__args, **self.__kwargs)\r\n  File \"/usr/local/lib/python2.7/dist-packages/libmproxy/script.py\", line 141, in run\r\n    fn(*args, **kwargs)\r\n  File \"mitmproxy/log_to_morph.py\", line 15, in response\r\n    'host': flow.request.headers[\"Host\"][0],\r\nIndexError: list index out of range\r\n\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nException in thread ScriptThread:\r\nTraceback (most recent call last):\r\n  File \"/usr/lib/python2.7/threading.py\", line 551, in __bootstrap_inner\r\n    self.run()\r\n  File \"/usr/lib/python2.7/threading.py\", line 504, in run\r\n    self.__target(*self.__args, **self.__kwargs)\r\n  File \"/usr/local/lib/python2.7/dist-packages/libmproxy/script.py\", line 141, in run\r\n    fn(*args, **kwargs)\r\n  File \"mitmproxy/log_to_morph.py\", line 15, in response\r\n    'host': flow.request.headers[\"Host\"][0],\r\nIndexError: list index out of range\r\n\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nError contacting Morph server: [Errno socket error] [Errno 111] Connection refused\r\nError contacting Morph server: [Errno socket error] [Errno 111] Connection refused\r\nError contacting Morph server: [Errno socket error] [Errno 111] Connection refused\r\nError contacting Morph server: [Errno socket error] [Errno 111] Connection refused\r\nError contacting Morph server: [Errno socket error] [Errno 111] Connection refused\r\nError contacting Morph server: [Errno socket error] [Errno 111] Connection refused\r\nError contacting Morph server: [Errno socket error] [Errno 111] Connection refused\r\nError contacting Morph server: [Errno socket error] [Errno 111] Connection refused\r\nError contacting Morph server: [Errno socket error] [Errno 111] Connection refused\r\nError contacting Morph server: [Errno socket error] [Errno 111] Connection refused\r\nError contacting Morph server: [Errno socket error] [Errno 111] Connection refused\r\nError contacting Morph server: [Errno socket error] [Errno 111] Connection refused\r\nError contacting Morph server: [Errno socket error] [Errno 111] Connection refused\r\nError contacting Morph server: [Errno socket error] [Errno 111] Connection refused\r\nError contacting Morph server: [Errno socket error] [Errno 111] Connection refused\r\nError contacting Morph server: [Errno socket error] [Errno -2] Name or service not known\r\nException in thread SlaveThread (0.0.0.0:8080):\r\nTraceback (most recent call last):\r\n  File \"/usr/lib/python2.7/threading.py\", line 551, in __bootstrap_inner\r\n  File \"/usr/local/lib/python2.7/dist-packages/libmproxy/controller.py\", line 77, in run\r\n  File \"/usr/local/lib/python2.7/dist-packages/netlib/tcp.py\", line 479, in serve_forever\r\n  File \"/usr/lib/python2.7/socket.py\", line 202, in accept\r\nerror: [Errno 24] Too many open files\r\n\r\nException in thread ScriptThread:\r\nTraceback (most recent call last):\r\n  File \"/usr/lib/python2.7/threading.py\", line 551, in __bootstrap_inner\r\n    self.run()\r\n  File \"/usr/lib/python2.7/threading.py\", line 504, in run\r\n    self.__target(*self.__args, **self.__kwargs)\r\n  File \"/usr/local/lib/python2.7/dist-packages/libmproxy/script.py\", line 141, in run\r\n    fn(*args, **kwargs)\r\n  File \"mitmproxy/log_to_morph.py\", line 15, in response\r\n    'host': flow.request.headers[\"Host\"][0],\r\nIndexError: list index out of range\r\n```"},"comment":{"url":"https://api.github.com/repos/openaustralia/morph/issues/comments/89099751","html_url":"https://github.com/openaustralia/morph/issues/521#issuecomment-89099751","issue_url":"https://api.github.com/repos/openaustralia/morph/issues/521","id":89099751,"user":{"login":"mlandauer","id":2221,"avatar_url":"https://avatars.githubusercontent.com/u/2221?v=3","gravatar_id":"","url":"https://api.github.com/users/mlandauer","html_url":"https://github.com/mlandauer","followers_url":"https://api.github.com/users/mlandauer/followers","following_url":"https://api.github.com/users/mlandauer/following{/other_user}","gists_url":"https://api.github.com/users/mlandauer/gists{/gist_id}","starred_url":"https://api.github.com/users/mlandauer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mlandauer/subscriptions","organizations_url":"https://api.github.com/users/mlandauer/orgs","repos_url":"https://api.github.com/users/mlandauer/repos","events_url":"https://api.github.com/users/mlandauer/events{/privacy}","received_events_url":"https://api.github.com/users/mlandauer/received_events","type":"User","site_admin":false},"created_at":"2015-04-03T00:57:38Z","updated_at":"2015-04-03T00:57:38Z","body":"I've noticed an interesting thing. In the connection_logs table before the last time that the proxy hung, the last set of records contain a long set of requests with `run_id` `NULL` and host `50.116.3.88:8080\" which looks mighty suspicious. It's like the proxy is going around in an endless loop where a special form of request is causing a request to itself which would cause another request to itself and so on, and surprise surprise ultimately leading to exhaustion of file handles."}}