{"action":"opened","issue":{"url":"https://api.github.com/repos/boto/boto/issues/3216","labels_url":"https://api.github.com/repos/boto/boto/issues/3216/labels{/name}","comments_url":"https://api.github.com/repos/boto/boto/issues/3216/comments","events_url":"https://api.github.com/repos/boto/boto/issues/3216/events","html_url":"https://github.com/boto/boto/issues/3216","id":86613473,"number":3216,"title":"Connection pool containing too many open connections","user":{"login":"ffdybuster","id":12152708,"avatar_url":"https://avatars.githubusercontent.com/u/12152708?v=3","gravatar_id":"","url":"https://api.github.com/users/ffdybuster","html_url":"https://github.com/ffdybuster","followers_url":"https://api.github.com/users/ffdybuster/followers","following_url":"https://api.github.com/users/ffdybuster/following{/other_user}","gists_url":"https://api.github.com/users/ffdybuster/gists{/gist_id}","starred_url":"https://api.github.com/users/ffdybuster/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ffdybuster/subscriptions","organizations_url":"https://api.github.com/users/ffdybuster/orgs","repos_url":"https://api.github.com/users/ffdybuster/repos","events_url":"https://api.github.com/users/ffdybuster/events{/privacy}","received_events_url":"https://api.github.com/users/ffdybuster/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-06-09T14:15:00Z","updated_at":"2015-06-09T14:15:00Z","closed_at":null,"body":"I'm trying to list all files in all S3 buckets:\r\n``` .py\r\nconn = boto.s3.connection.S3Connection(accessKey, secretKey)\r\nrs = conn.get_all_buckets()\r\nfor b in rs:\r\n    for k in b:\r\n        print(\"{0} {1}\".format(b.name, k.key))\r\n```\r\nUnfortunately, this doesn't work well. We have several thousands of buckets, and after around 1024 buckets, it hangs a bit and then finally stops:\r\n```\r\nTraceback (most recent call last):\r\n  File \"s3test.py\", line 20, in <module>\r\n  File \"/usr/local/lib/python2.7/dist-packages/boto/s3/bucketlistresultset.py\", line 34, in bucket_lister\r\n  File \"/usr/local/lib/python2.7/dist-packages/boto/s3/bucket.py\", line 472, in get_all_keys\r\n  File \"/usr/local/lib/python2.7/dist-packages/boto/s3/bucket.py\", line 398, in _get_all\r\n  File \"/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py\", line 664, in make_request\r\n  File \"/usr/local/lib/python2.7/dist-packages/boto/connection.py\", line 1071, in make_request\r\n  File \"/usr/local/lib/python2.7/dist-packages/boto/connection.py\", line 1030, in _mexe\r\nsocket.error: [Errno 24] Too many open files\r\n```\r\n(The ulimit for open files is 1024.)\r\n\r\nThe problem is that for every bucket, boto opens a new connection, which is then stored in the connection pool. But since the pool is indexed (among others) by the bucket hostname, eventually there will be thousands of open connections in the pool, triggering the above error.\r\n\r\nThis problem can be solved by starting to close (old) sessions as soon as the pool contains enough connections."}}