{"action":"created","issue":{"url":"https://api.github.com/repos/mongodb-labs/mongo-connector/issues/343","labels_url":"https://api.github.com/repos/mongodb-labs/mongo-connector/issues/343/labels{/name}","comments_url":"https://api.github.com/repos/mongodb-labs/mongo-connector/issues/343/comments","events_url":"https://api.github.com/repos/mongodb-labs/mongo-connector/issues/343/events","html_url":"https://github.com/mongodb-labs/mongo-connector/issues/343","id":112270508,"number":343,"title":"Users should be able to set the timeout and the chunk_size for elastic search bulk requests.","user":{"login":"Jinshui","id":7564813,"avatar_url":"https://avatars.githubusercontent.com/u/7564813?v=3","gravatar_id":"","url":"https://api.github.com/users/Jinshui","html_url":"https://github.com/Jinshui","followers_url":"https://api.github.com/users/Jinshui/followers","following_url":"https://api.github.com/users/Jinshui/following{/other_user}","gists_url":"https://api.github.com/users/Jinshui/gists{/gist_id}","starred_url":"https://api.github.com/users/Jinshui/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Jinshui/subscriptions","organizations_url":"https://api.github.com/users/Jinshui/orgs","repos_url":"https://api.github.com/users/Jinshui/repos","events_url":"https://api.github.com/users/Jinshui/events{/privacy}","received_events_url":"https://api.github.com/users/Jinshui/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2015-10-20T02:09:20Z","updated_at":"2015-10-21T03:31:42Z","closed_at":null,"body":"While dumping huge data to elasticsearch, the mongo-connector can't work normally because it often crashes due to connection timeout.  The default timeout is 10, and can't be changed. It should be an option of the mongo-connector command so that user can change it when necessary.\r\n```\r\n2015-10-20 01:48:04,992 [CRITICAL] mongo_connector.oplog_manager:543 - Exception during collection dump\r\nTraceback (most recent call last):\r\n  File \"/usr/lib/python2.6/site-packages/mongo_connector/oplog_manager.py\", line 495, in do_dump\r\n    upsert_all(dm)\r\n  File \"/usr/lib/python2.6/site-packages/mongo_connector/oplog_manager.py\", line 479, in upsert_all\r\n    dm.bulk_upsert(docs_to_dump(namespace), mapped_ns, long_ts)\r\n  File \"/usr/lib/python2.6/site-packages/mongo_connector/util.py\", line 32, in wrapped\r\n    return f(*args, **kwargs)\r\n  File \"/usr/lib/python2.6/site-packages/mongo_connector/doc_managers/elastic_doc_manager.py\", line 190, in bulk_upsert\r\n    for ok, resp in responses:\r\n  File \"/usr/lib/python2.6/site-packages/elasticsearch/helpers/__init__.py\", line 138, in streaming_bulk\r\n    raise e\r\nConnectionTimeout: ConnectionTimeout caused by - ReadTimeoutError(HTTPConnectionPool(host=u'172.31.1.254', port=9200): Read timed out. (read timeout=10))\r\n```\r\nThanks,"},"comment":{"url":"https://api.github.com/repos/mongodb-labs/mongo-connector/issues/comments/149771166","html_url":"https://github.com/mongodb-labs/mongo-connector/issues/343#issuecomment-149771166","issue_url":"https://api.github.com/repos/mongodb-labs/mongo-connector/issues/343","id":149771166,"user":{"login":"Jinshui","id":7564813,"avatar_url":"https://avatars.githubusercontent.com/u/7564813?v=3","gravatar_id":"","url":"https://api.github.com/users/Jinshui","html_url":"https://github.com/Jinshui","followers_url":"https://api.github.com/users/Jinshui/followers","following_url":"https://api.github.com/users/Jinshui/following{/other_user}","gists_url":"https://api.github.com/users/Jinshui/gists{/gist_id}","starred_url":"https://api.github.com/users/Jinshui/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Jinshui/subscriptions","organizations_url":"https://api.github.com/users/Jinshui/orgs","repos_url":"https://api.github.com/users/Jinshui/repos","events_url":"https://api.github.com/users/Jinshui/events{/privacy}","received_events_url":"https://api.github.com/users/Jinshui/received_events","type":"User","site_admin":false},"created_at":"2015-10-21T03:31:42Z","updated_at":"2015-10-21T03:31:42Z","body":"It seems both of the timeout and chunk_size can be configured by the configuration file as follows:\r\n```\r\n{\r\n  \"docManagers\": [\r\n    {\r\n      \"docManager\": \"elastic_doc_manager\",\r\n      \"targetURL\": \"localhost:9200\",\r\n      \"bulkSize\": 200,\r\n      \"autoCommitInterval\": 0,\r\n      \"args\": {\r\n        \"clientOptions\": {\"timeout\": 60}\r\n      }\r\n    }\r\n  ]\r\n}\r\n```\r\n\r\nCan anyone please confirm? I will test it later."}}