{"action":"created","issue":{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4597","labels_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4597/labels{/name}","comments_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4597/comments","events_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4597/events","html_url":"https://github.com/rethinkdb/rethinkdb/issues/4597","id":97931881,"number":4597,"title":"Heartbeat timeouts","user":{"login":"mcadam","id":3319629,"avatar_url":"https://avatars.githubusercontent.com/u/3319629?v=3","gravatar_id":"","url":"https://api.github.com/users/mcadam","html_url":"https://github.com/mcadam","followers_url":"https://api.github.com/users/mcadam/followers","following_url":"https://api.github.com/users/mcadam/following{/other_user}","gists_url":"https://api.github.com/users/mcadam/gists{/gist_id}","starred_url":"https://api.github.com/users/mcadam/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mcadam/subscriptions","organizations_url":"https://api.github.com/users/mcadam/orgs","repos_url":"https://api.github.com/users/mcadam/repos","events_url":"https://api.github.com/users/mcadam/events{/privacy}","received_events_url":"https://api.github.com/users/mcadam/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2015-07-29T13:10:51Z","updated_at":"2015-07-31T18:31:47Z","closed_at":null,"body":"Hello,\r\n\r\nI am having some problems while trying to run a Rethinkdb cluster in production using Kubernetes.\r\nI use your docker image from public registry. Once I launch a couple nodes which connect fine to each other, every 5 minutes or so I am having heartbeat timeouts.\r\n It is not usable as is in production as every time the connection timeouts, the connection is lost and the tables become unavailable until the server is connected again...\r\n\r\nUnder Kubernetes the docker version is 1.6, when you say the docker image is officially supported on docker version 1.7, could that cause my problem of heartbeat timeouts ?\r\n\r\nI tried on version 1.16.0-1 and version 2.0.4, same problem on both.\r\n\r\nThanks"},"comment":{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/comments/126777822","html_url":"https://github.com/rethinkdb/rethinkdb/issues/4597#issuecomment-126777822","issue_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4597","id":126777822,"user":{"login":"danielmewes","id":505365,"avatar_url":"https://avatars.githubusercontent.com/u/505365?v=3","gravatar_id":"","url":"https://api.github.com/users/danielmewes","html_url":"https://github.com/danielmewes","followers_url":"https://api.github.com/users/danielmewes/followers","following_url":"https://api.github.com/users/danielmewes/following{/other_user}","gists_url":"https://api.github.com/users/danielmewes/gists{/gist_id}","starred_url":"https://api.github.com/users/danielmewes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielmewes/subscriptions","organizations_url":"https://api.github.com/users/danielmewes/orgs","repos_url":"https://api.github.com/users/danielmewes/repos","events_url":"https://api.github.com/users/danielmewes/events{/privacy}","received_events_url":"https://api.github.com/users/danielmewes/received_events","type":"User","site_admin":false},"created_at":"2015-07-31T18:31:47Z","updated_at":"2015-07-31T18:31:47Z","body":"@mcadam Thanks for trying that out. I'm currently out of good ideas. My only guesses are that either one of the instances gets really slow occasionally or completely pauses execution (maybe something is happening on the hosts?), or that there is some firewall / router weirdness going on that doesn't affect ICMP packages. A third option could be a problem with our timer code, so that heartbeat timeouts are triggered either too early, or heartbeats are not sent from the other side quickly enough.\r\n\r\nIt's particularly odd that this happens even without any queries running. I don't think I've seen anything like that before.\r\nI know this is some work, but the best way to debug this at this point might be taking a network sniffer (e.g. ettercap or wireshark), and checking what's going on with the connections. In an otherwise completely idle cluster (make sure you don't have the web UI open either, since that would cause additional traffic), you should see a short message exchanged on the cluster connections (port 29015) every 2 seconds. If the interval sometimes gets significantly longer than 2 seconds, we might need to take a closer look at our timing code.\r\n\r\nRethinkDB 2.1 will also introduce a configuration option to increase the heartbeat timeout. However I doubt that this will help much in this case, given that the fact that it's currently being hit indicates a ~10 second stall in communication and/or processing. Even if a longer heartbeat timeout can help to avoid the disconnects (which I'm not sure it will), there will still be ~10 second stalls in query processing."}}