{"action":"created","issue":{"url":"https://api.github.com/repos/rancher/rancher/issues/2940","labels_url":"https://api.github.com/repos/rancher/rancher/issues/2940/labels{/name}","comments_url":"https://api.github.com/repos/rancher/rancher/issues/2940/comments","events_url":"https://api.github.com/repos/rancher/rancher/issues/2940/events","html_url":"https://github.com/rancher/rancher/issues/2940","id":120857118,"number":2940,"title":"Internal Rancher load balancer mysteriously stops working","user":{"login":"rzachariah","id":4552543,"avatar_url":"https://avatars.githubusercontent.com/u/4552543?v=3","gravatar_id":"","url":"https://api.github.com/users/rzachariah","html_url":"https://github.com/rzachariah","followers_url":"https://api.github.com/users/rzachariah/followers","following_url":"https://api.github.com/users/rzachariah/following{/other_user}","gists_url":"https://api.github.com/users/rzachariah/gists{/gist_id}","starred_url":"https://api.github.com/users/rzachariah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rzachariah/subscriptions","organizations_url":"https://api.github.com/users/rzachariah/orgs","repos_url":"https://api.github.com/users/rzachariah/repos","events_url":"https://api.github.com/users/rzachariah/events{/privacy}","received_events_url":"https://api.github.com/users/rzachariah/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-12-07T20:11:29Z","updated_at":"2015-12-09T18:26:34Z","closed_at":null,"body":"Hi there,\r\n\r\nI've set up a routing solution with two Rancher load balancers, a public one which, routes traffic from a public port, and an internal one, which does path based routing. I have these in two separate stacks. One of the stacks (the internal one) is portable from environment to environment. The other depends on ports available on the host.\r\n\r\nUsually, this works fine, but occasionally the internal load balancer stops working. It becomes unable to route traffic. When I restart it, it starts working again. I don't see anything in the log that indicates the problem. Do you have any suggestions for how to diagnose this?\r\n\r\nHere's the rancher LB log. Looks like nothing interesting happens until I restart the service today.\r\n\r\n11/25/2015 2:12:17 PMINFO: Updating monit\r\n11/25/2015 2:12:17 PMINFO: Downloading http://10.164.51.171:8080/v1//configcontent//monit current=\r\n11/25/2015 2:12:17 PMINFO: Running /var/lib/cattle/download/monit/monit-1-d166a713486fe4c0f039e152939d8d3b8ab8e6c2518e13d88f0b7ec68da46109/apply.sh\r\n11/25/2015 2:12:17 PMINFO: ROOT -> ./\r\n11/25/2015 2:12:17 PMINFO: ROOT -> ./etc/\r\n11/25/2015 2:12:17 PMINFO: ROOT -> ./etc/monit/\r\n11/25/2015 2:12:17 PMINFO: ROOT -> ./etc/monit/monitrc\r\n11/25/2015 2:12:17 PMINFO: Sending monit applied 1-d166a713486fe4c0f039e152939d8d3b8ab8e6c2518e13d88f0b7ec68da46109\r\n11/25/2015 2:12:17 PMINFO: Getting haproxy\r\n11/25/2015 2:12:17 PMINFO: Updating haproxy\r\n11/25/2015 2:12:17 PMINFO: Downloading http://10.164.51.171:8080/v1//configcontent//haproxy current=\r\n11/25/2015 2:12:17 PMINFO: Running /var/lib/cattle/download/haproxy/haproxy-1-02ea5d48cf5da5fc868cc4906e972d5178969de851be5b1e12dc5f3386e18378/apply.sh\r\n11/25/2015 2:12:17 PMINFO: ROOT -> ./\r\n11/25/2015 2:12:17 PMINFO: ROOT -> ./etc/\r\n11/25/2015 2:12:17 PMINFO: ROOT -> ./etc/haproxy/\r\n11/25/2015 2:12:17 PMINFO: ROOT -> ./etc/haproxy/certs/\r\n11/25/2015 2:12:17 PMINFO: ROOT -> ./etc/haproxy/certs/default.pem\r\n11/25/2015 2:12:17 PMINFO: ROOT -> ./etc/haproxy/certs/certs.pem\r\n11/25/2015 2:12:17 PMINFO: ROOT -> ./etc/haproxy/haproxy.cfg\r\n11/25/2015 2:12:17 PMINFO: ROOT -> ./etc/monit/\r\n11/25/2015 2:12:17 PMINFO: ROOT -> ./etc/monit/conf.d/\r\n11/25/2015 2:12:17 PMINFO: ROOT -> ./etc/monit/conf.d/haproxy\r\n11/25/2015 2:12:17 PMINFO: ROOT -> ./etc/default/\r\n11/25/2015 2:12:17 PMINFO: ROOT -> ./etc/default/haproxy\r\n11/25/2015 2:12:17 PMINFO: Sending haproxy applied 1-02ea5d48cf5da5fc868cc4906e972d5178969de851be5b1e12dc5f3386e18378\r\n11/25/2015 2:12:17 PMINFO: HOME -> ./\r\n11/25/2015 2:12:17 PMINFO: HOME -> ./etc/\r\n11/25/2015 2:12:17 PMINFO: HOME -> ./etc/cattle/\r\n11/25/2015 2:12:17 PMINFO: HOME -> ./etc/cattle/startup-env\r\n11/25/2015 2:12:17 PMINFO: ROOT -> ./\r\n11/25/2015 2:12:17 PMINFO: ROOT -> ./etc/\r\n11/25/2015 2:12:17 PMINFO: ROOT -> ./etc/init.d/\r\n11/25/2015 2:12:17 PMINFO: ROOT -> ./etc/init.d/agent-instance-startup\r\n11/25/2015 2:12:17 PMINFO: Sending agent-instance-startup applied 1-557156a6c83b3adc996b56beb2d4469f282a091c31fd17461c90ee76fa7fe6e0\r\n11/25/2015 2:12:17 PMmonit: generated unique Monit id 8acbdc5e5df331e582dfd74eda23a6cf and stored to '/var/lib/monit/id'\r\n11/25/2015 2:12:17 PMStarting monit daemon with http interface at [localhost:2812]\r\n12/7/2015 2:46:40 PM\r\nThe system is going down NOW!\r\n12/7/2015 2:46:40 PM\r\nSent SIGTERM to all processes\r\n12/7/2015 2:46:41 PM\r\nSent SIGKILL to all processes\r\n12/7/2015 2:46:41 PM\r\nRequesting system reboot\r\n12/7/2015 2:46:59 PMINFO: Downloading agent http://10.164.51.171:8080/v1/configcontent/configscripts\r\n12/7/2015 2:46:59 PMINFO: Updating configscripts\r\n12/7/2015 2:46:59 PMINFO: Downloading http://10.164.51.171:8080/v1//configcontent//configscripts current=\r\n12/7/2015 2:46:59 PMINFO: Running /var/lib/cattle/download/configscripts/configscripts-1-f5763391fb7914dcd14001f29ffe28d1167a3bfcc0ee0ec05d2ca9c722103c02/apply.sh\r\n12/7/2015 2:46:59 PMINFO: Sending configscripts applied 1-f5763391fb7914dcd14001f29ffe28d1167a3bfcc0ee0ec05d2ca9c722103c02\r\n12/7/2015 2:46:59 PMINFO: Updating agent-instance-startup\r\n12/7/2015 2:46:59 PMINFO: Downloading http://10.164.51.171:8080/v1//configcontent//agent-instance-startup current=\r\n12/7/2015 2:46:59 PMINFO: Running /var/lib/cattle/download/agent-instance-startup/agent-instance-startup-2-557156a6c83b3adc996b56beb2d4469f282a091c31fd17461c90ee76fa7fe6e0/apply.sh\r\n12/7/2015 2:46:59 PMINFO: Updating services\r\n12/7/2015 2:46:59 PMINFO: Downloading http://10.164.51.171:8080/v1//configcontent//services current=\r\n12/7/2015 2:46:59 PMINFO: Running /var/lib/cattle/download/services/services-1-061405f3edd960bfdfe1cfb8447be40eab5b4b608731608e224cc51c5dc30b91/apply.sh\r\n12/7/2015 2:46:59 PMINFO: HOME -> ./\r\n12/7/2015 2:46:59 PMINFO: HOME -> ./services\r\n12/7/2015 2:46:59 PMINFO: Sending services applied 1-061405f3edd960bfdfe1cfb8447be40eab5b4b608731608e224cc51c5dc30b91\r\n12/7/2015 2:46:59 PMINFO: Getting agent-instance-scripts\r\n12/7/2015 2:46:59 PMINFO: Updating agent-instance-scripts\r\n12/7/2015 2:46:59 PMINFO: Downloading http://10.164.51.171:8080/v1//configcontent//agent-instance-scripts current=\r\n12/7/2015 2:46:59 PMINFO: Running /var/lib/cattle/download/agent-instance-scripts/agent-instance-scripts-1-4b5124bd74cd423f98d57550b481ec77ec3a7135c6a650886ab95c043305d642/apply.sh\r\n12/7/2015 2:46:59 PMINFO: HOME -> ./\r\n12/7/2015 2:46:59 PMINFO: HOME -> ./events/\r\n12/7/2015 2:46:59 PMINFO: HOME -> ./events/config.update\r\n12/7/2015 2:46:59 PMINFO: HOME -> ./events/ping\r\n12/7/2015 2:46:59 PMINFO: Sending agent-instance-scripts applied 1-4b5124bd74cd423f98d57550b481ec77ec3a7135c6a650886ab95c043305d642\r\n12/7/2015 2:46:59 PMINFO: Getting monit\r\n12/7/2015 2:46:59 PMINFO: Updating monit\r\n12/7/2015 2:46:59 PMINFO: Downloading http://10.164.51.171:8080/v1//configcontent//monit current=\r\n12/7/2015 2:46:59 PMINFO: Running /var/lib/cattle/download/monit/monit-1-d166a713486fe4c0f039e152939d8d3b8ab8e6c2518e13d88f0b7ec68da46109/apply.sh\r\n12/7/2015 2:46:59 PMINFO: ROOT -> ./\r\n12/7/2015 2:46:59 PMINFO: ROOT -> ./etc/\r\n12/7/2015 2:46:59 PMINFO: ROOT -> ./etc/monit/\r\n12/7/2015 2:46:59 PMINFO: ROOT -> ./etc/monit/monitrc\r\n12/7/2015 2:46:59 PMINFO: Sending monit applied 1-d166a713486fe4c0f039e152939d8d3b8ab8e6c2518e13d88f0b7ec68da46109\r\n12/7/2015 2:46:59 PMINFO: Getting haproxy\r\n12/7/2015 2:46:59 PMINFO: Updating haproxy\r\n12/7/2015 2:46:59 PMINFO: Removing haproxy-34-39f1f6bdd2527b0410ecb7df9b0e2ca45693af0171597b58595d57a7c1083aa6\r\n12/7/2015 2:46:59 PMINFO: Downloading http://10.164.51.171:8080/v1//configcontent//haproxy current=\r\n12/7/2015 2:46:59 PMINFO: Running /var/lib/cattle/download/haproxy/haproxy-59-f2841f471f610f1abdb0bec6ce97912d8d19cd0ea731af73dd3c4ec7be331be2/apply.sh\r\n12/7/2015 2:46:59 PMINFO: ROOT -> ./\r\n12/7/2015 2:46:59 PMINFO: ROOT -> ./etc/\r\n12/7/2015 2:46:59 PMINFO: ROOT -> ./etc/haproxy/\r\n12/7/2015 2:46:59 PMINFO: ROOT -> ./etc/haproxy/certs/\r\n12/7/2015 2:46:59 PMINFO: ROOT -> ./etc/haproxy/certs/default.pem\r\n12/7/2015 2:46:59 PMINFO: ROOT -> ./etc/haproxy/certs/certs.pem\r\n12/7/2015 2:46:59 PMINFO: ROOT -> ./etc/haproxy/haproxy.cfg\r\n12/7/2015 2:46:59 PMINFO: ROOT -> ./etc/monit/\r\n12/7/2015 2:46:59 PMINFO: ROOT -> ./etc/monit/conf.d/\r\n12/7/2015 2:46:59 PMINFO: ROOT -> ./etc/monit/conf.d/haproxy\r\n12/7/2015 2:46:59 PMINFO: ROOT -> ./etc/default/\r\n12/7/2015 2:46:59 PMINFO: ROOT -> ./etc/default/haproxy\r\n12/7/2015 2:47:00 PMINFO: Sending haproxy applied 59-f2841f471f610f1abdb0bec6ce97912d8d19cd0ea731af73dd3c4ec7be331be2\r\n12/7/2015 2:47:00 PMINFO: HOME -> ./\r\n12/7/2015 2:47:00 PMINFO: HOME -> ./etc/\r\n12/7/2015 2:47:00 PMINFO: HOME -> ./etc/cattle/\r\n12/7/2015 2:47:00 PMINFO: HOME -> ./etc/cattle/startup-env\r\n12/7/2015 2:47:00 PMINFO: ROOT -> ./\r\n12/7/2015 2:47:00 PMINFO: ROOT -> ./etc/\r\n12/7/2015 2:47:00 PMINFO: ROOT -> ./etc/init.d/\r\n12/7/2015 2:47:00 PMINFO: ROOT -> ./etc/init.d/agent-instance-startup\r\n12/7/2015 2:47:00 PMINFO: Sending agent-instance-startup applied 2-557156a6c83b3adc996b56beb2d4469f282a091c31fd17461c90ee76fa7fe6e0\r\n12/7/2015 2:47:00 PMStarting monit daemon with http interface at [localhost:2812]"},"comment":{"url":"https://api.github.com/repos/rancher/rancher/issues/comments/163349425","html_url":"https://github.com/rancher/rancher/issues/2940#issuecomment-163349425","issue_url":"https://api.github.com/repos/rancher/rancher/issues/2940","id":163349425,"user":{"login":"cusspvz","id":3604053,"avatar_url":"https://avatars.githubusercontent.com/u/3604053?v=3","gravatar_id":"","url":"https://api.github.com/users/cusspvz","html_url":"https://github.com/cusspvz","followers_url":"https://api.github.com/users/cusspvz/followers","following_url":"https://api.github.com/users/cusspvz/following{/other_user}","gists_url":"https://api.github.com/users/cusspvz/gists{/gist_id}","starred_url":"https://api.github.com/users/cusspvz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cusspvz/subscriptions","organizations_url":"https://api.github.com/users/cusspvz/orgs","repos_url":"https://api.github.com/users/cusspvz/repos","events_url":"https://api.github.com/users/cusspvz/events{/privacy}","received_events_url":"https://api.github.com/users/cusspvz/received_events","type":"User","site_admin":false},"created_at":"2015-12-09T18:26:34Z","updated_at":"2015-12-09T18:26:34Z","body":"@rzachariah could you please tell me if it stops responding or it just delays your http requests?"}}