{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/etcd/issues/3333","labels_url":"https://api.github.com/repos/coreos/etcd/issues/3333/labels{/name}","comments_url":"https://api.github.com/repos/coreos/etcd/issues/3333/comments","events_url":"https://api.github.com/repos/coreos/etcd/issues/3333/events","html_url":"https://github.com/coreos/etcd/issues/3333","id":102006943,"number":3333,"title":"etcdserver: request timed out with requests from fleet ","user":{"login":"bobelev","id":3942342,"avatar_url":"https://avatars.githubusercontent.com/u/3942342?v=3","gravatar_id":"","url":"https://api.github.com/users/bobelev","html_url":"https://github.com/bobelev","followers_url":"https://api.github.com/users/bobelev/followers","following_url":"https://api.github.com/users/bobelev/following{/other_user}","gists_url":"https://api.github.com/users/bobelev/gists{/gist_id}","starred_url":"https://api.github.com/users/bobelev/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bobelev/subscriptions","organizations_url":"https://api.github.com/users/bobelev/orgs","repos_url":"https://api.github.com/users/bobelev/repos","events_url":"https://api.github.com/users/bobelev/events{/privacy}","received_events_url":"https://api.github.com/users/bobelev/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-08-19T21:47:22Z","updated_at":"2015-08-19T23:53:04Z","closed_at":null,"body":"fleet sometimes get timeouts for its requets. It happens at random time for a short time and causes fleet to reschedule all services.\r\n\r\nLogs from etcd and fleet are here —https://gist.github.com/bobelev/832197585721f6b968b9.\r\n\r\ncloud config\r\n```\r\ncoreos:\r\n  etcd2:\r\n    discovery: https://discovery.etcd.io/\r\n    advertise-client-urls: http://$private_ipv4:2379\r\n    initial-advertise-peer-urls: http://$private_ipv4:2380\r\n    listen-client-urls: http://$private_ipv4:2379,http://127.0.0.1:2379\r\n    listen-peer-urls: http://$private_ipv4:2380\r\n  fleet:\r\n    public-ip: $private_ipv4\r\n    etcd_servers: \"http://127.0.0.1:2379\"\r\n    etcd_request_timeout: 3\r\n  units:\r\n    - name: etcd2.service\r\n      command: start\r\n    - name: fleet.service\r\n      command: start\r\n```"},"comment":{"url":"https://api.github.com/repos/coreos/etcd/issues/comments/132828701","html_url":"https://github.com/coreos/etcd/issues/3333#issuecomment-132828701","issue_url":"https://api.github.com/repos/coreos/etcd/issues/3333","id":132828701,"user":{"login":"xiang90","id":4479947,"avatar_url":"https://avatars.githubusercontent.com/u/4479947?v=3","gravatar_id":"","url":"https://api.github.com/users/xiang90","html_url":"https://github.com/xiang90","followers_url":"https://api.github.com/users/xiang90/followers","following_url":"https://api.github.com/users/xiang90/following{/other_user}","gists_url":"https://api.github.com/users/xiang90/gists{/gist_id}","starred_url":"https://api.github.com/users/xiang90/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xiang90/subscriptions","organizations_url":"https://api.github.com/users/xiang90/orgs","repos_url":"https://api.github.com/users/xiang90/repos","events_url":"https://api.github.com/users/xiang90/events{/privacy}","received_events_url":"https://api.github.com/users/xiang90/received_events","type":"User","site_admin":false},"created_at":"2015-08-19T23:53:04Z","updated_at":"2015-08-19T23:53:04Z","body":"I see network layer issues (maybe in etcd's transportation layer) in the log. The timeouts are caused by the message dropping. Can you check if you have a stable network and no burst request load?"}}