{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/fleet/issues/804","labels_url":"https://api.github.com/repos/coreos/fleet/issues/804/labels{/name}","comments_url":"https://api.github.com/repos/coreos/fleet/issues/804/comments","events_url":"https://api.github.com/repos/coreos/fleet/issues/804/events","html_url":"https://github.com/coreos/fleet/issues/804","id":40930692,"number":804,"title":"Minimize Engine/AgentReconciler etcd requests","user":{"login":"bcwaldon","id":548759,"avatar_url":"https://avatars.githubusercontent.com/u/548759?v=3","gravatar_id":"","url":"https://api.github.com/users/bcwaldon","html_url":"https://github.com/bcwaldon","followers_url":"https://api.github.com/users/bcwaldon/followers","following_url":"https://api.github.com/users/bcwaldon/following{/other_user}","gists_url":"https://api.github.com/users/bcwaldon/gists{/gist_id}","starred_url":"https://api.github.com/users/bcwaldon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bcwaldon/subscriptions","organizations_url":"https://api.github.com/users/bcwaldon/orgs","repos_url":"https://api.github.com/users/bcwaldon/repos","events_url":"https://api.github.com/users/bcwaldon/events{/privacy}","received_events_url":"https://api.github.com/users/bcwaldon/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/coreos/fleet/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2014-08-22T17:30:02Z","updated_at":"2015-03-25T18:41:13Z","closed_at":null,"body":"I have a cluster running 2 units. The engine makes the following requests to gather the relevant info for a single reconciliation:\r\n\r\n```\r\nAug 22 17:27:29 core-01 fleet[1384]: I0822 17:27:29.002611 01384 reconciler.go:38] Polling Registry for actionable work\r\nAug 22 17:27:29 core-01 fleet[1384]: I0822 17:27:29.003835 01384 client.go:353] etcd: sending HTTP request GET http://localhost:4001/v2/keys/_coreos.com/fleet/job?consistent=true&recursive=true&sorted=true\r\nAug 22 17:27:29 core-01 fleet[1384]: I0822 17:27:29.004529 01384 client.go:360] etcd: recv response from GET http://localhost:4001/v2/keys/_coreos.com/fleet/job?consistent=true&recursive=true&sorted=true: 200 OK\r\nAug 22 17:27:29 core-01 fleet[1384]: I0822 17:27:29.005956 01384 client.go:353] etcd: sending HTTP request GET http://localhost:4001/v2/keys/_coreos.com/fleet/unit/e5a03e40306d918f3c0f6eb056a8e62acc1822d8?consistent=true&recursive=true&sorted=false\r\nAug 22 17:27:29 core-01 fleet[1384]: I0822 17:27:29.006835 01384 client.go:360] etcd: recv response from GET http://localhost:4001/v2/keys/_coreos.com/fleet/unit/e5a03e40306d918f3c0f6eb056a8e62acc1822d8?consistent=true&recursive=true&sorted=false: 200 OK\r\nAug 22 17:27:29 core-01 fleet[1384]: I0822 17:27:29.007362 01384 client.go:353] etcd: sending HTTP request GET http://localhost:4001/v2/keys/_coreos.com/fleet/unit/d9de011694dda111458a4fc9912ad629d86ad8af?consistent=true&recursive=true&sorted=false\r\nAug 22 17:27:29 core-01 fleet[1384]: I0822 17:27:29.008796 01384 client.go:360] etcd: recv response from GET http://localhost:4001/v2/keys/_coreos.com/fleet/unit/d9de011694dda111458a4fc9912ad629d86ad8af?consistent=true&recursive=true&sorted=false: 200 OK\r\nAug 22 17:27:29 core-01 fleet[1384]: I0822 17:27:29.009337 01384 client.go:353] etcd: sending HTTP request GET http://localhost:4001/v2/keys/_coreos.com/fleet/job?consistent=true&recursive=true&sorted=true\r\nAug 22 17:27:29 core-01 fleet[1384]: I0822 17:27:29.010080 01384 client.go:360] etcd: recv response from GET http://localhost:4001/v2/keys/_coreos.com/fleet/job?consistent=true&recursive=true&sorted=true: 200 OK\r\nAug 22 17:27:29 core-01 fleet[1384]: I0822 17:27:29.011552 01384 client.go:353] etcd: sending HTTP request GET http://localhost:4001/v2/keys/_coreos.com/fleet/state?consistent=true&recursive=true&sorted=false\r\nAug 22 17:27:29 core-01 fleet[1384]: I0822 17:27:29.012973 01384 client.go:360] etcd: recv response from GET http://localhost:4001/v2/keys/_coreos.com/fleet/state?consistent=true&recursive=true&sorted=false: 200 OK\r\nAug 22 17:27:29 core-01 fleet[1384]: I0822 17:27:29.013626 01384 client.go:353] etcd: sending HTTP request GET http://localhost:4001/v2/keys/_coreos.com/fleet/states?consistent=true&recursive=true&sorted=false\r\nAug 22 17:27:29 core-01 fleet[1384]: I0822 17:27:29.015100 01384 client.go:360] etcd: recv response from GET http://localhost:4001/v2/keys/_coreos.com/fleet/states?consistent=true&recursive=true&sorted=false: 200 OK\r\nAug 22 17:27:29 core-01 fleet[1384]: I0822 17:27:29.015385 01384 client.go:353] etcd: sending HTTP request GET http://localhost:4001/v2/keys/_coreos.com/fleet/machines?consistent=true&recursive=true&sorted=true\r\nAug 22 17:27:29 core-01 fleet[1384]: I0822 17:27:29.016772 01384 client.go:360] etcd: recv response from GET http://localhost:4001/v2/keys/_coreos.com/fleet/machines?consistent=true&recursive=true&sorted=true: 200 OK\r\nAug 22 17:27:29 core-01 fleet[1384]: I0822 17:27:29.017024 01384 engine.go:71] Engine completed reconciliation in 14.414077ms\r\n```\r\n\r\nThe agent ends up making a similarly-inefficient set of calls:\r\n\r\n```\r\nAug 22 17:41:06 core-01 fleet[1384]: I0822 17:41:06.000151 01384 reconcile.go:68] AgentReconciler tick\r\nAug 22 17:41:06 core-01 fleet[1384]: I0822 17:41:06.001583 01384 client.go:353] etcd: sending HTTP request GET http://localhost:4001/v2/keys/_coreos.com/fleet/job?consistent=true&recursive=true&sorted=true\r\nAug 22 17:41:06 core-01 fleet[1384]: I0822 17:41:06.009265 01384 client.go:360] etcd: recv response from GET http://localhost:4001/v2/keys/_coreos.com/fleet/job?consistent=true&recursive=true&sorted=true: 200 OK\r\nAug 22 17:41:06 core-01 fleet[1384]: I0822 17:41:06.010507 01384 client.go:353] etcd: sending HTTP request GET http://localhost:4001/v2/keys/_coreos.com/fleet/unit/e5a03e40306d918f3c0f6eb056a8e62acc1822d8?consistent=true&recursive=true&sorted=false\r\nAug 22 17:41:06 core-01 fleet[1384]: I0822 17:41:06.011013 01384 client.go:360] etcd: recv response from GET http://localhost:4001/v2/keys/_coreos.com/fleet/unit/e5a03e40306d918f3c0f6eb056a8e62acc1822d8?consistent=true&recursive=true&sorted=false: 200 OK\r\nAug 22 17:41:06 core-01 fleet[1384]: I0822 17:41:06.012216 01384 client.go:353] etcd: sending HTTP request GET http://localhost:4001/v2/keys/_coreos.com/fleet/unit/d9de011694dda111458a4fc9912ad629d86ad8af?consistent=true&recursive=true&sorted=false\r\nAug 22 17:41:06 core-01 fleet[1384]: I0822 17:41:06.015528 01384 client.go:360] etcd: recv response from GET http://localhost:4001/v2/keys/_coreos.com/fleet/unit/d9de011694dda111458a4fc9912ad629d86ad8af?consistent=true&recursive=true&sorted=false: 200 OK\r\nAug 22 17:41:06 core-01 fleet[1384]: I0822 17:41:06.018001 01384 client.go:353] etcd: sending HTTP request GET http://localhost:4001/v2/keys/_coreos.com/fleet/job?consistent=true&recursive=true&sorted=true\r\nAug 22 17:41:06 core-01 fleet[1384]: I0822 17:41:06.019370 01384 client.go:360] etcd: recv response from GET http://localhost:4001/v2/keys/_coreos.com/fleet/job?consistent=true&recursive=true&sorted=true: 200 OK\r\nAug 22 17:41:06 core-01 fleet[1384]: I0822 17:41:06.019657 01384 client.go:353] etcd: sending HTTP request GET http://localhost:4001/v2/keys/_coreos.com/fleet/state?consistent=true&recursive=true&sorted=false\r\nAug 22 17:41:06 core-01 fleet[1384]: I0822 17:41:06.022429 01384 client.go:360] etcd: recv response from GET http://localhost:4001/v2/keys/_coreos.com/fleet/state?consistent=true&recursive=true&sorted=false: 200 OK\r\nAug 22 17:41:06 core-01 fleet[1384]: I0822 17:41:06.022523 01384 client.go:353] etcd: sending HTTP request GET http://localhost:4001/v2/keys/_coreos.com/fleet/states?consistent=true&recursive=true&sorted=false\r\nAug 22 17:41:06 core-01 fleet[1384]: I0822 17:41:06.024085 01384 client.go:360] etcd: recv response from GET http://localhost:4001/v2/keys/_coreos.com/fleet/states?consistent=true&recursive=true&sorted=false: 200 OK\r\nAug 22 17:41:06 core-01 fleet[1384]: I0822 17:41:06.024225 01384 reconcile.go:265] Desired state \"launched\" matches current state of Job(foo.service), nothing to do\r\nAug 22 17:41:06 core-01 fleet[1384]: I0822 17:41:06.024233 01384 reconcile.go:265] Desired state \"launched\" matches current state of Job(ping.service), nothing to do\r\nAug 22 17:41:06 core-01 fleet[1384]: I0822 17:41:06.024240 01384 reconcile.go:44] AgentReconciler completed reconciliation in 22.798384ms\r\n```\r\n\r\nThere are two problems that need fixing:\r\n1. fleet gets each unit file individually - get all unit files with a single GET /unit?recursive=true\r\n2. fleet calls  GET /job?recursive=true twice - this should just be done once"},"comment":{"url":"https://api.github.com/repos/coreos/fleet/issues/comments/86163994","html_url":"https://github.com/coreos/fleet/issues/804#issuecomment-86163994","issue_url":"https://api.github.com/repos/coreos/fleet/issues/804","id":86163994,"user":{"login":"bcwaldon","id":548759,"avatar_url":"https://avatars.githubusercontent.com/u/548759?v=3","gravatar_id":"","url":"https://api.github.com/users/bcwaldon","html_url":"https://github.com/bcwaldon","followers_url":"https://api.github.com/users/bcwaldon/followers","following_url":"https://api.github.com/users/bcwaldon/following{/other_user}","gists_url":"https://api.github.com/users/bcwaldon/gists{/gist_id}","starred_url":"https://api.github.com/users/bcwaldon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bcwaldon/subscriptions","organizations_url":"https://api.github.com/users/bcwaldon/orgs","repos_url":"https://api.github.com/users/bcwaldon/repos","events_url":"https://api.github.com/users/bcwaldon/events{/privacy}","received_events_url":"https://api.github.com/users/bcwaldon/received_events","type":"User","site_admin":false},"created_at":"2015-03-25T18:41:13Z","updated_at":"2015-03-25T18:41:13Z","body":"@denderello Not any time soon. Feel free to work on this if you have the time."}}