{"action":"created","issue":{"url":"https://api.github.com/repos/hashicorp/vault/issues/673","labels_url":"https://api.github.com/repos/hashicorp/vault/issues/673/labels{/name}","comments_url":"https://api.github.com/repos/hashicorp/vault/issues/673/comments","events_url":"https://api.github.com/repos/hashicorp/vault/issues/673/events","html_url":"https://github.com/hashicorp/vault/issues/673","id":110109212,"number":673,"title":"Issue with lease revocation in v0.3","user":{"login":"nbrownus","id":957319,"avatar_url":"https://avatars.githubusercontent.com/u/957319?v=3","gravatar_id":"","url":"https://api.github.com/users/nbrownus","html_url":"https://github.com/nbrownus","followers_url":"https://api.github.com/users/nbrownus/followers","following_url":"https://api.github.com/users/nbrownus/following{/other_user}","gists_url":"https://api.github.com/users/nbrownus/gists{/gist_id}","starred_url":"https://api.github.com/users/nbrownus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nbrownus/subscriptions","organizations_url":"https://api.github.com/users/nbrownus/orgs","repos_url":"https://api.github.com/users/nbrownus/repos","events_url":"https://api.github.com/users/nbrownus/events{/privacy}","received_events_url":"https://api.github.com/users/nbrownus/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/hashicorp/vault/labels/waiting-reply","name":"waiting-reply","color":"c7def8"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-10-06T21:54:30Z","updated_at":"2015-10-06T23:51:44Z","closed_at":null,"body":"Looks like vault opens a socket for every revoke attempt. If you have many tens of thousands of leases to revoke it starves the system of sockets. I am currently running v0.3 but all these leases are a result of running v0.2 for a while. The error message is `connect: cannot assign requested address`.\r\n\r\nWhen `vault` does get a socket the revocation fails with `failed to revoke entry: secret is unsupported by this backend`. Not sure if this leaves the lease in `consul` and means `vault` will eventually attempt to revoke all the leases again at a later date."},"comment":{"url":"https://api.github.com/repos/hashicorp/vault/issues/comments/146036630","html_url":"https://github.com/hashicorp/vault/issues/673#issuecomment-146036630","issue_url":"https://api.github.com/repos/hashicorp/vault/issues/673","id":146036630,"user":{"login":"nbrownus","id":957319,"avatar_url":"https://avatars.githubusercontent.com/u/957319?v=3","gravatar_id":"","url":"https://api.github.com/users/nbrownus","html_url":"https://github.com/nbrownus","followers_url":"https://api.github.com/users/nbrownus/followers","following_url":"https://api.github.com/users/nbrownus/following{/other_user}","gists_url":"https://api.github.com/users/nbrownus/gists{/gist_id}","starred_url":"https://api.github.com/users/nbrownus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nbrownus/subscriptions","organizations_url":"https://api.github.com/users/nbrownus/orgs","repos_url":"https://api.github.com/users/nbrownus/repos","events_url":"https://api.github.com/users/nbrownus/events{/privacy}","received_events_url":"https://api.github.com/users/nbrownus/received_events","type":"User","site_admin":false},"created_at":"2015-10-06T23:51:44Z","updated_at":"2015-10-06T23:51:44Z","body":"This is server side lease revocation, backend is `consul`.\r\n\r\nI have a large number of leases because\r\n- My organization has a large number of secrets.\r\n- It is currently impossible to only allow a lease owner to renew a lease. Allowing for this decreases security #535 #455. Not allowing this results in lease bloat.\r\n- It is currently impossible to renew an auth token generated from the `app-id` auth mechanism #539. New auth tokens results in new secret leases.\r\n\r\n`vault` is currently opening >35k sockets to `consul` during these storms. I _could_ allow for more but that seems a little ridiculous.\r\n\r\nThe entire scenario:\r\n- `vault` leader attempts to revoke many leases. \r\n- `consul` hangs under the load, causing it to drop from the cluster.\r\n- New `vault` leader is elected, will likely fail because #648.\r\n- `vault` leader finishes the failed attempt to revoke leases but is unable to continue working because #648.\r\n- Human intervenes and restarts `vault`, 1 machine at a time.\r\n- Newly restarted and unsealed `vault` notices a lot of leases needing to be revoke, begins trying to revoke thousands of leases. This takes a good 10 to 15 minutes for me.\r\n- _If_ `consul` avoids hanging for long enough during the revocation storm and the newly unsealed `vault` instance can avoid #648, which seems to be a coin flip, I can move to the next machine. Rinse and repeat until `vault` is happy everywhere.\r\n\r\nv0.3 `vault` will attempt to revoke, fail, and re-attempt to revoke static secret leases until the maximum attempts to revoke has been reached. The failure means the next time `vault` restarts I get to do this all over again. This will affect anyone who was running v0.2 and upgrades to v0.3. Ideally `vault` would notice this fact and just delete the deprecated leases.\r\n\r\nGiven, I am not running on the best hardware in the world. I will eventually upgrade these machines but it seems like there is some low hanging fruit here that could get my setup running a lot better. v0.3 answers a lot of that by not creating leases for static secrets but if I were to use `vault` for a large number of dynamic leases the same issues would show up again."}}