{"action":"opened","issue":{"url":"https://api.github.com/repos/hashicorp/vault/issues/804","labels_url":"https://api.github.com/repos/hashicorp/vault/issues/804/labels{/name}","comments_url":"https://api.github.com/repos/hashicorp/vault/issues/804/comments","events_url":"https://api.github.com/repos/hashicorp/vault/issues/804/events","html_url":"https://github.com/hashicorp/vault/issues/804","id":119361125,"number":804,"title":"Vault with consul on large implementations ","user":{"login":"mo-mughrabi","id":772251,"avatar_url":"https://avatars.githubusercontent.com/u/772251?v=3","gravatar_id":"","url":"https://api.github.com/users/mo-mughrabi","html_url":"https://github.com/mo-mughrabi","followers_url":"https://api.github.com/users/mo-mughrabi/followers","following_url":"https://api.github.com/users/mo-mughrabi/following{/other_user}","gists_url":"https://api.github.com/users/mo-mughrabi/gists{/gist_id}","starred_url":"https://api.github.com/users/mo-mughrabi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mo-mughrabi/subscriptions","organizations_url":"https://api.github.com/users/mo-mughrabi/orgs","repos_url":"https://api.github.com/users/mo-mughrabi/repos","events_url":"https://api.github.com/users/mo-mughrabi/events{/privacy}","received_events_url":"https://api.github.com/users/mo-mughrabi/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-11-29T13:50:30Z","updated_at":"2015-11-29T13:50:30Z","closed_at":null,"body":"Hello, \r\n\r\nRecently we decided to give consul+vault a try to be implemented across our infrastructure. At the moment, we are trying to spin up a test site with consul+vault but am having number of issues that are not clear in the documentation. \r\n\r\nCurrently, in the test site, we have 3 consul servers and X number of instances as clients, we use an orchestrator to launch the instances and configure consul.\r\n\r\nHere is how server configuration looks like\r\n```\r\n{\r\n    \"acl_datacenter\": \"gce10\",\r\n    \"acl_default_policy\": \"deny\",\r\n    \"acl_master_token\": \"master-token\",\r\n    \"bind_addr\": \"0.0.0.0\",\r\n    \"bootstrap_expect\": 2,\r\n    \"client_addr\": \"0.0.0.0\",\r\n    \"data_dir\": \"/var/consul\",\r\n    \"datacenter\": \"gce1\",\r\n    \"dns_config\": {\r\n        \"allow_stale\": true\r\n    },\r\n    \"domain\": \"consul\",\r\n    \"enable_debug\": true,\r\n    \"encrypt\": \"encryption-key\",\r\n    \"log_level\": \"info\",\r\n    \"retry_interval\": \"30s\",\r\n    \"retry_join\": [\r\n        \"ip-address-consul-1\",\r\n        \"ip-address-consul-2\",\r\n        \"ip-address-consul-3\"\r\n    ],\r\n    \"server\": true,\r\n    \"ui_dir\": \"/usr/local/share/consul/ui\"\r\n}\r\n```\r\n\r\nAs for the client, this is a sample of how the service.json and config.json looks like\r\n\r\n`config.json`\r\n\r\n```\r\n{\r\n    \"acl_datacenter\": \"gce10\",\r\n    \"acl_default_policy\": \"deny\",\r\n    \"acl_token\": \"client-token\",\r\n    \"bind_addr\": \"0.0.0.0\",\r\n    \"data_dir\": \"/var/consul\",\r\n    \"datacenter\": \"gce1\",\r\n    \"enable_debug\": false,\r\n    \"encrypt\": \"encryption-key\",\r\n    \"log_level\": \"info\",\r\n    \"retry_interval\": \"30s\",\r\n    \"retry_join\": [\r\n        \"ip-address-consul-1\",\r\n        \"ip-address-consul-2\",\r\n        \"ip-address-consul-3\"\r\n    ],\r\n    \"server\": false,\r\n    \"ui_dir\": \"/usr/local/share/consul/ui\"\r\n}\r\n```\r\n\r\n`service.json`\r\n\r\n```\r\n{\r\n  \"services\": [{\"name\": \"api\", \"port\": 80}]\r\n}\r\n```\r\n\r\nAm able to see all servers and services using `consul members` they all visible, am also using `dnsmasq` to resolve dns for each of the services. \r\n\r\nWe also using `consul-template` in most services to render configuration using consul key/value and its working fine so far, and we are using the orchestrator layer to spin up services using `consul-template` to retrieve service information.\r\n\r\nNow our next step is to add vault to the mix, in order to secure some keys, for that we setup one vault server and registered the vault server as a service on consul. This is the vault server configuration \r\n\r\nconfig.hcl\r\n\r\n```\r\nbackend \"consul\" {\r\n  address = \"127.0.0.1:8500\"\r\n  advertise_addr = \"http://internal-network-ip-address:8500\"\r\n  path = \"vault\"\r\n}\r\n\r\nlistener \"tcp\" {\r\n address = \"internal-network-ip-address:8200\"\r\n tls_disable = 0\r\n}\r\n```\r\n\r\nThen, once consul is running on vault server and we launched vault in the above configuration. We tried to init vault, load consul backend and load an allow policy. \r\n\r\n```\r\nvault init\r\nvault unseal\r\nvault mount consul\r\nvault write consul/config/access \\\r\n    address=127.0.0.1:8500 \\\r\n    token=client-token\r\n\r\n```\r\n\r\n<img width=\"548\" alt=\"screen shot 2015-11-29 at 4 37 59 pm\" src=\"https://cloud.githubusercontent.com/assets/772251/11457357/92ad1eda-96b7-11e5-8e29-dc8236b7087d.png\">\r\n\r\nWhich generates the vault folder on consul.\r\n\r\nFrom the vault server, am able to write new keys, view vault status \r\n\r\n```\r\nvault status\r\nSealed: false\r\nKey Shares: 5\r\nKey Threshold: 3\r\nUnseal Progress: 0\r\n\r\nHigh-Availability Enabled: true\r\n\tMode: active\r\n\tLeader: http://internal-network-ip-address:8500\r\n\r\nvault read secret/database/password\r\nKey           \tValue\r\nlease_duration\t2592000\r\nvalue         \tdude\r\n```\r\n\r\nNow that I've setup vault server, I installed vault on other nodes to run as standby, using same vault config as the leader (above).\r\n\r\nI tried to create ACL in consul for the new node \"api\" which needs to read vault when running `consul-template`.\r\n<img width=\"643\" alt=\"screen shot 2015-11-29 at 4 44 15 pm\" src=\"https://cloud.githubusercontent.com/assets/772251/11457399/bad83c72-96b8-11e5-8827-e368ba01c411.png\">\r\n\r\nI used the newly created ACL token to and updated the node's consul config `acl_token=using-token-id-from-screenshot-above`\r\n\r\nbut when I try to run consul-template, it asks me to unseal the vault, after unsealing and running consul-template am getting following error \r\n\r\n```\r\nconsul-template -template settings.ctmpl:/opt/docker.env -once -retry 30s -token=423cea63-e546-c7f6-804f-898bd7afddfb \r\n2015/11/29 13:28:06 [ERR] (view) \"vault(secret/database)\" no secret exists at path \"\\\"vault(secret/database)\\\"\"\r\n2015/11/29 13:28:06 [ERR] (runner) watcher reported error: no secret exists at path \"\\\"vault(secret/database)\\\"\"\r\nConsul Template returned errors:\r\nno secret exists at path \"\\\"vault(secret/database)\\\"\"\r\n```\r\n\r\nHonestly, some parts of the configuration were not documented and found some clues here and there on mailing lists. I couldn't find a solid documentation on how to setup consul+vault on multi nodes\r\n\r\nPlease advise?"}}