{"action":"created","issue":{"url":"https://api.github.com/repos/cloudfoundry-incubator/lattice/issues/177","labels_url":"https://api.github.com/repos/cloudfoundry-incubator/lattice/issues/177/labels{/name}","comments_url":"https://api.github.com/repos/cloudfoundry-incubator/lattice/issues/177/comments","events_url":"https://api.github.com/repos/cloudfoundry-incubator/lattice/issues/177/events","html_url":"https://github.com/cloudfoundry-incubator/lattice/issues/177","id":101358562,"number":177,"title":"Problems deploying vagrant cluster, only first node runs tasks","user":{"login":"teodor-pripoae","id":393437,"avatar_url":"https://avatars.githubusercontent.com/u/393437?v=3","gravatar_id":"","url":"https://api.github.com/users/teodor-pripoae","html_url":"https://github.com/teodor-pripoae","followers_url":"https://api.github.com/users/teodor-pripoae/followers","following_url":"https://api.github.com/users/teodor-pripoae/following{/other_user}","gists_url":"https://api.github.com/users/teodor-pripoae/gists{/gist_id}","starred_url":"https://api.github.com/users/teodor-pripoae/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/teodor-pripoae/subscriptions","organizations_url":"https://api.github.com/users/teodor-pripoae/orgs","repos_url":"https://api.github.com/users/teodor-pripoae/repos","events_url":"https://api.github.com/users/teodor-pripoae/events{/privacy}","received_events_url":"https://api.github.com/users/teodor-pripoae/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-08-17T07:41:23Z","updated_at":"2015-08-18T19:53:09Z","closed_at":null,"body":"Hi,\r\n\r\nI'm trying to simulate a 3 machine cluster using vagrant and it seems that only the first node is able to run apps.\r\n\r\nMy setup is as follows:\r\n\r\n| machine   |    eth0         |    eth1       |     eth2        |\r\n|---------------|-----------------|---------------|-----------------|\r\n| c01           | 10.0.2.15       | 10.0.1.111    | 172.19.8.111      |\r\n| c02           | 10.0.2.15       | 10.0.1.112    | 172.19.8.112      |\r\n| c03           | 10.0.2.15       | 10.0.1.113    | 172.19.8.113      |\r\n\r\n- eth0 - It is added by vagrant as NAT interface for controlling the vm\r\n- eth1 - a bridged public ip, the vm will be accessible on the network, to simulate cloud public ip\r\n- eth2 - a private network ip, this will be used as a private network between vms\r\n\r\nVagrant setup:\r\n\r\n```ruby\r\n(1..$compute_nodes).each do |i|\r\n  hostname = \"c%02d\" % i\r\n  public_ip = \"10.0.1.#{i+110}\"\r\n  private_ip = \"172.19.8.#{i+110}\"\r\n\r\n  config.vm.define hostname do |node|\r\n    node.vm.box = \"lattice/ubuntu-trusty-64\"\r\n    node.vm.box_version = '0.3.0'\r\n\r\n    node.vm.provider \"virtualbox\" do |vb|\r\n      vb.memory = $compute_memory\r\n      vb.gui = false\r\n    end\r\n\r\n    node.vm.hostname = hostname\r\n\r\n    node.vm.network \"public_network\", bridge: \"en0\", ip: public_ip\r\n    node.vm.network \"private_network\", ip: private_ip\r\n\r\n    # ... provisioning with chef solo\r\n  end\r\nend\r\n```\r\n\r\nChef solo does the following:\r\n\r\n- allow all trafic on eth2 subnet `sudo ufw allow from 172.19.8.0/24`\r\n- deny all trafic on eth1\r\n- creates /var/lattice/setup/lattice-environment for each node, see below\r\n- downloads latest stable from s3\r\n- for c01 runs `install-from-tar collocated`, for c02, c03 runs `install-from-tar cell`\r\n- patches consul.json (see below) and then restarts consul agent\r\n\r\n```bash\r\n# C01\r\nPUBLIC_IP=172.19.8.111\r\nPRIVATE_IP=172.19.8.111\r\nCONSUL_SERVER_IP=172.19.8.111\r\nSYSTEM_IP=172.19.8.111\r\nSYSTEM_DOMAIN=vag.kuende.cf\r\nGARDEN_EXTERNAL_IP=172.19.8.111\r\nLATTICE_CELL_ID=c01\r\n# C02\r\nPUBLIC_IP=172.19.8.112\r\nPRIVATE_IP=172.19.8.112\r\nCONSUL_SERVER_IP=172.19.8.111\r\nSYSTEM_IP=172.19.8.111\r\nSYSTEM_DOMAIN=vag.kuende.cf\r\nGARDEN_EXTERNAL_IP=172.19.8.112\r\nLATTICE_CELL_ID=c02\r\n# C03\r\nPUBLIC_IP=172.19.8.113\r\nPRIVATE_IP=172.19.8.113\r\nCONSUL_SERVER_IP=172.19.8.111\r\nSYSTEM_IP=172.19.8.111\r\nSYSTEM_DOMAIN=vag.kuende.cf\r\nGARDEN_EXTERNAL_IP=172.19.8.113\r\nLATTICE_CELL_ID=c03\r\n```\r\n\r\nConsul config patch:\r\n\r\n```\r\n# it was showing consul address as 10.0.2.15:8301 (eth0)\r\n# and ltc visualize was showing only the first node\r\n# this example is for c01, similar for c02, c03\r\n{\r\n    \"advertise_addr\": \"172.19.8.111\",\r\n    \"bind_addr\": \"172.19.8.111\",\r\n    \"ports\": {\r\n        \"dns\": 53\r\n    }\r\n}\r\n```\r\n\r\nMy domain `kuende.cf` is set for `vag` and `*.vag` to c01 ip (172.19.8.111).\r\n\r\n\r\n```\r\n$ ltc target vag.kuende.cf\r\nCould not authenticate with the droplet store.\r\n\r\n$ echo '{\"target\":\"vag.kuende.cf\",\"dav_blob_store\":{\"host\": \"vag.kuende.cf\", \"port\": \"8444\"}}' > ~/.lattice/config.json\r\n\r\n$ ltc target\r\nTarget:\t\t\tvag.kuende.cf\r\nDroplet store:\tvag.kuende.cf:8444\r\n\r\n$ ltc cells\r\nCells\t\tZone\t\tMemory\t\tDisk\t\tApps\r\nc01\t\t\tz1\t\t\t2001M\t\t8675M\t\t0/0\r\nc02\t\t\tz1\t\t\t2001M\t\t8675M\t\t0/0\r\nc03\t\t\tz1\t\t\t2001M\t\t8675M\t\t0/0\r\n\r\n$ ltc create lattice-app cloudfoundry/lattice-app\r\n\r\n$ ltc visualize\r\nDistribution\r\nc01: •\r\nc02: empty\r\nc03: empty\r\n\r\n$ ltc scale lattice-app 3\r\nScaling lattice-app to 3 instances\r\n.....\r\nApp Scaled Successfully\r\n\r\n$ ltc visualize\r\nDistribution\r\nc01: •••\r\nc02: empty\r\nc03: empty\r\n```\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/cloudfoundry-incubator/lattice/issues/comments/132331719","html_url":"https://github.com/cloudfoundry-incubator/lattice/issues/177#issuecomment-132331719","issue_url":"https://api.github.com/repos/cloudfoundry-incubator/lattice/issues/177","id":132331719,"user":{"login":"teodor-pripoae","id":393437,"avatar_url":"https://avatars.githubusercontent.com/u/393437?v=3","gravatar_id":"","url":"https://api.github.com/users/teodor-pripoae","html_url":"https://github.com/teodor-pripoae","followers_url":"https://api.github.com/users/teodor-pripoae/followers","following_url":"https://api.github.com/users/teodor-pripoae/following{/other_user}","gists_url":"https://api.github.com/users/teodor-pripoae/gists{/gist_id}","starred_url":"https://api.github.com/users/teodor-pripoae/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/teodor-pripoae/subscriptions","organizations_url":"https://api.github.com/users/teodor-pripoae/orgs","repos_url":"https://api.github.com/users/teodor-pripoae/repos","events_url":"https://api.github.com/users/teodor-pripoae/events{/privacy}","received_events_url":"https://api.github.com/users/teodor-pripoae/received_events","type":"User","site_admin":false},"created_at":"2015-08-18T19:53:09Z","updated_at":"2015-08-18T19:53:09Z","body":"Hi,\r\n\r\nI've done a similar setup as digital ocean provides (the only difference is the eth0 which is added automatically by vagrant). 1 core for each vm, 2gb ram for each vm, 8gb disk for each vm, 1 public interface, 1 private interface.\r\n\r\nI looked over terraform config for DO, and it seems that cell and brain are connected on [ipv4_address](https://www.terraform.io/docs/providers/do/r/droplet.html#ipv4_address), which is the public ipv4 address. Since DO is offering private networking for intra-datacenter traffic, I don't see why it should communicate on the public interface. Also becouse private network in DO is not actually a **private** network, but just a network **shared** with all the DO customers in that specific datacenter, firewall rules should be enforced there.\r\n\r\nJust to make it clear, I'm using the same `install-from-tar` script that terraform is running, and providing the variables the same way that terraform is doing, but with chef. I think that the problem is due to lattice taking the first network interface available for clustering or ignoring firewall rules. I will do some more tests and I'll come later with an answer"}}