{"action":"created","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/6251","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/6251/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/6251/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/6251/events","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/6251","id":65559878,"number":6251,"title":"New GCE nodes are missing node-ip-range metadata","user":{"login":"mwuertinger","id":3739231,"avatar_url":"https://avatars.githubusercontent.com/u/3739231?v=3","gravatar_id":"","url":"https://api.github.com/users/mwuertinger","html_url":"https://github.com/mwuertinger","followers_url":"https://api.github.com/users/mwuertinger/followers","following_url":"https://api.github.com/users/mwuertinger/following{/other_user}","gists_url":"https://api.github.com/users/mwuertinger/gists{/gist_id}","starred_url":"https://api.github.com/users/mwuertinger/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mwuertinger/subscriptions","organizations_url":"https://api.github.com/users/mwuertinger/orgs","repos_url":"https://api.github.com/users/mwuertinger/repos","events_url":"https://api.github.com/users/mwuertinger/events{/privacy}","received_events_url":"https://api.github.com/users/mwuertinger/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-03-31T22:22:14Z","updated_at":"2015-03-31T23:20:57Z","closed_at":"2015-03-31T23:11:30Z","body":"# Summary\r\nAdding new node to a k8s cluster on Google Compute Engine (GCE) by modifying the associated instance group does not work. The newly created instances never enter the Ready state.\r\n\r\n# How to reproduce?\r\n- Download and extract latest ``kubernetes.tar.gz`` release (0.14.0) from Github.\r\n- ``MINION_DISK_SIZE=100GB cluster/kube-up.sh``\r\n- Wait till cluster becomes fully available.\r\n- ``cluster/kubectl.sh get minions``\r\n```\r\n    NAME                                          LABELS    STATUS\r\n    kubernetes-minion-daa5.c.<project>.internal   <none>    Ready\r\n    kubernetes-minion-e5h9.c.<project>.internal   <none>    Ready\r\n    kubernetes-minion-rzky.c.<project>.internal   <none>    Ready\r\n    kubernetes-minion-sbcr.c.<project>.internal   <none>    Ready\r\n```\r\n- Go to the developer console [1] and adjust the instance group size from 4 to 10 instances (I found no way to do this using ``gcloud``).\r\n- Wait a while\r\n- ``cluster/kubectl.sh get minions``\r\n```\r\nNAME                                          LABELS    STATUS\r\nkubernetes-minion-6g8n.c.<project>.internal   <none>    NotReady\r\nkubernetes-minion-c6g0.c.<project>.internal   <none>    NotReady\r\nkubernetes-minion-daa5.c.<project>.internal   <none>    Ready\r\nkubernetes-minion-e5h9.c.<project>.internal   <none>    Ready\r\nkubernetes-minion-er8q.c.<project>.internal   <none>    NotReady\r\nkubernetes-minion-iuer.c.<project>.internal   <none>    NotReady\r\nkubernetes-minion-j0ht.c.<project>.internal   <none>    NotReady\r\nkubernetes-minion-rzky.c.<project>.internal   <none>    Ready\r\nkubernetes-minion-sbcr.c.<project>.internal   <none>    Ready\r\nkubernetes-minion-zvt7.c.<project>.internal   <none>    NotReady\r\n```\r\n\r\n# Workaround\r\nThe minions can be saved by adding the ``node-ip-range`` metadata field to each of them using the following command:\r\n```bash\r\n  gcloud compute instances add-metadata \\\r\n    --zone $ZONE \\\r\n    $MINION_INSTANCE_NAME \\\r\n    --metadata node-ip-range=10.244.0.0/24\r\n```\r\n\r\n# Background information\r\n- The node template which is created by the ``create-node-template()`` function in ``cluster/gce/util.sh`` does not contain the ``node-ip-range`` metadata field.\r\n- The ``node-ip-range`` metadata field is added for all minions in a loop in the ``kube-up()`` function:\r\n```bash\r\n  # Create the routes and set IP ranges to instance metadata, 5 instances at a time.\r\n  for (( i=0; i<${#MINION_NAMES[@]}; i++)); do\r\n    create-route \"${MINION_NAMES[$i]}\" \"${MINION_IP_RANGES[$i]}\" &\r\n    add-instance-metadata \"${MINION_NAMES[$i]}\" \"node-ip-range=${MINION_IP_RANGES[$i]}\" &\r\n\r\n    if [ $i -ne 0 ] && [ $((i%5)) -eq 0 ]; then\r\n      echo Waiting for a batch of routes at $i...\r\n      wait-for-jobs\r\n    fi\r\n\r\n  done\r\n```\r\n- The variable ``MINION_IP_RANGES`` is determined in ``cluster/gce/config-default.sh``:\r\n```bash\r\nsubnets=()\r\n\r\nfor ((node_num=0; node_num<node_count; node_num++)); do\r\n  subnets+=(\"$next_node\"/\"${node_subnet_size}\")\r\n  next_node=$(increment_ipv4 $next_node $node_subnet_count)\r\ndone\r\n\r\nCLUSTER_IP_RANGE=\"10.244.0.0/16\"\r\nMINION_IP_RANGES=($(eval echo \"${subnets[@]}\"))\r\n```\r\n\r\n[1] `https://console.developers.google.com/project/<project>/compute/instanceGroups/list`"},"comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/comments/88280634","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/6251#issuecomment-88280634","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/6251","id":88280634,"user":{"login":"mwuertinger","id":3739231,"avatar_url":"https://avatars.githubusercontent.com/u/3739231?v=3","gravatar_id":"","url":"https://api.github.com/users/mwuertinger","html_url":"https://github.com/mwuertinger","followers_url":"https://api.github.com/users/mwuertinger/followers","following_url":"https://api.github.com/users/mwuertinger/following{/other_user}","gists_url":"https://api.github.com/users/mwuertinger/gists{/gist_id}","starred_url":"https://api.github.com/users/mwuertinger/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mwuertinger/subscriptions","organizations_url":"https://api.github.com/users/mwuertinger/orgs","repos_url":"https://api.github.com/users/mwuertinger/repos","events_url":"https://api.github.com/users/mwuertinger/events{/privacy}","received_events_url":"https://api.github.com/users/mwuertinger/received_events","type":"User","site_admin":false},"created_at":"2015-03-31T23:20:57Z","updated_at":"2015-03-31T23:20:57Z","body":"I created a pull request with a dirty hack that solves the problem for me. Although seeing little chance of this getting merged it might help other people running into the same issue until #5246 is fixed.\r\n\r\nhttps://github.com/GoogleCloudPlatform/kubernetes/pull/6254"}}