{"action":"created","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/6393","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/6393/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/6393/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/6393/events","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/pull/6393","id":66031684,"number":6393,"title":"Add PodSpec.NodeFailurePolicy = {Reschedule, Delete, Ignore}","user":{"login":"alex-mohr","id":8729097,"avatar_url":"https://avatars.githubusercontent.com/u/8729097?v=3","gravatar_id":"","url":"https://api.github.com/users/alex-mohr","html_url":"https://github.com/alex-mohr","followers_url":"https://api.github.com/users/alex-mohr/followers","following_url":"https://api.github.com/users/alex-mohr/following{/other_user}","gists_url":"https://api.github.com/users/alex-mohr/gists{/gist_id}","starred_url":"https://api.github.com/users/alex-mohr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alex-mohr/subscriptions","organizations_url":"https://api.github.com/users/alex-mohr/orgs","repos_url":"https://api.github.com/users/alex-mohr/repos","events_url":"https://api.github.com/users/alex-mohr/events{/privacy}","received_events_url":"https://api.github.com/users/alex-mohr/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/cla:%20yes","name":"cla: yes","color":"ECF7EC"}],"state":"open","locked":false,"assignee":{"login":"brendandburns","id":5751682,"avatar_url":"https://avatars.githubusercontent.com/u/5751682?v=3","gravatar_id":"","url":"https://api.github.com/users/brendandburns","html_url":"https://github.com/brendandburns","followers_url":"https://api.github.com/users/brendandburns/followers","following_url":"https://api.github.com/users/brendandburns/following{/other_user}","gists_url":"https://api.github.com/users/brendandburns/gists{/gist_id}","starred_url":"https://api.github.com/users/brendandburns/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brendandburns/subscriptions","organizations_url":"https://api.github.com/users/brendandburns/orgs","repos_url":"https://api.github.com/users/brendandburns/repos","events_url":"https://api.github.com/users/brendandburns/events{/privacy}","received_events_url":"https://api.github.com/users/brendandburns/received_events","type":"User","site_admin":false},"milestone":null,"comments":2,"created_at":"2015-04-02T23:28:20Z","updated_at":"2015-04-02T23:58:06Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/6393","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/pull/6393","diff_url":"https://github.com/GoogleCloudPlatform/kubernetes/pull/6393.diff","patch_url":"https://github.com/GoogleCloudPlatform/kubernetes/pull/6393.patch"},"body":"A start to updating PodSpec to include NodeFailurePolicy, which\r\ndescribes what the user would like the system to do when the pod is\r\nunable to run on its bound node, such as when that node is marked as\r\nfailed or removed from the cluster.\r\n\r\nNodeFailurePolicy may be Reschedule, Delete, or Ignore, with a\r\ndefault policy of Reschedule.\r\n\r\nDefault behavior will change such that pods reschedule in the presence\r\nof node failures by restarting on working nodes instead of being\r\ndeleted.  And if users do not want the default behavior, they're able\r\nto opt for deletion -- or completely disable those automatics by\r\nsetting the policy to Ignore and using some other mechanism to react.\r\n\r\nThis PR is only a sketch of the API and discusses the rationale and\r\nproposed plan.  After general consensus on the overall structure, I'll\r\nimplement it.  Plan: update the rest of pkg/api to support this field,\r\nadd inter-version conversion logic, update the existing pod eviction\r\ncode in cloudprovider/controller/nodecontroller.go to reschedule or\r\ndelete based on policy, add tests, and update docs to reflect new\r\nusage.\r\n\r\nI realize we've discussed similar topics before (most recently #5353,\r\nbut also #5334, #5335, and #260) with alternate approaches based on\r\nphilosophical perspectives as to what pods should be, but on further\r\nreflection, I think we can do better by our users by meeting them\r\nwhere they are and being pragmatic about allowing pods to survive\r\nfailures by rescheduling onto other nodes.\r\n\r\nIn particular, pod lifetime becomes bound to cluster lifetime rather\r\nthan the lifetime of a given node.  That change furthers our goal of\r\nhelping users to think about pods and containers rather than specific\r\nnodes.  Consider two specific use cases: (1) moving from node to\r\ncluster and (2) resizing a cluster.\r\n\r\n(1) If someone is using Kubelet (or docker or monit or supervisord) on\r\na particular node to keep its containers running, when they transition\r\nto Kubernetes, they receive a container cluster equivalent and\r\ntheir created pods benefit from the new environment.\r\n\r\n(2) With our current behavior, on static clusters composed of robust\r\nVMs, pods run for a long time.  But if the cluster is resized smaller,\r\npods on deleted nodes unexpectedly disappear.  With reschedulable\r\npods, they'll keep existing and running.\r\n\r\n\r\nOther benefits:\r\n* Names (IDs) remain unchanged as a pod moves around the cluster, which helps debugging and introspection.\r\n* Users can more-incrementally adopt Kubernetes concepts: they need only use pods and not replication controllers at first.\r\n* Replication controllers potentially become simpler: their config describes a factory that is able to create pods with specified properties, but the handling of specific names and ids is reflected in the pod.\r\n  * E.g. consider the difference between \"an EBS volume of 100GB\" and \"specific EBS volume Foo\".\r\n  * ... and we no longer need to create replication controllers of size 1.\r\n  * ... and it becomes feasible to have N reschedulable pods via 1 RC instead of N RCs.  (Or write a RCRC to replicate RCs of size 1.)\r\n\r\nYes, there will be costs to this change.  I rather-obviously think the\r\nbenefits outweigh those.  We won't be able to delete pods when it\r\nmight be convenient to do so.  And with some of the current overlay\r\nnetworks, in the short term a pod will change IP addresses because pod\r\nIPs are drawn from ranges bound to a particular host (though\r\npresumably we'll eventually remove that constraint).  It's important\r\nthat Kubernetes provide a great Container Cluster Construction Set --\r\nbut it's even more important that the out-of-the-box experience for\r\nvanilla Kubernetes be a great experience too.\r\n\r\n@smarterclayton, @erictune, @brendandburns because they've at various\r\ntimes seemed seemed to think it might be reasonable to reschedule\r\npods.  And @bgrant0607 for obvious reasons.\r\n"},"comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/comments/89084747","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/pull/6393#issuecomment-89084747","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/6393","id":89084747,"user":{"login":"smarterclayton","id":1163175,"avatar_url":"https://avatars.githubusercontent.com/u/1163175?v=3","gravatar_id":"","url":"https://api.github.com/users/smarterclayton","html_url":"https://github.com/smarterclayton","followers_url":"https://api.github.com/users/smarterclayton/followers","following_url":"https://api.github.com/users/smarterclayton/following{/other_user}","gists_url":"https://api.github.com/users/smarterclayton/gists{/gist_id}","starred_url":"https://api.github.com/users/smarterclayton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smarterclayton/subscriptions","organizations_url":"https://api.github.com/users/smarterclayton/orgs","repos_url":"https://api.github.com/users/smarterclayton/repos","events_url":"https://api.github.com/users/smarterclayton/events{/privacy}","received_events_url":"https://api.github.com/users/smarterclayton/received_events","type":"User","site_admin":false},"created_at":"2015-04-02T23:58:06Z","updated_at":"2015-04-02T23:58:06Z","body":"Some things it would affect:\r\n\r\n* initializers would not be able to run without knowledge that the node is down (scheduler would have to clear instance specific data on the pod)\r\n* a failed run once pod on a node might be rescheduled to another node (and started) before the kubelet reports the pod has failed.  (run once pods should not be allowed to be rescheduled?)\r\n* pods cannot reason about their past placement in external systems (can store podname+host)"}}