{"action":"created","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/10899","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/10899/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/10899/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/10899/events","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/10899","id":93660197,"number":10899,"title":"PD E2E tests fail in soak with `Pod cannot be started due to exceeded capacity`","user":{"login":"saad-ali","id":10052848,"avatar_url":"https://avatars.githubusercontent.com/u/10052848?v=3","gravatar_id":"","url":"https://api.github.com/users/saad-ali","html_url":"https://github.com/saad-ali","followers_url":"https://api.github.com/users/saad-ali/followers","following_url":"https://api.github.com/users/saad-ali/following{/other_user}","gists_url":"https://api.github.com/users/saad-ali/gists{/gist_id}","starred_url":"https://api.github.com/users/saad-ali/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saad-ali/subscriptions","organizations_url":"https://api.github.com/users/saad-ali/orgs","repos_url":"https://api.github.com/users/saad-ali/repos","events_url":"https://api.github.com/users/saad-ali/events{/privacy}","received_events_url":"https://api.github.com/users/saad-ali/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/priority/P1","name":"priority/P1","color":"eb6420"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/team/cluster","name":"team/cluster","color":"d2b48c"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/team/master","name":"team/master","color":"d2b48c"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/team/node","name":"team/node","color":"d2b48c"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/team/testing","name":"team/testing","color":"d2b48c"}],"state":"open","locked":false,"assignee":{"login":"saad-ali","id":10052848,"avatar_url":"https://avatars.githubusercontent.com/u/10052848?v=3","gravatar_id":"","url":"https://api.github.com/users/saad-ali","html_url":"https://github.com/saad-ali","followers_url":"https://api.github.com/users/saad-ali/followers","following_url":"https://api.github.com/users/saad-ali/following{/other_user}","gists_url":"https://api.github.com/users/saad-ali/gists{/gist_id}","starred_url":"https://api.github.com/users/saad-ali/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saad-ali/subscriptions","organizations_url":"https://api.github.com/users/saad-ali/orgs","repos_url":"https://api.github.com/users/saad-ali/repos","events_url":"https://api.github.com/users/saad-ali/events{/privacy}","received_events_url":"https://api.github.com/users/saad-ali/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/milestones/6","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/milestones/v1.0","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/milestones/6/labels","id":969317,"number":6,"title":"v1.0","description":"Issues that need to be resolved for v1.0 of Kubernetes.","creator":{"login":"goltermann","id":9358478,"avatar_url":"https://avatars.githubusercontent.com/u/9358478?v=3","gravatar_id":"","url":"https://api.github.com/users/goltermann","html_url":"https://github.com/goltermann","followers_url":"https://api.github.com/users/goltermann/followers","following_url":"https://api.github.com/users/goltermann/following{/other_user}","gists_url":"https://api.github.com/users/goltermann/gists{/gist_id}","starred_url":"https://api.github.com/users/goltermann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/goltermann/subscriptions","organizations_url":"https://api.github.com/users/goltermann/orgs","repos_url":"https://api.github.com/users/goltermann/repos","events_url":"https://api.github.com/users/goltermann/events{/privacy}","received_events_url":"https://api.github.com/users/goltermann/received_events","type":"User","site_admin":false},"open_issues":47,"closed_issues":688,"state":"open","created_at":"2015-02-06T20:27:16Z","updated_at":"2015-07-11T00:37:39Z","due_on":null,"closed_at":null},"comments":25,"created_at":"2015-07-08T00:30:29Z","updated_at":"2015-07-11T00:46:55Z","closed_at":null,"body":"PD tests fail with `Pod cannot be started due to exceeded capacity` in continuous soak-test\r\n\r\ne.g. jenkins: /job/kubernetes-soak-continuous-e2e/1845/\r\n\r\n```\r\nPod Disks should schedule a pod w/ a RW PD, remove it, then schedule it on another host\r\n\r\n/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/pd.go:113\r\nExpected error:\r\n    <*errors.errorString | 0xc2086d4620>: {\r\n        s: \"Giving up; pod went into failed status: \\n(*api.Pod){TypeMeta:(api.TypeMeta){Kind:(string) APIVersion:(string)} ObjectMeta:(api.ObjectMeta){Name:(string)pd-test-ab871a3d-2483-11e5-b9d9-42010af01555 GenerateName:(string) Namespace:(string)default SelfLink:(string)/api/v1/namespaces/default/pods/pd-test-ab871a3d-2483-11e5-b9d9-42010af01555 UID:(types.UID)b186db25-2483-11e5-a4e3-42010af01857 ResourceVersion:(string)12732 Generation:(int64)0 CreationTimestamp:(util.Time)2015-07-07 01:39:34 -0700 PDT DeletionTimestamp:(*util.Time)<nil> Labels:(map[string]string)<nil> Annotations:(map[string]string)<nil>} Spec:(api.PodSpec){Volumes:([]api.Volume)[{Name:(string)testpd VolumeSource:(api.VolumeSource){HostPath:(*api.HostPathVolumeSource)<nil> EmptyDir:(*api.EmptyDirVolumeSource)<nil> GCEPersistentDisk:(*api.GCEPersistentDiskVolumeSource){PDName:(string)e2e-soak-a72419da-2483-11e5-b9d9-42010af01555 FSType:(string)ext4 Partition:(int)0 ReadOnly:(bool)false} AWSElasticBlockStore:(*api.AWSElasticBlockStoreVolumeSource)<nil> GitRepo:(*api.GitRepoVolumeSource)<nil> Secret:(*api.SecretVolumeSource)<nil> NFS:(*api.NFSVolumeSource)<nil> ISCSI:(*api.ISCSIVolumeSource)<nil> Glusterfs:(*api.GlusterfsVolumeSource)<nil> PersistentVolumeClaim:(*api.PersistentVolumeClaimVolumeSource)<nil> RBD:(*api.RBDVolumeSource)<nil>}} {Name:(string)default-token-812kd VolumeSource:(api.VolumeSource){HostPath:(*api.HostPathVolumeSource)<nil> EmptyDir:(*api.EmptyDirVolumeSource)<nil> GCEPersistentDisk:(*api.GCEPersistentDiskVolumeSource)<nil> AWSElasticBlockStore:(*api.AWSElasticBlockStoreVolumeSource)<nil> GitRepo:(*api.GitRepoVolumeSource)<nil> Secret:(*api.SecretVolumeSource){SecretName:(string)default-token-812kd} NFS:(*api.NFSVolumeSource)<nil> ISCSI:(*api.ISCSIVolumeSource)<nil> Glusterfs:(*api.GlusterfsVolumeSource)<nil> PersistentVolumeClaim:(*api.PersistentVolumeClaimVolumeSource)<nil> RBD:(*api.RBDVolumeSource)<nil>}}] Containers:([]api.Container)[{Name:(string)testpd Image:(string)gcr.io/google_containers/pause Command:([]string)<nil> Args:([]string)<nil> WorkingDir:(string) Ports:([]api.ContainerPort)<nil> Env:([]api.EnvVar)<nil> Resources:(api.ResourceRequirements){Limits:(api.ResourceList)map[cpu:100m] Requests:(api.ResourceList)<nil>} VolumeMounts:([]api.VolumeMount)[{Name:(string)testpd ReadOnly:(bool)false MountPath:(string)/testpd} {Name:(string)default-token-812kd ReadOnly:(bool)true MountPath:(string)/var/run/secrets/kubernetes.io/serviceaccount}] LivenessProbe:(*api.Probe)<nil> ReadinessProbe:(*api.Probe)<nil> Lifecycle:(*api.Lifecycle)<nil> TerminationMessagePath:(string)/dev/termination-log ImagePullPolicy:(api.PullPolicy)IfNotPresent SecurityContext:(*api.SecurityContext)<nil>}] RestartPolicy:(api.RestartPolicy)Always TerminationGracePeriodSeconds:(*int64)<nil> ActiveDeadlineSeconds:(*int64)<nil> DNSPolicy:(api.DNSPolicy)ClusterFirst NodeSelector:(map[string]string)<nil> ServiceAccountName:(string)default NodeName:(string)e2e-soak-minion-urqn HostNetwork:(bool)false ImagePullSecrets:([]api.LocalObjectReference)<nil>} Status:(api.PodStatus){Phase:(api.PodPhase)Failed Conditions:([]api.PodCondition)<nil> Message:(string)Pod cannot be started due to exceeded capacity Reason:(string)CapacityExceeded HostIP:(string) PodIP:(string) StartTime:(*util.Time)2015-07-07 01:39:34 -0700 PDT ContainerStatuses:([]api.ContainerStatus)<nil>}}\",\r\n    }\r\n    Giving up; pod went into failed status: \r\n    (*api.Pod){TypeMeta:(api.TypeMeta){Kind:(string) APIVersion:(string)} ObjectMeta:(api.ObjectMeta){Name:(string)pd-test-ab871a3d-2483-11e5-b9d9-42010af01555 GenerateName:(string) Namespace:(string)default SelfLink:(string)/api/v1/namespaces/default/pods/pd-test-ab871a3d-2483-11e5-b9d9-42010af01555 UID:(types.UID)b186db25-2483-11e5-a4e3-42010af01857 ResourceVersion:(string)12732 Generation:(int64)0 CreationTimestamp:(util.Time)2015-07-07 01:39:34 -0700 PDT DeletionTimestamp:(*util.Time)<nil> Labels:(map[string]string)<nil> Annotations:(map[string]string)<nil>} Spec:(api.PodSpec){Volumes:([]api.Volume)[{Name:(string)testpd VolumeSource:(api.VolumeSource){HostPath:(*api.HostPathVolumeSource)<nil> EmptyDir:(*api.EmptyDirVolumeSource)<nil> GCEPersistentDisk:(*api.GCEPersistentDiskVolumeSource){PDName:(string)e2e-soak-a72419da-2483-11e5-b9d9-42010af01555 FSType:(string)ext4 Partition:(int)0 ReadOnly:(bool)false} AWSElasticBlockStore:(*api.AWSElasticBlockStoreVolumeSource)<nil> GitRepo:(*api.GitRepoVolumeSource)<nil> Secret:(*api.SecretVolumeSource)<nil> NFS:(*api.NFSVolumeSource)<nil> ISCSI:(*api.ISCSIVolumeSource)<nil> Glusterfs:(*api.GlusterfsVolumeSource)<nil> PersistentVolumeClaim:(*api.PersistentVolumeClaimVolumeSource)<nil> RBD:(*api.RBDVolumeSource)<nil>}} {Name:(string)default-token-812kd VolumeSource:(api.VolumeSource){HostPath:(*api.HostPathVolumeSource)<nil> EmptyDir:(*api.EmptyDirVolumeSource)<nil> GCEPersistentDisk:(*api.GCEPersistentDiskVolumeSource)<nil> AWSElasticBlockStore:(*api.AWSElasticBlockStoreVolumeSource)<nil> GitRepo:(*api.GitRepoVolumeSource)<nil> Secret:(*api.SecretVolumeSource){SecretName:(string)default-token-812kd} NFS:(*api.NFSVolumeSource)<nil> ISCSI:(*api.ISCSIVolumeSource)<nil> Glusterfs:(*api.GlusterfsVolumeSource)<nil> PersistentVolumeClaim:(*api.PersistentVolumeClaimVolumeSource)<nil> RBD:(*api.RBDVolumeSource)<nil>}}] Containers:([]api.Container)[{Name:(string)testpd Image:(string)gcr.io/google_containers/pause Command:([]string)<nil> Args:([]string)<nil> WorkingDir:(string) Ports:([]api.ContainerPort)<nil> Env:([]api.EnvVar)<nil> Resources:(api.ResourceRequirements){Limits:(api.ResourceList)map[cpu:100m] Requests:(api.ResourceList)<nil>} VolumeMounts:([]api.VolumeMount)[{Name:(string)testpd ReadOnly:(bool)false MountPath:(string)/testpd} {Name:(string)default-token-812kd ReadOnly:(bool)true MountPath:(string)/var/run/secrets/kubernetes.io/serviceaccount}] LivenessProbe:(*api.Probe)<nil> ReadinessProbe:(*api.Probe)<nil> Lifecycle:(*api.Lifecycle)<nil> TerminationMessagePath:(string)/dev/termination-log ImagePullPolicy:(api.PullPolicy)IfNotPresent SecurityContext:(*api.SecurityContext)<nil>}] RestartPolicy:(api.RestartPolicy)Always TerminationGracePeriodSeconds:(*int64)<nil> ActiveDeadlineSeconds:(*int64)<nil> DNSPolicy:(api.DNSPolicy)ClusterFirst NodeSelector:(map[string]string)<nil> ServiceAccountName:(string)default NodeName:(string)e2e-soak-minion-urqn HostNetwork:(bool)false ImagePullSecrets:([]api.LocalObjectReference)<nil>} Status:(api.PodStatus){Phase:(api.PodPhase)Failed Conditions:([]api.PodCondition)<nil> Message:(string)Pod cannot be started due to exceeded capacity Reason:(string)CapacityExceeded HostIP:(string) PodIP:(string) StartTime:(*util.Time)2015-07-07 01:39:34 -0700 PDT ContainerStatuses:([]api.ContainerStatus)<nil>}}\r\nnot to have occurred\r\n\r\n\r\nPod Disks should schedule a pod w/ a readonly PD on two hosts, then remove both.\r\n\r\n/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/pd.go:172\r\nExpected error:\r\n    <*errors.errorString | 0xc209024410>: {\r\n        s: \"Giving up; pod went into failed status: \\n(*api.Pod){TypeMeta:(api.TypeMeta){Kind:(string) APIVersion:(string)} ObjectMeta:(api.ObjectMeta){Name:(string)pd-test-4e63f3d7-2487-11e5-b9d9-42010af01555 GenerateName:(string) Namespace:(string)default SelfLink:(string)/api/v1/namespaces/default/pods/pd-test-4e63f3d7-2487-11e5-b9d9-42010af01555 UID:(types.UID)60556283-2487-11e5-a4e3-42010af01857 ResourceVersion:(string)15075 Generation:(int64)0 CreationTimestamp:(util.Time)2015-07-07 02:05:56 -0700 PDT DeletionTimestamp:(*util.Time)<nil> Labels:(map[string]string)<nil> Annotations:(map[string]string)<nil>} Spec:(api.PodSpec){Volumes:([]api.Volume)[{Name:(string)testpd VolumeSource:(api.VolumeSource){HostPath:(*api.HostPathVolumeSource)<nil> EmptyDir:(*api.EmptyDirVolumeSource)<nil> GCEPersistentDisk:(*api.GCEPersistentDiskVolumeSource){PDName:(string)e2e-soak-486ab5a2-2487-11e5-b9d9-42010af01555 FSType:(string)ext4 Partition:(int)0 ReadOnly:(bool)true} AWSElasticBlockStore:(*api.AWSElasticBlockStoreVolumeSource)<nil> GitRepo:(*api.GitRepoVolumeSource)<nil> Secret:(*api.SecretVolumeSource)<nil> NFS:(*api.NFSVolumeSource)<nil> ISCSI:(*api.ISCSIVolumeSource)<nil> Glusterfs:(*api.GlusterfsVolumeSource)<nil> PersistentVolumeClaim:(*api.PersistentVolumeClaimVolumeSource)<nil> RBD:(*api.RBDVolumeSource)<nil>}} {Name:(string)default-token-812kd VolumeSource:(api.VolumeSource){HostPath:(*api.HostPathVolumeSource)<nil> EmptyDir:(*api.EmptyDirVolumeSource)<nil> GCEPersistentDisk:(*api.GCEPersistentDiskVolumeSource)<nil> AWSElasticBlockStore:(*api.AWSElasticBlockStoreVolumeSource)<nil> GitRepo:(*api.GitRepoVolumeSource)<nil> Secret:(*api.SecretVolumeSource){SecretName:(string)default-token-812kd} NFS:(*api.NFSVolumeSource)<nil> ISCSI:(*api.ISCSIVolumeSource)<nil> Glusterfs:(*api.GlusterfsVolumeSource)<nil> PersistentVolumeClaim:(*api.PersistentVolumeClaimVolumeSource)<nil> RBD:(*api.RBDVolumeSource)<nil>}}] Containers:([]api.Container)[{Name:(string)testpd Image:(string)gcr.io/google_containers/pause Command:([]string)<nil> Args:([]string)<nil> WorkingDir:(string) Ports:([]api.ContainerPort)<nil> Env:([]api.EnvVar)<nil> Resources:(api.ResourceRequirements){Limits:(api.ResourceList)map[cpu:100m] Requests:(api.ResourceList)<nil>} VolumeMounts:([]api.VolumeMount)[{Name:(string)testpd ReadOnly:(bool)false MountPath:(string)/testpd} {Name:(string)default-token-812kd ReadOnly:(bool)true MountPath:(string)/var/run/secrets/kubernetes.io/serviceaccount}] LivenessProbe:(*api.Probe)<nil> ReadinessProbe:(*api.Probe)<nil> Lifecycle:(*api.Lifecycle)<nil> TerminationMessagePath:(string)/dev/termination-log ImagePullPolicy:(api.PullPolicy)IfNotPresent SecurityContext:(*api.SecurityContext)<nil>}] RestartPolicy:(api.RestartPolicy)Always TerminationGracePeriodSeconds:(*int64)<nil> ActiveDeadlineSeconds:(*int64)<nil> DNSPolicy:(api.DNSPolicy)ClusterFirst NodeSelector:(map[string]string)<nil> ServiceAccountName:(string)default NodeName:(string)e2e-soak-minion-urqn HostNetwork:(bool)false ImagePullSecrets:([]api.LocalObjectReference)<nil>} Status:(api.PodStatus){Phase:(api.PodPhase)Failed Conditions:([]api.PodCondition)<nil> Message:(string)Pod cannot be started due to exceeded capacity Reason:(string)CapacityExceeded HostIP:(string) PodIP:(string) StartTime:(*util.Time)2015-07-07 02:05:56 -0700 PDT ContainerStatuses:([]api.ContainerStatus)<nil>}}\",\r\n    }\r\n    Giving up; pod went into failed status: \r\n    (*api.Pod){TypeMeta:(api.TypeMeta){Kind:(string) APIVersion:(string)} ObjectMeta:(api.ObjectMeta){Name:(string)pd-test-4e63f3d7-2487-11e5-b9d9-42010af01555 GenerateName:(string) Namespace:(string)default SelfLink:(string)/api/v1/namespaces/default/pods/pd-test-4e63f3d7-2487-11e5-b9d9-42010af01555 UID:(types.UID)60556283-2487-11e5-a4e3-42010af01857 ResourceVersion:(string)15075 Generation:(int64)0 CreationTimestamp:(util.Time)2015-07-07 02:05:56 -0700 PDT DeletionTimestamp:(*util.Time)<nil> Labels:(map[string]string)<nil> Annotations:(map[string]string)<nil>} Spec:(api.PodSpec){Volumes:([]api.Volume)[{Name:(string)testpd VolumeSource:(api.VolumeSource){HostPath:(*api.HostPathVolumeSource)<nil> EmptyDir:(*api.EmptyDirVolumeSource)<nil> GCEPersistentDisk:(*api.GCEPersistentDiskVolumeSource){PDName:(string)e2e-soak-486ab5a2-2487-11e5-b9d9-42010af01555 FSType:(string)ext4 Partition:(int)0 ReadOnly:(bool)true} AWSElasticBlockStore:(*api.AWSElasticBlockStoreVolumeSource)<nil> GitRepo:(*api.GitRepoVolumeSource)<nil> Secret:(*api.SecretVolumeSource)<nil> NFS:(*api.NFSVolumeSource)<nil> ISCSI:(*api.ISCSIVolumeSource)<nil> Glusterfs:(*api.GlusterfsVolumeSource)<nil> PersistentVolumeClaim:(*api.PersistentVolumeClaimVolumeSource)<nil> RBD:(*api.RBDVolumeSource)<nil>}} {Name:(string)default-token-812kd VolumeSource:(api.VolumeSource){HostPath:(*api.HostPathVolumeSource)<nil> EmptyDir:(*api.EmptyDirVolumeSource)<nil> GCEPersistentDisk:(*api.GCEPersistentDiskVolumeSource)<nil> AWSElasticBlockStore:(*api.AWSElasticBlockStoreVolumeSource)<nil> GitRepo:(*api.GitRepoVolumeSource)<nil> Secret:(*api.SecretVolumeSource){SecretName:(string)default-token-812kd} NFS:(*api.NFSVolumeSource)<nil> ISCSI:(*api.ISCSIVolumeSource)<nil> Glusterfs:(*api.GlusterfsVolumeSource)<nil> PersistentVolumeClaim:(*api.PersistentVolumeClaimVolumeSource)<nil> RBD:(*api.RBDVolumeSource)<nil>}}] Containers:([]api.Container)[{Name:(string)testpd Image:(string)gcr.io/google_containers/pause Command:([]string)<nil> Args:([]string)<nil> WorkingDir:(string) Ports:([]api.ContainerPort)<nil> Env:([]api.EnvVar)<nil> Resources:(api.ResourceRequirements){Limits:(api.ResourceList)map[cpu:100m] Requests:(api.ResourceList)<nil>} VolumeMounts:([]api.VolumeMount)[{Name:(string)testpd ReadOnly:(bool)false MountPath:(string)/testpd} {Name:(string)default-token-812kd ReadOnly:(bool)true MountPath:(string)/var/run/secrets/kubernetes.io/serviceaccount}] LivenessProbe:(*api.Probe)<nil> ReadinessProbe:(*api.Probe)<nil> Lifecycle:(*api.Lifecycle)<nil> TerminationMessagePath:(string)/dev/termination-log ImagePullPolicy:(api.PullPolicy)IfNotPresent SecurityContext:(*api.SecurityContext)<nil>}] RestartPolicy:(api.RestartPolicy)Always TerminationGracePeriodSeconds:(*int64)<nil> ActiveDeadlineSeconds:(*int64)<nil> DNSPolicy:(api.DNSPolicy)ClusterFirst NodeSelector:(map[string]string)<nil> ServiceAccountName:(string)default NodeName:(string)e2e-soak-minion-urqn HostNetwork:(bool)false ImagePullSecrets:([]api.LocalObjectReference)<nil>} Status:(api.PodStatus){Phase:(api.PodPhase)Failed Conditions:([]api.PodCondition)<nil> Message:(string)Pod cannot be started due to exceeded capacity Reason:(string)CapacityExceeded HostIP:(string) PodIP:(string) StartTime:(*util.Time)2015-07-07 02:05:56 -0700 PDT ContainerStatuses:([]api.ContainerStatus)<nil>}}\r\nnot to have occurred\r\n\r\n```"},"comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/comments/120556948","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/10899#issuecomment-120556948","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/10899","id":120556948,"user":{"login":"saad-ali","id":10052848,"avatar_url":"https://avatars.githubusercontent.com/u/10052848?v=3","gravatar_id":"","url":"https://api.github.com/users/saad-ali","html_url":"https://github.com/saad-ali","followers_url":"https://api.github.com/users/saad-ali/followers","following_url":"https://api.github.com/users/saad-ali/following{/other_user}","gists_url":"https://api.github.com/users/saad-ali/gists{/gist_id}","starred_url":"https://api.github.com/users/saad-ali/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saad-ali/subscriptions","organizations_url":"https://api.github.com/users/saad-ali/orgs","repos_url":"https://api.github.com/users/saad-ali/repos","events_url":"https://api.github.com/users/saad-ali/events{/privacy}","received_events_url":"https://api.github.com/users/saad-ali/received_events","type":"User","site_admin":false},"created_at":"2015-07-11T00:46:55Z","updated_at":"2015-07-11T00:46:55Z","body":"Ok, so this all makes sense now.\r\n\r\nThere is a `Network when a minion node becomes unreachable` test that runs before this PD test. It forces one node to go offline, which causes all the system pods to get rescheduled to the other node. And then the PD test tries to schedule its pod on the over-committed node and freaks out when it is unable to.\r\n\r\nSo changing the default namespace for the PD test pods should fix this. But, a better fix might be to somehow trigger a rebalance when starting the PD test. @bprashanth suggested maybe deleting and  existing RCs.\r\n\r\ntldr: expected behavior/test issue\r\n\r\n```\r\n\u001B[0mNodes\u001B[0m \u001B[90mNetwork\u001B[0m \u001B[0mwhen a minion node becomes unreachable\u001B[0m \r\n  \u001B[1m[replication controller] recreates pods scheduled on the unreachable minion node AND allows scheduling of pods on a minion after it rejoins the cluster\u001B[0m\r\n  \u001B[37m/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/resize_nodes.go:557\u001B[0m\r\n[BeforeEach] Nodes\r\n  /go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/resize_nodes.go:412\r\n>>> testContext.KubeConfig: /var/lib/jenkins/jobs/kubernetes-soak-weekly-deploy/workspace/.kube/config\r\nINFO: Waiting up to 2m0s for service account default to be provisioned in ns e2e-tests-resize-nodes-ou7zq\r\nINFO: Get service account default in ns e2e-tests-resize-nodes-ou7zq failed, ignoring for 5s: serviceaccounts \"default\" not found\r\nINFO: Service account default in ns e2e-tests-resize-nodes-ou7zq with secrets found. (5.015614174s)\r\n[BeforeEach] when a minion node becomes unreachable\r\n  /go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/resize_nodes.go:506\r\n[It] [replication controller] recreates pods scheduled on the unreachable minion node AND allows scheduling of pods on a minion after it rejoins the cluster\r\n  /go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/resize_nodes.go:557\r\n\u001B[1mSTEP\u001B[0m: creating replication controller my-hostname-net\r\nINFO: Pod name my-hostname-net: Found 0 pods out of 2\r\nINFO: Pod name my-hostname-net: Found 2 pods out of 2\r\n\u001B[1mSTEP\u001B[0m: ensuring each pod is running\r\nINFO: Waiting up to 5m0s for pod my-hostname-net-m4bja                                   status to be running\r\nINFO: Found pod 'my-hostname-net-m4bja' on node 'soak-gce-weekly-minion-0u8y'\r\nINFO: Waiting up to 5m0s for pod my-hostname-net-pelkk                                   status to be running\r\nINFO: Found pod 'my-hostname-net-pelkk' on node 'soak-gce-weekly-minion-otq4'\r\n\u001B[1mSTEP\u001B[0m: trying to dial each unique pod\r\nINFO: Controller my-hostname-net: Got expected result from replica 1 [my-hostname-net-m4bja]: \"my-hostname-net-m4bja\", 1 of 2 required successes so far\r\nINFO: Controller my-hostname-net: Got expected result from replica 2 [my-hostname-net-pelkk]: \"my-hostname-net-pelkk\", 2 of 2 required successes so far\r\n\u001B[1mSTEP\u001B[0m: choose a node with at least one pod - we will block some network traffic on this node\r\n\u001B[1mSTEP\u001B[0m: block network traffic from node soak-gce-weekly-minion-0u8y\r\nINFO: Getting external IP address for soak-gce-weekly-minion-0u8y\r\n\u001B[1mSTEP\u001B[0m: block network traffic from node soak-gce-weekly-minion-0u8y to the master\r\nINFO: Using SSH key: /var/lib/jenkins/jobs/kubernetes-soak-weekly-deploy/workspace/.ssh/google_compute_engine\r\nINFO: Waiting for node soak-gce-weekly-minion-0u8y to be not ready\r\nINFO: Waiting up to 2m0s for node soak-gce-weekly-minion-0u8y readiness to be false\r\nINFO: Node soak-gce-weekly-minion-0u8y condition 1/1: type: Ready, status: True\r\nINFO: Node soak-gce-weekly-minion-0u8y condition 1/1: type: Ready, status: True\r\nINFO: Node soak-gce-weekly-minion-0u8y condition 1/1: type: Ready, status: True\r\nINFO: Node soak-gce-weekly-minion-0u8y condition 1/1: type: Ready, status: True\r\nINFO: Node soak-gce-weekly-minion-0u8y condition 1/1: type: Ready, status: True\r\nINFO: Node soak-gce-weekly-minion-0u8y condition 1/1: type: Ready, status: True\r\nINFO: Node soak-gce-weekly-minion-0u8y condition 1/1: type: Ready, status: True\r\nINFO: Node soak-gce-weekly-minion-0u8y condition 1/1: type: Ready, status: Unknown\r\nINFO: Successfully found node soak-gce-weekly-minion-0u8y readiness to be false\r\nINFO: Waiting for pod my-hostname-net-m4bja to be removed\r\nINFO: Waiting for pod my-hostname-net-m4bja to disappear\r\nINFO: Pod my-hostname-net-m4bja still exists\r\nINFO: Waiting for pod my-hostname-net-m4bja to disappear\r\nINFO: Pod my-hostname-net-m4bja still exists\r\nINFO: Waiting for pod my-hostname-net-m4bja to disappear\r\nINFO: Pod my-hostname-net-m4bja still exists\r\nINFO: Waiting for pod my-hostname-net-m4bja to disappear\r\nINFO: Pod my-hostname-net-m4bja still exists\r\nINFO: Waiting for pod my-hostname-net-m4bja to disappear\r\nINFO: Pod my-hostname-net-m4bja still exists\r\nINFO: Waiting for pod my-hostname-net-m4bja to disappear\r\nINFO: Pod my-hostname-net-m4bja still exists\r\nINFO: Waiting for pod my-hostname-net-m4bja to disappear\r\nINFO: Pod my-hostname-net-m4bja still exists\r\nINFO: Waiting for pod my-hostname-net-m4bja to disappear\r\nINFO: Pod my-hostname-net-m4bja still exists\r\nINFO: Waiting for pod my-hostname-net-m4bja to disappear\r\nINFO: Pod my-hostname-net-m4bja still exists\r\nINFO: Waiting for pod my-hostname-net-m4bja to disappear\r\nINFO: Pod my-hostname-net-m4bja still exists\r\nINFO: Waiting for pod my-hostname-net-m4bja to disappear\r\nINFO: Pod my-hostname-net-m4bja no longer exists\r\n\u001B[1mSTEP\u001B[0m: verifying whether the pod from the unreachable node is recreated\r\nINFO: Pod name my-hostname-net: Found 2 pods out of 2\r\n\u001B[1mSTEP\u001B[0m: ensuring each pod is running\r\nINFO: Waiting up to 5m0s for pod my-hostname-net-aaqma                                   status to be running\r\nINFO: Waiting for pod my-hostname-net-aaqma                                   in namespace 'e2e-tests-resize-nodes-ou7zq' status to be 'running'(found phase: \"Pending\", readiness: false) (3.928301ms elapsed)\r\nINFO: Found pod 'my-hostname-net-aaqma' on node 'soak-gce-weekly-minion-otq4'\r\nINFO: Waiting up to 5m0s for pod my-hostname-net-pelkk                                   status to be running\r\nINFO: Found pod 'my-hostname-net-pelkk' on node 'soak-gce-weekly-minion-otq4'\r\n\u001B[1mSTEP\u001B[0m: trying to dial each unique pod\r\nINFO: Controller my-hostname-net: Got expected result from replica 1 [my-hostname-net-aaqma]: \"my-hostname-net-aaqma\", 1 of 2 required successes so far\r\nINFO: Controller my-hostname-net: Got expected result from replica 2 [my-hostname-net-pelkk]: \"my-hostname-net-pelkk\", 2 of 2 required successes so far\r\n\u001B[1mSTEP\u001B[0m: Unblock network traffic from node soak-gce-weekly-minion-0u8y to the master\r\nINFO: Using SSH key: /var/lib/jenkins/jobs/kubernetes-soak-weekly-deploy/workspace/.ssh/google_compute_engine\r\nINFO: Waiting for node soak-gce-weekly-minion-0u8y to be ready\r\nINFO: Waiting up to 2m0s for node soak-gce-weekly-minion-0u8y readiness to be true\r\nINFO: Node soak-gce-weekly-minion-0u8y condition 1/1: type: Ready, status: Unknown\r\nINFO: Node soak-gce-weekly-minion-0u8y condition 1/1: type: Ready, status: Unknown\r\nINFO: Node soak-gce-weekly-minion-0u8y condition 1/1: type: Ready, status: Unknown\r\nINFO: Node soak-gce-weekly-minion-0u8y condition 1/1: type: Ready, status: Unknown\r\nINFO: Node soak-gce-weekly-minion-0u8y condition 1/1: type: Ready, status: Unknown\r\nINFO: Node soak-gce-weekly-minion-0u8y condition 1/1: type: Ready, status: Unknown\r\nINFO: Node soak-gce-weekly-minion-0u8y condition 1/1: type: Ready, status: Unknown\r\nINFO: Node soak-gce-weekly-minion-0u8y condition 1/1: type: Ready, status: Unknown\r\nINFO: Node soak-gce-weekly-minion-0u8y condition 1/1: type: Ready, status: Unknown\r\nINFO: Node soak-gce-weekly-minion-0u8y condition 1/1: type: Ready, status: Unknown\r\nINFO: Node soak-gce-weekly-minion-0u8y condition 1/1: type: Ready, status: Unknown\r\nINFO: Node soak-gce-weekly-minion-0u8y condition 1/1: type: Ready, status: Unknown\r\nINFO: Node soak-gce-weekly-minion-0u8y condition 1/1: type: Ready, status: Unknown\r\nINFO: Node soak-gce-weekly-minion-0u8y condition 1/1: type: Ready, status: Unknown\r\nINFO: Node soak-gce-weekly-minion-0u8y condition 1/1: type: Ready, status: Unknown\r\nINFO: Node soak-gce-weekly-minion-0u8y condition 1/1: type: Ready, status: Unknown\r\nINFO: Node soak-gce-weekly-minion-0u8y condition 1/1: type: Ready, status: True\r\nINFO: Successfully found node soak-gce-weekly-minion-0u8y readiness to be true\r\n\u001B[1mSTEP\u001B[0m: verify whether new pods can be created on the re-attached node\r\nINFO: Created pod additionalpod on node soak-gce-weekly-minion-0u8y\r\nINFO: Pod name additionalpod: Found 1 pods out of 1\r\n\u001B[1mSTEP\u001B[0m: ensuring each pod is running\r\nINFO: Waiting up to 5m0s for pod additionalpod                                           status to be running\r\nINFO: Waiting for pod additionalpod                                           in namespace 'e2e-tests-resize-nodes-ou7zq' status to be 'running'(found phase: \"Pending\", readiness: false) (20.470499ms elapsed)\r\nINFO: Found pod 'additionalpod' on node 'soak-gce-weekly-minion-0u8y'\r\n\u001B[1mSTEP\u001B[0m: trying to dial each unique pod\r\nINFO: Controller additionalpod: Got expected result from replica 1 [additionalpod]: \"additionalpod\", 1 of 1 required successes so far\r\n[AfterEach] Nodes\r\n  /go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/resize_nodes.go:423\r\n\u001B[1mSTEP\u001B[0m: checking whether all nodes are healthy\r\nINFO: Node soak-gce-weekly-minion-0u8y condition 1/1: type: Ready, status: True\r\nINFO: Node soak-gce-weekly-minion-otq4 condition 1/1: type: Ready, status: True\r\n\u001B[1mSTEP\u001B[0m: destroying namespace for this suite e2e-tests-resize-nodes-ou7zq\r\n\r\n\u001B[32m• [SLOW TEST:376.324 seconds]\u001B[0m\r\nNodes\r\n\u001B[90m/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/resize_nodes.go:560\u001B[0m\r\n  Network\r\n  \u001B[90m/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/resize_nodes.go:559\u001B[0m\r\n    when a minion node becomes unreachable\r\n    \u001B[90m/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/resize_nodes.go:558\u001B[0m\r\n      [replication controller] recreates pods scheduled on the unreachable minion node AND allows scheduling of pods on a minion after it rejoins the cluster\r\n      \u001B[90m/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/resize_nodes.go:557\u001B[0m\r\n\r\n```"}}