{"action":"created","issue":{"url":"https://api.github.com/repos/openshift/origin/issues/6318","labels_url":"https://api.github.com/repos/openshift/origin/issues/6318/labels{/name}","comments_url":"https://api.github.com/repos/openshift/origin/issues/6318/comments","events_url":"https://api.github.com/repos/openshift/origin/issues/6318/events","html_url":"https://github.com/openshift/origin/issues/6318","id":122234022,"number":6318,"title":"[cluster metrics] failed to pull the origin-metrics-cassandra image during configure the metrics deployer ","user":{"login":"thincal","id":1206509,"avatar_url":"https://avatars.githubusercontent.com/u/1206509?v=3","gravatar_id":"","url":"https://api.github.com/users/thincal","html_url":"https://github.com/thincal","followers_url":"https://api.github.com/users/thincal/followers","following_url":"https://api.github.com/users/thincal/following{/other_user}","gists_url":"https://api.github.com/users/thincal/gists{/gist_id}","starred_url":"https://api.github.com/users/thincal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thincal/subscriptions","organizations_url":"https://api.github.com/users/thincal/orgs","repos_url":"https://api.github.com/users/thincal/repos","events_url":"https://api.github.com/users/thincal/events{/privacy}","received_events_url":"https://api.github.com/users/thincal/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/openshift/origin/labels/component/imageregistry","name":"component/imageregistry","color":"0052cc"},{"url":"https://api.github.com/repos/openshift/origin/labels/component/metrics","name":"component/metrics","color":"0052cc"},{"url":"https://api.github.com/repos/openshift/origin/labels/priority/P2","name":"priority/P2","color":"fbca04"}],"state":"open","locked":false,"assignee":{"login":"soltysh","id":576341,"avatar_url":"https://avatars.githubusercontent.com/u/576341?v=3","gravatar_id":"","url":"https://api.github.com/users/soltysh","html_url":"https://github.com/soltysh","followers_url":"https://api.github.com/users/soltysh/followers","following_url":"https://api.github.com/users/soltysh/following{/other_user}","gists_url":"https://api.github.com/users/soltysh/gists{/gist_id}","starred_url":"https://api.github.com/users/soltysh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/soltysh/subscriptions","organizations_url":"https://api.github.com/users/soltysh/orgs","repos_url":"https://api.github.com/users/soltysh/repos","events_url":"https://api.github.com/users/soltysh/events{/privacy}","received_events_url":"https://api.github.com/users/soltysh/received_events","type":"User","site_admin":false},"milestone":null,"comments":15,"created_at":"2015-12-15T09:54:53Z","updated_at":"2015-12-21T14:07:03Z","closed_at":null,"body":"I am following the steps from [metrics deployer] (https://docs.openshift.org/latest/install_config/cluster_metrics.html#metrics-deployer), the heapster and hawkular-metrics can be successfully pulled from integrated registry, but origin-metrics-cassandra image pulling failed.\r\n\r\nThere are some additional information:\r\n\r\n- image prefix assigned in ```metrics.yaml``` as bellow\r\n\r\n```\r\n  name: IMAGE_PREFIX\r\n  value: \"172.30.117.244:5000/openshift/origin-\"\r\n```\r\n\r\n- required image streams are already prepared\r\n\r\n```\r\n$ oc get is\r\nNAME                              DOCKER REPO                                                     TAGS                         UPDATED\r\norigin-metrics-cassandra          172.30.117.244:5000/openshift/origin-metrics-cassandra          latest                       22 minutes ago\r\norigin-metrics-deployer           172.30.117.244:5000/openshift/origin-metrics-deployer           latest                       25 hours ago\r\norigin-metrics-hawkular-metrics   172.30.117.244:5000/openshift/origin-metrics-hawkular-metrics   latest                       23 hours ago\r\norigin-metrics-heapster           172.30.117.244:5000/openshift/origin-metrics-heapster           latest                       24 hours ago\r\n```\r\n\r\n- deploy the metrics components\r\n\r\n```\r\noc process -f metrics.yaml -v \\\r\n   HAWKULAR_METRICS_HOSTNAME=metrics.example.com oc create -f -\r\n```\r\n\r\n- after a while the openshift event reports bellow failure\r\n```\r\nFailed to pull image \"172.30.117.244:5000/openshift/origin-metrics-cassandra:latest\": image pull failed for 172.30.117.244:5000/openshift/origin-metrics-cassandra:latest, this may be because there are no credentials on this request. details: (Error: image openshift/origin-metrics-cassandra:latest not found)\r\n```\r\nAfter inspect the info about the ```hawkular-cassandra``` pod, ```imagePullSecrets```is not found, but the ```heapster```pod does have.\r\n\r\nSo I think the missed ```imagePullSecrets``` might cause the pull failure, could you help with it ? many thanks.\r\n\r\n```hawkular-cassandra``` pod's info:\r\n```\r\n$ oc get pod hawkular-cassandra-1-sfn6k -o yaml\r\napiVersion: v1\r\nkind: Pod\r\nmetadata:\r\n  annotations:\r\n    kubernetes.io/created-by: |\r\n      {\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicationController\",\"namespace\":\"openshift-infra\",\"name\":\"hawkular-cassandra-1\",\"uid\":\"f2134aba-a2fc-11e5-a1e2-8cdcd44e7386\",\"apiVersion\":\"v1\",\"resourceVersion\":\"195345\"}}\r\n    openshift.io/scc: restricted\r\n  creationTimestamp: 2015-12-15T07:24:58Z\r\n  generateName: hawkular-cassandra-1-\r\n  labels:\r\n    metrics-infra: hawkular-cassandra\r\n    name: hawkular-cassandra-1\r\n    type: hawkular-cassandra\r\n  name: hawkular-cassandra-1-sfn6k\r\n  namespace: openshift-infra\r\n  resourceVersion: \"199465\"\r\n  selfLink: /api/v1/namespaces/openshift-infra/pods/hawkular-cassandra-1-sfn6k\r\n  uid: f215e656-a2fc-11e5-a1e2-8cdcd44e7386\r\nspec:\r\n  containers:\r\n  - command:\r\n    - /opt/apache-cassandra/bin/cassandra-docker.sh\r\n    - --seed_provider_classname=org.hawkular.openshift.cassandra.OpenshiftSeedProvider\r\n    - --cluster_name=hawkular-metrics\r\n    - --data_volume=/cassandra_data\r\n    - --internode_encryption=all\r\n    - --require_node_auth=true\r\n    - --enable_client_encryption=true\r\n    - --require_client_auth=true\r\n    - --keystore_file=/secret/cassandra.keystore\r\n    - --keystore_password_file=/secret/cassandra.keystore.password\r\n    - --truststore_file=/secret/cassandra.truststore\r\n    - --truststore_password_file=/secret/cassandra.truststore.password\r\n    - --cassandra_pem_file=/secret/cassandra.pem\r\n    env:\r\n    - name: CASSANDRA_MASTER\r\n      value: \"true\"\r\n    - name: POD_NAMESPACE\r\n      valueFrom:\r\n        fieldRef:\r\n          apiVersion: v1\r\n          fieldPath: metadata.namespace\r\n    image: 172.30.117.244:5000/openshift/origin-metrics-cassandra:latest\r\n    imagePullPolicy: Always\r\n    lifecycle:\r\n      postStart:\r\n        exec:\r\n          command:\r\n          - /opt/apache-cassandra/bin/cassandra-docker-ready.sh\r\n      preStop:\r\n        exec:\r\n          command:\r\n          - /opt/apache-cassandra/bin/cassandra-docker-pre-stop.sh\r\n    name: hawkular-cassandra-1\r\n    ports:\r\n    - containerPort: 9042\r\n      name: cql-port\r\n      protocol: TCP\r\n    - containerPort: 9160\r\n      name: thift-port\r\n      protocol: TCP\r\n    - containerPort: 7000\r\n      name: tcp-port\r\n      protocol: TCP\r\n    - containerPort: 7001\r\n      name: ssl-port\r\n      protocol: TCP\r\n    resources: {}\r\n    securityContext:\r\n      privileged: false\r\n      runAsUser: 1000020000\r\n      seLinuxOptions:\r\n        level: s0:c5,c0\r\n    terminationMessagePath: /dev/termination-log\r\n    volumeMounts:\r\n    - mountPath: /cassandra_data\r\n      name: cassandra-data\r\n    - mountPath: /secret\r\n      name: hawkular-cassandra-secrets\r\n    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount\r\n      name: cassandra-token-ze2o2\r\n      readOnly: true\r\n  dnsPolicy: ClusterFirst\r\n  host: node1.example.com\r\n  nodeName: node1.example.com\r\n  restartPolicy: Always\r\n  securityContext:\r\n    seLinuxOptions:\r\n      level: s0:c5,c0\r\n  serviceAccount: cassandra\r\n  serviceAccountName: cassandra\r\n  terminationGracePeriodSeconds: 30\r\n  volumes:\r\n  - name: cassandra-data\r\n    persistentVolumeClaim:\r\n      claimName: metrics-cassandra-1\r\n  - name: hawkular-cassandra-secrets\r\n    secret:\r\n      secretName: hawkular-cassandra-secrets\r\n  - name: cassandra-token-ze2o2\r\n    secret:\r\n      secretName: cassandra-token-ze2o2\r\nstatus:\r\n  conditions:\r\n  - lastProbeTime: null\r\n    lastTransitionTime: 2015-12-15T07:24:58Z\r\n    message: 'containers with unready status: [hawkular-cassandra-1]'\r\n    reason: ContainersNotReady\r\n    status: \"False\"\r\n    type: Ready\r\n  containerStatuses:\r\n  - image: 172.30.117.244:5000/openshift/origin-metrics-cassandra:latest\r\n    imageID: \"\"\r\n    lastState: {}\r\n    name: hawkular-cassandra-1\r\n    ready: false\r\n    restartCount: 0\r\n    state:\r\n      waiting:\r\n        message: Back-off pulling image \"172.30.117.244:5000/openshift/origin-metrics-cassandra:latest\"\r\n        reason: ImagePullBackOff\r\n  hostIP: 192.168.0.144\r\n  phase: Pending\r\n  podIP: 10.1.0.58\r\n  startTime: 2015-12-15T07:24:58Z\r\n```"},"comment":{"url":"https://api.github.com/repos/openshift/origin/issues/comments/166312464","html_url":"https://github.com/openshift/origin/issues/6318#issuecomment-166312464","issue_url":"https://api.github.com/repos/openshift/origin/issues/6318","id":166312464,"user":{"login":"mwringe","id":691166,"avatar_url":"https://avatars.githubusercontent.com/u/691166?v=3","gravatar_id":"","url":"https://api.github.com/users/mwringe","html_url":"https://github.com/mwringe","followers_url":"https://api.github.com/users/mwringe/followers","following_url":"https://api.github.com/users/mwringe/following{/other_user}","gists_url":"https://api.github.com/users/mwringe/gists{/gist_id}","starred_url":"https://api.github.com/users/mwringe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mwringe/subscriptions","organizations_url":"https://api.github.com/users/mwringe/orgs","repos_url":"https://api.github.com/users/mwringe/repos","events_url":"https://api.github.com/users/mwringe/events{/privacy}","received_events_url":"https://api.github.com/users/mwringe/received_events","type":"User","site_admin":false},"created_at":"2015-12-21T14:07:03Z","updated_at":"2015-12-21T14:07:03Z","body":"There is a section in the origin-metrics troubleshooting guide on how to check the DNS server: https://github.com/openshift/origin-metrics/blob/master/docs/troubleshooting.adoc#openshift-dns-server\r\n\r\nThe origin-metrics components rely on the DNS server running properly"}}