{"action":"created","issue":{"url":"https://api.github.com/repos/mesosphere/marathon/issues/1161","labels_url":"https://api.github.com/repos/mesosphere/marathon/issues/1161/labels{/name}","comments_url":"https://api.github.com/repos/mesosphere/marathon/issues/1161/comments","events_url":"https://api.github.com/repos/mesosphere/marathon/issues/1161/events","html_url":"https://github.com/mesosphere/marathon/issues/1161","id":57086646,"number":1161,"title":"Marathon/Mesos does not pull docker images.","user":{"login":"more-free","id":2353434,"avatar_url":"https://avatars.githubusercontent.com/u/2353434?v=3","gravatar_id":"","url":"https://api.github.com/users/more-free","html_url":"https://github.com/more-free","followers_url":"https://api.github.com/users/more-free/followers","following_url":"https://api.github.com/users/more-free/following{/other_user}","gists_url":"https://api.github.com/users/more-free/gists{/gist_id}","starred_url":"https://api.github.com/users/more-free/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/more-free/subscriptions","organizations_url":"https://api.github.com/users/more-free/orgs","repos_url":"https://api.github.com/users/more-free/repos","events_url":"https://api.github.com/users/more-free/events{/privacy}","received_events_url":"https://api.github.com/users/more-free/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-02-09T20:41:40Z","updated_at":"2015-02-09T22:59:07Z","closed_at":null,"body":"Hi,\r\n\r\nRecently we are using Marathon / Mesos to deploy some of our own docker containers (from our own registry), but it fails to pull the images from registry  -- it always fails deploying (from Mesos UI we see failed tasks) unless we manually run \"docker pull\" on slave nodes to pull images in advance.\r\n\r\nIt looks to me , seems Mesos fails to pull images ?  Any clue about this  ?  Or is there any log I can check ? Thanks !\r\n\r\nThe versions we use are :\r\nMarathon 0.8.0\r\nMesos 0.21\r\n\r\nand we use json to deploy with Marathon.\r\n"},"comment":{"url":"https://api.github.com/repos/mesosphere/marathon/issues/comments/73609457","html_url":"https://github.com/mesosphere/marathon/issues/1161#issuecomment-73609457","issue_url":"https://api.github.com/repos/mesosphere/marathon/issues/1161","id":73609457,"user":{"login":"more-free","id":2353434,"avatar_url":"https://avatars.githubusercontent.com/u/2353434?v=3","gravatar_id":"","url":"https://api.github.com/users/more-free","html_url":"https://github.com/more-free","followers_url":"https://api.github.com/users/more-free/followers","following_url":"https://api.github.com/users/more-free/following{/other_user}","gists_url":"https://api.github.com/users/more-free/gists{/gist_id}","starred_url":"https://api.github.com/users/more-free/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/more-free/subscriptions","organizations_url":"https://api.github.com/users/more-free/orgs","repos_url":"https://api.github.com/users/more-free/repos","events_url":"https://api.github.com/users/more-free/events{/privacy}","received_events_url":"https://api.github.com/users/more-free/received_events","type":"User","site_admin":false},"created_at":"2015-02-09T22:59:07Z","updated_at":"2015-02-09T22:59:07Z","body":"Hi,\r\n\r\nthanks very much for your response !  Actually we don't have our own executor,  we use the default \"CMD\" to run docker.  From the Mesos UI,  we saw that the deployment failed immediately.  Also, the log on mesos-slave shows that it fails to pull docker image.  But actually our docker image is there and there is no firewall / authentication / etc. to prevent mesos-slave from accessing our docker registry (and we can manually pull the docker image on mesos-slave with exactly the same command shown in the log). \r\n\r\nI paste some of the log on mesos-slave below,  please help if you have any clue about this issue...  Thanks very much !!\r\n\r\n\r\n\r\n\r\n\r\nI0209 20:24:18.364290   823 gc.cpp:84] Unscheduling '/var/run/mesos/slaves/20150207-012846-1912885258-5050-1164-S8/frameworks/20141216-070346-1912885258-5050-32095-0000' from gc\r\nI0209 20:24:18.364780   823 gc.cpp:84] Unscheduling '/var/run/mesos/meta/slaves/20150207-012846-1912885258-5050-1164-S8/frameworks/20141216-070346-1912885258-5050-32095-0000' from gc\r\nI0209 20:24:18.365309   823 slave.cpp:1193] Launching task  s-app-3.9ff7a871-b099-11e4-9b55-56847afe9799 for framework 20141216-070346-1912885258-5050-32095-0000\r\nI0209 20:24:18.368279   823 slave.cpp:3997] Launching executor  s-app-3.9ff7a871-b099-11e4-9b55-56847afe9799 of framework 20141216-070346-1912885258-5050-32095-0000 in work directory '/var/run/mesos/slaves/20150207-012846-1912885258-5050-1164-S8/frameworks/20141216-070346-1912885258-5050-32095-0000/executors/ s-app-3.9ff7a871-b099-11e4-9b55-56847afe9799/runs/42cc4e7f-99b9-448e-84dc-0259f0ab867b'\r\nI0209 20:24:18.372236   823 slave.cpp:1316] Queuing task ' s-app-3.9ff7a871-b099-11e4-9b55-56847afe9799' for executor  s-app-3.9ff7a871-b099-11e4-9b55-56847afe9799 of framework '20141216-070346-1912885258-5050-32095-0000\r\nI0209 20:24:18.378255   822 docker.cpp:928] Starting container '42cc4e7f-99b9-448e-84dc-0259f0ab867b' for task ' s-app-3.9ff7a871-b099-11e4-9b55-56847afe9799' (and executor ' s-app-3.9ff7a871-b099-11e4-9b55-56847afe9799') of framework '20141216-070346-1912885258-5050-32095-0000'\r\nI0209 20:24:19.975740   818 docker.cpp:1502] Destroying container '42cc4e7f-99b9-448e-84dc-0259f0ab867b'\r\nE0209 20:24:19.976316   821 slave.cpp:2787] Container '42cc4e7f-99b9-448e-84dc-0259f0ab867b' for executor ' s-app-3.9ff7a871-b099-11e4-9b55-56847afe9799' of framework '20141216-070346-1912885258-5050-32095-0000' failed to start: Failed to 'docker pull docker-registry.ops.yahoo.com:4080/apurvak/yql_test_apple_weather_tenant:04d8c8615ac7': exit status = exited with status 1 stderr = 2015/02/09 20:24:19 Tag 04d8c8615ac7 not found in repository docker-registry.ops.yahoo.com:4080/apurvak/yql_test_apple_weather_tenant\r\nI0209 20:24:19.976426   818 docker.cpp:1546] Destroying Container '42cc4e7f-99b9-448e-84dc-0259f0ab867b' in PULLING state\r\nE0209 20:24:19.978085   818 slave.cpp:2882] Termination of executor ' s-app-3.9ff7a871-b099-11e4-9b55-56847afe9799' of framework '20141216-070346-1912885258-5050-32095-0000' failed: Unknown container: 42cc4e7f-99b9-448e-84dc-0259f0ab867b\r\nE0209 20:24:19.978721   823 slave.cpp:3134] Failed to unmonitor container for executor  s-app-3.9ff7a871-b099-11e4-9b55-56847afe9799 of framework 20141216-070346-1912885258-5050-32095-0000: Not monitored\r\nI0209 20:24:19.980541   818 slave.cpp:2215] Handling status update TASK_FAILED (UUID: 1037e83c-7e47-4905-9c7a-b936e363a851) for task  s-app-3.9ff7a871-b099-11e4-9b55-56847afe9799 of framework 20141216-070346-1912885258-5050-32095-0000 from @0.0.0.0:0\r\nW0209 20:24:19.981329   819 docker.cpp:1184] Ignoring updating unknown container: 42cc4e7f-99b9-448e-84dc-0259f0ab867b\r\nI0209 20:24:19.982303   818 status_update_manager.cpp:317] Received status update TASK_FAILED (UUID: 1037e83c-7e47-4905-9c7a-b936e363a851) for task  s-app-3.9ff7a871-b099-11e4-9b55-56847afe9799 of framework 20141216-070346-1912885258-5050-32095-0000\r\nI0209 20:24:19.983125   818 status_update_manager.hpp:346] Checkpointing UPDATE for status update TASK_FAILED (UUID: 1037e83c-7e47-4905-9c7a-b936e363a851) for task  s-app-3.9ff7a871-b099-11e4-9b55-56847afe9799 of framework 20141216-070346-1912885258-5050-32095-0000\r\nI0209 20:24:19.987365   818 slave.cpp:2458] Forwarding the update TASK_FAILED (UUID: 1037e83c-7e47-4905-9c7a-b936e363a851) for task  s-app-3.9ff7a871-b099-11e4-9b55-56847afe9799 of framework 20141216-070346-1912885258-5050-32095-0000 to master@10.80.4.114:5050\r\nI0209 20:24:20.003286   819 status_update_manager.cpp:389] Received status update acknowledgement (UUID: 1037e83c-7e47-4905-9c7a-b936e363a851) for task  s-app-3.9ff7a871-b099-11e4-9b55-56847afe9799 of framework 20141216-070346-1912885258-5050-32095-0000\r\nI0209 20:24:20.003672   819 status_update_manager.hpp:346] Checkpointing ACK for status update TASK_FAILED (UUID: 1037e83c-7e47-4905-9c7a-b936e363a851) for task  s-app-3.9ff7a871-b099-11e4-9b55-56847afe9799 of framework 20141216-070346-1912885258-5050-32095-0000\r\nI0209 20:24:20.007709   819 slave.cpp:3007] Cleaning up executor ' s-app-3.9ff7a871-b099-11e4-9b55-56847afe9799' of framework 20141216-070346-1912885258-5050-32095-0000\r\nI0209 20:24:20.008559   820 gc.cpp:56] Scheduling '/var/run/mesos/slaves/20150207-012846-1912885258-5050-1164-S8/frameworks/20141216-070346-1912885258-5050-32095-0000/executors/ s-app-3.9ff7a871-b099-11e4-9b55-56847afe9799/runs/42cc4e7f-99b9-448e-84dc-0259f0ab867b' for gc 6.99999990292741days in the future\r\nI0209 20:24:20.008646   819 slave.cpp:3084] Cleaning up framework 20141216-070346-1912885258-5050-32095-0000\r\nI0209 20:24:20.009016   820 gc.cpp:56] Scheduling '/var/run/mesos/slaves/20150207-012846-1912885258-5050-1164-S8/frameworks/20141216-070346-1912885258-5050-32095-0000/executors/ s-app-3.9ff7a871-b099-11e4-9b55-56847afe9799' for gc 6.99999990141333days in the future\r\nI0209 20:24:20.009558   825 status_update_manager.cpp:279] Closing status update streams for framework 20141216-070346-1912885258-5050-32095-0000\r\nI0209 20:24:20.009974   820 gc.cpp:56] Scheduling '/var/run/mesos/meta/slaves/20150207-012846-1912885258-5050-1164-S8/frameworks/20141216-070346-1912885258-5050-32095-0000/executors/ s-app-3.9ff7a871-b099-11e4-9b55-56847afe9799/runs/42cc4e7f-99b9-448e-84dc-0259f0ab867b' for gc 6.9999999007763days in the future\r\nI0209 20:24:20.010691   820 gc.cpp:56] Scheduling '/var/run/mesos/meta/slaves/20150207-012846-1912885258-5050-1164-S8/frameworks/20141216-070346-1912885258-5050-32095-0000/executors/ s-app-3.9ff7a871-b099-11e4-9b55-56847afe9799' for gc 6.99999990023111days in the future\r\nI0209 20:24:20.011175   820 gc.cpp:56] Scheduling '/var/run/mesos/slaves/20150207-012846-1912885258-5050-1164-S8/frameworks/20141216-070346-1912885258-5050-32095-0000' for gc 6.99999988994074days in the future\r\nI0209 20:24:20.011530   820 gc.cpp:56] Scheduling '/var/run/mesos/meta/slaves/20150207-012846-1912885258-5050-1164-S8/frameworks/20141216-070346-1912885258-5050-32095-0000' for gc 6.99999988933926days in the future\r\nI0209 20:25:15.613370   819 slave.cpp:3321] Current usage 1.49%. Max allowed age: 6.195929403155035days\r\n\r\n\r\n"}}