{"action":"closed","issue":{"url":"https://api.github.com/repos/mesosphere/marathon/issues/1620","labels_url":"https://api.github.com/repos/mesosphere/marathon/issues/1620/labels{/name}","comments_url":"https://api.github.com/repos/mesosphere/marathon/issues/1620/comments","events_url":"https://api.github.com/repos/mesosphere/marathon/issues/1620/events","html_url":"https://github.com/mesosphere/marathon/issues/1620","id":85680853,"number":1620,"title":"deployment got stuck when marathon updating app","user":{"login":"mirake","id":8209454,"avatar_url":"https://avatars.githubusercontent.com/u/8209454?v=3","gravatar_id":"","url":"https://api.github.com/users/mirake","html_url":"https://github.com/mirake","followers_url":"https://api.github.com/users/mirake/followers","following_url":"https://api.github.com/users/mirake/following{/other_user}","gists_url":"https://api.github.com/users/mirake/gists{/gist_id}","starred_url":"https://api.github.com/users/mirake/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mirake/subscriptions","organizations_url":"https://api.github.com/users/mirake/orgs","repos_url":"https://api.github.com/users/mirake/repos","events_url":"https://api.github.com/users/mirake/events{/privacy}","received_events_url":"https://api.github.com/users/mirake/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/mesosphere/marathon/labels/bug","name":"bug","color":"fc2929"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-06-06T02:33:01Z","updated_at":"2015-06-08T13:37:32Z","closed_at":"2015-06-08T13:37:31Z","body":"I use marathon 0.8.0 and mesos 0.21.0 to deploy docker containers. I found a weird problem. Sometimes, when I call \"PUT /v2/apps/<my-app-id>\" to change the docker image to another tag. the marathon web UI shows \"Deploying\" next to the appid, but nothing is going to change, the new container is not created, and the deployment never ends. However this situation does not occur everytime, sometimes the deployment finishes successfully and everything looks great.\r\n\r\nMy cluster has about 300 containers, running at 21 2xlarge AWS EC2 instances. I believe there's enough resource, because the mesos master web UI shows CPU(Offered 7.5, Idle 58.75) and Mem(Offered 28GB, Idle 488.2GB).\r\n\r\nmy request data is:\r\nPUT http://xxxxxx:8080/v2/apps/test1/\r\n{\r\n    \"id\": \"test1\",\r\n    \"container\": {\r\n        \"type\": \"DOCKER\",\r\n        \"docker\": {\r\n            \"image\": \"mirake/httpserver:v2\",\r\n            \"network\": \"BRIDGE\",\r\n            \"portMappings\": [\r\n                {\r\n                    \"containerPort\": 80,\r\n                    \"protocol\": \"tcp\"\r\n                }\r\n            ]\r\n        }\r\n    }    \r\n}\r\n\r\n\r\nHere is the failed deployment log from  /var/log/syslog:\r\nJun  4 01:39:37 ip-172-31-8-66 mesos-master[1146]: I0604 01:39:37.680768  1300 master.cpp:3843] Sending 2 offers to framework 20150327-042900-1459822508-5050-13001-0000 (marathon) at scheduler-c8048229-a361-40e5-9b8a-3e2909d0cd36@172.31.8.66:46002\r\nJun  4 01:39:37 ip-172-31-8-66 mesos-master[1146]: I0604 01:39:37.682083  1300 master.cpp:3843] Sending 3 offers to framework 20150525-140738-1107828652-5050-1146-0000 (chronos-2.3.3) at scheduler-572d489d-5135-471d-97e5-021f3c795930@172.31.3.87:60381\r\nJun  4 01:39:37 ip-172-31-8-66 mesos-master[1146]: I0604 01:39:37.688146  1303 master.cpp:2344] Processing reply for offers: [ 20150525-140738-1107828652-5050-1146-O5506576 ] on slave 20150521-093427-1577787308-5050-26116-S34 at slave(1)@172.31.2.96:5051 (ip-172-31-2-96.cn-north-1.compute.internal) for framework 20150525-140738-1107828652-5050-1146-0000 (chronos-2.3.3) at scheduler-572d489d-5135-471d-97e5-021f3c795930@172.31.3.87:60381\r\nJun  4 01:39:37 ip-172-31-8-66 mesos-master[1146]: I0604 01:39:37.689216  1303 master.cpp:2344] Processing reply for offers: [ 20150525-140738-1107828652-5050-1146-O5506577 ] on slave 20150525-140738-1107828652-5050-1146-S15 at slave(1)@172.31.1.26:5051 (ip-172-31-1-26.cn-north-1.compute.internal) for framework 20150525-140738-1107828652-5050-1146-0000 (chronos-2.3.3) at scheduler-572d489d-5135-471d-97e5-021f3c795930@172.31.3.87:60381\r\nJun  4 01:39:37 ip-172-31-8-66 mesos-master[1146]: I0604 01:39:37.689972  1303 master.cpp:2344] Processing reply for offers: [ 20150525-140738-1107828652-5050-1146-O5506578 ] on slave 20150521-093427-1577787308-5050-26116-S18 at slave(1)@172.31.4.249:5051 (ip-172-31-4-249.cn-north-1.compute.internal) for framework 20150525-140738-1107828652-5050-1146-0000 (chronos-2.3.3) at scheduler-572d489d-5135-471d-97e5-021f3c795930@172.31.3.87:60381\r\nJun  4 01:39:37 ip-172-31-8-66 mesos-master[1146]: I0604 01:39:37.691040  1305 hierarchical_allocator_process.hpp:563] Recovered cpus(*):1.75; mem(*):23694; ports(*):[31538-32000, 31426-31526, 31532-31532, 31534-31534, 31014-31421, 31009-31012]; disk(*):14896 (total allocatable: cpus(*):1.75; mem(*):23694; ports(*):[31538-32000, 31426-31526, 31532-31532, 31534-31534, 31014-31421, 31009-31012]; disk(*):14896) on slave 20150521-093427-1577787308-5050-26116-S34 from framework 20150525-140738-1107828652-5050-1146-0000\r\nJun  4 01:39:37 ip-172-31-8-66 mesos-master[1146]: I0604 01:39:37.692083  1303 hierarchical_allocator_process.hpp:563] Recovered cpus(*):7.25; mem(*):28366; disk(*):14896; ports(*):[31003-32000] (total allocatable: cpus(*):7.25; mem(*):28366; disk(*):14896; ports(*):[31003-32000]) on slave 20150525-140738-1107828652-5050-1146-S15 from framework 20150525-140738-1107828652-5050-1146-0000\r\nJun  4 01:39:37 ip-172-31-8-66 mesos-master[1146]: I0604 01:39:37.692740  1303 hierarchical_allocator_process.hpp:563] Recovered mem(*):22734; ports(*):[31003-31003, 31032-32000, 31030-31030, 31020-31022, 31001-31001]; disk(*):14896; cpus(*):1.75 (total allocatable: mem(*):22734; ports(*):[31003-31003, 31032-32000, 31030-31030, 31020-31022, 31001-31001]; disk(*):14896; cpus(*):1.75) on slave 20150521-093427-1577787308-5050-26116-S18 from framework 20150525-140738-1107828652-5050-1146-0000\r\nJun  4 01:39:37 ip-172-31-8-66 marathon[8933]: [2015-06-04 01:39:37,808] INFO Computed new deployment plan: DeploymentPlan(2015-06-04T01:39:35.444Z, (Step(List(Restart(App(/test1, None))))))) (mesosphere.marathon.upgrade.DeploymentPlan$:246)\r\nJun  4 01:39:37 ip-172-31-8-66 marathon[8933]: [INFO] [06/04/2015 01:39:37.809] [marathon-akka.actor.default-dispatcher-198] [akka://marathon/user/$b] POSTing to all endpoints.\r\nJun  4 01:39:37 ip-172-31-8-66 marathon[8933]: [INFO] [06/04/2015 01:39:37.809] [marathon-akka.actor.default-dispatcher-198] [akka://marathon/user/MarathonScheduler/$a/UpgradeManager/2ce6efd1-b7f1-4b55-b6eb-a2b282b9e4a7/$a] For minimumHealthCapacity 1.0 of /test1 leave 1 tasks running, maximum capacity 2, killing 0 tasks immediately\r\nJun  4 01:39:37 ip-172-31-8-66 marathon[8933]: [INFO] [06/04/2015 01:39:37.809] [marathon-akka.actor.default-dispatcher-198] [akka://marathon/user/MarathonScheduler/$a/UpgradeManager/2ce6efd1-b7f1-4b55-b6eb-a2b282b9e4a7/$a] Reconciliating tasks during app /test1 restart: queuing 1 new tasks\r\nJun  4 01:39:38 ip-172-31-8-66 marathon[8933]: [INFO] [06/04/2015 01:39:38.004] [pool-2-thread-395] [akka://marathon/user/$b] Sending POST to:http://172.31.14.9:8080/api/v1/appevents/\r\nJun  4 01:39:38 ip-172-31-8-66 mesos-master[1146]: I0604 01:39:38.168552  1306 master.cpp:2344] Processing reply for offers: [ 20150525-140738-1107828652-5050-1146-O5506574 ] on slave 20150521-093427-1577787308-5050-26116-S29 at slave(1)@172.31.1.104:5051 (ip-172-31-1-104.cn-north-1.compute.internal) for framework 20150327-042900-1459822508-5050-13001-0000 (marathon) at scheduler-c8048229-a361-40e5-9b8a-3e2909d0cd36@172.31.8.66:46002\r\nJun  4 01:39:38 ip-172-31-8-66 mesos-master[1146]: I0604 01:39:38.169561  1306 master.cpp:2344] Processing reply for offers: [ 20150525-140738-1107828652-5050-1146-O5506575 ] on slave 20150521-093427-1577787308-5050-26116-S31 at slave(1)@172.31.14.11:5051 (ip-172-31-14-11.cn-north-1.compute.internal) for framework 20150327-042900-1459822508-5050-13001-0000 (marathon) at scheduler-c8048229-a361-40e5-9b8a-3e2909d0cd36@172.31.8.66:46002\r\nJun  4 01:39:38 ip-172-31-8-66 mesos-master[1146]: I0604 01:39:38.170581  1306 hierarchical_allocator_process.hpp:563] Recovered mem(*):20942; ports(*):[31112-31112, 31007-31008, 31021-31110, 31002-31004, 31115-32000]; disk(*):14896 (total allocatable: mem(*):20942; ports(*):[31112-31112, 31007-31008, 31021-31110, 31002-31004, 31115-32000]; disk(*):14896) on slave 20150521-093427-1577787308-5050-26116-S29 from framework 20150327-042900-1459822508-5050-13001-0000\r\nJun  4 01:39:38 ip-172-31-8-66 mesos-master[1146]: I0604 01:39:38.171366  1306 hierarchical_allocator_process.hpp:563] Recovered mem(*):22478; ports(*):[31026-32000, 31003-31003, 31001-31001, 31022-31022, 31005-31005]; disk(*):14896; cpus(*):1.5 (total allocatable: mem(*):22478; ports(*):[31026-32000, 31003-31003, 31001-31001, 31022-31022, 31005-31005]; disk(*):14896; cpus(*):1.5) on slave 20150521-093427-1577787308-5050-26116-S31 from framework 20150327-042900-1459822508-5050-13001-0000\r\nJun  4 01:39:38 ip-172-31-8-66 marathon[8933]: [2015-06-04 01:39:38,556] INFO 172.31.3.143 -  -  [04/Jun/2015:01:39:38 +0000] \"GET / HTTP/1.1\" 200 413 \"-\" \"ELB-HealthChecker/1.0\" (mesosphere.chaos.http.ChaosRequestLog:15)\r\nJun  4 01:39:38 ip-172-31-8-66 mesos-master[1146]: I0604 01:39:38.689867  1301 master.cpp:3843] Sending 10 offers to framework 20150327-042900-1459822508-5050-13001-0000 (marathon) at scheduler-c8048229-a361-40e5-9b8a-3e2909d0cd36@172.31.8.66:46002\r\nJun  4 01:39:38 ip-172-31-8-66 mesos-master[1146]: I0604 01:39:38.691584  1301 master.cpp:3843] Sending 9 offers to framework 20150525-140738-1107828652-5050-1146-0000 (chronos-2.3.3) at scheduler-572d489d-5135-471d-97e5-021f3c795930@172.31.3.87:60381\r\nJun  4 01:39:38 ip-172-31-8-66 mesos-master[1146]: I0604 01:39:38.698211  1300 master.cpp:2344] Processing reply for offers: [ 20150525-140738-1107828652-5050-1146-O5506589 ] on slave 20150521-093427-1577787308-5050-26116-S17 at slave(1)@172.31.8.29:5051 (ip-172-31-8-29.cn-north-1.compute.internal) for framework 20150525-140738-1107828652-5050-1146-0000 (chronos-2.3.3) at scheduler-572d489d-5135-471d-97e5-021f3c795930@172.31.3.87:60381\r\nJun  4 01:39:38 ip-172-31-8-66 mesos-master[1146]: I0604 01:39:38.698962  1300 master.cpp:2344] Processing reply for offers: [ 20150525-140738-1107828652-5050-1146-O5506590 ] on slave 20150521-093427-1577787308-5050-26116-S41 at slave(1)@172.31.1.0:5051 (ip-172-31-1-0.cn-north-1.compute.internal) for framework 20150525-140738-1107828652-5050-1146-0000 (chronos-2.3.3) at scheduler-572d489d-5135-471d-97e5-021f3c795930@172.31.3.87:60381\r\nJun  4 01:39:38 ip-172-31-8-66 mesos-master[1146]: I0604 01:39:38.699630  1300 master.cpp:2344] Processing reply for offers: [ 20150525-140738-1107828652-5050-1146-O5506591 ] on slave 20150521-093427-1577787308-5050-26116-S51 at slave(1)@172.31.13.17:5051 (ip-172-31-13-17.cn-north-1.compute.internal) for framework 20150525-140738-1107828652-5050-1146-0000 (chronos-2.3.3) at scheduler-572d489d-5135-471d-97e5-021f3c795930@172.31.3.87:60381\r\n\r\n\r\n\r\nIf the deployment succeeds, the log is like this:\r\nJun  4 01:42:41 ip-172-31-8-66 marathon[8933]: [2015-06-04 01:42:41,882] INFO Computed new deployment plan: DeploymentPlan(2015-06-04T01:42:39.420Z, (Step(List(Restart(App(/test1, None))))))) (mesosphere.marathon.upgrade.DeploymentPlan$:246)\r\nJun  4 01:42:41 ip-172-31-8-66 marathon[8933]: [INFO] [06/04/2015 01:42:41.883] [marathon-akka.actor.default-dispatcher-193] [akka://marathon/user/$b] POSTing to all endpoints.\r\nJun  4 01:42:41 ip-172-31-8-66 marathon[8933]: [INFO] [06/04/2015 01:42:41.883] [marathon-akka.actor.default-dispatcher-205] [akka://marathon/user/MarathonScheduler/$a/UpgradeManager/9b50aaab-4c2f-40e5-a88d-4ff89e362f75/$a] For minimumHealthCapacity 1.0 of /test1 leave 1 tasks running, maximum capacity 2, killing 0 tasks immediately\r\nJun  4 01:42:41 ip-172-31-8-66 marathon[8933]: [INFO] [06/04/2015 01:42:41.884] [marathon-akka.actor.default-dispatcher-205] [akka://marathon/user/MarathonScheduler/$a/UpgradeManager/9b50aaab-4c2f-40e5-a88d-4ff89e362f75/$a] Reconciliating tasks during app /test1 restart: queuing 1 new tasks\r\nJun  4 01:42:41 ip-172-31-8-66 marathon[8933]: [INFO] [06/04/2015 01:42:41.885] [pool-2-thread-404] [akka://marathon/user/$b] Sending POST to:http://172.31.14.9:8080/api/v1/appevents/\r\nJun  4 01:42:41 ip-172-31-8-66 mesos-master[1146]: I0604 01:42:41.954910  1305 master.cpp:3843] Sending 5 offers to framework 20150327-042900-1459822508-5050-13001-0000 (marathon) at scheduler-c8048229-a361-40e5-9b8a-3e2909d0cd36@172.31.8.66:46002\r\nJun  4 01:42:42 ip-172-31-8-66 mesos-master[1146]: I0604 01:42:42.485059  1304 master.cpp:2344] Processing reply for offers: [ 20150525-140738-1107828652-5050-1146-O5507980 ] on slave 20150525-140235-1577787308-5050-31169-S0 at slave(1)@172.31.12.241:5051 (ip-172-31-12-241.cn-north-1.compute.internal) for framework 20150327-042900-1459822508-5050-13001-0000 (marathon) at scheduler-c8048229-a361-40e5-9b8a-3e2909d0cd36@172.31.8.66:46002\r\nJun  4 01:42:42 ip-172-31-8-66 mesos-master[1146]: I0604 01:42:42.486330  1304 master.cpp:2344] Processing reply for offers: [ 20150525-140738-1107828652-5050-1146-O5507981 ] on slave 20150525-140738-1107828652-5050-1146-S10 at slave(1)@172.31.2.0:5051 (ip-172-31-2-0.cn-north-1.compute.internal) for framework 20150327-042900-1459822508-5050-13001-0000 (marathon) at scheduler-c8048229-a361-40e5-9b8a-3e2909d0cd36@172.31.8.66:46002\r\nJun  4 01:42:42 ip-172-31-8-66 mesos-master[1146]: I0604 01:42:42.486871  1304 master.cpp:2344] Processing reply for offers: [ 20150525-140738-1107828652-5050-1146-O5507982 ] on slave 20150525-140235-1577787308-5050-31169-S1 at slave(1)@172.31.11.10:5051 (ip-172-31-11-10.cn-north-1.compute.internal) for framework 20150327-042900-1459822508-5050-13001-0000 (marathon) at scheduler-c8048229-a361-40e5-9b8a-3e2909d0cd36@172.31.8.66:46002\r\nJun  4 01:42:42 ip-172-31-8-66 mesos-master[1146]: I0604 01:42:42.487386  1304 master.cpp:2344] Processing reply for offers: [ 20150525-140738-1107828652-5050-1146-O5507983 ] on slave 20150521-093427-1577787308-5050-26116-S29 at slave(1)@172.31.1.104:5051 (ip-172-31-1-104.cn-north-1.compute.internal) for framework 20150327-042900-1459822508-5050-13001-0000 (marathon) at scheduler-c8048229-a361-40e5-9b8a-3e2909d0cd36@172.31.8.66:46002\r\nJun  4 01:42:42 ip-172-31-8-66 mesos-master[1146]: I0604 01:42:42.487921  1304 master.cpp:2344] Processing reply for offers: [ 20150525-140738-1107828652-5050-1146-O5507984 ] on slave 20150521-093427-1577787308-5050-26116-S46 at slave(1)@172.31.14.38:5051 (ip-172-31-14-38.cn-north-1.compute.internal) for framework 20150327-042900-1459822508-5050-13001-0000 (marathon) at scheduler-c8048229-a361-40e5-9b8a-3e2909d0cd36@172.31.8.66:46002\r\nJun  4 01:42:42 ip-172-31-8-66 mesos-master[1146]: I0604 01:42:42.488325  1304 master.hpp:877] Adding task test1.fdb993b7-0a5a-11e5-a6cc-02416b28d26a with resources cpus(*):1; mem(*):64; ports(*):[31011-31011] on slave 20150525-140235-1577787308-5050-31169-S0 (ip-172-31-12-241.cn-north-1.compute.internal)\r\nJun  4 01:42:42 ip-172-31-8-66 mesos-master[1146]: I0604 01:42:42.488514  1304 master.cpp:2503] Launching task test1.fdb993b7-0a5a-11e5-a6cc-02416b28d26a of framework 20150327-042900-1459822508-5050-13001-0000 (marathon) at scheduler-c8048229-a361-40e5-9b8a-3e2909d0cd36@172.31.8.66:46002 with resources cpus(*):1; mem(*):64; ports(*):[31011-31011] on slave 20150525-140235-1577787308-5050-31169-S0 at slave(1)@172.31.12.241:5051 (ip-172-31-12-241.cn-north-1.compute.internal)\r\nJun  4 01:42:42 ip-172-31-8-66 mesos-master[1146]: I0604 01:42:42.489491  1300 hierarchical_allocator_process.hpp:563] Recovered cpus(*):4.5; mem(*):26510; ports(*):[31014-32000, 31008-31009]; disk(*):14896 (total allocatable: cpus(*):4.5; mem(*):26510; ports(*):[31014-32000, 31008-31009]; disk(*):14896) on slave 20150525-140235-1577787308-5050-31169-S0 from framework 20150327-042900-1459822508-5050-13001-0000\r\n.....\r\nJun  4 01:42:43 ip-172-31-8-66 mesos-master[1146]: I0604 01:42:43.257398  1307 master.cpp:3446] Forwarding status update TASK_RUNNING (UUID: f105520a-4462-44bf-8375-07aa0ce878a2) for task test1.fdb993b7-0a5a-11e5-a6cc-02416b28d26a of framework 20150327-042900-1459822508-5050-13001-0000\r\nJun  4 01:42:43 ip-172-31-8-66 mesos-master[1146]: I0604 01:42:43.257822  1307 master.cpp:3418] Status update TASK_RUNNING (UUID: f105520a-4462-44bf-8375-07aa0ce878a2) for task test1.fdb993b7-0a5a-11e5-a6cc-02416b28d26a of framework 20150327-042900-1459822508-5050-13001-0000 from slave 20150525-140235-1577787308-5050-31169-S0 at slave(1)@172.31.12.241:5051 (ip-172-31-12-241.cn-north-1.compute.internal)\r\nJun  4 01:42:43 ip-172-31-8-66 mesos-master[1146]: I0604 01:42:43.257927  1307 master.cpp:4693] Updating the latest state of task test1.fdb993b7-0a5a-11e5-a6cc-02416b28d26a of framework 20150327-042900-1459822508-5050-13001-0000 to TASK_RUNNING\r\nJun  4 01:42:43 ip-172-31-8-66 marathon[8933]: [2015-06-04 01:42:43,469] INFO Received status update for task test1.fdb993b7-0a5a-11e5-a6cc-02416b28d26a: TASK_RUNNING () (mesosphere.marathon.MarathonScheduler:148)\r\nJun  4 01:42:43 ip-172-31-8-66 mesos-master[1146]: I0604 01:42:43.480370  1301 master.cpp:2918] Forwarding status update acknowledgement f105520a-4462-44bf-8375-07aa0ce878a2 for task test1.fdb993b7-0a5a-11e5-a6cc-02416b28d26a of framework 20150327-042900-1459822508-5050-13001-0000 (marathon) at scheduler-c8048229-a361-40e5-9b8a-3e2909d0cd36@172.31.8.66:46002 to slave 20150525-140235-1577787308-5050-31169-S0 at slave(1)@172.31.12.241:5051 (ip-172-31-12-241.cn-north-1.compute.internal)\r\n........\r\nJun  4 01:42:43 ip-172-31-8-66 marathon[8933]: [INFO] [06/04/2015 01:42:43.487] [marathon-akka.actor.default-dispatcher-172] [akka://marathon/user/MarathonScheduler/$a/UpgradeManager/9b50aaab-4c2f-40e5-a88d-4ff89e362f75/$a] Killing old task test1.eb0cff95-0a5a-11e5-a6cc-02416b28d26a because test1.fdb993b7-0a5a-11e5-a6cc-02416b28d26a became reachable\r\nJun  4 01:42:43 ip-172-31-8-66 marathon[8933]: [INFO] [06/04/2015 01:42:43.487] [marathon-akka.actor.default-dispatcher-172] [akka://marathon/user/MarathonScheduler/$a/UpgradeManager/9b50aaab-4c2f-40e5-a88d-4ff89e362f75/$a] All tasks for /test1 are healthy\r\n........\r\nJun  4 01:42:43 ip-172-31-8-66 mesos-master[1146]: I0604 01:42:43.489122  1301 master.cpp:2716] Asked to kill task test1.eb0cff95-0a5a-11e5-a6cc-02416b28d26a of framework 20150327-042900-1459822508-5050-13001-0000\r\nJun  4 01:42:43 ip-172-31-8-66 mesos-master[1146]: I0604 01:42:43.489231  1301 master.cpp:2814] Telling slave 20150521-093427-1577787308-5050-26116-S34 at slave(1)@172.31.2.96:5051 (ip-172-31-2-96.cn-north-1.compute.internal) to kill task test1.eb0cff95-0a5a-11e5-a6cc-02416b28d26a of framework 20150327-042900-1459822508-5050-13001-0000 (marathon) at scheduler-c8048229-a361-40e5-9b8a-3e2909d0cd36@172.31.8.66:46002\r\n.......\r\nJun  4 01:42:43 ip-172-31-8-66 mesos-master[1146]: I0604 01:42:43.695261  1303 master.cpp:3446] Forwarding status update TASK_KILLED (UUID: 1b72909b-a72f-4bf0-9cfe-446cb657508d) for task test1.eb0cff95-0a5a-11e5-a6cc-02416b28d26a of framework 20150327-042900-1459822508-5050-13001-0000\r\nJun  4 01:42:43 ip-172-31-8-66 mesos-master[1146]: I0604 01:42:43.695633  1303 master.cpp:3418] Status update TASK_KILLED (UUID: 1b72909b-a72f-4bf0-9cfe-446cb657508d) for task test1.eb0cff95-0a5a-11e5-a6cc-02416b28d26a of framework 20150327-042900-1459822508-5050-13001-0000 from slave 20150521-093427-1577787308-5050-26116-S34 at slave(1)@172.31.2.96:5051 (ip-172-31-2-96.cn-north-1.compute.internal)\r\nJun  4 01:42:43 ip-172-31-8-66 mesos-master[1146]: I0604 01:42:43.695735  1303 master.cpp:4693] Updating the latest state of task test1.eb0cff95-0a5a-11e5-a6cc-02416b28d26a of framework 20150327-042900-1459822508-5050-13001-0000 to TASK_KILLED\r\nJun  4 01:42:43 ip-172-31-8-66 mesos-master[1146]: I0604 01:42:43.696563  1303 hierarchical_allocator_process.hpp:563] Recovered cpus(*):1; mem(*):64; ports(*):[31538-31538] (total allocatable: cpus(*):2.5; mem(*):23502; ports(*):[31532-31532, 31534-31534, 31014-31421, 31009-31012, 31003-31003, 31427-31526, 31538-32000]; disk(*):14896) on slave 20150521-093427-1577787308-5050-26116-S34 from framework 20150327-042900-1459822508-5050-13001-0000\r\nJun  4 01:42:43 ip-172-31-8-66 marathon[8933]: [2015-06-04 01:42:43,696] INFO Received status update for task test1.eb0cff95-0a5a-11e5-a6cc-02416b28d26a: TASK_KILLED (Command terminated with signal Terminated) (mesosphere.marathon.MarathonScheduler:148)\r\nJun  4 01:42:43 ip-172-31-8-66 mesos-master[1146]: I0604 01:42:43.699671  1307 master.cpp:4752] Removing task test1.eb0cff95-0a5a-11e5-a6cc-02416b28d26a with resources cpus(*):1; mem(*):64; ports(*):[31538-31538] of framework 20150327-042900-1459822508-5050-13001-0000 on slave 20150521-093427-1577787308-5050-26116-S34 at slave(1)@172.31.2.96:5051 (ip-172-31-2-96.cn-north-1.compute.internal)\r\nJun  4 01:42:43 ip-172-31-8-66 mesos-master[1146]: I0604 01:42:43.700086  1307 master.cpp:2918] Forwarding status update acknowledgement 1b72909b-a72f-4bf0-9cfe-446cb657508d for task test1.eb0cff95-0a5a-11e5-a6cc-02416b28d26a of framework 20150327-042900-1459822508-5050-13001-0000 (marathon) at scheduler-c8048229-a361-40e5-9b8a-3e2909d0cd36@172.31.8.66:46002 to slave 20150521-093427-1577787308-5050-26116-S34 at slave(1)@172.31.2.96:5051 (ip-172-31-2-96.cn-north-1.compute.internal)\r\nJun  4 01:42:43 ip-172-31-8-66 marathon[8933]: [2015-06-04 01:42:43,699] INFO Task test1.eb0cff95-0a5a-11e5-a6cc-02416b28d26a expunged and removed from TaskTracker (mesosphere.marathon.tasks.TaskTracker:107)\r\n"}}