{"action":"created","issue":{"url":"https://api.github.com/repos/mesosphere/marathon/issues/1005","labels_url":"https://api.github.com/repos/mesosphere/marathon/issues/1005/labels{/name}","comments_url":"https://api.github.com/repos/mesosphere/marathon/issues/1005/comments","events_url":"https://api.github.com/repos/mesosphere/marathon/issues/1005/events","html_url":"https://github.com/mesosphere/marathon/issues/1005","id":54416087,"number":1005,"title":"Failed tasks are not rescheduled","user":{"login":"bobrik","id":89186,"avatar_url":"https://avatars.githubusercontent.com/u/89186?v=3","gravatar_id":"","url":"https://api.github.com/users/bobrik","html_url":"https://github.com/bobrik","followers_url":"https://api.github.com/users/bobrik/followers","following_url":"https://api.github.com/users/bobrik/following{/other_user}","gists_url":"https://api.github.com/users/bobrik/gists{/gist_id}","starred_url":"https://api.github.com/users/bobrik/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bobrik/subscriptions","organizations_url":"https://api.github.com/users/bobrik/orgs","repos_url":"https://api.github.com/users/bobrik/repos","events_url":"https://api.github.com/users/bobrik/events{/privacy}","received_events_url":"https://api.github.com/users/bobrik/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/mesosphere/marathon/labels/blocker","name":"blocker","color":"333333"},{"url":"https://api.github.com/repos/mesosphere/marathon/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":{"login":"drexin","id":433020,"avatar_url":"https://avatars.githubusercontent.com/u/433020?v=3","gravatar_id":"","url":"https://api.github.com/users/drexin","html_url":"https://github.com/drexin","followers_url":"https://api.github.com/users/drexin/followers","following_url":"https://api.github.com/users/drexin/following{/other_user}","gists_url":"https://api.github.com/users/drexin/gists{/gist_id}","starred_url":"https://api.github.com/users/drexin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/drexin/subscriptions","organizations_url":"https://api.github.com/users/drexin/orgs","repos_url":"https://api.github.com/users/drexin/repos","events_url":"https://api.github.com/users/drexin/events{/privacy}","received_events_url":"https://api.github.com/users/drexin/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/mesosphere/marathon/milestones/10","labels_url":"https://api.github.com/repos/mesosphere/marathon/milestones/10/labels","id":808564,"number":10,"title":"0.7.7","description":"","creator":{"login":"ConnorDoyle","id":379372,"avatar_url":"https://avatars.githubusercontent.com/u/379372?v=3","gravatar_id":"","url":"https://api.github.com/users/ConnorDoyle","html_url":"https://github.com/ConnorDoyle","followers_url":"https://api.github.com/users/ConnorDoyle/followers","following_url":"https://api.github.com/users/ConnorDoyle/following{/other_user}","gists_url":"https://api.github.com/users/ConnorDoyle/gists{/gist_id}","starred_url":"https://api.github.com/users/ConnorDoyle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ConnorDoyle/subscriptions","organizations_url":"https://api.github.com/users/ConnorDoyle/orgs","repos_url":"https://api.github.com/users/ConnorDoyle/repos","events_url":"https://api.github.com/users/ConnorDoyle/events{/privacy}","received_events_url":"https://api.github.com/users/ConnorDoyle/received_events","type":"User","site_admin":false},"open_issues":34,"closed_issues":26,"state":"open","created_at":"2014-09-30T17:06:30Z","updated_at":"2015-01-15T11:37:16Z","due_on":"2014-12-23T08:00:00Z","closed_at":null},"comments":3,"created_at":"2015-01-15T06:13:59Z","updated_at":"2015-01-15T13:31:16Z","closed_at":null,"body":"See: https://github.com/mesosphere/marathon/issues/995#issuecomment-69914753\r\n\r\nTried to run docker image that takes 1.8gb on disk. Marathon scheduled task, task moved to state `FAILED` after some time.\r\n\r\nSlave logs:\r\n\r\n```\r\nE0114 13:11:53.822912     7 slave.cpp:2787] Container '0c4d000a-afc3-4deb-aeb6-e3c8b42892b2' for executor 'topface_prod-test_app.c4f59dac-9bee-11e4-a315-56847afe9799' of framework '20150113-074256-3892422848-5050-1-0003' failed to start:  future discarded\r\nE0114 13:11:53.823235     7 slave.cpp:2882] Termination of executor 'topface_prod-test_app.c4f59dac-9bee-11e4-a315-56847afe9799' of framework '20150113-074256-3892422848-5050-1-0003' failed: Unknown container: 0c4d000a-afc3-4deb-aeb6-e3c8b42892b2\r\nE0114 13:11:53.823521    11 slave.cpp:3134] Failed to unmonitor container for executor topface_prod-test_app.c4f59dac-9bee-11e4-a315-56847afe9799 of framework 20150113-074256-3892422848-5050-1-0003: Not monitored\r\nW0114 13:11:53.845448     7 docker.cpp:1184] Ignoring updating unknown container: 0c4d000a-afc3-4deb-aeb6-e3c8b42892b\r\n```\r\n\r\nMarathon logs:\r\n\r\n```\r\n[2015-01-14 13:11:53,924] INFO Received status update for task topface_prod-test_app.c4f59dac-9bee-11e4-a315-56847afe9799: TASK_FAILED (Abnormal executor termination) (mesosphere.marathon.MarathonScheduler:148)\r\n[2015-01-14 13:11:53,929] INFO Task launch delay for [/topface/prod-test/app] is now [1] seconds (mesosphere.util.RateLimiter:34)\r\n[2015-01-14 13:11:53,931] INFO Task topface_prod-test_app.c4f59dac-9bee-11e4-a315-56847afe9799 expunged and removed from TaskTracker (mesosphere.marathon.tasks.TaskTracker:101)\r\n[2015-01-14 13:11:53,931] INFO Sending event notification. (mesosphere.marathon.MarathonScheduler:261)\r\n[2015-01-14 13:11:53,959] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999180 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:53,959] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999519 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:53,960] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999997589 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:53,960] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999563 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:53,960] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999576 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:53,961] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999612 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:53,961] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999572 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:53,961] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999445 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:53,962] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999595 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:53,962] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999641 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:53,962] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999544 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:53,962] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999637 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:53,963] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999477 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:53,963] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999421 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:53,963] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999470 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:53,963] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999548 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:53,964] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999606 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:53,964] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999620 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:53,964] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999601 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:53,965] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999527 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:53,965] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999642 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:53,965] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999645 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:53,965] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999441 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:53,966] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999557 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:53,966] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999601 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:53,966] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999623 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:53,967] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999559 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:53,967] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999454 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:54,759] INFO 172.17.0.29 -  -  [14/Jan/2015:13:11:54 +0000] \"GET /v2/tasks HTTP/1.1\" 200 12 \"-\" \"Go 1.1 package http\" (mesosphere.chaos.http.ChaosRequestLog:15)\r\n[2015-01-14 13:11:55,005] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999072 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:55,005] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999603 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:55,005] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999570 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:55,006] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999523 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:55,006] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999481 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:55,006] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999643 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:55,006] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999635 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:55,007] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999522 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:55,007] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999668 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:55,007] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999649 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:55,008] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999680 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:55,008] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999585 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:55,008] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999650 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:55,008] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999579 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:55,009] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999630 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:55,009] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999550 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:55,009] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999660 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:55,009] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999648 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:55,010] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999656 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:55,010] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999643 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:55,010] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999636 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:55,010] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999595 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:55,011] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999666 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:55,011] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999662 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:55,011] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999663 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:55,011] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999654 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:55,012] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999652 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:55,012] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999646 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:55,012] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999678 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:55,012] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999638 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:55,013] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999662 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:55,013] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999638 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n[2015-01-14 13:11:55,013] INFO Delaying /topface/prod-test/app due to backoff. Time left: 999999590 nanoseconds. (mesosphere.marathon.tasks.TaskQueue:108)\r\n```\r\n\r\nLogs continue to assure me that task is going to be rescheduled in ~1s, but that is not happening:\r\n\r\n```\r\nweb13 ~ # docker logs 4c | fgrep -v 'due to backoff' | fgrep -v v2/tasks | tail -n5\r\n[2015-01-14 13:11:53,929] INFO Task launch delay for [/topface/prod-test/app] is now [1] seconds (mesosphere.util.RateLimiter:34)\r\n[2015-01-14 13:11:53,931] INFO Task topface_prod-test_app.c4f59dac-9bee-11e4-a315-56847afe9799 expunged and removed from TaskTracker (mesosphere.marathon.tasks.TaskTracker:101)\r\n[2015-01-14 13:11:53,931] INFO Sending event notification. (mesosphere.marathon.MarathonScheduler:261)\r\n[2015-01-14 13:16:36,419] INFO Syncing tasks for all apps (mesosphere.marathon.SchedulerActions:397)\r\n[2015-01-14 13:16:36,419] INFO Requesting task reconciliation with the Mesos master (mesosphere.marathon.SchedulerActions:419)\r\n```"},"comment":{"url":"https://api.github.com/repos/mesosphere/marathon/issues/comments/70085413","html_url":"https://github.com/mesosphere/marathon/issues/1005#issuecomment-70085413","issue_url":"https://api.github.com/repos/mesosphere/marathon/issues/1005","id":70085413,"user":{"login":"bobrik","id":89186,"avatar_url":"https://avatars.githubusercontent.com/u/89186?v=3","gravatar_id":"","url":"https://api.github.com/users/bobrik","html_url":"https://github.com/bobrik","followers_url":"https://api.github.com/users/bobrik/followers","following_url":"https://api.github.com/users/bobrik/following{/other_user}","gists_url":"https://api.github.com/users/bobrik/gists{/gist_id}","starred_url":"https://api.github.com/users/bobrik/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bobrik/subscriptions","organizations_url":"https://api.github.com/users/bobrik/orgs","repos_url":"https://api.github.com/users/bobrik/repos","events_url":"https://api.github.com/users/bobrik/events{/privacy}","received_events_url":"https://api.github.com/users/bobrik/received_events","type":"User","site_admin":false},"created_at":"2015-01-15T13:31:16Z","updated_at":"2015-01-15T13:31:16Z","body":"I'm building docker image to test it."}}