{"action":"created","issue":{"url":"https://api.github.com/repos/alibaba/mpich2-yarn/issues/37","labels_url":"https://api.github.com/repos/alibaba/mpich2-yarn/issues/37/labels{/name}","comments_url":"https://api.github.com/repos/alibaba/mpich2-yarn/issues/37/comments","events_url":"https://api.github.com/repos/alibaba/mpich2-yarn/issues/37/events","html_url":"https://github.com/alibaba/mpich2-yarn/issues/37","id":45976613,"number":37,"title":"error in AppMaster.stderr","user":{"login":"schmidb","id":2534842,"avatar_url":"https://avatars.githubusercontent.com/u/2534842?v=3","gravatar_id":"","url":"https://api.github.com/users/schmidb","html_url":"https://github.com/schmidb","followers_url":"https://api.github.com/users/schmidb/followers","following_url":"https://api.github.com/users/schmidb/following{/other_user}","gists_url":"https://api.github.com/users/schmidb/gists{/gist_id}","starred_url":"https://api.github.com/users/schmidb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/schmidb/subscriptions","organizations_url":"https://api.github.com/users/schmidb/orgs","repos_url":"https://api.github.com/users/schmidb/repos","events_url":"https://api.github.com/users/schmidb/events{/privacy}","received_events_url":"https://api.github.com/users/schmidb/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2014-10-16T11:47:58Z","updated_at":"2015-11-07T14:59:56Z","closed_at":null,"body":"Hi,\r\n\r\nnow I have time again to test mpich2-yarn on Amazon EMR. I have the following problem\r\n\r\nAppMaster.stderr : Total file length is 11170 bytes.\r\n\r\n...\r\n14/10/16 11:43:55 INFO server.ApplicationMaster: Setting up container command\r\n14/10/16 11:43:55 INFO server.ApplicationMaster: Executing command: [${JAVA_HOME}/bin/java -Xmx1024m org.apache.hadoop.yarn.mpi.server.Container 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr ]\r\n14/10/16 11:43:55 INFO impl.ContainerManagementProtocolProxy: Opening proxy : ip-172-31-42-163.ec2.internal:9103\r\n14/10/16 11:43:55 INFO impl.NMClientAsyncImpl: Processing Event EventType: START_CONTAINER for Container container_1413458972800_0006_01_000003\r\n14/10/16 11:43:55 INFO impl.ContainerManagementProtocolProxy: Opening proxy : ip-172-31-42-165.ec2.internal:9103\r\n14/10/16 11:43:55 INFO handler.MPINMAsyncHandler: onContainerStarted invoked.\r\n14/10/16 11:43:55 INFO handler.MPINMAsyncHandler: onContainerStarted invoked.\r\n14/10/16 11:43:57 INFO server.MPDListenerImpl: Try to report status.\r\n14/10/16 11:43:57 INFO server.MPDListenerImpl: container_1413458972800_0006_01_000003 report status INITIALIZED\r\n14/10/16 11:43:57 INFO server.MPDListenerImpl: Try to report status.\r\n14/10/16 11:43:57 INFO server.MPDListenerImpl: container_1413458972800_0006_01_000002 report status INITIALIZED\r\n14/10/16 11:43:57 INFO server.MPDListenerImpl: Try to report status.\r\n14/10/16 11:43:57 INFO server.MPDListenerImpl: container_1413458972800_0006_01_000003 report status ERROR_FINISHED\r\n14/10/16 11:43:58 ERROR server.ApplicationMaster: error occurs while starting MPD\r\norg.apache.hadoop.yarn.mpi.util.MPDException: Container container_1413458972800_0006_01_000003 error\r\n\tat org.apache.hadoop.yarn.mpi.server.MPDListenerImpl.isAllMPDStarted(MPDListenerImpl.java:121)\r\n\tat org.apache.hadoop.yarn.mpi.server.ApplicationMaster.run(ApplicationMaster.java:733)\r\n\tat org.apache.hadoop.yarn.mpi.server.ApplicationMaster.main(ApplicationMaster.java:170)\r\norg.apache.hadoop.yarn.mpi.util.MPDException: Container container_1413458972800_0006_01_000003 error\r\n\tat org.apache.hadoop.yarn.mpi.server.MPDListenerImpl.isAllMPDStarted(MPDListenerImpl.java:121)\r\n\tat org.apache.hadoop.yarn.mpi.server.ApplicationMaster.run(ApplicationMaster.java:733)\r\n\tat org.apache.hadoop.yarn.mpi.server.ApplicationMaster.main(ApplicationMaster.java:170)\r\n14/10/16 11:43:58 INFO server.ApplicationMaster: Application completed. Stopping running containers\r\n14/10/16 11:43:58 INFO impl.ContainerManagementProtocolProxy: Closing proxy : ip-172-31-42-163.ec2.internal:9103\r\n14/10/16 11:43:58 INFO impl.ContainerManagementProtocolProxy: Closing proxy : ip-172-31-42-165.ec2.internal:9103\r\n14/10/16 11:43:58 INFO server.ApplicationMaster: Application completed. Signalling finish to RM\r\n14/10/16 11:43:58 INFO impl.AMRMClientImpl: Waiting for application to be successfully unregistered.\r\n14/10/16 11:43:58 INFO server.ApplicationMaster: AMRM, NM two services stopped\r\n14/10/16 11:43:58 INFO impl.AMRMClientAsyncImpl: Interrupted while waiting for queue\r\njava.lang.InterruptedException\r\n\tat java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)\r\n\tat java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052)\r\n\tat java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)\r\n\tat org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:275)\r\n14/10/16 11:43:58 INFO server.ApplicationMaster: Finalizing.\r\n14/10/16 11:43:58 INFO server.ApplicationMaster: Application Master failed. exiting\r\n...\r\n\r\nAny idea how to fix that? Key-less ssh to nodes now works.\r\n\r\nThanks\r\nMarkus"},"comment":{"url":"https://api.github.com/repos/alibaba/mpich2-yarn/issues/comments/154712763","html_url":"https://github.com/alibaba/mpich2-yarn/issues/37#issuecomment-154712763","issue_url":"https://api.github.com/repos/alibaba/mpich2-yarn/issues/37","id":154712763,"user":{"login":"hadimansouri","id":8458562,"avatar_url":"https://avatars.githubusercontent.com/u/8458562?v=3","gravatar_id":"","url":"https://api.github.com/users/hadimansouri","html_url":"https://github.com/hadimansouri","followers_url":"https://api.github.com/users/hadimansouri/followers","following_url":"https://api.github.com/users/hadimansouri/following{/other_user}","gists_url":"https://api.github.com/users/hadimansouri/gists{/gist_id}","starred_url":"https://api.github.com/users/hadimansouri/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hadimansouri/subscriptions","organizations_url":"https://api.github.com/users/hadimansouri/orgs","repos_url":"https://api.github.com/users/hadimansouri/repos","events_url":"https://api.github.com/users/hadimansouri/events{/privacy}","received_events_url":"https://api.github.com/users/hadimansouri/received_events","type":"User","site_admin":false},"created_at":"2015-11-07T14:59:56Z","updated_at":"2015-11-07T14:59:56Z","body":"hi\r\nIs there any progress on this issue?\r\nI have same problem. Thanks to help me!"}}