When the number of tasks are too large in spark-on-yarn-client mode, containers are failed to launch
<p>I'm testing spark-sql on hive running some sophisticated sql statements. When the largest number of my jobs are too large, say over 70000 in my situation, the application will stuck for a while and then throw the following error messages:</p>

<blockquote>
  <p>6/07/06 17:53:20 WARN YarnSchedulerBackend$YarnSchedulerEndpoint:
  Container marked as failed: container_1466092136376_0229_01_000002 on
  host: 10.17.124.127. Exit status: 1. Diagnostics: Exception from
  container-launch. Container id: container_1466092136376_0229_01_000002
  Exit code: 1 Stack trace: ExitCodeException exitCode=1:   at
  org.apache.hadoop.util.Shell.runCommand(Shell.java:545)   at
  org.apache.hadoop.util.Shell.run(Shell.java:456)  at
  org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
    at
  org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
    at
  org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
    at
  org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)     at
  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)</p>
  
  <p>Container exited with a non-zero exit code 1</p>
  
  <p>16/07/06 17:53:20 WARN NettyRpcEndpointRef: Error sending message
  [message = RemoveExecutor(1,Container marked as failed:
  container_1466092136376_0229_01_000002 on host: 10.17.124.127. Exit
  status: 1. Diagnostics: Exception from container-launch. Container id:
  container_1466092136376_0229_01_000002 Exit code: 1 Stack trace:
  ExitCodeException exitCode=1:     at
  org.apache.hadoop.util.Shell.runCommand(Shell.java:545)   at
  org.apache.hadoop.util.Shell.run(Shell.java:456)  at
  org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
    at
  org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
    at
  org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
    at
  org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)     at
  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)</p>
  
  <p>Container exited with a non-zero exit code 1 )] in 1 attempts
  org.apache.spark.SparkException: Could not find CoarseGrainedScheduler
  or it has been stopped.   at
  org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:161)
    at
  org.apache.spark.rpc.netty.Dispatcher.postLocalMessage(Dispatcher.scala:126)
    at org.apache.spark.rpc.netty.NettyRpcEnv.ask(NettyRpcEnv.scala:227)
    at
  org.apache.spark.rpc.netty.NettyRpcEndpointRef.ask(NettyRpcEnv.scala:511)
    at
  org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:100)
    at
  org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:77)
    at
  org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.removeExecutor(CoarseGrainedSchedulerBackend.scala:359)
    at
  org.apache.spark.scheduler.cluster.YarnSchedulerBackend$YarnSchedulerEndpoint$$anonfun$receive$1.applyOrElse(YarnSchedulerBackend.scala:176)
    at
  org.apache.spark.rpc.netty.Inbox$$anonfun$process$1.apply$mcV$sp(Inbox.scala:116)
    at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:204)     at
  org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:100)     at
  org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)
    at
  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at
  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)</p>
</blockquote>

<p>how to avoid the problem?</p>