{"action":"created","issue":{"url":"https://api.github.com/repos/awslabs/emr-bootstrap-actions/issues/38","labels_url":"https://api.github.com/repos/awslabs/emr-bootstrap-actions/issues/38/labels{/name}","comments_url":"https://api.github.com/repos/awslabs/emr-bootstrap-actions/issues/38/comments","events_url":"https://api.github.com/repos/awslabs/emr-bootstrap-actions/issues/38/events","html_url":"https://github.com/awslabs/emr-bootstrap-actions/issues/38","id":54042314,"number":38,"title":"install-spark with -x for 1.2.0.a does not set spark.default.parallelism correctly","user":{"login":"jonchase","id":900627,"avatar_url":"https://avatars.githubusercontent.com/u/900627?v=3","gravatar_id":"","url":"https://api.github.com/users/jonchase","html_url":"https://github.com/jonchase","followers_url":"https://api.github.com/users/jonchase/followers","following_url":"https://api.github.com/users/jonchase/following{/other_user}","gists_url":"https://api.github.com/users/jonchase/gists{/gist_id}","starred_url":"https://api.github.com/users/jonchase/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jonchase/subscriptions","organizations_url":"https://api.github.com/users/jonchase/orgs","repos_url":"https://api.github.com/users/jonchase/repos","events_url":"https://api.github.com/users/jonchase/events{/privacy}","received_events_url":"https://api.github.com/users/jonchase/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-01-12T11:14:27Z","updated_at":"2015-01-14T11:20:29Z","closed_at":null,"body":"When I run s3://support.elasticmapreduce/spark/install-spark with -x, -v, 1.2.0.a, -g, \r\n\r\nMy `/home/hadoop/spark/conf/spark-defaults.conf` looks like: \r\n\r\n```\r\nspark.eventLog.enabled  true\r\nspark.eventLog.dir      hdfs:///spark-logs\r\nspark.executor.extraJavaOptions         -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70\r\nspark.metrics.conf      /home/hadoop/spark/conf/ganglia.metrics.properties\r\nspark.executor.instances\r\nspark.executor.cores\r\nspark.executor.memory   10112M\r\nspark.default.parallelism       *\r\n```\r\n\r\nThe last line contains an invalid value (`spark.default.parallelism       *`).  When I try to run a Spark job, I get:\r\n\r\n```\r\nException in thread \"main\" java.lang.NumberFormatException: For input string: \"*\"\r\n        at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)\r\n        at java.lang.Integer.parseInt(Integer.java:569)\r\n        at java.lang.Integer.parseInt(Integer.java:615)\r\n        at scala.collection.immutable.StringLike$class.toInt(StringLike.scala:229)\r\n        at scala.collection.immutable.StringOps.toInt(StringOps.scala:31)\r\n        at org.apache.spark.SparkConf$$anonfun$getInt$2.apply(SparkConf.scala:184)\r\n        at org.apache.spark.SparkConf$$anonfun$getInt$2.apply(SparkConf.scala:184)\r\n        at scala.Option.map(Option.scala:145)\r\n        at org.apache.spark.SparkConf.getInt(SparkConf.scala:184)\r\n        at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.defaultParallelism(CoarseGrainedSchedulerBackend.scala:271)\r\n        at org.apache.spark.scheduler.TaskSchedulerImpl.defaultParallelism(TaskSchedulerImpl.scala:402)\r\n        at org.apache.spark.SparkContext.defaultParallelism(SparkContext.scala:1455)\r\n        at org.apache.spark.SparkContext.defaultMinPartitions(SparkContext.scala:1462)\r\n        at org.apache.spark.SparkContext.textFile$default$2(SparkContext.scala:539)\r\n        at org.apache.spark.api.java.JavaSparkContext.textFile(JavaSparkContext.scala:184)\r\n```\r\n\r\nIt looks like the issue is in `maximize-spark-default-config`.  When I download it to my master and run it manually, there are some issues:\r\n\r\n```\r\n+ VCOREREFERENCE=http://support.elasticmapreduce.s3.amazonaws.com/spark/vcorereference.tsv\r\n+ CONFIGURESPARK=http://support.elasticmapreduce.s3.amazonaws.com/spark/configure-spark.bash\r\n+ echo 'Configuring Spark default configuration to the max memory and vcore setting given configured number of cores nodes at cluster creation'\r\nConfiguring Spark default configuration to the max memory and vcore setting given configured number of cores nodes at cluster creation\r\n+ /usr/share/aws/emr/scripts/configure-hadoop -y yarn.scheduler.minimum-allocation-mb=256\r\nProcessing default file /home/hadoop/conf/yarn-site.xml with overwrite yarn.scheduler.minimum-allocation-mb=256\r\n'yarn.scheduler.minimum-allocation-mb': new value '256' overwriting '256'\r\nSaved /home/hadoop/conf/yarn-site.xml with overwrites. Original saved to /home/hadoop/conf/yarn-site.xml.old\r\n++ grep /mnt/var/lib/info/job-flow.json -e CORE -A 4\r\n++ grep -e requestedInstanceCount\r\n++ cut -d: -f2\r\n++ sed 's/\\s\\+//g'\r\n+ NUM_CORE_NODES=\r\n+ '[' -lt 2 ']'\r\n./f: line 27: [: -lt: unary operator expected\r\n++ grep /mnt/var/lib/info/job-flow.json -e CORE -A 4\r\n++ grep -e instanceType\r\n++ cut '-d\"' -f4\r\n++ sed 's/\\s\\+//g'\r\n+ CORE_INSTANCE_TYPE=\r\n+ '[' == '' ']'\r\n./f: line 35: [: ==: unary operator expected\r\n+ wget http://support.elasticmapreduce.s3.amazonaws.com/spark/vcorereference.tsv\r\n--2015-01-12 10:58:39--  http://support.elasticmapreduce.s3.amazonaws.com/spark/vcorereference.tsv\r\nResolving support.elasticmapreduce.s3.amazonaws.com (support.elasticmapreduce.s3.amazonaws.com)... 54.231.32.217\r\nConnecting to support.elasticmapreduce.s3.amazonaws.com (support.elasticmapreduce.s3.amazonaws.com)|54.231.32.217|:80... connected.\r\nHTTP request sent, awaiting response... 200 OK\r\nLength: 470 [text/tab-separated-values]\r\nSaving to: ‘vcorereference.tsv.3’\r\n\r\nvcorereference.tsv.3                                            100%[========================================================================================================================================================>]     470  --.-KB/s   in 0s\r\n\r\n2015-01-12 10:58:40 (57.0 MB/s) - ‘vcorereference.tsv.3’ saved [470/470]\r\n\r\n++ grep vcorereference.tsv -e\r\n++ cut -f2\r\ngrep: option requires an argument -- 'e'\r\nUsage: grep [OPTION]... PATTERN [FILE]...\r\nTry `grep --help' for more information.\r\n+ NUM_VCORES=\r\n++ grep /home/hadoop/conf/yarn-site.xml -e 'yarn\\.scheduler\\.maximum-allocation-mb'\r\n++ sed 's/.*<value>\\(.*\\).*<\\/value>.*/\\1/g'\r\n+ MAX_YARN_MEMORY=11520\r\n++ expr 11520 - 1024 - 384\r\n+ EXEC_MEMORY=10112\r\n+ EXEC_MEMORY+=M\r\n+ echo 'num vcores '\r\nnum vcores\r\n+ echo 'num core nodes '\r\nnum core nodes\r\n++ expr '*'\r\n+ PARALLEL='*'\r\n+ echo 'parallel *'\r\nparallel *\r\n+ wget http://support.elasticmapreduce.s3.amazonaws.com/spark/configure-spark.bash\r\n--2015-01-12 10:58:40--  http://support.elasticmapreduce.s3.amazonaws.com/spark/configure-spark.bash\r\nResolving support.elasticmapreduce.s3.amazonaws.com (support.elasticmapreduce.s3.amazonaws.com)... 54.231.32.217\r\nConnecting to support.elasticmapreduce.s3.amazonaws.com (support.elasticmapreduce.s3.amazonaws.com)|54.231.32.217|:80... connected.\r\nHTTP request sent, awaiting response... 200 OK\r\nLength: 584 [binary/octet-stream]\r\nSaving to: ‘configure-spark.bash.3’\r\n\r\nconfigure-spark.bash.3                                          100%[========================================================================================================================================================>]     584  --.-KB/s   in 0s\r\n\r\n2015-01-12 10:58:40 (69.6 MB/s) - ‘configure-spark.bash.3’ saved [584/584]\r\n\r\n+ exit 0\r\n```\r\n\r\n(btw, I commented out the call to configure-spark.bash at the end).\r\n\r\nIt looks like:\r\n* `NUM_CORE_NODES` is set to empty string (expected?)\r\n* syntax error on  `'[' -lt 2 ']'`\r\n    `line 27: [: -lt: unary operator expected`\r\n* another on `+ '[' == '' ']'`\r\n   `line 35: [: ==: unary operator expected`\r\n* `NUM_VCORES` is an empty string (expected?)\r\n* `PARALLEL` is set to `'*'` - this is what is causing my current issue\r\n\r\nI can hack this for now, but it'd be great to see a fix this pushed out (or clarification if I'm doing something wrong).  "},"comment":{"url":"https://api.github.com/repos/awslabs/emr-bootstrap-actions/issues/comments/69902601","html_url":"https://github.com/awslabs/emr-bootstrap-actions/issues/38#issuecomment-69902601","issue_url":"https://api.github.com/repos/awslabs/emr-bootstrap-actions/issues/38","id":69902601,"user":{"login":"jonchase","id":900627,"avatar_url":"https://avatars.githubusercontent.com/u/900627?v=3","gravatar_id":"","url":"https://api.github.com/users/jonchase","html_url":"https://github.com/jonchase","followers_url":"https://api.github.com/users/jonchase/followers","following_url":"https://api.github.com/users/jonchase/following{/other_user}","gists_url":"https://api.github.com/users/jonchase/gists{/gist_id}","starred_url":"https://api.github.com/users/jonchase/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jonchase/subscriptions","organizations_url":"https://api.github.com/users/jonchase/orgs","repos_url":"https://api.github.com/users/jonchase/repos","events_url":"https://api.github.com/users/jonchase/events{/privacy}","received_events_url":"https://api.github.com/users/jonchase/received_events","type":"User","site_admin":false},"created_at":"2015-01-14T11:20:29Z","updated_at":"2015-01-14T11:20:29Z","body":"This works on 3.3.1 for NUM_CORE_NODES:\r\n\r\n  \"instanceGroupName\": \"core\",\r\n      \"instanceRole\": \"Core\",\r\n      \"marketType\": \"Spot\",\r\n      \"instanceType\": \"m3.2xlarge\",\r\n      \"requestedInstanceCount\": 10,\r\n      \"bidPrice\": \".25\"\r\n\r\n\r\ngrep /mnt/var/lib/info/job-flow.json -i -e \"CORE\" -A 4 | grep -e requestedInstanceCount | cut -d':' -f2 | sed  's/\\s\\+//g' | cut -d',' -f1\r\n10"}}