{"action":"created","issue":{"url":"https://api.github.com/repos/bloomberg/chef-bach/issues/200","labels_url":"https://api.github.com/repos/bloomberg/chef-bach/issues/200/labels{/name}","comments_url":"https://api.github.com/repos/bloomberg/chef-bach/issues/200/comments","events_url":"https://api.github.com/repos/bloomberg/chef-bach/issues/200/events","html_url":"https://github.com/bloomberg/chef-bach/pull/200","id":96256425,"number":200,"title":"Spark Fixes","user":{"login":"cbaenziger","id":4192761,"avatar_url":"https://avatars.githubusercontent.com/u/4192761?v=3","gravatar_id":"","url":"https://api.github.com/users/cbaenziger","html_url":"https://github.com/cbaenziger","followers_url":"https://api.github.com/users/cbaenziger/followers","following_url":"https://api.github.com/users/cbaenziger/following{/other_user}","gists_url":"https://api.github.com/users/cbaenziger/gists{/gist_id}","starred_url":"https://api.github.com/users/cbaenziger/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cbaenziger/subscriptions","organizations_url":"https://api.github.com/users/cbaenziger/orgs","repos_url":"https://api.github.com/users/cbaenziger/repos","events_url":"https://api.github.com/users/cbaenziger/events{/privacy}","received_events_url":"https://api.github.com/users/cbaenziger/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/bloomberg/chef-bach/labels/Breaking","name":"Breaking","color":"000000"},{"url":"https://api.github.com/repos/bloomberg/chef-bach/labels/Ready%20to%20Review","name":"Ready to Review","color":"cc317c"},{"url":"https://api.github.com/repos/bloomberg/chef-bach/labels/Spark","name":"Spark","color":"fad8c7"},{"url":"https://api.github.com/repos/bloomberg/chef-bach/labels/YARN","name":"YARN","color":"0052cc"}],"state":"open","locked":false,"assignee":{"login":"cbaenziger","id":4192761,"avatar_url":"https://avatars.githubusercontent.com/u/4192761?v=3","gravatar_id":"","url":"https://api.github.com/users/cbaenziger","html_url":"https://github.com/cbaenziger","followers_url":"https://api.github.com/users/cbaenziger/followers","following_url":"https://api.github.com/users/cbaenziger/following{/other_user}","gists_url":"https://api.github.com/users/cbaenziger/gists{/gist_id}","starred_url":"https://api.github.com/users/cbaenziger/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cbaenziger/subscriptions","organizations_url":"https://api.github.com/users/cbaenziger/orgs","repos_url":"https://api.github.com/users/cbaenziger/repos","events_url":"https://api.github.com/users/cbaenziger/events{/privacy}","received_events_url":"https://api.github.com/users/cbaenziger/received_events","type":"User","site_admin":false},"milestone":null,"comments":11,"created_at":"2015-07-21T08:15:17Z","updated_at":"2015-10-09T20:00:09Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/bloomberg/chef-bach/pulls/200","html_url":"https://github.com/bloomberg/chef-bach/pull/200","diff_url":"https://github.com/bloomberg/chef-bach/pull/200.diff","patch_url":"https://github.com/bloomberg/chef-bach/pull/200.patch"},"body":"This provides an update to how we deploy Spark with the following fixes:\r\n\r\n**Update SPARK environment for SPARK-8332 and for LZO use**:\r\n* Fix for [SPARK-8332](https://issues.apache.org/jira/browse/SPARK-8332) -- simply do not include Jackson JARs in the `CLASSPATH`\r\n* Include an `LD_LIBRARY_PATH` directive for Spark to find LZO and not fatally bail out (replaced in later commit with a `spark-defaults.conf`)\r\n* Include `SPARK_LOCAL_IP` and `SPARK_PUBLIC_DNS` to inform the machine to what it should bind and the hostname which it should advertise\r\n\r\n**Add in Spark dynamic allocation and logging for eventual Spark history server use**:\r\n* Create a `spark-defaults.conf` with some reasonable defaults\r\n* Create an HDFS `/spark-history` directory for eventual Spark History Server use\r\n* Use `spark.{driver,executor}.extraLibraryPath` for necessary Java libraries instead of `LD_LIBRARY_PATH`\r\n* Add in some GC logging for executors\r\n* Consolidate shuffle files since we use XFS by default\r\n* Increase the number of connections per host to a reasonable number for our default cluster sizes\r\n\r\n**Add in cgroups for YARN**:\r\n* Configured per Apache Hadoop [documentation](http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/NodeManagerCgroups.html)\r\n* Verified by ensuring containers has entries under `/sys/fs/cgroup/cpu/hadoop-yarn/<container ID>`\r\n* Does not set `yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage` which is useful for benchmarking or deterministically long running processes\r\n\r\n**Enables long-running job log aggregation**:\r\n* Set rolling log aggregation to 30 minutes\r\n* Set deletion of old logs to 31 days\r\n\r\n**Enables dominant resource fairness**:\r\n* ~~Configures fair-scheduler to use `drf` instead of `fair` scheduling algorithm~~\r\n* Sets YARN to use cgroups to keep absolute percentage of CPU usage at `node['bcpc']['hadoop']['yarn']['nodemanager']['avail_vcpu']['ratio']` while advertising desired number of vCPU's; one can still set either `node['bcpc']['hadoop']['yarn']['nodemanager']['avail_vcpu']['ratio']` and/or `node['bcpc']['hadoop']['yarn']['nodemanager']['avail_vcpu']['count']` and get working behavior calculated using `node['cpu']['total']`.\r\n\r\n**__BREAKING__: Changes `rm_port`**:\r\n* To normalize attributes into `nodemanager` and `resourcemanager` sections in the attributes file:\r\n  ````\r\n  default[\"bcpc\"][\"hadoop\"][\"yarn\"][\"rm_port\"] = 8032\r\n  ````\r\n  Changed to **breaking** attribute:\r\n  ````\r\n  default[\"bcpc\"][\"hadoop\"][\"yarn\"][\"resourcemanager\"][\"port\"] = 8032\r\n  ````"},"comment":{"url":"https://api.github.com/repos/bloomberg/chef-bach/issues/comments/146973674","html_url":"https://github.com/bloomberg/chef-bach/pull/200#issuecomment-146973674","issue_url":"https://api.github.com/repos/bloomberg/chef-bach/issues/200","id":146973674,"user":{"login":"kiiranh","id":4454094,"avatar_url":"https://avatars.githubusercontent.com/u/4454094?v=3","gravatar_id":"","url":"https://api.github.com/users/kiiranh","html_url":"https://github.com/kiiranh","followers_url":"https://api.github.com/users/kiiranh/followers","following_url":"https://api.github.com/users/kiiranh/following{/other_user}","gists_url":"https://api.github.com/users/kiiranh/gists{/gist_id}","starred_url":"https://api.github.com/users/kiiranh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kiiranh/subscriptions","organizations_url":"https://api.github.com/users/kiiranh/orgs","repos_url":"https://api.github.com/users/kiiranh/repos","events_url":"https://api.github.com/users/kiiranh/events{/privacy}","received_events_url":"https://api.github.com/users/kiiranh/received_events","type":"User","site_admin":false},"created_at":"2015-10-09T20:00:09Z","updated_at":"2015-10-09T20:00:09Z","body":" Thanks @cbaenziger. I guess this is good to mainstream"}}