{"action":"created","comment":{"url":"https://api.github.com/repos/apache/spark/pulls/comments/46336258","id":46336258,"diff_hunk":"@@ -0,0 +1,87 @@\n+/*\n+ * Licensed to the Apache Software Foundation (ASF) under one or more\n+ * contributor license agreements.  See the NOTICE file distributed with\n+ * this work for additional information regarding copyright ownership.\n+ * The ASF licenses this file to You under the Apache License, Version 2.0\n+ * (the \"License\"); you may not use this file except in compliance with\n+ * the License.  You may obtain a copy of the License at\n+ *\n+ *    http://www.apache.org/licenses/LICENSE-2.0\n+ *\n+ * Unless required by applicable law or agreed to in writing, software\n+ * distributed under the License is distributed on an \"AS IS\" BASIS,\n+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n+ * See the License for the specific language governing permissions and\n+ * limitations under the License.\n+ */\n+\n+package org.apache.spark.scheduler.cluster\n+\n+import org.scalatest.BeforeAndAfter\n+\n+import org.apache.spark.{Logging, SparkConf, SparkContext, SparkFunSuite}\n+\n+/**\n+ * Test the integration with [[SchedulerExtensionServices]]\n+ */\n+class ExtensionServiceIntegrationSuite extends SparkFunSuite\n+  with BeforeAndAfter\n+  with Logging {\n+\n+  val applicationId = new StubApplicationId(0, 1111L)\n+  val attemptId = new StubApplicationAttemptId(applicationId, 1)\n+  var sparkCtx: SparkContext = _\n+\n+  /*\n+   * Setup phase creates the spark context\n+   */\n+  before {\n+    val sparkConf = new SparkConf()\n+    sparkConf.set(SchedulerExtensionServices.SPARK_YARN_SERVICES,\n+      classOf[SimpleExtensionService].getName())\n+    sparkConf.setMaster(\"local\").setAppName(\"ExtensionServiceIntegrationSuite\")\n+    sparkCtx = new SparkContext(sparkConf)","path":"yarn/src/test/scala/org/apache/spark/scheduler/cluster/ExtensionServiceIntegrationSuite.scala","position":43,"original_position":43,"commit_id":"3663a9181a6a99e93e5c7a487111b5c105992336","original_commit_id":"3663a9181a6a99e93e5c7a487111b5c105992336","user":{"login":"vanzin","id":1694083,"avatar_url":"https://avatars.githubusercontent.com/u/1694083?v=3","gravatar_id":"","url":"https://api.github.com/users/vanzin","html_url":"https://github.com/vanzin","followers_url":"https://api.github.com/users/vanzin/followers","following_url":"https://api.github.com/users/vanzin/following{/other_user}","gists_url":"https://api.github.com/users/vanzin/gists{/gist_id}","starred_url":"https://api.github.com/users/vanzin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vanzin/subscriptions","organizations_url":"https://api.github.com/users/vanzin/orgs","repos_url":"https://api.github.com/users/vanzin/repos","events_url":"https://api.github.com/users/vanzin/events{/privacy}","received_events_url":"https://api.github.com/users/vanzin/received_events","type":"User","site_admin":false},"body":"You can probably use `LocalSparkContext` instead of rolling your own code for cleaning this up.","created_at":"2015-12-01T20:46:08Z","updated_at":"2015-12-01T20:46:08Z","html_url":"https://github.com/apache/spark/pull/9182#discussion_r46336258","pull_request_url":"https://api.github.com/repos/apache/spark/pulls/9182","_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/comments/46336258"},"html":{"href":"https://github.com/apache/spark/pull/9182#discussion_r46336258"},"pull_request":{"href":"https://api.github.com/repos/apache/spark/pulls/9182"}}},"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/9182","id":48212744,"html_url":"https://github.com/apache/spark/pull/9182","diff_url":"https://github.com/apache/spark/pull/9182.diff","patch_url":"https://github.com/apache/spark/pull/9182.patch","issue_url":"https://api.github.com/repos/apache/spark/issues/9182","number":9182,"state":"open","locked":false,"title":"[SPARK-11314] [YARN] add service API and test service for Yarn Cluster schedulers","user":{"login":"steveloughran","id":162090,"avatar_url":"https://avatars.githubusercontent.com/u/162090?v=3","gravatar_id":"","url":"https://api.github.com/users/steveloughran","html_url":"https://github.com/steveloughran","followers_url":"https://api.github.com/users/steveloughran/followers","following_url":"https://api.github.com/users/steveloughran/following{/other_user}","gists_url":"https://api.github.com/users/steveloughran/gists{/gist_id}","starred_url":"https://api.github.com/users/steveloughran/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/steveloughran/subscriptions","organizations_url":"https://api.github.com/users/steveloughran/orgs","repos_url":"https://api.github.com/users/steveloughran/repos","events_url":"https://api.github.com/users/steveloughran/events{/privacy}","received_events_url":"https://api.github.com/users/steveloughran/received_events","type":"User","site_admin":false},"body":"This is purely the yarn/src/main and yarn/src/test bits of the YARN ATS integration: the extension model to load and run implementations of `SchedulerExtensionService` in the yarn cluster scheduler process —and to stop them afterwards.\r\n\r\nThere's duplication between the two schedulers, yarn-client and yarn-cluster, at least in terms of setting everything up, because the common superclass, `YarnSchedulerBackend` is in spark-core, and the extension services need the YARN app/attempt IDs.\r\n\r\nIf you look at how the the extension services are loaded, the case class `SchedulerExtensionServiceBinding` is used to pass in config info -currently just the spark context and the yarn IDs, of which one, the attemptID, will be null when running client-side. I'm passing in a case class to ensure that it would be possible in future to add extra arguments to the binding class, yet, as the method signature will not have changed, still be able to load existing services.\r\n\r\nThere's no functional extension service here, just one for testing. The real tests come in the bigger pull requests. At the same time, there's no restriction of this extension service purely to the ATS history publisher. Anything else that wants to listen to the spark context and publish events could use this, and I'd also consider writing one for the YARN-913 registry service, so that the URLs of the web UI would be locatable through that (low priority; would make more sense if integrated with a REST client).\r\n\r\nThere's no minicluster test. Given the test execution overhead of setting up minicluster tests, it'd  probably be better to add an extension service into one of the existing tests.","created_at":"2015-10-20T15:01:23Z","updated_at":"2015-12-01T20:46:08Z","closed_at":null,"merged_at":null,"merge_commit_sha":"178f1c45e0af79000e9d384fc3101207fdc691f4","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/apache/spark/pulls/9182/commits","review_comments_url":"https://api.github.com/repos/apache/spark/pulls/9182/comments","review_comment_url":"https://api.github.com/repos/apache/spark/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/spark/issues/9182/comments","statuses_url":"https://api.github.com/repos/apache/spark/statuses/3663a9181a6a99e93e5c7a487111b5c105992336","head":{"label":"steveloughran:stevel/feature/SPARK-1537-service","ref":"stevel/feature/SPARK-1537-service","sha":"3663a9181a6a99e93e5c7a487111b5c105992336","user":{"login":"steveloughran","id":162090,"avatar_url":"https://avatars.githubusercontent.com/u/162090?v=3","gravatar_id":"","url":"https://api.github.com/users/steveloughran","html_url":"https://github.com/steveloughran","followers_url":"https://api.github.com/users/steveloughran/followers","following_url":"https://api.github.com/users/steveloughran/following{/other_user}","gists_url":"https://api.github.com/users/steveloughran/gists{/gist_id}","starred_url":"https://api.github.com/users/steveloughran/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/steveloughran/subscriptions","organizations_url":"https://api.github.com/users/steveloughran/orgs","repos_url":"https://api.github.com/users/steveloughran/repos","events_url":"https://api.github.com/users/steveloughran/events{/privacy}","received_events_url":"https://api.github.com/users/steveloughran/received_events","type":"User","site_admin":false},"repo":{"id":32387210,"name":"spark","full_name":"steveloughran/spark","owner":{"login":"steveloughran","id":162090,"avatar_url":"https://avatars.githubusercontent.com/u/162090?v=3","gravatar_id":"","url":"https://api.github.com/users/steveloughran","html_url":"https://github.com/steveloughran","followers_url":"https://api.github.com/users/steveloughran/followers","following_url":"https://api.github.com/users/steveloughran/following{/other_user}","gists_url":"https://api.github.com/users/steveloughran/gists{/gist_id}","starred_url":"https://api.github.com/users/steveloughran/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/steveloughran/subscriptions","organizations_url":"https://api.github.com/users/steveloughran/orgs","repos_url":"https://api.github.com/users/steveloughran/repos","events_url":"https://api.github.com/users/steveloughran/events{/privacy}","received_events_url":"https://api.github.com/users/steveloughran/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/steveloughran/spark","description":"Mirror of Apache Spark","fork":true,"url":"https://api.github.com/repos/steveloughran/spark","forks_url":"https://api.github.com/repos/steveloughran/spark/forks","keys_url":"https://api.github.com/repos/steveloughran/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/steveloughran/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/steveloughran/spark/teams","hooks_url":"https://api.github.com/repos/steveloughran/spark/hooks","issue_events_url":"https://api.github.com/repos/steveloughran/spark/issues/events{/number}","events_url":"https://api.github.com/repos/steveloughran/spark/events","assignees_url":"https://api.github.com/repos/steveloughran/spark/assignees{/user}","branches_url":"https://api.github.com/repos/steveloughran/spark/branches{/branch}","tags_url":"https://api.github.com/repos/steveloughran/spark/tags","blobs_url":"https://api.github.com/repos/steveloughran/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/steveloughran/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/steveloughran/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/steveloughran/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/steveloughran/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/steveloughran/spark/languages","stargazers_url":"https://api.github.com/repos/steveloughran/spark/stargazers","contributors_url":"https://api.github.com/repos/steveloughran/spark/contributors","subscribers_url":"https://api.github.com/repos/steveloughran/spark/subscribers","subscription_url":"https://api.github.com/repos/steveloughran/spark/subscription","commits_url":"https://api.github.com/repos/steveloughran/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/steveloughran/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/steveloughran/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/steveloughran/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/steveloughran/spark/contents/{+path}","compare_url":"https://api.github.com/repos/steveloughran/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/steveloughran/spark/merges","archive_url":"https://api.github.com/repos/steveloughran/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/steveloughran/spark/downloads","issues_url":"https://api.github.com/repos/steveloughran/spark/issues{/number}","pulls_url":"https://api.github.com/repos/steveloughran/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/steveloughran/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/steveloughran/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/steveloughran/spark/labels{/name}","releases_url":"https://api.github.com/repos/steveloughran/spark/releases{/id}","created_at":"2015-03-17T10:36:33Z","updated_at":"2015-03-17T10:36:52Z","pushed_at":"2015-12-01T20:36:36Z","git_url":"git://github.com/steveloughran/spark.git","ssh_url":"git@github.com:steveloughran/spark.git","clone_url":"https://github.com/steveloughran/spark.git","svn_url":"https://github.com/steveloughran/spark","homepage":null,"size":134265,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"2db4662fe2d72749c06ad5f11f641a388343f77c","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":17165658,"name":"spark","full_name":"apache/spark","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/spark","description":"Mirror of Apache Spark","fork":false,"url":"https://api.github.com/repos/apache/spark","forks_url":"https://api.github.com/repos/apache/spark/forks","keys_url":"https://api.github.com/repos/apache/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/spark/teams","hooks_url":"https://api.github.com/repos/apache/spark/hooks","issue_events_url":"https://api.github.com/repos/apache/spark/issues/events{/number}","events_url":"https://api.github.com/repos/apache/spark/events","assignees_url":"https://api.github.com/repos/apache/spark/assignees{/user}","branches_url":"https://api.github.com/repos/apache/spark/branches{/branch}","tags_url":"https://api.github.com/repos/apache/spark/tags","blobs_url":"https://api.github.com/repos/apache/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/spark/languages","stargazers_url":"https://api.github.com/repos/apache/spark/stargazers","contributors_url":"https://api.github.com/repos/apache/spark/contributors","subscribers_url":"https://api.github.com/repos/apache/spark/subscribers","subscription_url":"https://api.github.com/repos/apache/spark/subscription","commits_url":"https://api.github.com/repos/apache/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/spark/contents/{+path}","compare_url":"https://api.github.com/repos/apache/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/spark/merges","archive_url":"https://api.github.com/repos/apache/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/spark/downloads","issues_url":"https://api.github.com/repos/apache/spark/issues{/number}","pulls_url":"https://api.github.com/repos/apache/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/spark/labels{/name}","releases_url":"https://api.github.com/repos/apache/spark/releases{/id}","created_at":"2014-02-25T08:00:08Z","updated_at":"2015-12-01T20:30:06Z","pushed_at":"2015-12-01T20:36:37Z","git_url":"git://github.com/apache/spark.git","ssh_url":"git@github.com:apache/spark.git","clone_url":"https://github.com/apache/spark.git","svn_url":"https://github.com/apache/spark","homepage":null,"size":388562,"stargazers_count":6412,"watchers_count":6412,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":5922,"mirror_url":"git://git.apache.org/spark.git","open_issues_count":378,"forks":5922,"open_issues":378,"watchers":6412,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/9182"},"html":{"href":"https://github.com/apache/spark/pull/9182"},"issue":{"href":"https://api.github.com/repos/apache/spark/issues/9182"},"comments":{"href":"https://api.github.com/repos/apache/spark/issues/9182/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/spark/pulls/9182/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/spark/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/spark/pulls/9182/commits"},"statuses":{"href":"https://api.github.com/repos/apache/spark/statuses/3663a9181a6a99e93e5c7a487111b5c105992336"}}}}