{"action":"created","comment":{"url":"https://api.github.com/repos/apache/flink/pulls/comments/41269987","id":41269987,"diff_hunk":"@@ -70,18 +82,65 @@\n \tprivate final int maxConnections;\n \n \t/**\n+\t * Shutdown hook thread to ensure deletion of the storage directory (or <code>null</code> if\n+\t * {@link RecoveryMode#STANDALONE})\n+\t */\n+\tprivate final Thread shutdownHook;\n+\n+\t/**\n \t * Instantiates a new BLOB server and binds it to a free network port.\n \t * \n \t * @throws IOException\n \t *         thrown if the BLOB server cannot bind to a free network port\n \t */\n \tpublic BlobServer(Configuration config) throws IOException {\n+\t\tcheckNotNull(config, \"Configuration\");\n+\n+\t\tthis.recoveryMode = RecoveryMode.fromConfig(config);\n \n \t\t// configure and create the storage directory\n \t\tString storageDirectory = config.getString(ConfigConstants.BLOB_STORAGE_DIRECTORY_KEY, null);\n \t\tthis.storageDir = BlobUtils.initStorageDirectory(storageDirectory);\n \t\tLOG.info(\"Created BLOB server storage directory {}\", storageDir);\n \n+\t\tif (recoveryMode == RecoveryMode.STANDALONE) {\n+\t\t\trecoveryBasePath = null;\n+\t\t}\n+\t\telse {\n+\t\t\t// Initialize file state backend for recovery\n+\t\t\tString stateBackend = config.getString(ConfigConstants.STATE_BACKEND,\n+\t\t\t\t\tConfigConstants.DEFAULT_STATE_BACKEND);\n+\n+\t\t\tif (!stateBackend.toLowerCase().equals(\"filesystem\")) {\n+\t\t\t\tthrow new IllegalConfigurationException(String.format(\"Illegal state backend \" +\n+\t\t\t\t\t\t\t\t\"configuration '%s'. Please configure 'FILESYSTEM' as state \" +\n+\t\t\t\t\t\t\t\t\"backend and specify the recovery path via '%s' key.\",\n+\t\t\t\t\t\tstateBackend, ConfigConstants.STATE_BACKEND_FS_RECOVERY_PATH));\n+\t\t\t}","path":"flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobServer.java","position":98,"original_position":98,"commit_id":"14c90cf1ce427ff88a5b42e1fe3f9cab966352c5","original_commit_id":"00ea6b02f15f00486c9541af08b8197c41dd94f7","user":{"login":"uce","id":1756620,"avatar_url":"https://avatars.githubusercontent.com/u/1756620?v=3","gravatar_id":"","url":"https://api.github.com/users/uce","html_url":"https://github.com/uce","followers_url":"https://api.github.com/users/uce/followers","following_url":"https://api.github.com/users/uce/following{/other_user}","gists_url":"https://api.github.com/users/uce/gists{/gist_id}","starred_url":"https://api.github.com/users/uce/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/uce/subscriptions","organizations_url":"https://api.github.com/users/uce/orgs","repos_url":"https://api.github.com/users/uce/repos","events_url":"https://api.github.com/users/uce/events{/privacy}","received_events_url":"https://api.github.com/users/uce/received_events","type":"User","site_admin":false},"body":"State backend enum is part of the streaming module, which is not a dependency of the runtime atm. I will coordinate with @aljoscha and check what a good place for it is with the streaming rewrite going on.","created_at":"2015-10-06T14:22:05Z","updated_at":"2015-10-06T14:22:05Z","html_url":"https://github.com/apache/flink/pull/1227#discussion_r41269987","pull_request_url":"https://api.github.com/repos/apache/flink/pulls/1227","_links":{"self":{"href":"https://api.github.com/repos/apache/flink/pulls/comments/41269987"},"html":{"href":"https://github.com/apache/flink/pull/1227#discussion_r41269987"},"pull_request":{"href":"https://api.github.com/repos/apache/flink/pulls/1227"}}},"pull_request":{"url":"https://api.github.com/repos/apache/flink/pulls/1227","id":46796105,"html_url":"https://github.com/apache/flink/pull/1227","diff_url":"https://github.com/apache/flink/pull/1227.diff","patch_url":"https://github.com/apache/flink/pull/1227.patch","issue_url":"https://api.github.com/repos/apache/flink/issues/1227","number":1227,"state":"open","locked":false,"title":"[FLINK-2805] [blobmanager] Write JARs to file state backend for recovery","user":{"login":"uce","id":1756620,"avatar_url":"https://avatars.githubusercontent.com/u/1756620?v=3","gravatar_id":"","url":"https://api.github.com/users/uce","html_url":"https://github.com/uce","followers_url":"https://api.github.com/users/uce/followers","following_url":"https://api.github.com/users/uce/following{/other_user}","gists_url":"https://api.github.com/users/uce/gists{/gist_id}","starred_url":"https://api.github.com/users/uce/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/uce/subscriptions","organizations_url":"https://api.github.com/users/uce/orgs","repos_url":"https://api.github.com/users/uce/repos","events_url":"https://api.github.com/users/uce/events{/privacy}","received_events_url":"https://api.github.com/users/uce/received_events","type":"User","site_admin":false},"body":"This is a follow up to #1153. I've taken two changes from #1153 for convenience. Other than that, this is independent.\r\n\r\nWhen running the `BlobServer` in `RecoveryMode#ZOOKEEPER`, this will upload the JARs to the configured file system backend (e.g. HDFS). **Important**: it introduces a hard dependency to have a configured file state backend when running the blob server with recovery. This is in line with #1153.\r\n\r\nThis JAR copying only happens on the server side, e.g. the client uploads to the server and the server uploads it to the state backend. Same when requesting a locally non-existing blob: the client requests from the server and the server downloads if not available and then answers the client.\r\n\r\nThere are other ways to implement this, but this one was minimally invasive and fully circumvents any Akka actor threads for downloading/uploading. A more invasive change could allow to directly interact with the state backend on the client side [task manager] as well. This would spread the load better across the cluster in case of a DFS and save unnecessary network transfers.","created_at":"2015-10-05T15:34:55Z","updated_at":"2015-10-06T14:22:05Z","closed_at":null,"merged_at":null,"merge_commit_sha":"94de28f115493f9588379d3841499201cb3ec214","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/apache/flink/pulls/1227/commits","review_comments_url":"https://api.github.com/repos/apache/flink/pulls/1227/comments","review_comment_url":"https://api.github.com/repos/apache/flink/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/flink/issues/1227/comments","statuses_url":"https://api.github.com/repos/apache/flink/statuses/14c90cf1ce427ff88a5b42e1fe3f9cab966352c5","head":{"label":"uce:hdfs_jars-2805","ref":"hdfs_jars-2805","sha":"14c90cf1ce427ff88a5b42e1fe3f9cab966352c5","user":{"login":"uce","id":1756620,"avatar_url":"https://avatars.githubusercontent.com/u/1756620?v=3","gravatar_id":"","url":"https://api.github.com/users/uce","html_url":"https://github.com/uce","followers_url":"https://api.github.com/users/uce/followers","following_url":"https://api.github.com/users/uce/following{/other_user}","gists_url":"https://api.github.com/users/uce/gists{/gist_id}","starred_url":"https://api.github.com/users/uce/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/uce/subscriptions","organizations_url":"https://api.github.com/users/uce/orgs","repos_url":"https://api.github.com/users/uce/repos","events_url":"https://api.github.com/users/uce/events{/privacy}","received_events_url":"https://api.github.com/users/uce/received_events","type":"User","site_admin":false},"repo":{"id":20699435,"name":"flink","full_name":"uce/flink","owner":{"login":"uce","id":1756620,"avatar_url":"https://avatars.githubusercontent.com/u/1756620?v=3","gravatar_id":"","url":"https://api.github.com/users/uce","html_url":"https://github.com/uce","followers_url":"https://api.github.com/users/uce/followers","following_url":"https://api.github.com/users/uce/following{/other_user}","gists_url":"https://api.github.com/users/uce/gists{/gist_id}","starred_url":"https://api.github.com/users/uce/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/uce/subscriptions","organizations_url":"https://api.github.com/users/uce/orgs","repos_url":"https://api.github.com/users/uce/repos","events_url":"https://api.github.com/users/uce/events{/privacy}","received_events_url":"https://api.github.com/users/uce/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/uce/flink","description":"Mirror of Apache Flink","fork":true,"url":"https://api.github.com/repos/uce/flink","forks_url":"https://api.github.com/repos/uce/flink/forks","keys_url":"https://api.github.com/repos/uce/flink/keys{/key_id}","collaborators_url":"https://api.github.com/repos/uce/flink/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/uce/flink/teams","hooks_url":"https://api.github.com/repos/uce/flink/hooks","issue_events_url":"https://api.github.com/repos/uce/flink/issues/events{/number}","events_url":"https://api.github.com/repos/uce/flink/events","assignees_url":"https://api.github.com/repos/uce/flink/assignees{/user}","branches_url":"https://api.github.com/repos/uce/flink/branches{/branch}","tags_url":"https://api.github.com/repos/uce/flink/tags","blobs_url":"https://api.github.com/repos/uce/flink/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/uce/flink/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/uce/flink/git/refs{/sha}","trees_url":"https://api.github.com/repos/uce/flink/git/trees{/sha}","statuses_url":"https://api.github.com/repos/uce/flink/statuses/{sha}","languages_url":"https://api.github.com/repos/uce/flink/languages","stargazers_url":"https://api.github.com/repos/uce/flink/stargazers","contributors_url":"https://api.github.com/repos/uce/flink/contributors","subscribers_url":"https://api.github.com/repos/uce/flink/subscribers","subscription_url":"https://api.github.com/repos/uce/flink/subscription","commits_url":"https://api.github.com/repos/uce/flink/commits{/sha}","git_commits_url":"https://api.github.com/repos/uce/flink/git/commits{/sha}","comments_url":"https://api.github.com/repos/uce/flink/comments{/number}","issue_comment_url":"https://api.github.com/repos/uce/flink/issues/comments{/number}","contents_url":"https://api.github.com/repos/uce/flink/contents/{+path}","compare_url":"https://api.github.com/repos/uce/flink/compare/{base}...{head}","merges_url":"https://api.github.com/repos/uce/flink/merges","archive_url":"https://api.github.com/repos/uce/flink/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/uce/flink/downloads","issues_url":"https://api.github.com/repos/uce/flink/issues{/number}","pulls_url":"https://api.github.com/repos/uce/flink/pulls{/number}","milestones_url":"https://api.github.com/repos/uce/flink/milestones{/number}","notifications_url":"https://api.github.com/repos/uce/flink/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/uce/flink/labels{/name}","releases_url":"https://api.github.com/repos/uce/flink/releases{/id}","created_at":"2014-06-10T19:56:02Z","updated_at":"2015-06-18T14:47:26Z","pushed_at":"2015-10-06T14:18:05Z","git_url":"git://github.com/uce/flink.git","ssh_url":"git@github.com:uce/flink.git","clone_url":"https://github.com/uce/flink.git","svn_url":"https://github.com/uce/flink","homepage":null,"size":64016,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"4938ff09f692d7b8b1c3af16125d2216eb5c623c","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":20587599,"name":"flink","full_name":"apache/flink","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/flink","description":"Mirror of Apache Flink","fork":false,"url":"https://api.github.com/repos/apache/flink","forks_url":"https://api.github.com/repos/apache/flink/forks","keys_url":"https://api.github.com/repos/apache/flink/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/flink/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/flink/teams","hooks_url":"https://api.github.com/repos/apache/flink/hooks","issue_events_url":"https://api.github.com/repos/apache/flink/issues/events{/number}","events_url":"https://api.github.com/repos/apache/flink/events","assignees_url":"https://api.github.com/repos/apache/flink/assignees{/user}","branches_url":"https://api.github.com/repos/apache/flink/branches{/branch}","tags_url":"https://api.github.com/repos/apache/flink/tags","blobs_url":"https://api.github.com/repos/apache/flink/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/flink/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/flink/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/flink/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/flink/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/flink/languages","stargazers_url":"https://api.github.com/repos/apache/flink/stargazers","contributors_url":"https://api.github.com/repos/apache/flink/contributors","subscribers_url":"https://api.github.com/repos/apache/flink/subscribers","subscription_url":"https://api.github.com/repos/apache/flink/subscription","commits_url":"https://api.github.com/repos/apache/flink/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/flink/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/flink/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/flink/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/flink/contents/{+path}","compare_url":"https://api.github.com/repos/apache/flink/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/flink/merges","archive_url":"https://api.github.com/repos/apache/flink/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/flink/downloads","issues_url":"https://api.github.com/repos/apache/flink/issues{/number}","pulls_url":"https://api.github.com/repos/apache/flink/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/flink/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/flink/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/flink/labels{/name}","releases_url":"https://api.github.com/repos/apache/flink/releases{/id}","created_at":"2014-06-07T07:00:10Z","updated_at":"2015-10-05T15:49:25Z","pushed_at":"2015-10-06T14:18:07Z","git_url":"git://github.com/apache/flink.git","ssh_url":"git@github.com:apache/flink.git","clone_url":"https://github.com/apache/flink.git","svn_url":"https://github.com/apache/flink","homepage":null,"size":2589455,"stargazers_count":657,"watchers_count":657,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":447,"mirror_url":"git://git.apache.org/incubator-flink.git","open_issues_count":74,"forks":447,"open_issues":74,"watchers":657,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/flink/pulls/1227"},"html":{"href":"https://github.com/apache/flink/pull/1227"},"issue":{"href":"https://api.github.com/repos/apache/flink/issues/1227"},"comments":{"href":"https://api.github.com/repos/apache/flink/issues/1227/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/flink/pulls/1227/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/flink/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/flink/pulls/1227/commits"},"statuses":{"href":"https://api.github.com/repos/apache/flink/statuses/14c90cf1ce427ff88a5b42e1fe3f9cab966352c5"}}}}