{"action":"created","comment":{"url":"https://api.github.com/repos/druid-io/druid/pulls/comments/37663246","id":37663246,"diff_hunk":"@@ -13,22 +13,137 @@ Druid extensions can be specified in the `common.runtime.properties`. There are\n \n If you add your extension jar to the classpath at runtime, Druid will load it into the system.  This mechanism is relatively easy to reason about, but it also means that you have to ensure that all dependency jars on the classpath are compatible.  That is, Druid makes no provisions while using this method to maintain class loader isolation so you must make sure that the jars on your classpath are mutually compatible.\n \n-### Specify maven coordinates\n+### Add to the extension directory\n \n-Druid has the ability to automatically load extension jars from maven at runtime.  With this mechanism, Druid also loads up the dependencies of the extension jar into an isolated class loader.  That means that your extension can depend on a different version of a library that Druid also uses and both can co-exist.\n+If you don't want to fiddle with classpath, you can create an extension directory and tell Druid to load extensions from there.\n \n-### I want classloader isolation, but I don't want my production machines downloading their own dependencies.  What should I do?\n+To let Druid load your extensions, follow the steps below\n \n-If you want to take advantage of the maven-based classloader isolation but you are also rightly frightened by the prospect of each of your production machines downloading their own dependencies on deploy, this section is for you.\n+1) Specify `druid.extensions.extensionsDir` (root directory for normal Druid extensions) and `druid.extensions.hadoopDependenciesDir` (root directory for hadoop related dependencies). If you don' specify these properties, Druid will use their default values, see [Configuration](../configuration/index.html).\n \n-The trick to doing this is\n+2) Prepare normal extension directories under root extension directory.  Under the root extension directory, you should create sub-directories for each extension you might want to load.  Inside each sub-directory, you can put extension related files in it.\n \n-1) Specify a local directory for `druid.extensions.localRepository`\n+Example:\n \n-2) Run the `tools pull-deps` command to pull all the specified dependencies down into your local repository\n+Suppose you specify `druid.extensions.extensionsDir=/usr/local/druid/druid_extensions`, and your Druid might load normal extensions ```druid-examples```, ```druid-kafka-eight``` and ```mysql-metadata-storage```.\n \n-3) Bundle up the local repository along with your other Druid stuff into whatever you use for a deployable artifact\n+Then under ```druid_extensions```, it should look like this,\n \n-4) Run Your druid processes with `druid.extensions.remoteRepositories=[]` and a local repository set to wherever your bundled \"local\" repository is located\n+```\n+druid_extensions/\n+├── druid-examples\n+│   ├── commons-beanutils-1.8.3.jar\n+│   ├── commons-digester-1.8.jar\n+│   ├── commons-logging-1.1.1.jar\n+│   ├── commons-validator-1.4.0.jar\n+│   ├── druid-examples-0.8.0-rc1.jar\n+│   ├── twitter4j-async-3.0.3.jar\n+│   ├── twitter4j-core-3.0.3.jar\n+│   └── twitter4j-stream-3.0.3.jar\n+├── druid-kafka-eight\n+│   ├── druid-kafka-eight-0.7.3.jar\n+│   ├── jline-0.9.94.jar\n+│   ├── jopt-simple-3.2.jar\n+│   ├── kafka-clients-0.8.2.1.jar\n+│   ├── kafka_2.10-0.8.2.1.jar\n+│   ├── log4j-1.2.16.jar\n+│   ├── lz4-1.3.0.jar\n+│   ├── metrics-core-2.2.0.jar\n+│   ├── netty-3.7.0.Final.jar\n+│   ├── scala-library-2.10.4.jar\n+│   ├── slf4j-log4j12-1.6.1.jar\n+│   ├── snappy-java-1.1.1.6.jar\n+│   ├── zkclient-0.3.jar\n+│   └── zookeeper-3.4.6.jar\n+└── mysql-metadata-storage\n+    ├── jdbi-2.32.jar\n+    ├── mysql-connector-java-5.1.34.jar\n+    └── mysql-metadata-storage-0.8.0-rc1.jar\n+```\n \n-The Druid processes will then only load up jars from the local repository and will not try to go out onto the internet to find the maven dependencies.\n+As you can see, there are three sub-directories ```druid-examples```, ```druid-kafka-eight``` and ```mysql-metadata-storage``` under ```druid_extensions```, each sub-directory denotes an extension that Druid might load.\n+\n+3) Prepare hadoop dependencies directories under root hadoop dependency directory. The file structure for hadoop dependencies is almost same as normal Druid extensions described above, except that there is an extra layer of folder that specifies which version of hadoop it contains. \n+\n+Example:\n+\n+Suppose you specify `druid.extensions.hadoopDependenciesDir=/usr/local/druid/hadoop_druid_dependencies`, and you want to prepare both `hadoop-client` 2.3.0 and 2.4.0,\n+\n+Then under ```hadoop_druid_dependencies```, it should look like this,","path":"docs/content/operations/including-extensions.md","position":70,"original_position":70,"commit_id":"4450881efb1a139bf592f690ca7c2c249223ad07","original_commit_id":"4450881efb1a139bf592f690ca7c2c249223ad07","user":{"login":"fjy","id":428325,"avatar_url":"https://avatars.githubusercontent.com/u/428325?v=3","gravatar_id":"","url":"https://api.github.com/users/fjy","html_url":"https://github.com/fjy","followers_url":"https://api.github.com/users/fjy/followers","following_url":"https://api.github.com/users/fjy/following{/other_user}","gists_url":"https://api.github.com/users/fjy/gists{/gist_id}","starred_url":"https://api.github.com/users/fjy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fjy/subscriptions","organizations_url":"https://api.github.com/users/fjy/orgs","repos_url":"https://api.github.com/users/fjy/repos","events_url":"https://api.github.com/users/fjy/events{/privacy}","received_events_url":"https://api.github.com/users/fjy/received_events","type":"User","site_admin":false},"body":"we'll need to update the docs around how to get Druid working with different versions of Hadoop","created_at":"2015-08-21T18:27:36Z","updated_at":"2015-08-21T18:27:36Z","html_url":"https://github.com/druid-io/druid/pull/1638#discussion_r37663246","pull_request_url":"https://api.github.com/repos/druid-io/druid/pulls/1638","_links":{"self":{"href":"https://api.github.com/repos/druid-io/druid/pulls/comments/37663246"},"html":{"href":"https://github.com/druid-io/druid/pull/1638#discussion_r37663246"},"pull_request":{"href":"https://api.github.com/repos/druid-io/druid/pulls/1638"}}},"pull_request":{"url":"https://api.github.com/repos/druid-io/druid/pulls/1638","id":42724507,"html_url":"https://github.com/druid-io/druid/pull/1638","diff_url":"https://github.com/druid-io/druid/pull/1638.diff","patch_url":"https://github.com/druid-io/druid/pull/1638.patch","issue_url":"https://api.github.com/repos/druid-io/druid/issues/1638","number":1638,"state":"open","locked":false,"title":"Remove Maven client at runtime + Provide a way to load Druid extensions through local file system","user":{"login":"guobingkun","id":11789051,"avatar_url":"https://avatars.githubusercontent.com/u/11789051?v=3","gravatar_id":"","url":"https://api.github.com/users/guobingkun","html_url":"https://github.com/guobingkun","followers_url":"https://api.github.com/users/guobingkun/followers","following_url":"https://api.github.com/users/guobingkun/following{/other_user}","gists_url":"https://api.github.com/users/guobingkun/gists{/gist_id}","starred_url":"https://api.github.com/users/guobingkun/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/guobingkun/subscriptions","organizations_url":"https://api.github.com/users/guobingkun/orgs","repos_url":"https://api.github.com/users/guobingkun/repos","events_url":"https://api.github.com/users/guobingkun/events{/privacy}","received_events_url":"https://api.github.com/users/guobingkun/received_events","type":"User","site_admin":false},"body":"This is a follow up of https://groups.google.com/forum/#!searchin/druid-development/proposal/druid-development/7KJCsQ9GvGo/u3XMyNLmeAIJ\r\n\r\nThis pull request does the following, \r\n1) Remove maven client from downloading extensions at runtime\r\n2) Provide a way to load Druid extensions and hadoop dependencies through file system.\r\n3) Refactor pull-deps so that it can download extensions into extension directories.\r\n4) Add documents on how to use this new extension loading mechanism\r\n\r\n","created_at":"2015-08-18T17:17:55Z","updated_at":"2015-08-21T18:27:36Z","closed_at":null,"merged_at":null,"merge_commit_sha":"bfb83dd489ac67b2ce419d599dd5fdc3807f964f","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/druid-io/druid/pulls/1638/commits","review_comments_url":"https://api.github.com/repos/druid-io/druid/pulls/1638/comments","review_comment_url":"https://api.github.com/repos/druid-io/druid/pulls/comments{/number}","comments_url":"https://api.github.com/repos/druid-io/druid/issues/1638/comments","statuses_url":"https://api.github.com/repos/druid-io/druid/statuses/4450881efb1a139bf592f690ca7c2c249223ad07","head":{"label":"guobingkun:remove_maven_client_code","ref":"remove_maven_client_code","sha":"4450881efb1a139bf592f690ca7c2c249223ad07","user":{"login":"guobingkun","id":11789051,"avatar_url":"https://avatars.githubusercontent.com/u/11789051?v=3","gravatar_id":"","url":"https://api.github.com/users/guobingkun","html_url":"https://github.com/guobingkun","followers_url":"https://api.github.com/users/guobingkun/followers","following_url":"https://api.github.com/users/guobingkun/following{/other_user}","gists_url":"https://api.github.com/users/guobingkun/gists{/gist_id}","starred_url":"https://api.github.com/users/guobingkun/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/guobingkun/subscriptions","organizations_url":"https://api.github.com/users/guobingkun/orgs","repos_url":"https://api.github.com/users/guobingkun/repos","events_url":"https://api.github.com/users/guobingkun/events{/privacy}","received_events_url":"https://api.github.com/users/guobingkun/received_events","type":"User","site_admin":false},"repo":{"id":33376574,"name":"druid","full_name":"guobingkun/druid","owner":{"login":"guobingkun","id":11789051,"avatar_url":"https://avatars.githubusercontent.com/u/11789051?v=3","gravatar_id":"","url":"https://api.github.com/users/guobingkun","html_url":"https://github.com/guobingkun","followers_url":"https://api.github.com/users/guobingkun/followers","following_url":"https://api.github.com/users/guobingkun/following{/other_user}","gists_url":"https://api.github.com/users/guobingkun/gists{/gist_id}","starred_url":"https://api.github.com/users/guobingkun/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/guobingkun/subscriptions","organizations_url":"https://api.github.com/users/guobingkun/orgs","repos_url":"https://api.github.com/users/guobingkun/repos","events_url":"https://api.github.com/users/guobingkun/events{/privacy}","received_events_url":"https://api.github.com/users/guobingkun/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/guobingkun/druid","description":"Real²time Exploratory Analytics on Large Datasets","fork":true,"url":"https://api.github.com/repos/guobingkun/druid","forks_url":"https://api.github.com/repos/guobingkun/druid/forks","keys_url":"https://api.github.com/repos/guobingkun/druid/keys{/key_id}","collaborators_url":"https://api.github.com/repos/guobingkun/druid/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/guobingkun/druid/teams","hooks_url":"https://api.github.com/repos/guobingkun/druid/hooks","issue_events_url":"https://api.github.com/repos/guobingkun/druid/issues/events{/number}","events_url":"https://api.github.com/repos/guobingkun/druid/events","assignees_url":"https://api.github.com/repos/guobingkun/druid/assignees{/user}","branches_url":"https://api.github.com/repos/guobingkun/druid/branches{/branch}","tags_url":"https://api.github.com/repos/guobingkun/druid/tags","blobs_url":"https://api.github.com/repos/guobingkun/druid/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/guobingkun/druid/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/guobingkun/druid/git/refs{/sha}","trees_url":"https://api.github.com/repos/guobingkun/druid/git/trees{/sha}","statuses_url":"https://api.github.com/repos/guobingkun/druid/statuses/{sha}","languages_url":"https://api.github.com/repos/guobingkun/druid/languages","stargazers_url":"https://api.github.com/repos/guobingkun/druid/stargazers","contributors_url":"https://api.github.com/repos/guobingkun/druid/contributors","subscribers_url":"https://api.github.com/repos/guobingkun/druid/subscribers","subscription_url":"https://api.github.com/repos/guobingkun/druid/subscription","commits_url":"https://api.github.com/repos/guobingkun/druid/commits{/sha}","git_commits_url":"https://api.github.com/repos/guobingkun/druid/git/commits{/sha}","comments_url":"https://api.github.com/repos/guobingkun/druid/comments{/number}","issue_comment_url":"https://api.github.com/repos/guobingkun/druid/issues/comments{/number}","contents_url":"https://api.github.com/repos/guobingkun/druid/contents/{+path}","compare_url":"https://api.github.com/repos/guobingkun/druid/compare/{base}...{head}","merges_url":"https://api.github.com/repos/guobingkun/druid/merges","archive_url":"https://api.github.com/repos/guobingkun/druid/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/guobingkun/druid/downloads","issues_url":"https://api.github.com/repos/guobingkun/druid/issues{/number}","pulls_url":"https://api.github.com/repos/guobingkun/druid/pulls{/number}","milestones_url":"https://api.github.com/repos/guobingkun/druid/milestones{/number}","notifications_url":"https://api.github.com/repos/guobingkun/druid/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/guobingkun/druid/labels{/name}","releases_url":"https://api.github.com/repos/guobingkun/druid/releases{/id}","created_at":"2015-04-03T18:24:06Z","updated_at":"2015-05-07T19:33:13Z","pushed_at":"2015-08-20T18:23:21Z","git_url":"git://github.com/guobingkun/druid.git","ssh_url":"git@github.com:guobingkun/druid.git","clone_url":"https://github.com/guobingkun/druid.git","svn_url":"https://github.com/guobingkun/druid","homepage":"http://druid.io","size":95501,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"druid-io:master","ref":"master","sha":"1ecec1da5a2df0be89ac4630578108d554f7cdb8","user":{"login":"druid-io","id":4203474,"avatar_url":"https://avatars.githubusercontent.com/u/4203474?v=3","gravatar_id":"","url":"https://api.github.com/users/druid-io","html_url":"https://github.com/druid-io","followers_url":"https://api.github.com/users/druid-io/followers","following_url":"https://api.github.com/users/druid-io/following{/other_user}","gists_url":"https://api.github.com/users/druid-io/gists{/gist_id}","starred_url":"https://api.github.com/users/druid-io/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/druid-io/subscriptions","organizations_url":"https://api.github.com/users/druid-io/orgs","repos_url":"https://api.github.com/users/druid-io/repos","events_url":"https://api.github.com/users/druid-io/events{/privacy}","received_events_url":"https://api.github.com/users/druid-io/received_events","type":"Organization","site_admin":false},"repo":{"id":6358188,"name":"druid","full_name":"druid-io/druid","owner":{"login":"druid-io","id":4203474,"avatar_url":"https://avatars.githubusercontent.com/u/4203474?v=3","gravatar_id":"","url":"https://api.github.com/users/druid-io","html_url":"https://github.com/druid-io","followers_url":"https://api.github.com/users/druid-io/followers","following_url":"https://api.github.com/users/druid-io/following{/other_user}","gists_url":"https://api.github.com/users/druid-io/gists{/gist_id}","starred_url":"https://api.github.com/users/druid-io/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/druid-io/subscriptions","organizations_url":"https://api.github.com/users/druid-io/orgs","repos_url":"https://api.github.com/users/druid-io/repos","events_url":"https://api.github.com/users/druid-io/events{/privacy}","received_events_url":"https://api.github.com/users/druid-io/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/druid-io/druid","description":"Column oriented distributed data store ideal for powering interactive applications","fork":false,"url":"https://api.github.com/repos/druid-io/druid","forks_url":"https://api.github.com/repos/druid-io/druid/forks","keys_url":"https://api.github.com/repos/druid-io/druid/keys{/key_id}","collaborators_url":"https://api.github.com/repos/druid-io/druid/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/druid-io/druid/teams","hooks_url":"https://api.github.com/repos/druid-io/druid/hooks","issue_events_url":"https://api.github.com/repos/druid-io/druid/issues/events{/number}","events_url":"https://api.github.com/repos/druid-io/druid/events","assignees_url":"https://api.github.com/repos/druid-io/druid/assignees{/user}","branches_url":"https://api.github.com/repos/druid-io/druid/branches{/branch}","tags_url":"https://api.github.com/repos/druid-io/druid/tags","blobs_url":"https://api.github.com/repos/druid-io/druid/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/druid-io/druid/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/druid-io/druid/git/refs{/sha}","trees_url":"https://api.github.com/repos/druid-io/druid/git/trees{/sha}","statuses_url":"https://api.github.com/repos/druid-io/druid/statuses/{sha}","languages_url":"https://api.github.com/repos/druid-io/druid/languages","stargazers_url":"https://api.github.com/repos/druid-io/druid/stargazers","contributors_url":"https://api.github.com/repos/druid-io/druid/contributors","subscribers_url":"https://api.github.com/repos/druid-io/druid/subscribers","subscription_url":"https://api.github.com/repos/druid-io/druid/subscription","commits_url":"https://api.github.com/repos/druid-io/druid/commits{/sha}","git_commits_url":"https://api.github.com/repos/druid-io/druid/git/commits{/sha}","comments_url":"https://api.github.com/repos/druid-io/druid/comments{/number}","issue_comment_url":"https://api.github.com/repos/druid-io/druid/issues/comments{/number}","contents_url":"https://api.github.com/repos/druid-io/druid/contents/{+path}","compare_url":"https://api.github.com/repos/druid-io/druid/compare/{base}...{head}","merges_url":"https://api.github.com/repos/druid-io/druid/merges","archive_url":"https://api.github.com/repos/druid-io/druid/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/druid-io/druid/downloads","issues_url":"https://api.github.com/repos/druid-io/druid/issues{/number}","pulls_url":"https://api.github.com/repos/druid-io/druid/pulls{/number}","milestones_url":"https://api.github.com/repos/druid-io/druid/milestones{/number}","notifications_url":"https://api.github.com/repos/druid-io/druid/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/druid-io/druid/labels{/name}","releases_url":"https://api.github.com/repos/druid-io/druid/releases{/id}","created_at":"2012-10-23T19:08:07Z","updated_at":"2015-08-21T12:38:56Z","pushed_at":"2015-08-21T18:19:13Z","git_url":"git://github.com/druid-io/druid.git","ssh_url":"git@github.com:druid-io/druid.git","clone_url":"https://github.com/druid-io/druid.git","svn_url":"https://github.com/druid-io/druid","homepage":"http://druid.io","size":175278,"stargazers_count":1946,"watchers_count":1946,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":433,"mirror_url":null,"open_issues_count":163,"forks":433,"open_issues":163,"watchers":1946,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/druid-io/druid/pulls/1638"},"html":{"href":"https://github.com/druid-io/druid/pull/1638"},"issue":{"href":"https://api.github.com/repos/druid-io/druid/issues/1638"},"comments":{"href":"https://api.github.com/repos/druid-io/druid/issues/1638/comments"},"review_comments":{"href":"https://api.github.com/repos/druid-io/druid/pulls/1638/comments"},"review_comment":{"href":"https://api.github.com/repos/druid-io/druid/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/druid-io/druid/pulls/1638/commits"},"statuses":{"href":"https://api.github.com/repos/druid-io/druid/statuses/4450881efb1a139bf592f690ca7c2c249223ad07"}}}}