{"action":"created","comment":{"url":"https://api.github.com/repos/apache/storm/pulls/comments/47687409","id":47687409,"diff_hunk":"@@ -0,0 +1,224 @@\n+# Introduction\n+\n+The purpose of this document is to provide a description of the Resource Aware Scheduler for the Storm distributed real-time computation system.  This document will provide you with both a high level description of the resource aware scheduler in Storm\n+\n+## Using Resource Aware Scheduler\n+\n+The user can switch to using the Resource Aware Scheduler by setting the following in *conf/storm.yaml*\n+\n+    storm.scheduler: “backtype.storm.scheduler.resource.ResourceAwareScheduler”\n+\n+\n+## API Overview\n+\n+For a Storm Topology, the user can now specify the amount of resources a topology component (i.e. Spout or Bolt) is required to run a single instance of the component.  The user can specify the resource requirement for a topology component by using the following API calls.\n+\n+### Setting Memory Requirement\n+\n+API to set component memory requirement:\n+\n+    public T setMemoryLoad(Number onHeap, Number offHeap)\n+\n+Parameters:\n+* Number onHeap – The amount of on heap memory an instance of this component will consume in megabytes\n+* Number offHeap – The amount of off heap memory an instance of this component will consume in megabytes\n+\n+The user also has to option to just specify the on heap memory requirement if the component does not have an off heap memory need.\n+\n+    public T setMemoryLoad(Number onHeap)\n+\n+Parameters:\n+* Number onHeap – The amount of on heap memory an instance of this component will consume\n+\n+If no value is provided for offHeap, 0.0 will be used. If no value is provided for onHeap, or if the API is never called for a component, the default value will be used.\n+\n+Example of Usage:\n+\n+    SpoutDeclarer s1 = builder.setSpout(\"word\", new TestWordSpout(), 10);\n+    s1.setMemoryLoad(1024.0, 512.0);\n+    builder.setBolt(\"exclaim1\", new ExclamationBolt(), 3)\n+                .shuffleGrouping(\"word\").setMemoryLoad(512.0);\n+\n+The entire memory requested for this topology is 16.5 GB. That is from 10 spouts with 1GB on heap memory and 0.5 GB off heap memory each and 3 bolts with 0.5 GB on heap memory each.\n+\n+### Setting CPU Requirement\n+\n+\n+API to set component CPU requirement:\n+\n+    public T setCPULoad(Double amount)\n+\n+Parameters:\n+* Number amount – The amount of on CPU an instance of this component will consume.\n+\n+Currently, the amount of CPU resources a component requires or is available on a node is represented by a point system. CPU usage is a difficult concept to define. Different CPU architectures perform differently depending on the task at hand. They are so complex that expressing all of that in a single precise portable number is impossible. Instead we take a convention over configuration approach and are primarily concerned with rough level of CPU usage while still providing the possibility to specify amounts more fine grained.\n+\n+By convention a CPU core typically will get 100 points. If you feel that your processors are more or less powerful you can adjust this accordingly. Heavy tasks that are CPU bound will get 100 points, as they can consume an entire core. Medium tasks should get 50, light tasks 25, and tiny tasks 10. In some cases you have a task that spawns other threads to help with processing. These tasks may need to go above 100 points to express the amount of CPU they are using. If these conventions are followed the common case for a single threaded task the reported Capacity * 100 should be the number of CPU points that the task needs.\n+\n+Example of Usage:\n+\n+    SpoutDeclarer s1 = builder.setSpout(\"word\", new TestWordSpout(), 10);\n+    s1.setCPULoad(15.0);\n+    builder.setBolt(\"exclaim1\", new ExclamationBolt(), 3)\n+                .shuffleGrouping(\"word\").setCPULoad(10.0);\n+\n+###\tLimiting the Heap Size per Worker (JVM) Process\n+\n+\n+    public void setTopologyWorkerMaxHeapSize(Number size)\n+\n+Parameters:\n+* Number size – The memory limit a worker process will be allocated in megabytes\n+\n+The user can limit the amount of memory resources the resource aware scheduler that is allocated to a single worker on a per topology basis by using the above API.  This API is in place so that the users can spread executors to multiple workers.  However, spreading workers to multiple workers may increase the communication latency since executors will not be able to use Disruptor Queue for intra-process communication.\n+\n+Example of Usage:\n+\n+    Config conf = new Config();\n+    conf.setTopologyWorkerMaxHeapSize(512.0);\n+\n+### Setting Available Resources on Node\n+\n+A storm administrator can specify node resource availability by modifying the *conf/storm.yaml* file located in the storm home directory of that node.\n+\n+A storm administrator can specify how much available memory a node has in megabytes adding the following to *storm.yaml*\n+\n+    supervisor.memory.capacity.mb: [amount<Double>]\n+\n+A storm administrator can also specify how much available CPU resources a node has available adding the following to *storm.yaml*\n+\n+    supervisor.cpu.capacity: [amount<Double>]\n+\n+\n+Note: that the amount the user can specify for the available CPU is represented using a point system like discussed earlier.\n+\n+Example of Usage:\n+\n+    supervisor.memory.capacity.mb: 20480.0\n+    supervisor.cpu.capacity: 100.0\n+\n+\n+### Other Configurations\n+\n+The user can set some default configurations for the Resource Aware Scheduler in *conf/storm.yaml*:\n+\n+    //default value if on heap memory requirement is not specified for a component \n+    topology.component.resources.onheap.memory.mb: 128.0\n+\n+    //default value if off heap memory requirement is not specified for a component \n+    topology.component.resources.offheap.memory.mb: 0.0\n+\n+    //default value if CPU requirement is not specified for a component \n+    topology.component.cpu.pcore.percent: 10.0\n+\n+    //default value for the max heap size for a worker  \n+    topology.worker.max.heap.size.mb: 768.0\n+\n+# Topology Priorities and Per User Resource \n+\n+The next step for the Resource Aware Scheduler or RAS is to enable it to have multitenant capabilities since many Storm users typically share a Storm cluster.  Resource Aware Scheduler needs to be able to allocate resources on a per user basis.  Each user can be guaranteed a certain amount of resources to run his or her topologies and the Resource Aware Scheduler should meet those guarantees when possible.  When the Storm cluster has extra free resources, Resource Aware Scheduler needs to be able allocate additional resources to user in a fair manner. The importance of topologies can also vary.  Topologies can be used for actual production or just experimentation, thus Resource Aware Scheduler should take into account the importance of a topology when determining the order in which to schedule topologies or when to evict topologies","path":"docs/documentation/Resource_Aware_Scheduler_overview.md","position":119,"original_position":119,"commit_id":"9a542a6934426eb43f996ac7437e40013643bd43","original_commit_id":"9a542a6934426eb43f996ac7437e40013643bd43","user":{"login":"jerrypeng","id":3613359,"avatar_url":"https://avatars.githubusercontent.com/u/3613359?v=3","gravatar_id":"","url":"https://api.github.com/users/jerrypeng","html_url":"https://github.com/jerrypeng","followers_url":"https://api.github.com/users/jerrypeng/followers","following_url":"https://api.github.com/users/jerrypeng/following{/other_user}","gists_url":"https://api.github.com/users/jerrypeng/gists{/gist_id}","starred_url":"https://api.github.com/users/jerrypeng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jerrypeng/subscriptions","organizations_url":"https://api.github.com/users/jerrypeng/orgs","repos_url":"https://api.github.com/users/jerrypeng/repos","events_url":"https://api.github.com/users/jerrypeng/events{/privacy}","received_events_url":"https://api.github.com/users/jerrypeng/received_events","type":"User","site_admin":false},"body":"will fix","created_at":"2015-12-15T19:42:09Z","updated_at":"2015-12-15T19:42:09Z","html_url":"https://github.com/apache/storm/pull/921#discussion_r47687409","pull_request_url":"https://api.github.com/repos/apache/storm/pulls/921","_links":{"self":{"href":"https://api.github.com/repos/apache/storm/pulls/comments/47687409"},"html":{"href":"https://github.com/apache/storm/pull/921#discussion_r47687409"},"pull_request":{"href":"https://api.github.com/repos/apache/storm/pulls/921"}}},"pull_request":{"url":"https://api.github.com/repos/apache/storm/pulls/921","id":52622402,"html_url":"https://github.com/apache/storm/pull/921","diff_url":"https://github.com/apache/storm/pull/921.diff","patch_url":"https://github.com/apache/storm/pull/921.patch","issue_url":"https://api.github.com/repos/apache/storm/issues/921","number":921,"state":"open","locked":false,"title":"[STORM-898] - Add priorities and per user resource guarantees to Resource Aware Scheduler","user":{"login":"jerrypeng","id":3613359,"avatar_url":"https://avatars.githubusercontent.com/u/3613359?v=3","gravatar_id":"","url":"https://api.github.com/users/jerrypeng","html_url":"https://github.com/jerrypeng","followers_url":"https://api.github.com/users/jerrypeng/followers","following_url":"https://api.github.com/users/jerrypeng/following{/other_user}","gists_url":"https://api.github.com/users/jerrypeng/gists{/gist_id}","starred_url":"https://api.github.com/users/jerrypeng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jerrypeng/subscriptions","organizations_url":"https://api.github.com/users/jerrypeng/orgs","repos_url":"https://api.github.com/users/jerrypeng/repos","events_url":"https://api.github.com/users/jerrypeng/events{/privacy}","received_events_url":"https://api.github.com/users/jerrypeng/received_events","type":"User","site_admin":false},"body":"https://issues.apache.org/jira/browse/STORM-898\r\n\r\nAllows the Resource Aware Scheduler to how multi-tenant capabilites. In a multi-tenant environment we would like to be able to give individual users a guarantee of how much CPU/Memory/Network they will be able to use in a cluster. We would also like to know which topologies a user feels are the most important to keep running if there are not enough resources to run all of their topologies.\r\n\r\nEach user should be able to specify if their topology is production, staging, or development.\r\n\r\nBased on topology priorities and resource guarantees, we need to be able to evict topologies that are less important so that users can have there resource guarantees satisfied\r\n\r\nEach topology can also be scheduled by a different strategy.  Users can define custom scheduling strategies and the user can specify on a per topology basis what strategy to use for a topology\r\n\r\nFor more details please read documentation:\r\n\r\nhttps://issues.apache.org/jira/secure/attachment/12775719/Resource%20Aware%20Scheduler%20for%20Storm.pdf\r\n\r\nI have also recently updated the documentation","created_at":"2015-12-04T02:24:51Z","updated_at":"2015-12-15T19:42:10Z","closed_at":null,"merged_at":null,"merge_commit_sha":"b5ea2b469c0585930d974c45060a83dfd7760745","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/apache/storm/pulls/921/commits","review_comments_url":"https://api.github.com/repos/apache/storm/pulls/921/comments","review_comment_url":"https://api.github.com/repos/apache/storm/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/storm/issues/921/comments","statuses_url":"https://api.github.com/repos/apache/storm/statuses/9a542a6934426eb43f996ac7437e40013643bd43","head":{"label":"jerrypeng:STORM-898","ref":"STORM-898","sha":"9a542a6934426eb43f996ac7437e40013643bd43","user":{"login":"jerrypeng","id":3613359,"avatar_url":"https://avatars.githubusercontent.com/u/3613359?v=3","gravatar_id":"","url":"https://api.github.com/users/jerrypeng","html_url":"https://github.com/jerrypeng","followers_url":"https://api.github.com/users/jerrypeng/followers","following_url":"https://api.github.com/users/jerrypeng/following{/other_user}","gists_url":"https://api.github.com/users/jerrypeng/gists{/gist_id}","starred_url":"https://api.github.com/users/jerrypeng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jerrypeng/subscriptions","organizations_url":"https://api.github.com/users/jerrypeng/orgs","repos_url":"https://api.github.com/users/jerrypeng/repos","events_url":"https://api.github.com/users/jerrypeng/events{/privacy}","received_events_url":"https://api.github.com/users/jerrypeng/received_events","type":"User","site_admin":false},"repo":{"id":37741458,"name":"storm","full_name":"jerrypeng/storm","owner":{"login":"jerrypeng","id":3613359,"avatar_url":"https://avatars.githubusercontent.com/u/3613359?v=3","gravatar_id":"","url":"https://api.github.com/users/jerrypeng","html_url":"https://github.com/jerrypeng","followers_url":"https://api.github.com/users/jerrypeng/followers","following_url":"https://api.github.com/users/jerrypeng/following{/other_user}","gists_url":"https://api.github.com/users/jerrypeng/gists{/gist_id}","starred_url":"https://api.github.com/users/jerrypeng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jerrypeng/subscriptions","organizations_url":"https://api.github.com/users/jerrypeng/orgs","repos_url":"https://api.github.com/users/jerrypeng/repos","events_url":"https://api.github.com/users/jerrypeng/events{/privacy}","received_events_url":"https://api.github.com/users/jerrypeng/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/jerrypeng/storm","description":"Mirror of Apache Storm","fork":true,"url":"https://api.github.com/repos/jerrypeng/storm","forks_url":"https://api.github.com/repos/jerrypeng/storm/forks","keys_url":"https://api.github.com/repos/jerrypeng/storm/keys{/key_id}","collaborators_url":"https://api.github.com/repos/jerrypeng/storm/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/jerrypeng/storm/teams","hooks_url":"https://api.github.com/repos/jerrypeng/storm/hooks","issue_events_url":"https://api.github.com/repos/jerrypeng/storm/issues/events{/number}","events_url":"https://api.github.com/repos/jerrypeng/storm/events","assignees_url":"https://api.github.com/repos/jerrypeng/storm/assignees{/user}","branches_url":"https://api.github.com/repos/jerrypeng/storm/branches{/branch}","tags_url":"https://api.github.com/repos/jerrypeng/storm/tags","blobs_url":"https://api.github.com/repos/jerrypeng/storm/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/jerrypeng/storm/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/jerrypeng/storm/git/refs{/sha}","trees_url":"https://api.github.com/repos/jerrypeng/storm/git/trees{/sha}","statuses_url":"https://api.github.com/repos/jerrypeng/storm/statuses/{sha}","languages_url":"https://api.github.com/repos/jerrypeng/storm/languages","stargazers_url":"https://api.github.com/repos/jerrypeng/storm/stargazers","contributors_url":"https://api.github.com/repos/jerrypeng/storm/contributors","subscribers_url":"https://api.github.com/repos/jerrypeng/storm/subscribers","subscription_url":"https://api.github.com/repos/jerrypeng/storm/subscription","commits_url":"https://api.github.com/repos/jerrypeng/storm/commits{/sha}","git_commits_url":"https://api.github.com/repos/jerrypeng/storm/git/commits{/sha}","comments_url":"https://api.github.com/repos/jerrypeng/storm/comments{/number}","issue_comment_url":"https://api.github.com/repos/jerrypeng/storm/issues/comments{/number}","contents_url":"https://api.github.com/repos/jerrypeng/storm/contents/{+path}","compare_url":"https://api.github.com/repos/jerrypeng/storm/compare/{base}...{head}","merges_url":"https://api.github.com/repos/jerrypeng/storm/merges","archive_url":"https://api.github.com/repos/jerrypeng/storm/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/jerrypeng/storm/downloads","issues_url":"https://api.github.com/repos/jerrypeng/storm/issues{/number}","pulls_url":"https://api.github.com/repos/jerrypeng/storm/pulls{/number}","milestones_url":"https://api.github.com/repos/jerrypeng/storm/milestones{/number}","notifications_url":"https://api.github.com/repos/jerrypeng/storm/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/jerrypeng/storm/labels{/name}","releases_url":"https://api.github.com/repos/jerrypeng/storm/releases{/id}","created_at":"2015-06-19T19:23:06Z","updated_at":"2015-11-23T22:33:09Z","pushed_at":"2015-12-11T07:27:12Z","git_url":"git://github.com/jerrypeng/storm.git","ssh_url":"git@github.com:jerrypeng/storm.git","clone_url":"https://github.com/jerrypeng/storm.git","svn_url":"https://github.com/jerrypeng/storm","homepage":null,"size":26859,"stargazers_count":2,"watchers_count":2,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":true,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":2,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"6babbb08c43a32a7624e81dfc3a3bfe52db02a33","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":14135470,"name":"storm","full_name":"apache/storm","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/storm","description":"Mirror of Apache Storm","fork":false,"url":"https://api.github.com/repos/apache/storm","forks_url":"https://api.github.com/repos/apache/storm/forks","keys_url":"https://api.github.com/repos/apache/storm/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/storm/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/storm/teams","hooks_url":"https://api.github.com/repos/apache/storm/hooks","issue_events_url":"https://api.github.com/repos/apache/storm/issues/events{/number}","events_url":"https://api.github.com/repos/apache/storm/events","assignees_url":"https://api.github.com/repos/apache/storm/assignees{/user}","branches_url":"https://api.github.com/repos/apache/storm/branches{/branch}","tags_url":"https://api.github.com/repos/apache/storm/tags","blobs_url":"https://api.github.com/repos/apache/storm/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/storm/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/storm/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/storm/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/storm/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/storm/languages","stargazers_url":"https://api.github.com/repos/apache/storm/stargazers","contributors_url":"https://api.github.com/repos/apache/storm/contributors","subscribers_url":"https://api.github.com/repos/apache/storm/subscribers","subscription_url":"https://api.github.com/repos/apache/storm/subscription","commits_url":"https://api.github.com/repos/apache/storm/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/storm/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/storm/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/storm/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/storm/contents/{+path}","compare_url":"https://api.github.com/repos/apache/storm/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/storm/merges","archive_url":"https://api.github.com/repos/apache/storm/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/storm/downloads","issues_url":"https://api.github.com/repos/apache/storm/issues{/number}","pulls_url":"https://api.github.com/repos/apache/storm/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/storm/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/storm/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/storm/labels{/name}","releases_url":"https://api.github.com/repos/apache/storm/releases{/id}","created_at":"2013-11-05T08:00:14Z","updated_at":"2015-12-15T13:53:34Z","pushed_at":"2015-12-15T17:00:49Z","git_url":"git://github.com/apache/storm.git","ssh_url":"git@github.com:apache/storm.git","clone_url":"https://github.com/apache/storm.git","svn_url":"https://github.com/apache/storm","homepage":null,"size":155734,"stargazers_count":2640,"watchers_count":2640,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":true,"forks_count":1972,"mirror_url":"git://git.apache.org/storm.git","open_issues_count":89,"forks":1972,"open_issues":89,"watchers":2640,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/storm/pulls/921"},"html":{"href":"https://github.com/apache/storm/pull/921"},"issue":{"href":"https://api.github.com/repos/apache/storm/issues/921"},"comments":{"href":"https://api.github.com/repos/apache/storm/issues/921/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/storm/pulls/921/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/storm/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/storm/pulls/921/commits"},"statuses":{"href":"https://api.github.com/repos/apache/storm/statuses/9a542a6934426eb43f996ac7437e40013643bd43"}}}}