{"action":"opened","issue":{"url":"https://api.github.com/repos/druid-io/druid/issues/1636","labels_url":"https://api.github.com/repos/druid-io/druid/issues/1636/labels{/name}","comments_url":"https://api.github.com/repos/druid-io/druid/issues/1636/comments","events_url":"https://api.github.com/repos/druid-io/druid/issues/1636/events","html_url":"https://github.com/druid-io/druid/issues/1636","id":101537865,"number":1636,"title":"Setting Processing Buffer Size in Config Has No Effect","user":{"login":"MrAlias","id":5543599,"avatar_url":"https://avatars.githubusercontent.com/u/5543599?v=3","gravatar_id":"","url":"https://api.github.com/users/MrAlias","html_url":"https://github.com/MrAlias","followers_url":"https://api.github.com/users/MrAlias/followers","following_url":"https://api.github.com/users/MrAlias/following{/other_user}","gists_url":"https://api.github.com/users/MrAlias/gists{/gist_id}","starred_url":"https://api.github.com/users/MrAlias/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MrAlias/subscriptions","organizations_url":"https://api.github.com/users/MrAlias/orgs","repos_url":"https://api.github.com/users/MrAlias/repos","events_url":"https://api.github.com/users/MrAlias/events{/privacy}","received_events_url":"https://api.github.com/users/MrAlias/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-08-18T00:25:50Z","updated_at":"2015-08-18T00:25:50Z","closed_at":null,"body":"I'm attempting to running an instance of Druid-0.8.0 as a historical node on a VM with limited memory.\r\n\r\nThe service is failing to initialize with the following error repeated:\r\n\r\n```\r\n1) Not enough direct memory.  Please adjust -XX:MaxDirectMemorySize, druid.processing.buffer.sizeBytes, or druid.processing.numThreads: maxDirectMemory[528,154,624], memoryNeeded[2,147,483,648] = druid.processing.buffer.sizeBytes[1,073,741,824] * ( druid.processing.numThreads[1] + 1 )\r\n  at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:106)\r\n  at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:106)\r\n  while locating io.druid.collections.StupidPool<java.nio.ByteBuffer> annotated with @io.druid.guice.annotations.Global()\r\n    for parameter 1 at io.druid.query.groupby.GroupByQueryEngine.<init>(GroupByQueryEngine.java:75)\r\n  at io.druid.guice.QueryRunnerFactoryModule.configure(QueryRunnerFactoryModule.java:83)\r\n  while locating io.druid.query.groupby.GroupByQueryEngine\r\n    for parameter 0 at io.druid.query.groupby.GroupByQueryRunnerFactory.<init>(GroupByQueryRunnerFactory.java:79)\r\n  at io.druid.guice.QueryRunnerFactoryModule.configure(QueryRunnerFactoryModule.java:80)\r\n  while locating io.druid.query.groupby.GroupByQueryRunnerFactory\r\n  while locating io.druid.query.QueryRunnerFactory annotated with @com.google.inject.multibindings.Element(setName=,uniqueId=26, type=MAPBINDER)\r\n  at io.druid.guice.DruidBinders.queryRunnerFactoryBinder(DruidBinders.java:36)\r\n  while locating java.util.Map<java.lang.Class<? extends io.druid.query.Query>, io.druid.query.QueryRunnerFactory>\r\n    for parameter 0 at io.druid.query.DefaultQueryRunnerFactoryConglomerate.<init>(DefaultQueryRunnerFactoryConglomerate.java:34)\r\n  while locating io.druid.query.DefaultQueryRunnerFactoryConglomerate\r\n  at io.druid.guice.StorageNodeModule.configure(StorageNodeModule.java:53)\r\n  while locating io.druid.query.QueryRunnerFactoryConglomerate\r\n    for parameter 1 at io.druid.server.coordination.ServerManager.<init>(ServerManager.java:105)\r\n  at io.druid.cli.CliHistorical$1.configure(CliHistorical.java:76)\r\n  while locating io.druid.server.coordination.ServerManager\r\n  at io.druid.cli.CliHistorical$1.configure(CliHistorical.java:78)\r\n  while locating io.druid.query.QuerySegmentWalker\r\n    for parameter 3 at io.druid.server.QueryResource.<init>(QueryResource.java:90)\r\n  while locating io.druid.server.QueryResource\r\n\r\n```\r\n\r\nI can solve this error by adjusting the `-XX:MaxDirectMemorySize` size to be larger than the `memoryNeeded` size specified in the error.  However, when I try to reduce or even change the `druid.processing.buffer.sizeBytes` configuration value to fix the problem, this has no effect.  Regardless of what I set this value to in my properties file, the error message will report the value to be `1,073,741,824`.\r\n\r\nIn digging through the source it looks like this come from the `intermediateComputeSizeBytes` method of the [`DruidProcessingConfig`](https://github.com/druid-io/druid/blob/0.8.0/processing/src/main/java/io/druid/query/DruidProcessingConfig.java) class, which by default returns a static value (`1024 * 1024 * 1024`).  I'm likely missing some implementation  details of this class and method, so I can't say for certain why this is happening, but as it is the value set in the properties file has no effect on the reported buffer size for the storage of intermediate results."}}