{"action":"created","comment":{"url":"https://api.github.com/repos/druid-io/druid/pulls/comments/23703425","id":23703425,"diff_hunk":"@@ -384,55 +411,127 @@ public int add(InputRow row) throws IndexSizeExceededException\n       throw new IAE(\"Cannot add row[%s] because it is below the minTimestamp[%s]\", row, new DateTime(minTimestamp));\n     }\n \n+    String[][] dims = null;\n     final List<String> rowDimensions = row.getDimensions();\n \n-    String[][] dims;\n-    List<String[]> overflow = null;\n-    synchronized (dimensionOrder) {\n-      dims = new String[dimensionOrder.size()][];\n-      for (String dimension : rowDimensions) {\n-        List<String> dimensionValues = row.getDimension(dimension);\n-\n-        // Set column capabilities as data is coming in\n-        ColumnCapabilitiesImpl capabilities = columnCapabilities.get(dimension);\n-        if (capabilities == null) {\n-          capabilities = new ColumnCapabilitiesImpl();\n-          capabilities.setType(ValueType.STRING);\n-          columnCapabilities.put(dimension, capabilities);\n+    for (String dimension : rowDimensions) {\n+      List<String> dimensionValues = row.getDimension(dimension);\n+      final Boolean raceWinner;\n+      if (dimensionOrder.containsKey(dimension)) {\n+        // Common case, don't even try to lock\n+        raceWinner = false;\n+      }else{\n+        final Lock dimLock = dimensionSynchronizer.get(dimension);\n+        dimLock.lock();\n+        try {\n+          // If we didn't win the race, populate cache values in the same codepath below as if we never tried to lock\n+          if(raceWinner = !dimensionOrder.containsKey(dimension)){\n+            // we won a race, add a new dimension\n+            // Set column capabilities as data is coming in if it wasn't set in constructor\n+            ColumnCapabilitiesImpl capabilities = columnCapabilities.get(dimension);\n+            if (capabilities == null) {\n+              capabilities = new ColumnCapabilitiesImpl();\n+              capabilities.setType(ValueType.STRING);\n+              ColumnCapabilitiesImpl other = columnCapabilities.putIfAbsent(dimension, capabilities);\n+              if (other != null) {\n+                throw new ISE(\"Bad columnCapabilites state\");\n+              }\n+            }\n+            if (dimensionValues.size() > 1) {\n+              capabilities.setHasMultipleValues(true);\n+            }\n+\n+            final Integer orderID = orderIncrementer.getAndIncrement();","path":"processing/src/main/java/io/druid/segment/incremental/IncrementalIndex.java","position":218,"original_position":218,"commit_id":"7d9106a40e4cc988ef829381d3a8e8403e7327fc","original_commit_id":"7d9106a40e4cc988ef829381d3a8e8403e7327fc","user":{"login":"drcrallen","id":8213081,"avatar_url":"https://avatars.githubusercontent.com/u/8213081?v=3","gravatar_id":"","url":"https://api.github.com/users/drcrallen","html_url":"https://github.com/drcrallen","followers_url":"https://api.github.com/users/drcrallen/followers","following_url":"https://api.github.com/users/drcrallen/following{/other_user}","gists_url":"https://api.github.com/users/drcrallen/gists{/gist_id}","starred_url":"https://api.github.com/users/drcrallen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/drcrallen/subscriptions","organizations_url":"https://api.github.com/users/drcrallen/orgs","repos_url":"https://api.github.com/users/drcrallen/repos","events_url":"https://api.github.com/users/drcrallen/events{/privacy}","received_events_url":"https://api.github.com/users/drcrallen/received_events","type":"User","site_admin":false},"body":" Because it was originally an index into an array. There is no reason it has to be an integer and Long is probably more future proof, but also less like the previous behavior. Is there any objection to using Long for (mathematical) integers that are not an index into an array?","created_at":"2015-01-28T17:14:06Z","updated_at":"2015-01-28T17:14:06Z","html_url":"https://github.com/druid-io/druid/pull/1065#discussion_r23703425","pull_request_url":"https://api.github.com/repos/druid-io/druid/pulls/1065","_links":{"self":{"href":"https://api.github.com/repos/druid-io/druid/pulls/comments/23703425"},"html":{"href":"https://github.com/druid-io/druid/pull/1065#discussion_r23703425"},"pull_request":{"href":"https://api.github.com/repos/druid-io/druid/pulls/1065"}}},"pull_request":{"url":"https://api.github.com/repos/druid-io/druid/pulls/1065","id":27939623,"html_url":"https://github.com/druid-io/druid/pull/1065","diff_url":"https://github.com/druid-io/druid/pull/1065.diff","patch_url":"https://github.com/druid-io/druid/pull/1065.patch","issue_url":"https://api.github.com/repos/druid-io/druid/issues/1065","number":1065,"state":"open","locked":false,"title":"Remove most locks from IncrementalIndex and fix OffHeapIncrementalIndex","user":{"login":"drcrallen","id":8213081,"avatar_url":"https://avatars.githubusercontent.com/u/8213081?v=3","gravatar_id":"","url":"https://api.github.com/users/drcrallen","html_url":"https://github.com/drcrallen","followers_url":"https://api.github.com/users/drcrallen/followers","following_url":"https://api.github.com/users/drcrallen/following{/other_user}","gists_url":"https://api.github.com/users/drcrallen/gists{/gist_id}","starred_url":"https://api.github.com/users/drcrallen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/drcrallen/subscriptions","organizations_url":"https://api.github.com/users/drcrallen/orgs","repos_url":"https://api.github.com/users/drcrallen/repos","events_url":"https://api.github.com/users/drcrallen/events{/privacy}","received_events_url":"https://api.github.com/users/drcrallen/received_events","type":"User","site_admin":false},"body":"* A placeholder value of -1 is used as an index value to determine race winners.\r\n* Consumers of index values have been updated to handle -1 values.","created_at":"2015-01-23T17:18:09Z","updated_at":"2015-01-28T17:14:07Z","closed_at":null,"merged_at":null,"merge_commit_sha":"ef21e86b6715a5015999a148cf6ef28f426dd9cc","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/druid-io/druid/pulls/1065/commits","review_comments_url":"https://api.github.com/repos/druid-io/druid/pulls/1065/comments","review_comment_url":"https://api.github.com/repos/druid-io/druid/pulls/comments/{number}","comments_url":"https://api.github.com/repos/druid-io/druid/issues/1065/comments","statuses_url":"https://api.github.com/repos/druid-io/druid/statuses/7d9106a40e4cc988ef829381d3a8e8403e7327fc","head":{"label":"metamx:IncrementalIndexConcurrency","ref":"IncrementalIndexConcurrency","sha":"7d9106a40e4cc988ef829381d3a8e8403e7327fc","user":{"login":"metamx","id":299329,"avatar_url":"https://avatars.githubusercontent.com/u/299329?v=3","gravatar_id":"","url":"https://api.github.com/users/metamx","html_url":"https://github.com/metamx","followers_url":"https://api.github.com/users/metamx/followers","following_url":"https://api.github.com/users/metamx/following{/other_user}","gists_url":"https://api.github.com/users/metamx/gists{/gist_id}","starred_url":"https://api.github.com/users/metamx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/metamx/subscriptions","organizations_url":"https://api.github.com/users/metamx/orgs","repos_url":"https://api.github.com/users/metamx/repos","events_url":"https://api.github.com/users/metamx/events{/privacy}","received_events_url":"https://api.github.com/users/metamx/received_events","type":"Organization","site_admin":false},"repo":{"id":28248872,"name":"druid","full_name":"metamx/druid","owner":{"login":"metamx","id":299329,"avatar_url":"https://avatars.githubusercontent.com/u/299329?v=3","gravatar_id":"","url":"https://api.github.com/users/metamx","html_url":"https://github.com/metamx","followers_url":"https://api.github.com/users/metamx/followers","following_url":"https://api.github.com/users/metamx/following{/other_user}","gists_url":"https://api.github.com/users/metamx/gists{/gist_id}","starred_url":"https://api.github.com/users/metamx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/metamx/subscriptions","organizations_url":"https://api.github.com/users/metamx/orgs","repos_url":"https://api.github.com/users/metamx/repos","events_url":"https://api.github.com/users/metamx/events{/privacy}","received_events_url":"https://api.github.com/users/metamx/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/metamx/druid","description":"Real²time Exploratory Analytics on Large Datasets","fork":true,"url":"https://api.github.com/repos/metamx/druid","forks_url":"https://api.github.com/repos/metamx/druid/forks","keys_url":"https://api.github.com/repos/metamx/druid/keys{/key_id}","collaborators_url":"https://api.github.com/repos/metamx/druid/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/metamx/druid/teams","hooks_url":"https://api.github.com/repos/metamx/druid/hooks","issue_events_url":"https://api.github.com/repos/metamx/druid/issues/events{/number}","events_url":"https://api.github.com/repos/metamx/druid/events","assignees_url":"https://api.github.com/repos/metamx/druid/assignees{/user}","branches_url":"https://api.github.com/repos/metamx/druid/branches{/branch}","tags_url":"https://api.github.com/repos/metamx/druid/tags","blobs_url":"https://api.github.com/repos/metamx/druid/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/metamx/druid/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/metamx/druid/git/refs{/sha}","trees_url":"https://api.github.com/repos/metamx/druid/git/trees{/sha}","statuses_url":"https://api.github.com/repos/metamx/druid/statuses/{sha}","languages_url":"https://api.github.com/repos/metamx/druid/languages","stargazers_url":"https://api.github.com/repos/metamx/druid/stargazers","contributors_url":"https://api.github.com/repos/metamx/druid/contributors","subscribers_url":"https://api.github.com/repos/metamx/druid/subscribers","subscription_url":"https://api.github.com/repos/metamx/druid/subscription","commits_url":"https://api.github.com/repos/metamx/druid/commits{/sha}","git_commits_url":"https://api.github.com/repos/metamx/druid/git/commits{/sha}","comments_url":"https://api.github.com/repos/metamx/druid/comments{/number}","issue_comment_url":"https://api.github.com/repos/metamx/druid/issues/comments/{number}","contents_url":"https://api.github.com/repos/metamx/druid/contents/{+path}","compare_url":"https://api.github.com/repos/metamx/druid/compare/{base}...{head}","merges_url":"https://api.github.com/repos/metamx/druid/merges","archive_url":"https://api.github.com/repos/metamx/druid/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/metamx/druid/downloads","issues_url":"https://api.github.com/repos/metamx/druid/issues{/number}","pulls_url":"https://api.github.com/repos/metamx/druid/pulls{/number}","milestones_url":"https://api.github.com/repos/metamx/druid/milestones{/number}","notifications_url":"https://api.github.com/repos/metamx/druid/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/metamx/druid/labels{/name}","releases_url":"https://api.github.com/repos/metamx/druid/releases{/id}","created_at":"2014-12-19T22:30:25Z","updated_at":"2015-01-27T03:34:58Z","pushed_at":"2015-01-28T00:12:54Z","git_url":"git://github.com/metamx/druid.git","ssh_url":"git@github.com:metamx/druid.git","clone_url":"https://github.com/metamx/druid.git","svn_url":"https://github.com/metamx/druid","homepage":"http://druid.io","size":88896,"stargazers_count":8,"watchers_count":8,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":8,"mirror_url":null,"open_issues_count":0,"forks":8,"open_issues":0,"watchers":8,"default_branch":"master"}},"base":{"label":"druid-io:master","ref":"master","sha":"cd9ba1f85225c1d54b646e2fdb1cb8ace08d3b45","user":{"login":"druid-io","id":4203474,"avatar_url":"https://avatars.githubusercontent.com/u/4203474?v=3","gravatar_id":"","url":"https://api.github.com/users/druid-io","html_url":"https://github.com/druid-io","followers_url":"https://api.github.com/users/druid-io/followers","following_url":"https://api.github.com/users/druid-io/following{/other_user}","gists_url":"https://api.github.com/users/druid-io/gists{/gist_id}","starred_url":"https://api.github.com/users/druid-io/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/druid-io/subscriptions","organizations_url":"https://api.github.com/users/druid-io/orgs","repos_url":"https://api.github.com/users/druid-io/repos","events_url":"https://api.github.com/users/druid-io/events{/privacy}","received_events_url":"https://api.github.com/users/druid-io/received_events","type":"Organization","site_admin":false},"repo":{"id":6358188,"name":"druid","full_name":"druid-io/druid","owner":{"login":"druid-io","id":4203474,"avatar_url":"https://avatars.githubusercontent.com/u/4203474?v=3","gravatar_id":"","url":"https://api.github.com/users/druid-io","html_url":"https://github.com/druid-io","followers_url":"https://api.github.com/users/druid-io/followers","following_url":"https://api.github.com/users/druid-io/following{/other_user}","gists_url":"https://api.github.com/users/druid-io/gists{/gist_id}","starred_url":"https://api.github.com/users/druid-io/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/druid-io/subscriptions","organizations_url":"https://api.github.com/users/druid-io/orgs","repos_url":"https://api.github.com/users/druid-io/repos","events_url":"https://api.github.com/users/druid-io/events{/privacy}","received_events_url":"https://api.github.com/users/druid-io/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/druid-io/druid","description":"Real²time Exploratory Analytics on Large Datasets","fork":false,"url":"https://api.github.com/repos/druid-io/druid","forks_url":"https://api.github.com/repos/druid-io/druid/forks","keys_url":"https://api.github.com/repos/druid-io/druid/keys{/key_id}","collaborators_url":"https://api.github.com/repos/druid-io/druid/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/druid-io/druid/teams","hooks_url":"https://api.github.com/repos/druid-io/druid/hooks","issue_events_url":"https://api.github.com/repos/druid-io/druid/issues/events{/number}","events_url":"https://api.github.com/repos/druid-io/druid/events","assignees_url":"https://api.github.com/repos/druid-io/druid/assignees{/user}","branches_url":"https://api.github.com/repos/druid-io/druid/branches{/branch}","tags_url":"https://api.github.com/repos/druid-io/druid/tags","blobs_url":"https://api.github.com/repos/druid-io/druid/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/druid-io/druid/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/druid-io/druid/git/refs{/sha}","trees_url":"https://api.github.com/repos/druid-io/druid/git/trees{/sha}","statuses_url":"https://api.github.com/repos/druid-io/druid/statuses/{sha}","languages_url":"https://api.github.com/repos/druid-io/druid/languages","stargazers_url":"https://api.github.com/repos/druid-io/druid/stargazers","contributors_url":"https://api.github.com/repos/druid-io/druid/contributors","subscribers_url":"https://api.github.com/repos/druid-io/druid/subscribers","subscription_url":"https://api.github.com/repos/druid-io/druid/subscription","commits_url":"https://api.github.com/repos/druid-io/druid/commits{/sha}","git_commits_url":"https://api.github.com/repos/druid-io/druid/git/commits{/sha}","comments_url":"https://api.github.com/repos/druid-io/druid/comments{/number}","issue_comment_url":"https://api.github.com/repos/druid-io/druid/issues/comments/{number}","contents_url":"https://api.github.com/repos/druid-io/druid/contents/{+path}","compare_url":"https://api.github.com/repos/druid-io/druid/compare/{base}...{head}","merges_url":"https://api.github.com/repos/druid-io/druid/merges","archive_url":"https://api.github.com/repos/druid-io/druid/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/druid-io/druid/downloads","issues_url":"https://api.github.com/repos/druid-io/druid/issues{/number}","pulls_url":"https://api.github.com/repos/druid-io/druid/pulls{/number}","milestones_url":"https://api.github.com/repos/druid-io/druid/milestones{/number}","notifications_url":"https://api.github.com/repos/druid-io/druid/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/druid-io/druid/labels{/name}","releases_url":"https://api.github.com/repos/druid-io/druid/releases{/id}","created_at":"2012-10-23T19:08:07Z","updated_at":"2015-01-28T14:47:50Z","pushed_at":"2015-01-27T22:54:18Z","git_url":"git://github.com/druid-io/druid.git","ssh_url":"git@github.com:druid-io/druid.git","clone_url":"https://github.com/druid-io/druid.git","svn_url":"https://github.com/druid-io/druid","homepage":"http://druid.io","size":153468,"stargazers_count":1340,"watchers_count":1340,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":291,"mirror_url":null,"open_issues_count":126,"forks":291,"open_issues":126,"watchers":1340,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/druid-io/druid/pulls/1065"},"html":{"href":"https://github.com/druid-io/druid/pull/1065"},"issue":{"href":"https://api.github.com/repos/druid-io/druid/issues/1065"},"comments":{"href":"https://api.github.com/repos/druid-io/druid/issues/1065/comments"},"review_comments":{"href":"https://api.github.com/repos/druid-io/druid/pulls/1065/comments"},"review_comment":{"href":"https://api.github.com/repos/druid-io/druid/pulls/comments/{number}"},"commits":{"href":"https://api.github.com/repos/druid-io/druid/pulls/1065/commits"},"statuses":{"href":"https://api.github.com/repos/druid-io/druid/statuses/7d9106a40e4cc988ef829381d3a8e8403e7327fc"}}}}