{"action":"created","comment":{"url":"https://api.github.com/repos/apache/spark/pulls/comments/25969626","id":25969626,"diff_hunk":"@@ -257,21 +262,56 @@ private[spark] object SizeEstimator extends Logging {\n     val parent = getClassInfo(cls.getSuperclass)\n     var shellSize = parent.shellSize\n     var pointerFields = parent.pointerFields\n+    \n+    // we have only four sizes: 1, 2, 4, 8. pointerSize can be 4 or 8.\n+    // we can just use an Array.fill(8 + 1) to hold all the size counting, but we\n+    // can also use a HashMap to store size counting; Here we uses Array.\n+    val sizeCount = Array.fill(fieldSizes.max + 1)(0)\n \n+    // iterate through the fields of this class and gather information.\n     for (field <- cls.getDeclaredFields) {\n       if (!Modifier.isStatic(field.getModifiers)) {\n         val fieldClass = field.getType\n         if (fieldClass.isPrimitive) {\n-          shellSize += primitiveSize(fieldClass)\n+          sizeCount(primitiveSize(fieldClass)) += 1\n         } else {\n           field.setAccessible(true) // Enable future get()'s on this field\n-          shellSize += pointerSize\n+          sizeCount(pointerSize) += 1\n           pointerFields = field :: pointerFields\n         }\n       }\n     }\n \n-    shellSize = alignSize(shellSize)\n+    // Based on the simulated field layout code in Aleksey Shipilev's report:\n+    // http://cr.openjdk.java.net/~shade/papers/2013-shipilev-fieldlayout-latest.pdf\n+    // The code is in Figure 9. Aleksey Shiplev also leaders the development of JOL (Java Object \n+    // Layout) tool. The simulated code is also in the JOL. The simplified idea of field layout \n+    // consists of these parts (see more details in the report):\n+    // 1. field alignment: HotSpot lay out the fields aligned by their size.","path":"core/src/main/scala/org/apache/spark/util/SizeEstimator.scala","position":62,"original_position":62,"commit_id":"a55f3e1655b3b6ebef1f0d1bc615654a6803f7c8","original_commit_id":"a55f3e1655b3b6ebef1f0d1bc615654a6803f7c8","user":{"login":"shivaram","id":143893,"avatar_url":"https://avatars.githubusercontent.com/u/143893?v=3","gravatar_id":"","url":"https://api.github.com/users/shivaram","html_url":"https://github.com/shivaram","followers_url":"https://api.github.com/users/shivaram/followers","following_url":"https://api.github.com/users/shivaram/following{/other_user}","gists_url":"https://api.github.com/users/shivaram/gists{/gist_id}","starred_url":"https://api.github.com/users/shivaram/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shivaram/subscriptions","organizations_url":"https://api.github.com/users/shivaram/orgs","repos_url":"https://api.github.com/users/shivaram/repos","events_url":"https://api.github.com/users/shivaram/events{/privacy}","received_events_url":"https://api.github.com/users/shivaram/received_events","type":"User","site_admin":false},"body":"reword to - `HotSpot lays out fields aligned by their size`","created_at":"2015-03-06T19:19:30Z","updated_at":"2015-03-06T19:19:30Z","html_url":"https://github.com/apache/spark/pull/4783#discussion_r25969626","pull_request_url":"https://api.github.com/repos/apache/spark/pulls/4783","_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/comments/25969626"},"html":{"href":"https://github.com/apache/spark/pull/4783#discussion_r25969626"},"pull_request":{"href":"https://api.github.com/repos/apache/spark/pulls/4783"}}},"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/4783","id":30054781,"html_url":"https://github.com/apache/spark/pull/4783","diff_url":"https://github.com/apache/spark/pull/4783.diff","patch_url":"https://github.com/apache/spark/pull/4783.patch","issue_url":"https://api.github.com/repos/apache/spark/issues/4783","number":4783,"state":"open","locked":false,"title":"[SPARK-6030][CORE] Don't alignSize the computed shellSize for classes in the getClassInfo method","user":{"login":"advancedxy","id":807537,"avatar_url":"https://avatars.githubusercontent.com/u/807537?v=3","gravatar_id":"","url":"https://api.github.com/users/advancedxy","html_url":"https://github.com/advancedxy","followers_url":"https://api.github.com/users/advancedxy/followers","following_url":"https://api.github.com/users/advancedxy/following{/other_user}","gists_url":"https://api.github.com/users/advancedxy/gists{/gist_id}","starred_url":"https://api.github.com/users/advancedxy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/advancedxy/subscriptions","organizations_url":"https://api.github.com/users/advancedxy/orgs","repos_url":"https://api.github.com/users/advancedxy/repos","events_url":"https://api.github.com/users/advancedxy/events{/privacy}","received_events_url":"https://api.github.com/users/advancedxy/received_events","type":"User","site_admin":false},"body":"SizeEstimator gives wrong result for Integer on 64bit JVM with UseCompressedOops on, this pr fixes that. For more details, please refer [SPARK-6030](https://issues.apache.org/jira/browse/SPARK-6030)\r\n@sryza, I noticed there is a pr to expose SizeEstimator, maybe that should be waited by this pr get merged if we confirm this problem.\r\nAnd @shivaram would you mind to review this pr since you contribute related code. Also cc to @srowen and @mateiz ","created_at":"2015-02-26T06:58:40Z","updated_at":"2015-03-06T19:19:30Z","closed_at":null,"merged_at":null,"merge_commit_sha":"ef254a06ab5e63d086e665c886c80afc2f7a200f","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/apache/spark/pulls/4783/commits","review_comments_url":"https://api.github.com/repos/apache/spark/pulls/4783/comments","review_comment_url":"https://api.github.com/repos/apache/spark/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/spark/issues/4783/comments","statuses_url":"https://api.github.com/repos/apache/spark/statuses/a55f3e1655b3b6ebef1f0d1bc615654a6803f7c8","head":{"label":"advancedxy:SPARK-6030","ref":"SPARK-6030","sha":"a55f3e1655b3b6ebef1f0d1bc615654a6803f7c8","user":{"login":"advancedxy","id":807537,"avatar_url":"https://avatars.githubusercontent.com/u/807537?v=3","gravatar_id":"","url":"https://api.github.com/users/advancedxy","html_url":"https://github.com/advancedxy","followers_url":"https://api.github.com/users/advancedxy/followers","following_url":"https://api.github.com/users/advancedxy/following{/other_user}","gists_url":"https://api.github.com/users/advancedxy/gists{/gist_id}","starred_url":"https://api.github.com/users/advancedxy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/advancedxy/subscriptions","organizations_url":"https://api.github.com/users/advancedxy/orgs","repos_url":"https://api.github.com/users/advancedxy/repos","events_url":"https://api.github.com/users/advancedxy/events{/privacy}","received_events_url":"https://api.github.com/users/advancedxy/received_events","type":"User","site_admin":false},"repo":{"id":18693947,"name":"spark","full_name":"advancedxy/spark","owner":{"login":"advancedxy","id":807537,"avatar_url":"https://avatars.githubusercontent.com/u/807537?v=3","gravatar_id":"","url":"https://api.github.com/users/advancedxy","html_url":"https://github.com/advancedxy","followers_url":"https://api.github.com/users/advancedxy/followers","following_url":"https://api.github.com/users/advancedxy/following{/other_user}","gists_url":"https://api.github.com/users/advancedxy/gists{/gist_id}","starred_url":"https://api.github.com/users/advancedxy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/advancedxy/subscriptions","organizations_url":"https://api.github.com/users/advancedxy/orgs","repos_url":"https://api.github.com/users/advancedxy/repos","events_url":"https://api.github.com/users/advancedxy/events{/privacy}","received_events_url":"https://api.github.com/users/advancedxy/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/advancedxy/spark","description":"Mirror of Apache Spark","fork":true,"url":"https://api.github.com/repos/advancedxy/spark","forks_url":"https://api.github.com/repos/advancedxy/spark/forks","keys_url":"https://api.github.com/repos/advancedxy/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/advancedxy/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/advancedxy/spark/teams","hooks_url":"https://api.github.com/repos/advancedxy/spark/hooks","issue_events_url":"https://api.github.com/repos/advancedxy/spark/issues/events{/number}","events_url":"https://api.github.com/repos/advancedxy/spark/events","assignees_url":"https://api.github.com/repos/advancedxy/spark/assignees{/user}","branches_url":"https://api.github.com/repos/advancedxy/spark/branches{/branch}","tags_url":"https://api.github.com/repos/advancedxy/spark/tags","blobs_url":"https://api.github.com/repos/advancedxy/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/advancedxy/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/advancedxy/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/advancedxy/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/advancedxy/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/advancedxy/spark/languages","stargazers_url":"https://api.github.com/repos/advancedxy/spark/stargazers","contributors_url":"https://api.github.com/repos/advancedxy/spark/contributors","subscribers_url":"https://api.github.com/repos/advancedxy/spark/subscribers","subscription_url":"https://api.github.com/repos/advancedxy/spark/subscription","commits_url":"https://api.github.com/repos/advancedxy/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/advancedxy/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/advancedxy/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/advancedxy/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/advancedxy/spark/contents/{+path}","compare_url":"https://api.github.com/repos/advancedxy/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/advancedxy/spark/merges","archive_url":"https://api.github.com/repos/advancedxy/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/advancedxy/spark/downloads","issues_url":"https://api.github.com/repos/advancedxy/spark/issues{/number}","pulls_url":"https://api.github.com/repos/advancedxy/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/advancedxy/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/advancedxy/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/advancedxy/spark/labels{/name}","releases_url":"https://api.github.com/repos/advancedxy/spark/releases{/id}","created_at":"2014-04-12T02:00:27Z","updated_at":"2015-02-14T07:50:36Z","pushed_at":"2015-03-05T05:16:51Z","git_url":"git://github.com/advancedxy/spark.git","ssh_url":"git@github.com:advancedxy/spark.git","clone_url":"https://github.com/advancedxy/spark.git","svn_url":"https://github.com/advancedxy/spark","homepage":null,"size":88416,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"d20559b157743981b9c09e286f2aaff8cbefab59","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":17165658,"name":"spark","full_name":"apache/spark","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/spark","description":"Mirror of Apache Spark","fork":false,"url":"https://api.github.com/repos/apache/spark","forks_url":"https://api.github.com/repos/apache/spark/forks","keys_url":"https://api.github.com/repos/apache/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/spark/teams","hooks_url":"https://api.github.com/repos/apache/spark/hooks","issue_events_url":"https://api.github.com/repos/apache/spark/issues/events{/number}","events_url":"https://api.github.com/repos/apache/spark/events","assignees_url":"https://api.github.com/repos/apache/spark/assignees{/user}","branches_url":"https://api.github.com/repos/apache/spark/branches{/branch}","tags_url":"https://api.github.com/repos/apache/spark/tags","blobs_url":"https://api.github.com/repos/apache/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/spark/languages","stargazers_url":"https://api.github.com/repos/apache/spark/stargazers","contributors_url":"https://api.github.com/repos/apache/spark/contributors","subscribers_url":"https://api.github.com/repos/apache/spark/subscribers","subscription_url":"https://api.github.com/repos/apache/spark/subscription","commits_url":"https://api.github.com/repos/apache/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/spark/contents/{+path}","compare_url":"https://api.github.com/repos/apache/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/spark/merges","archive_url":"https://api.github.com/repos/apache/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/spark/downloads","issues_url":"https://api.github.com/repos/apache/spark/issues{/number}","pulls_url":"https://api.github.com/repos/apache/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/spark/labels{/name}","releases_url":"https://api.github.com/repos/apache/spark/releases{/id}","created_at":"2014-02-25T08:00:08Z","updated_at":"2015-03-06T19:19:13Z","pushed_at":"2015-03-06T14:43:27Z","git_url":"git://github.com/apache/spark.git","ssh_url":"git@github.com:apache/spark.git","clone_url":"https://github.com/apache/spark.git","svn_url":"https://github.com/apache/spark","homepage":null,"size":1352463,"stargazers_count":3099,"watchers_count":3099,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":2685,"mirror_url":"git://git.apache.org/spark.git","open_issues_count":278,"forks":2685,"open_issues":278,"watchers":3099,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/4783"},"html":{"href":"https://github.com/apache/spark/pull/4783"},"issue":{"href":"https://api.github.com/repos/apache/spark/issues/4783"},"comments":{"href":"https://api.github.com/repos/apache/spark/issues/4783/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/spark/pulls/4783/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/spark/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/spark/pulls/4783/commits"},"statuses":{"href":"https://api.github.com/repos/apache/spark/statuses/a55f3e1655b3b6ebef1f0d1bc615654a6803f7c8"}}}}