{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/7752","labels_url":"https://api.github.com/repos/apache/spark/issues/7752/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/7752/comments","events_url":"https://api.github.com/repos/apache/spark/issues/7752/events","html_url":"https://github.com/apache/spark/pull/7752","id":97984522,"number":7752,"title":"[SPARK-9404][SQL] unsafe array data","user":{"login":"cloud-fan","id":3182036,"avatar_url":"https://avatars.githubusercontent.com/u/3182036?v=3","gravatar_id":"","url":"https://api.github.com/users/cloud-fan","html_url":"https://github.com/cloud-fan","followers_url":"https://api.github.com/users/cloud-fan/followers","following_url":"https://api.github.com/users/cloud-fan/following{/other_user}","gists_url":"https://api.github.com/users/cloud-fan/gists{/gist_id}","starred_url":"https://api.github.com/users/cloud-fan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cloud-fan/subscriptions","organizations_url":"https://api.github.com/users/cloud-fan/orgs","repos_url":"https://api.github.com/users/cloud-fan/repos","events_url":"https://api.github.com/users/cloud-fan/events{/privacy}","received_events_url":"https://api.github.com/users/cloud-fan/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":40,"created_at":"2015-07-29T17:14:06Z","updated_at":"2015-08-02T09:06:51Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/7752","html_url":"https://github.com/apache/spark/pull/7752","diff_url":"https://github.com/apache/spark/pull/7752.diff","patch_url":"https://github.com/apache/spark/pull/7752.patch"},"body":"This PR adds a UnsafeArrayData, current we encode it in this way:\r\n\r\nfirst 4 bytes is the # elements\r\nthen each 4 byte is the start offset of the element, unless it is negative, in which case the element is null.\r\nfollowed by the elements themselves\r\n\r\nan example:  [10, 11, 12, 13, null, 14] will be encoded as:\r\n5, 28, 32, 36, 40, -44, 44, 10, 11, 12, 13, 14\r\n\r\nNote that, when we read a UnsafeArrayData from bytes, we can read the first 4 bytes as numElements and take the rest(first 4 bytes skipped) as value region.\r\n\r\nTODO:\r\n\r\n1. add doc\r\n2. add tests\r\n3. add unsafe map(it's much easier than array, only need to add a new writer)"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/127000670","html_url":"https://github.com/apache/spark/pull/7752#issuecomment-127000670","issue_url":"https://api.github.com/repos/apache/spark/issues/7752","id":127000670,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2015-08-02T09:06:51Z","updated_at":"2015-08-02T09:06:51Z","body":"  [Test build #39411 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39411/console) for   PR 7752 at commit [`20d1039`](https://github.com/apache/spark/commit/20d103973d0256ad26a258a2c8205a7347754496).\n * This patch **fails Spark unit tests**.\n * This patch merges cleanly.\n * This patch adds the following public classes _(experimental)_:\n  * `public class UnsafeArrayData extends ArrayData `\n  * `  public static class ArrayWriter `\n  * `public class UnsafeWriters `\n  * `  public static class DecimalWriter `\n  * `  public static class UTF8StringWriter `\n  * `  public static class BinaryWriter `\n  * `  public static class StructWriter `\n  * `  public static class IntervalWriter `\n  * `  public static class ArrayWriter `\n  * `case class FromUnsafe(child: Expression) extends UnaryExpression`\n  * `      class SpecificSafeProjection extends $`\n  * `case class FromUTCTimestamp(left: Expression, right: Expression)`\n  * `case class ToUTCTimestamp(left: Expression, right: Expression)`\n  * `case class DateDiff(endDate: Expression, startDate: Expression)`\n  * `case class InitCap(child: Expression) extends UnaryExpression with ImplicitCastInputTypes `\n"}}