{"action":"created","issue":{"url":"https://api.github.com/repos/facebook/presto/issues/3827","labels_url":"https://api.github.com/repos/facebook/presto/issues/3827/labels{/name}","comments_url":"https://api.github.com/repos/facebook/presto/issues/3827/comments","events_url":"https://api.github.com/repos/facebook/presto/issues/3827/events","html_url":"https://github.com/facebook/presto/issues/3827","id":113252127,"number":3827,"title":"NullPointerException when reading parquet data from S3","user":{"login":"Cidan","id":1133628,"avatar_url":"https://avatars.githubusercontent.com/u/1133628?v=3","gravatar_id":"","url":"https://api.github.com/users/Cidan","html_url":"https://github.com/Cidan","followers_url":"https://api.github.com/users/Cidan/followers","following_url":"https://api.github.com/users/Cidan/following{/other_user}","gists_url":"https://api.github.com/users/Cidan/gists{/gist_id}","starred_url":"https://api.github.com/users/Cidan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Cidan/subscriptions","organizations_url":"https://api.github.com/users/Cidan/orgs","repos_url":"https://api.github.com/users/Cidan/repos","events_url":"https://api.github.com/users/Cidan/events{/privacy}","received_events_url":"https://api.github.com/users/Cidan/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-10-25T21:04:42Z","updated_at":"2015-10-25T23:31:43Z","closed_at":null,"body":"I've exported about 350 million records from our Cassandra database to S3 using Spark in parquet format. I then created an external table in our stand alone Hive/Hadoop cluster that points to this data, however Presto is throwing errors when certain columns are selected.\r\n\r\nOur setup is like so:\r\n\r\n* n Cassandra servers running Spark on top of them.\r\n* n Hadoop servers running hive 0.13.1\r\n* 30 Presto workers\r\n\r\nHere is a partial schema. Unfortunately I am unable to provide the full schema, however these columns should suffice for the purposes of this issue.\r\n\r\n```\r\n            Column             |  Type   | Null | Partition Key | Comment\r\n-------------------------------+---------+------+---------------+---------\r\n puid                          | varchar | true | false         |\r\n ts                            | bigint  | true | false         |\r\n uid                           | varchar | true | false         |\r\n```\r\n\r\nThe error I get is:\r\n```\r\n2015-10-25T20:04:29.855Z        ERROR   SplitRunner-35-35       com.facebook.presto.execution.TaskExecutor      Error processing Split 20151025_200427_00176_bu25r.1.6-1 {path=s3n://xxx/part-r-00104-899794d4-cc41-4a52-8cbf-5e7f89671242.gz.parquet, start=0, length=18262360, hosts=[localhost], database=default, table=xxx, forceLocalScheduling=false, partitionName=<UNPARTITIONED>, effectivePredicate=TupleDomain:ALL} (start = 1445803468298, wall = 1557 ms, cpu = 0 ms, calls = 1)\r\njava.lang.NullPointerException\r\n        at com.facebook.presto.spi.RecordPageSource.getNextPage(RecordPageSource.java:118)\r\n        at com.facebook.presto.operator.TableScanOperator.getOutput(TableScanOperator.java:239)\r\n        at com.facebook.presto.operator.Driver.processInternal(Driver.java:380)\r\n        at com.facebook.presto.operator.Driver.processFor(Driver.java:303)\r\n        at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:587)\r\n        at com.facebook.presto.execution.TaskExecutor$PrioritizedSplitRunner.process(TaskExecutor.java:505)\r\n        at com.facebook.presto.execution.TaskExecutor$Runner.run(TaskExecutor.java:640)\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n        at java.lang.Thread.run(Thread.java:745)\r\n```\r\n\r\nHere are the various queries and their results:\r\n\r\n| Query | Result |\r\n--- | ---\r\n`select puid from xxx` | works\r\n`select count(puid) from xxx` | works\r\n`select uid,ts,puid from xxx where uid is null` | works? (shows results with ts 0, incorrect)\r\n`select count(uid) from xxx where uid is null` | works? returns 0, which is untrue\r\n`select uid from xxx` | error\r\n`select uid from xxx where uid is not null` | error\r\n`select uid,puid,ts from xxx` | error\r\n\r\nA few things to note:\r\n* This error happens multiple times per query, each error with a different parquet file in S3 (part-r-nnnnn)\r\n* The same error happens with multiple different types of queries.\r\n* These same files can be read back via Spark or Hive, parsed and transformed, without issue.\r\n* The UID isn't the only column this happens on, it's just the one I picked for this example.\r\n* This data is sparse; some rows might be missing a column.\r\n\r\nThe pattern here is when Presto tries to load a UID that isn't a NULL, it gets the above error.\r\n\r\nIf I can provide any more information, please let me know. Thanks!"},"comment":{"url":"https://api.github.com/repos/facebook/presto/issues/comments/150987274","html_url":"https://github.com/facebook/presto/issues/3827#issuecomment-150987274","issue_url":"https://api.github.com/repos/facebook/presto/issues/3827","id":150987274,"user":{"login":"zhenxiao","id":1274155,"avatar_url":"https://avatars.githubusercontent.com/u/1274155?v=3","gravatar_id":"","url":"https://api.github.com/users/zhenxiao","html_url":"https://github.com/zhenxiao","followers_url":"https://api.github.com/users/zhenxiao/followers","following_url":"https://api.github.com/users/zhenxiao/following{/other_user}","gists_url":"https://api.github.com/users/zhenxiao/gists{/gist_id}","starred_url":"https://api.github.com/users/zhenxiao/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zhenxiao/subscriptions","organizations_url":"https://api.github.com/users/zhenxiao/orgs","repos_url":"https://api.github.com/users/zhenxiao/repos","events_url":"https://api.github.com/users/zhenxiao/events{/privacy}","received_events_url":"https://api.github.com/users/zhenxiao/received_events","type":"User","site_admin":false},"created_at":"2015-10-25T23:31:43Z","updated_at":"2015-10-25T23:31:43Z","body":"@Cidan great you get it working. Aha, that setting is enable access parquet files by column name, instead of by column ordinal position. Seems like you do not have complex types at all. I think ORC is always accessing columns by column index, not sure about the details, let's ask @dain \r\nJust curious, did you evert do alter table rename to your table or alter table add columns to your table? Or there should not be problems for either column index access or column name access"}}