{"action":"closed","issue":{"url":"https://api.github.com/repos/CommBank/ebenezer/issues/93","labels_url":"https://api.github.com/repos/CommBank/ebenezer/issues/93/labels{/name}","comments_url":"https://api.github.com/repos/CommBank/ebenezer/issues/93/comments","events_url":"https://api.github.com/repos/CommBank/ebenezer/issues/93/events","html_url":"https://github.com/CommBank/ebenezer/issues/93","id":63875059,"number":93,"title":"Parquet reading throws ParquetDecodingException . ","user":{"login":"Vidhyaparvathy","id":7449760,"avatar_url":"https://avatars.githubusercontent.com/u/7449760?v=3","gravatar_id":"","url":"https://api.github.com/users/Vidhyaparvathy","html_url":"https://github.com/Vidhyaparvathy","followers_url":"https://api.github.com/users/Vidhyaparvathy/followers","following_url":"https://api.github.com/users/Vidhyaparvathy/following{/other_user}","gists_url":"https://api.github.com/users/Vidhyaparvathy/gists{/gist_id}","starred_url":"https://api.github.com/users/Vidhyaparvathy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Vidhyaparvathy/subscriptions","organizations_url":"https://api.github.com/users/Vidhyaparvathy/orgs","repos_url":"https://api.github.com/users/Vidhyaparvathy/repos","events_url":"https://api.github.com/users/Vidhyaparvathy/events{/privacy}","received_events_url":"https://api.github.com/users/Vidhyaparvathy/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/CommBank/ebenezer/labels/in%20progress","name":"in progress","color":"ededed"}],"state":"closed","locked":false,"assignee":{"login":"rowandavies","id":468519,"avatar_url":"https://avatars.githubusercontent.com/u/468519?v=3","gravatar_id":"","url":"https://api.github.com/users/rowandavies","html_url":"https://github.com/rowandavies","followers_url":"https://api.github.com/users/rowandavies/followers","following_url":"https://api.github.com/users/rowandavies/following{/other_user}","gists_url":"https://api.github.com/users/rowandavies/gists{/gist_id}","starred_url":"https://api.github.com/users/rowandavies/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rowandavies/subscriptions","organizations_url":"https://api.github.com/users/rowandavies/orgs","repos_url":"https://api.github.com/users/rowandavies/repos","events_url":"https://api.github.com/users/rowandavies/events{/privacy}","received_events_url":"https://api.github.com/users/rowandavies/received_events","type":"User","site_admin":false},"milestone":null,"comments":11,"created_at":"2015-03-24T01:30:42Z","updated_at":"2015-04-12T23:00:37Z","closed_at":"2015-04-12T23:00:37Z","body":"We were trying to find the delta between two tables which has similar column names . One table had data in text ( a table from default) and the other had it in parquet (table from pr). The parquet table  was the result of an old maestro job which does not have any \"optional\" or \"required\"  in the thrift .  When we ran the job it showed an error like , required binary \"columnname\" cannot be converted to optional binary  \"columnname \". We tried specifying all columns as \"required\" in the thrift and ran it again . Got the same error . Tried running with \"optional\" in thrift , and got the same error . \r\n```\r\nat parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:189)\r\n    at parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:130)\r\n    at parquet.hadoop.mapred.DeprecatedParquetInputFormat$RecordReaderWrapper.<init>(DeprecatedParquetInputFormat.java:94)\r\n    at parquet.hadoop.mapred.DeprecatedParquetInputFormat.getRecordReader(DeprecatedParquetInputFormat.java:47)\r\n    at cascading.tap.hadoop.io.MultiInputFormat$1.operate(MultiInputFormat.java:253)\r\n    at cascading.tap.hadoop.io.MultiInputFormat$1.operate(MultiInputFormat.java:248)\r\n    at cascading.util.Util.retry(Util.java:762)\r\n    at cascading.tap.hadoop.io.MultiInputFormat.getRecordReader(MultiInputFormat.java:247)\r\n    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:394)\r\n    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)\r\n    at org.apache.hadoop.mapred.Child$4.run(Child.java:268)\r\n    at java.security.AccessController.doPrivileged(Native Method)\r\n    at javax.security.auth.Subject.doAs(Subject.java:415)\r\n    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)\r\n    at org.apache.hadoop.mapred.Child.main(Child.java:262)\r\nCaused by: java.lang.NullPointerException\r\n    at parquet.io.RecordReaderImplementation.<init>(RecordReaderImplementation.java:264)\r\n    at parquet.io.MessageColumnIO.getRecordReader(MessageColumnIO.java:60)\r\n    at parquet.io.MessageColumnIO.getRecordReader(MessageColumnIO.java:74)\r\n    at parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:110)\r\n    at parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:174)\r\n    ... 14 more\r\n2015-03-24 10:24:06,252 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1\r\n2015-03-24 10:24:06,255 WARN org.apache.hadoop.mapred.Child: Error running child\r\nparquet.io.ParquetDecodingException: Can not read value at 0 in block -1 in file \r\n    at parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:189)\r\n    at parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:130)\r\n    at parquet.hadoop.mapred.DeprecatedParquetInputFormat$RecordReaderWrapper.<init>(DeprecatedParquetInputFormat.java:94)\r\n    at parquet.hadoop.mapred.DeprecatedParquetInputFormat.getRecordReader(DeprecatedParquetInputFormat.java:47)\r\n    at cascading.tap.hadoop.io.MultiInputFormat$1.operate(MultiInputFormat.java:253)\r\n    at cascading.tap.hadoop.io.MultiInputFormat$1.operate(MultiInputFormat.java:248)\r\n    at cascading.util.Util.retry(Util.java:762)\r\n    at cascading.tap.hadoop.io.MultiInputFormat.getRecordReader(MultiInputFormat.java:247)\r\n    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:394)\r\n    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)\r\n    at org.apache.hadoop.mapred.Child$4.run(Child.java:268)\r\n    at java.security.AccessController.doPrivileged(Native Method)\r\n    at javax.security.auth.Subject.doAs(Subject.java:415)\r\n    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)\r\n    at org.apache.hadoop.mapred.Child.main(Child.java:262)\r\nCaused by: java.lang.NullPointerException\r\n    at parquet.io.RecordReaderImplementation.<init>(RecordReaderImplementation.java:264)\r\n    at parquet.io.MessageColumnIO.getRecordReader(MessageColumnIO.java:60)\r\n    at parquet.io.MessageColumnIO.getRecordReader(MessageColumnIO.java:74)\r\n    at parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:110)\r\n    at parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:174)\r\n    ... 14 more\r\n2015-03-24 10:24:06,259 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task\r\n```\r\n"}}