{"action":"created","issue":{"url":"https://api.github.com/repos/twitter/scalding/issues/1202","labels_url":"https://api.github.com/repos/twitter/scalding/issues/1202/labels{/name}","comments_url":"https://api.github.com/repos/twitter/scalding/issues/1202/comments","events_url":"https://api.github.com/repos/twitter/scalding/issues/1202/events","html_url":"https://github.com/twitter/scalding/issues/1202","id":58150490,"number":1202,"title":"Scalding, Parquet, Thrift, and Scrooge: Column Projection is Wonky","user":{"login":"Gastove","id":3824934,"avatar_url":"https://avatars.githubusercontent.com/u/3824934?v=3","gravatar_id":"","url":"https://api.github.com/users/Gastove","html_url":"https://github.com/Gastove","followers_url":"https://api.github.com/users/Gastove/followers","following_url":"https://api.github.com/users/Gastove/following{/other_user}","gists_url":"https://api.github.com/users/Gastove/gists{/gist_id}","starred_url":"https://api.github.com/users/Gastove/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Gastove/subscriptions","organizations_url":"https://api.github.com/users/Gastove/orgs","repos_url":"https://api.github.com/users/Gastove/repos","events_url":"https://api.github.com/users/Gastove/events{/privacy}","received_events_url":"https://api.github.com/users/Gastove/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-02-19T00:17:49Z","updated_at":"2015-02-19T00:30:52Z","closed_at":null,"body":"I have a file called common.thrift that defines types and structs we use all over the place. It looks a bit like this:\r\n\r\n```\r\ntypedef string ID\r\n\r\nenum IdType {\r\n  APID = 1,\r\n  UDID = 2\r\n}\r\n\r\nstruct Identity {\r\n  1: Id id\r\n  2: IdType id_type\r\n}\r\n```\r\nThen we have a record of an app open, so:\r\n\r\n```\r\ninclude \"common.thrift\"\r\n\r\nstruct Open {\r\n  1: common.Identity identity\r\n  2: common.ID app_id\r\n}\r\n```\r\n\r\nI'm compiling my thrift definitions with Scrooge 3.16.3 for Scala 2.10.x and libthrift for Java. \r\n\r\nFor my next trick, I use a Cascading job to generate a big ol' pile of Thrift objects, stored in Parquet. This goes swimmingly. Now, over in Scala-land again, I define a source for my Thrift objects:\r\n\r\n```\r\ncase class Opens(\r\n  path: String,\r\n  override val withFilter: Option[FilterPredicate] = None,\r\n  override val withColumns: Set[String] = Set()\r\n) extends FixedPathParquetScrooge[Open](path)\r\n```\r\n\r\nAnd try to read it, using column projection, in roughly the simplest way possible:\r\n\r\n```\r\nclass PrintOpens(args: Args) extends Job(args) {\r\n\r\n  val path = // path to parquet file in hdfs\r\n  TypedPipe.from(Opens(path, withColumns=Set(\"identity/**\")))\r\n    .map{ o => o.identity.id }\r\n    .write(TypedTsv(conf.getString(\"output_root\") + \"/column_subselect_out\"))\r\n}\r\n```\r\n\r\nWhen I try to run this (using `hadoop jar my-shaded-jar.jar com.twitter.scalding.Tool com.my.jar.PrintOpens --hdfs`), I get:\r\n\r\n```\r\nCaused by: parquet.io.InvalidRecordException: required group identity = 1 {\r\n  required binary id (UTF8) = 1;\r\n  required binary id_type (ENUM) = 2;\r\n  required int32 id_type = 2;\r\n} found: expected optional group identity {\r\n  optional binary id (UTF8);\r\n  optional binary id_type (ENUM);\r\n}\r\n```\r\n\r\nIf I swap out `FixedPathParquetScrooge` for `FixedPathParquetThrift` and use the libthrift-generated Java definition of an Open, this works perfectly.\r\n\r\nThoughts?"},"comment":{"url":"https://api.github.com/repos/twitter/scalding/issues/comments/74978656","html_url":"https://github.com/twitter/scalding/issues/1202#issuecomment-74978656","issue_url":"https://api.github.com/repos/twitter/scalding/issues/1202","id":74978656,"user":{"login":"tsdeng","id":169678,"avatar_url":"https://avatars.githubusercontent.com/u/169678?v=3","gravatar_id":"","url":"https://api.github.com/users/tsdeng","html_url":"https://github.com/tsdeng","followers_url":"https://api.github.com/users/tsdeng/followers","following_url":"https://api.github.com/users/tsdeng/following{/other_user}","gists_url":"https://api.github.com/users/tsdeng/gists{/gist_id}","starred_url":"https://api.github.com/users/tsdeng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tsdeng/subscriptions","organizations_url":"https://api.github.com/users/tsdeng/orgs","repos_url":"https://api.github.com/users/tsdeng/repos","events_url":"https://api.github.com/users/tsdeng/events{/privacy}","received_events_url":"https://api.github.com/users/tsdeng/received_events","type":"User","site_admin":false},"created_at":"2015-02-19T00:30:52Z","updated_at":"2015-02-19T00:30:52Z","body":"This problem was addressed in parquet master already. It's just not been released.\r\nIt's triggered when you have fields that are not \"optional\" nor \"required\".\r\nIf you annotate your fields explicitly with \"optional\" or \"required\", this problem should go away"}}