{"action":"created","issue":{"url":"https://api.github.com/repos/twitter/scalding/issues/1154","labels_url":"https://api.github.com/repos/twitter/scalding/issues/1154/labels{/name}","comments_url":"https://api.github.com/repos/twitter/scalding/issues/1154/comments","events_url":"https://api.github.com/repos/twitter/scalding/issues/1154/events","html_url":"https://github.com/twitter/scalding/issues/1154","id":53823395,"number":1154,"title":"Problems with tuples and map-side reduce.","user":{"login":"wbertelsen","id":1421662,"avatar_url":"https://avatars.githubusercontent.com/u/1421662?v=3","gravatar_id":"","url":"https://api.github.com/users/wbertelsen","html_url":"https://github.com/wbertelsen","followers_url":"https://api.github.com/users/wbertelsen/followers","following_url":"https://api.github.com/users/wbertelsen/following{/other_user}","gists_url":"https://api.github.com/users/wbertelsen/gists{/gist_id}","starred_url":"https://api.github.com/users/wbertelsen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wbertelsen/subscriptions","organizations_url":"https://api.github.com/users/wbertelsen/orgs","repos_url":"https://api.github.com/users/wbertelsen/repos","events_url":"https://api.github.com/users/wbertelsen/events{/privacy}","received_events_url":"https://api.github.com/users/wbertelsen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-01-09T01:23:09Z","updated_at":"2015-01-09T01:39:38Z","closed_at":null,"body":"At Foursquare all our hadoop jobs consume and output thrift (https://thrift.apache.org/) records full of optional fields. \r\n\r\nI've run into an issue where, under certain conditions when putting my thrift objects in a tuple, by the time the come back out of the Scalding/Cascading tupleing process all their fields are missing. Specifically this seems tied to instances where Scalding will opt to do a map-side reduce - In the example below everything is fine once I add a .forceToReducers call.\r\n\r\nie, if I do:\r\n```scala \r\n val thriftData: TypedPipe[(ThriftKey, ThriftValue)] = ...\r\n  val processedThriftData = {\r\n    thriftData.groupBy(_._1.optionalSubKeyOrDefault).head.toTypedPipe.flatMap {\r\n      case (subKey, (thriftKey, thriftValue)) => process(thriftKey, thriftValue)\r\n    }\r\n  }\r\n```\r\nBy the time my data gets to `process` the fields of both `thriftKey` and `thriftValue` will be missing.\r\n\r\non the other hand if i do:\r\n```scala \r\n val thriftData: TypedPipe[(ThriftKey, ThriftValue)] = ...\r\n  val processedThriftData = {\r\n    thriftData.groupBy(_._1.optionalSubKeyOrDefault).forceToReducers.head.toTypedPipe.flatMap {\r\n      case (subKey, (thriftKey, thriftValue)) => process(thriftKey, thriftValue)\r\n    }\r\n  }\r\n```\r\neverything is dandy in `process`.\r\n\r\nI think the missing fields is a symptom of something throwing away the `thriftValue` bytes but still successfully deserializing it back into a valid `ThriftValue`.\r\n\r\nMy suspicion is that without the `forceToReducers` when `.head` gets called it ends up doing a `sumByLocalKeys`, which uses a `singleConverter` which screws up my data.\r\n\r\nI think there is a bug here, but I could also be doing something dumb. Any ideas?"},"comment":{"url":"https://api.github.com/repos/twitter/scalding/issues/comments/69281456","html_url":"https://github.com/twitter/scalding/issues/1154#issuecomment-69281456","issue_url":"https://api.github.com/repos/twitter/scalding/issues/1154","id":69281456,"user":{"login":"ianoc","id":446652,"avatar_url":"https://avatars.githubusercontent.com/u/446652?v=3","gravatar_id":"","url":"https://api.github.com/users/ianoc","html_url":"https://github.com/ianoc","followers_url":"https://api.github.com/users/ianoc/followers","following_url":"https://api.github.com/users/ianoc/following{/other_user}","gists_url":"https://api.github.com/users/ianoc/gists{/gist_id}","starred_url":"https://api.github.com/users/ianoc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ianoc/subscriptions","organizations_url":"https://api.github.com/users/ianoc/orgs","repos_url":"https://api.github.com/users/ianoc/repos","events_url":"https://api.github.com/users/ianoc/events{/privacy}","received_events_url":"https://api.github.com/users/ianoc/received_events","type":"User","site_admin":false},"created_at":"2015-01-09T01:39:38Z","updated_at":"2015-01-09T01:39:38Z","body":"Do you have the chill thrift serializer wired in ? Kryo fields serializer\r\ncan have problems on occasion. Also do you know if any values have this bad\r\nproperty ?\r\n\r\nOn Thursday, January 8, 2015, Will Bertelsen <notifications@github.com>\r\nwrote:\r\n\r\n> At Foursquare all our hadoop jobs consume and output thrift (\r\n> https://thrift.apache.org/) records full of optional fields.\r\n>\r\n> I've run into an issue where, under certain conditions when putting my\r\n> thrift objects in a tuple, by the time it comes back out of the\r\n> Scalding/Cascading tupleing process all the fields are missing.\r\n> Specifically this seems tied to instances where Scalding will opt to do a\r\n> map side reduce. In the example below everything is fine if I add a\r\n> .forceToReducers call.\r\n>\r\n> ie, if I do:\r\n>\r\n>  val thriftData: TypedPipe[(ThriftKey, ThriftValue)] = ...\r\n>   val processedThriftData = {\r\n>     thriftData.groupBy(_._1.optionalSubKeyOrDefault).head.toTypedPipe.flatMap {\r\n>       case (subKey, (thriftKey, thriftValue)) => process(thriftKey, thriftValue)\r\n>     }\r\n>   }\r\n>\r\n> By the time my data gets to my process function the subfields of my\r\n> thriftKey will still be there, but the subfields of my thriftValue wont.\r\n>\r\n> on the other hand if i do:\r\n>\r\n>  val thriftData: TypedPipe[(ThriftKey, ThriftValue)] = ...\r\n>   val processedThriftData = {\r\n>     thriftData.groupBy(_._1.optionalSubKeyOrDefault).forceToReducers.head.toTypedPipe.flatMap {\r\n>       case (subKey, (thriftKey, thriftValue)) => process(thriftKey, thriftValue)\r\n>     }\r\n>   }\r\n>\r\n> everything is dandy in process.\r\n>\r\n> I think the missing fields is a symptom of something throwing away the\r\n> thriftValue bytes but still successfully deserializing it back into a\r\n> valid ThriftValue.\r\n>\r\n> My suspicion is that without the forceToReducers when .head gets called\r\n> it ends up doing a sumByLocalKeys, which uses a singleConverter, and this\r\n> throws out the second value of my tuple.\r\n>\r\n> I think there is a bug here, but I could also be doing something dumb. Any\r\n> ideas?\r\n>\r\n> —\r\n> Reply to this email directly or view it on GitHub\r\n> <https://github.com/twitter/scalding/issues/1154>.\r\n>"}}