{"action":"created","issue":{"url":"https://api.github.com/repos/ScaleUnlimited/cascading.avro/issues/36","labels_url":"https://api.github.com/repos/ScaleUnlimited/cascading.avro/issues/36/labels{/name}","comments_url":"https://api.github.com/repos/ScaleUnlimited/cascading.avro/issues/36/comments","events_url":"https://api.github.com/repos/ScaleUnlimited/cascading.avro/issues/36/events","html_url":"https://github.com/ScaleUnlimited/cascading.avro/pull/36","id":56814788,"number":36,"title":"Unified AvroSchemes with depth parameter, made conversion logic pluggable, supported unions","user":{"login":"silasdavis","id":99715,"avatar_url":"https://avatars.githubusercontent.com/u/99715?v=3","gravatar_id":"","url":"https://api.github.com/users/silasdavis","html_url":"https://github.com/silasdavis","followers_url":"https://api.github.com/users/silasdavis/followers","following_url":"https://api.github.com/users/silasdavis/following{/other_user}","gists_url":"https://api.github.com/users/silasdavis/gists{/gist_id}","starred_url":"https://api.github.com/users/silasdavis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/silasdavis/subscriptions","organizations_url":"https://api.github.com/users/silasdavis/orgs","repos_url":"https://api.github.com/users/silasdavis/repos","events_url":"https://api.github.com/users/silasdavis/events{/privacy}","received_events_url":"https://api.github.com/users/silasdavis/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2015-02-06T14:00:46Z","updated_at":"2015-02-17T19:14:25Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/ScaleUnlimited/cascading.avro/pulls/36","html_url":"https://github.com/ScaleUnlimited/cascading.avro/pull/36","diff_url":"https://github.com/ScaleUnlimited/cascading.avro/pull/36.diff","patch_url":"https://github.com/ScaleUnlimited/cascading.avro/pull/36.patch"},"body":"So I appreciate this is a rather large pull request. I've added various tests and I've tried to not break any existing APIs/contracts. I can look at splitting some of the changes out if it's really necessary, but some of them are linked and quite a few aspects of the project felt like they could use a cleanup.\r\n\r\n**Version bumps:**\r\n- Hadoop 1.0.3 -> 2.4.0\r\n- Cascading 2.1.6 -> 2.6.1\r\n- Avro 1.7.4 -> 1.7.7\r\n\r\nThe old versions of Avro in particular were causing problems. For my usage I also required newer version of Cascading and Hadoop. It it is possibly and necessary to maintain different Hadoop/Cascading version I suggest we do so on separate branches.\r\n\r\n**Changes:**\r\n- Unified the notions of AvroScheme and PackedAvroScheme with a depth paramter (how far should we unpack records into TupleEntries). This is mostly to support ShallowAvroScheme which is a TupleEntry representing an outer Avro record, but leaves any first children Avro record fields as Avro records. Adding AvroSpecificRecordSerialization to the JobConf `io.serializations` means we get intermediate serializability when doing this.\r\n- Made the conversion logic between Avro and 'Cascading' types pluggable. There are various ways we may want to represent logical Avro records in Cascading, I've split and tidied up this logic so that you can provide your own classes implementing the interface AvroConverter<TRecordFrom,TRecordTo>, or you subclass one of the conversion classes (themselves sharing common traversal logic, pivoting on a Schema, in AvroConverterBase).\r\n- Supported fairly arbitary union types. It is conceivable that with a weird-enough union type (such as one containing a string array and a string map) that the conversion won't give you what you want, but I think it most cases it should be adequate and certainly improves on what was there.\r\n- Increased the flexibility of Avro conversion logic with respect to `byte[]`, `List`, `TupleEntry`, `Map` etc by capturing the many-to-many nature of the type mappings in `TypeMappings`\r\n- Standardised by representing Avro maps and arrays as Tuples, and records as TupleEntries in Cascading land. Provided helpers to convert to maps. This means intermediate serialisations (such as TempHfs and others) don't break (requiring extra stages in pipeline) when dealing with GroupBy and other aggregations (Cascading doesn't know how to serialise lone java Maps and Lists - but you can embed them in Avro records if you like).\r\n- Made CascadingToAvro serialize into SpecificRecords when they are available via `SpecificData.get().getClass(schema)`. Just downcast the the converted `IndexedRecord`.\r\n- Used functionality in newer Avro to remove reflection from AvroSpecificRecordSerialization\r\n- Updated some test logic\r\n- Added some tests"},"comment":{"url":"https://api.github.com/repos/ScaleUnlimited/cascading.avro/issues/comments/74731630","html_url":"https://github.com/ScaleUnlimited/cascading.avro/pull/36#issuecomment-74731630","issue_url":"https://api.github.com/repos/ScaleUnlimited/cascading.avro/issues/36","id":74731630,"user":{"login":"kkrugler","id":66217,"avatar_url":"https://avatars.githubusercontent.com/u/66217?v=3","gravatar_id":"","url":"https://api.github.com/users/kkrugler","html_url":"https://github.com/kkrugler","followers_url":"https://api.github.com/users/kkrugler/followers","following_url":"https://api.github.com/users/kkrugler/following{/other_user}","gists_url":"https://api.github.com/users/kkrugler/gists{/gist_id}","starred_url":"https://api.github.com/users/kkrugler/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kkrugler/subscriptions","organizations_url":"https://api.github.com/users/kkrugler/orgs","repos_url":"https://api.github.com/users/kkrugler/repos","events_url":"https://api.github.com/users/kkrugler/events{/privacy}","received_events_url":"https://api.github.com/users/kkrugler/received_events","type":"User","site_admin":false},"created_at":"2015-02-17T19:14:25Z","updated_at":"2015-02-17T19:14:25Z","body":"Agreed re version update - go to 2.6.0? And bump Cascading version dependency from 2.5.5 to 2.6.3? I'm thinking we should save the 3.0 bump for when it's a release that targets Cascading 3.x"}}