{"action":"created","issue":{"url":"https://api.github.com/repos/geotrellis/geotrellis/issues/1138","labels_url":"https://api.github.com/repos/geotrellis/geotrellis/issues/1138/labels{/name}","comments_url":"https://api.github.com/repos/geotrellis/geotrellis/issues/1138/comments","events_url":"https://api.github.com/repos/geotrellis/geotrellis/issues/1138/events","html_url":"https://github.com/geotrellis/geotrellis/issues/1138","id":95230677,"number":1138,"title":"Use of Kryo for persistence","user":{"login":"echeipesh","id":1158084,"avatar_url":"https://avatars.githubusercontent.com/u/1158084?v=3","gravatar_id":"","url":"https://api.github.com/users/echeipesh","html_url":"https://github.com/echeipesh","followers_url":"https://api.github.com/users/echeipesh/followers","following_url":"https://api.github.com/users/echeipesh/following{/other_user}","gists_url":"https://api.github.com/users/echeipesh/gists{/gist_id}","starred_url":"https://api.github.com/users/echeipesh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/echeipesh/subscriptions","organizations_url":"https://api.github.com/users/echeipesh/orgs","repos_url":"https://api.github.com/users/echeipesh/repos","events_url":"https://api.github.com/users/echeipesh/events{/privacy}","received_events_url":"https://api.github.com/users/echeipesh/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-07-15T16:06:01Z","updated_at":"2015-09-11T17:36:16Z","closed_at":"2015-09-11T17:36:16Z","body":"Just spent a good chunk of time debugging a problem that can be summarized as:\r\n\r\nKryo serializer/de-serializer does not behave as expected when running program through `spark-submit` vs through `sbt run`. To clarify: in order to use `sbt run` `provided` tag must be removed from `spark-core` in dependencies list.\r\n\r\nThe error can have two forms: \r\n**If classes being serialized with kryo are registered** \r\n\r\nWhile deserializing `(LayerId, KeyIndex[K])` in AttributeStore.\r\nSerialization from `spark-submit` kryo will be one lower than then serialization from `sbt run` kryo in the   first byte. (ex: `Array[Byte](51, 1)` vs `Array[Byte](52, 1))`. The first byte represents and ID kryo has assigned to the class when it is registered and the offset by one is preserved if the order of registration is changed.\r\n\r\n**If classes being serialized are not registered**\r\n\r\nUnregistered classes forces Kyro to use full class names when serializing and the  `(LayerId, KeyIndex[K])` tuple is deserialized without issue. (the serialized bytes match up exactly).\r\n\r\nHowever when reading `(K, Tile)` through the tile reader several things are possible:\r\n\"Unregistered class ID 0000 encountered\" when some other common types are registered with Kryo.\r\nor \"scala.Tuple1 cannot be cast to scala.Tuple2\" when no additional classes are registered with Kryo.\r\n\r\nIn both cases running the program through `spark-submit` allows it to correctly deserialize both the metadata and the tile tuples.\r\n\r\n**Why**\r\nWe pressgang the spark `KryoSerializer` into our service here. A good guess is that it somehow goes through different initialization between the two scenarios. \r\n\r\n"},"comment":{"url":"https://api.github.com/repos/geotrellis/geotrellis/issues/comments/139608958","html_url":"https://github.com/geotrellis/geotrellis/issues/1138#issuecomment-139608958","issue_url":"https://api.github.com/repos/geotrellis/geotrellis/issues/1138","id":139608958,"user":{"login":"echeipesh","id":1158084,"avatar_url":"https://avatars.githubusercontent.com/u/1158084?v=3","gravatar_id":"","url":"https://api.github.com/users/echeipesh","html_url":"https://github.com/echeipesh","followers_url":"https://api.github.com/users/echeipesh/followers","following_url":"https://api.github.com/users/echeipesh/following{/other_user}","gists_url":"https://api.github.com/users/echeipesh/gists{/gist_id}","starred_url":"https://api.github.com/users/echeipesh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/echeipesh/subscriptions","organizations_url":"https://api.github.com/users/echeipesh/orgs","repos_url":"https://api.github.com/users/echeipesh/repos","events_url":"https://api.github.com/users/echeipesh/events{/privacy}","received_events_url":"https://api.github.com/users/echeipesh/received_events","type":"User","site_admin":false},"created_at":"2015-09-11T17:36:16Z","updated_at":"2015-09-11T17:36:16Z","body":"Resolved with use of Avro for catalogs"}}