{"action":"closed","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/397","labels_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/397/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/397/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/397/events","html_url":"https://github.com/elastic/elasticsearch-hadoop/issues/397","id":62182633,"number":397,"title":"Keep date formatting behavior aligned with ES API for spark serialization","user":{"login":"mgreene","id":269244,"avatar_url":"https://avatars.githubusercontent.com/u/269244?v=3","gravatar_id":"","url":"https://api.github.com/users/mgreene","html_url":"https://github.com/mgreene","followers_url":"https://api.github.com/users/mgreene/followers","following_url":"https://api.github.com/users/mgreene/following{/other_user}","gists_url":"https://api.github.com/users/mgreene/gists{/gist_id}","starred_url":"https://api.github.com/users/mgreene/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mgreene/subscriptions","organizations_url":"https://api.github.com/users/mgreene/orgs","repos_url":"https://api.github.com/users/mgreene/repos","events_url":"https://api.github.com/users/mgreene/events{/privacy}","received_events_url":"https://api.github.com/users/mgreene/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/labels/cascading","name":"cascading","color":"e102d8"},{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/labels/enhancement","name":"enhancement","color":"fbca04"},{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/labels/hive","name":"hive","color":"02d7e1"},{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/labels/pig","name":"pig","color":"0b02e1"},{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/labels/spark","name":"spark","color":"006b75"},{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/labels/storm","name":"storm","color":"f7c6c7"},{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/labels/v2.1.0.Beta4","name":"v2.1.0.Beta4","color":"bfdadc"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-03-16T19:15:49Z","updated_at":"2015-04-28T12:34:59Z","closed_at":"2015-04-28T12:34:59Z","body":"I've noticed that when streaming documents into spark with fields that have type `date`, the format that is passed to the spark job itself is that of the `java.util.Date` default. This occurs even if the underlying document has UTC millis as its format. The problem we encountered is when we reindex our data, we stream in one format, and write out a different one.\r\n\r\nI tracked this behavior down to this serializer: https://github.com/elastic/elasticsearch-hadoop/blob/master/spark/src/main/scala/org/elasticsearch/spark/serialization/ScalaValueReader.scala#L100-L101\r\n\r\nI bring this up because I think this is more than a implementation preference. The ES API itself returns the format in which it was indexed with so I think that's justification for making sure the serializer behaves the same way.\r\n\r\nThe behavior in the spark connector also differs from that of the hadoop implementation, which is aligned with how the REST API behaves."}}