{"action":"created","issue":{"url":"https://api.github.com/repos/databricks/spark-avro/issues/92","labels_url":"https://api.github.com/repos/databricks/spark-avro/issues/92/labels{/name}","comments_url":"https://api.github.com/repos/databricks/spark-avro/issues/92/comments","events_url":"https://api.github.com/repos/databricks/spark-avro/issues/92/events","html_url":"https://github.com/databricks/spark-avro/issues/92","id":111489893,"number":92,"title":"spark-avro 2.0.1 generates strange schema (spark-avro 1.0.0 is fine)","user":{"login":"alexnastetsky","id":9036753,"avatar_url":"https://avatars.githubusercontent.com/u/9036753?v=3","gravatar_id":"","url":"https://api.github.com/users/alexnastetsky","html_url":"https://github.com/alexnastetsky","followers_url":"https://api.github.com/users/alexnastetsky/followers","following_url":"https://api.github.com/users/alexnastetsky/following{/other_user}","gists_url":"https://api.github.com/users/alexnastetsky/gists{/gist_id}","starred_url":"https://api.github.com/users/alexnastetsky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alexnastetsky/subscriptions","organizations_url":"https://api.github.com/users/alexnastetsky/orgs","repos_url":"https://api.github.com/users/alexnastetsky/repos","events_url":"https://api.github.com/users/alexnastetsky/events{/privacy}","received_events_url":"https://api.github.com/users/alexnastetsky/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-10-14T20:48:32Z","updated_at":"2015-10-15T02:09:29Z","closed_at":null,"body":"I save my dataframe to avro with spark-avro 1.0.0 and it looks like this (using avro-tools tojson):\r\n\r\n{\"field1\":\"value1\",\"field2\":976200}\r\n{\"field1\":\"value2\",\"field2\":976200}\r\n{\"field1\":\"value3\",\"field2\":614100}\r\n\r\nBut when I use spark-avro 2.0.1, it looks like this:\r\n\r\n{\"field1\":{\"string\":\"value1\"},\"field2\":{\"long\":976200}}\r\n{\"field1\":{\"string\":\"value2\"},\"field2\":{\"long\":976200}}\r\n{\"field1\":{\"string\":\"value3\"},\"field2\":{\"long\":614100}}\r\n\r\nAt this point I'd be happy to use spark-avro 1.0.0, except that it doesn't seem to support specifying a compression codec (I want deflate)."},"comment":{"url":"https://api.github.com/repos/databricks/spark-avro/issues/comments/148256547","html_url":"https://github.com/databricks/spark-avro/issues/92#issuecomment-148256547","issue_url":"https://api.github.com/repos/databricks/spark-avro/issues/92","id":148256547,"user":{"login":"alexnastetsky","id":9036753,"avatar_url":"https://avatars.githubusercontent.com/u/9036753?v=3","gravatar_id":"","url":"https://api.github.com/users/alexnastetsky","html_url":"https://github.com/alexnastetsky","followers_url":"https://api.github.com/users/alexnastetsky/followers","following_url":"https://api.github.com/users/alexnastetsky/following{/other_user}","gists_url":"https://api.github.com/users/alexnastetsky/gists{/gist_id}","starred_url":"https://api.github.com/users/alexnastetsky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alexnastetsky/subscriptions","organizations_url":"https://api.github.com/users/alexnastetsky/orgs","repos_url":"https://api.github.com/users/alexnastetsky/repos","events_url":"https://api.github.com/users/alexnastetsky/events{/privacy}","received_events_url":"https://api.github.com/users/alexnastetsky/received_events","type":"User","site_admin":false},"created_at":"2015-10-15T02:09:29Z","updated_at":"2015-10-15T02:09:29Z","body":"The reason it thinks it's nullable=true in 2.0.1 is because the DefaultSource class was changed from CreatableRelationProvider to HadoopFsRelationProvider, and in ResolvedDataSource#apply in the HadoopFsRelationProvider case, it uses dataSchema.asNullable to create the AvroRelation."}}