{"action":"created","issue":{"url":"https://api.github.com/repos/mtth/hdfs/issues/31","labels_url":"https://api.github.com/repos/mtth/hdfs/issues/31/labels{/name}","comments_url":"https://api.github.com/repos/mtth/hdfs/issues/31/comments","events_url":"https://api.github.com/repos/mtth/hdfs/issues/31/events","html_url":"https://github.com/mtth/hdfs/issues/31","id":109159400,"number":31,"title":"AvroWrite exponentional file size","user":{"login":"zoltan-fedor","id":3887295,"avatar_url":"https://avatars.githubusercontent.com/u/3887295?v=3","gravatar_id":"","url":"https://api.github.com/users/zoltan-fedor","html_url":"https://github.com/zoltan-fedor","followers_url":"https://api.github.com/users/zoltan-fedor/followers","following_url":"https://api.github.com/users/zoltan-fedor/following{/other_user}","gists_url":"https://api.github.com/users/zoltan-fedor/gists{/gist_id}","starred_url":"https://api.github.com/users/zoltan-fedor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zoltan-fedor/subscriptions","organizations_url":"https://api.github.com/users/zoltan-fedor/orgs","repos_url":"https://api.github.com/users/zoltan-fedor/repos","events_url":"https://api.github.com/users/zoltan-fedor/events{/privacy}","received_events_url":"https://api.github.com/users/zoltan-fedor/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-09-30T19:13:02Z","updated_at":"2015-10-01T00:10:24Z","closed_at":null,"body":"I have found a very interesting issue with AvroWriter, which breaks the whole writing process.\r\nBasically the files size start growing exponentially.\r\nWhen I export 100 records, the file size is 10Kb, when I export 200 records (of the same), the file size is 670Kb!!!\r\n\r\nI use the standard AvroWriter example (except added a long string value to make the data bigger).\r\nHow to reproduce:\r\nThe below code exports 2 records. just take those 2 records and copy-paste them until you don't have 200 records of the same and then run the export. Observe that the file size became 670Kb.\r\nRemove half of the records, leave only 100 records. Export again, observe that the file size is 10Kb!\r\n\r\n```\r\n\"\"\"Avro extension example.\"\"\"\r\nfrom hdfs import InsecureClient\r\nfrom hdfs.ext.avro import AvroReader, AvroWriter\r\nimport os\r\n\r\nos.environ['http_proxy'] = ''\r\nos.environ['user'] = 'my_user'\r\n\r\n# Get the default alias' client.\r\nclient = InsecureClient('http://xxx:50070', user='my_user')\r\n\r\n# Some sample data.\r\nrecords = [\r\n    {'name': 'Ann', 'age': 23, 'string': 'asdasd asklhj doiuiroe t43785 iofdjkvnuhyt789u judffkguirety9 jdfvhjkfcvhuierytuiy348957hjdfh jkdfgirhytiery5689347589'},\r\n    {'name': 'Bob', 'age': 22, 'string': 'nkdfn kjfhy tuier vgfhe89734 89hIUHIUJKBNUIh iuhiuh48 9 5u7489jkldjvlcdjvoiejuioweufoksdfsdf'},\r\n]\r\n\r\n# Write an Avro File to HDFS (since our records' schema is very simple, we let\r\n# the writer infer it automatically, otherwise we would pass it as argument).\r\n# codec can be null, deflate or snappy\r\nwith AvroWriter(client, '/user/my_user/names.avro', overwrite=True) as writer:\r\n    for record in records:\r\n        writer.write(record)\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/mtth/hdfs/issues/comments/144578994","html_url":"https://github.com/mtth/hdfs/issues/31#issuecomment-144578994","issue_url":"https://api.github.com/repos/mtth/hdfs/issues/31","id":144578994,"user":{"login":"mtth","id":1216372,"avatar_url":"https://avatars.githubusercontent.com/u/1216372?v=3","gravatar_id":"","url":"https://api.github.com/users/mtth","html_url":"https://github.com/mtth","followers_url":"https://api.github.com/users/mtth/followers","following_url":"https://api.github.com/users/mtth/following{/other_user}","gists_url":"https://api.github.com/users/mtth/gists{/gist_id}","starred_url":"https://api.github.com/users/mtth/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mtth/subscriptions","organizations_url":"https://api.github.com/users/mtth/orgs","repos_url":"https://api.github.com/users/mtth/repos","events_url":"https://api.github.com/users/mtth/events{/privacy}","received_events_url":"https://api.github.com/users/mtth/received_events","type":"User","site_admin":false},"created_at":"2015-10-01T00:10:24Z","updated_at":"2015-10-01T00:10:24Z","body":"Great, I can reproduce it! Will look into it tonight."}}