Reading xml failing due to size -can it be read line by line in python?
<p>The following python code works fine when I use it on a test dataset of a few thousand records but when I try it with an xml of about 400MB it runs out of memory. Is there a way to get it to read line by line?</p>

<pre><code>import urllib
import xml.etree.ElementTree as ET

url = raw_input('Enter - ')

value,count,total, counts =0,0,0, dict()

print "Retrieving: ", url
file=urllib.urlopen(url)
data=file.read()
print 'Retrieved',len(data),'characters'
xml=ET.fromstring(data)
tags=xml.findall('.//Postcode')
for tag in tags:
#    print tag.text
    count+=int(tag.text)
print 'Count: ', len(tags)
print 'Sum: ', count
</code></pre>

<p>This will write to sqlite3 but fails with a memory error on testing (before it gets to the DB writing process that's not included in the sample code above).</p>

<p>The data I am trying to read can be freely downloaded from <a href="http://data.gov.au/dataset/abn-bulk-extract" rel="nofollow">http://data.gov.au/dataset/abn-bulk-extract</a></p>