Why does my script returns Memory Error?
<p>I'm trying to do a statistics about one web page. This page has categories and products in there. I don't download informations about this products, I'm only counting them. </p>

<p>The point is that I'm getting either <code>Memory Error</code> error or just <code>Some text like Script ends with code -1073741819</code> (the number is exact). </p>

<p>I've tried to print <code>size</code> of variable <code>category_urls</code> after each loop and it does not increases.</p>

<p><strong>EDIT:</strong>
The memory error raises when the category which is being counted is too big (about 60 000 urls).</p>

<p>The main loop is simple:</p>

<pre><code>for category in categories:
    count_category(category)
</code></pre>

<p>I suppose that after each iteration, the memory should be released but I can't see any release when I look at <code>Task Manager</code> -> <code>Memory</code> tab (Python.exe). I see that the memory consumption is higher and higher.</p>

<p>In case it helps to solve the problem:</p>

<pre><code>def count_category(url):
    category_urls = list(get_category_urls(url))
    mLib.printToFile('database/count.txt',str(len(category_urls)))
    set_spracovanie_kategorie(url) # This fnc just writes category url into text file

def get_category_urls(url):
    log('Getting category urls: {}'.format(url))
    urls = []
    next_url = url

    i=1
    while next_url:
        root = load_root(next_url)
        urls.extend(get_products_on_page(root))
        for x in urls:
            if 'weballow' in x:
                yield x
        next_url = next_page(root, url) (next page is defined below)
        # if next_url == False:
        #     return urls
        i+=1

def get_products_on_page(root):
    hrefs = root.xpath('//div[@id="product-contain"]//h2/a/@href')
    return hrefs
</code></pre>

<p><strong>AND LXML LOADING FUNCTIONS:</strong></p>

<pre><code>class RedirectException(Exception):
    pass

def load_url(url):
    r = requests.get(url,allow_redirects=False)
    if r.status_code == 301:
        raise RedirectException
    html = r.text
    return html

def load_root(url):
    html = load_url(url)
    return etree.fromstring(html, etree.HTMLParser())
</code></pre>

<p><strong>NEXT PAGE:</strong></p>

<pre><code>def next_page(root, url):
    next = root.xpath('//a[@class="next"]/@href')
    if len(next) &gt; 0:
        return urljoin(url, next[0])
    return False
</code></pre>

<p>Could you give me and advice what to do? </p>