Python3: Program occupies much more memory than expected
<h2>Short</h2>

<p>My python program occupies much more memory than expected or returned by memory profiling tools. I need a strategy to find the memory leak and fix it.</p>

<h2>Detailed</h2>

<p>I am running a python3 script on a 64bit Linux machine. Almost all code is bundled within one object:</p>

<pre><code>obj = MyObject(*myArguments)
result = obj.doSomething()
print(result)
</code></pre>

<p>During the creation of <code>obj</code>, the program reads a text file with a size of ca. 100MB. Since I save the information in multiple ways, I expect the whole object to occupy a couple of hundret MB memory. </p>

<p>Indeed, measuring its size with <code>asizeof.asized(obj)</code> from the package <a href="https://pythonhosted.org/Pympler/" rel="nofollow noreferrer">pympler</a> returns around 123MB. However, <code>top</code> tells me that my program occupies around <strong>1GB</strong> memory. </p>

<p>I understand that local variables in methods will occupy further RAM. However, looking into my code I see that none of these local variables can be that big. I double checked this using <code>asizeof.asized</code> again.</p>

<p>It is not a major concern to me that the script needs 1GB of memory. However, I execute some methods in parallel (in 12 pocesses): </p>

<pre><code>class MyObject()

    def doSomething(arg):
        # do something

    def myParallelMethod(args)
        with sharedmem.MapReduce() as pool:
            result = pool.map(self.doSomething, args)
        return result
</code></pre>

<p>This makes the the total memory usage become <strong>8GB</strong>, even though I put all large objects in shared memory:</p>

<pre><code>self.myLargeNumPyArray = sharedmem.copy(self.myLargeNumPyArray)
</code></pre>

<p>I assured with test programs that the memory is really shared. </p>

<p>Checking with <code>asizeof</code>, I obtained in each subprocess that</p>

<ul>
<li><code>asizeof.asized(self)</code> is 1MB (i.e. much smaller than the "original" object - maybe due to the shared memory, which is not counted double)</li>
<li><code>asizeof.asized(myOneAndOnlyBigLocalVariable)</code> is 230MB.</li>
</ul>

<p>All in all my program should occupy not much more than 123MB + 12*230MB = 2.8GB &lt;&lt; 8GB. So, why does the program require so much memory?</p>

<p>One explanation could be that there are some hidden parts (garbage?) in my object that is being copied, when the program is run in parallel.</p>

<p><strong>Does anyone know a strategy to find out, where the memory leak is? How could I fix it?</strong> </p>

<p>I have read many threads regarding memory profiling, e.g. <a href="https://stackoverflow.com/questions/8656124/profiling-memory-in-python-3">Profiling memory in python 3</a>, <a href="https://stackoverflow.com/questions/7758699/is-there-any-working-memory-profiler-for-python3">Is there any working memory profiler for Python3</a>, <a href="https://stackoverflow.com/questions/110259/which-python-memory-profiler-is-recommended">Which Python memory profiler is recommended?</a>, or <a href="https://stackoverflow.com/questions/552744/how-do-i-profile-memory-usage-in-python">How do I profile memory usage in Python?</a>, but all the recomended tools do not explain the memory usage.</p>

<hr>

<h2>Update</h2>

<p>I have been asked to provide a minimal example of the code. The code below shows the same problems with memory consumption in the parallel part as my original one. I have already figured out the issue with the non-parallel part of my code, which was that I had a large numpy array with data type <code>object</code> as object variable. Due to this data type, the array cannot be put into shared memory and <code>asized</code> only returns the shallow size. Thanks to @user2357112 for helping me to figure this out!</p>

<p>I would therefore like to concentrate on the issue in the parallel part: Inserting values into <code>queue</code> in the method <code>singleSourceShortestPaths</code> (below marked with a comment) changes the memory consumption from around 1.5GB to <strong>10GB</strong>. Are there any ideas for how to explain this behaviour?</p>

<pre><code>import numpy as np
from heapdict import heapdict
from pympler import asizeof
import sharedmem

class RoadNetwork():

    strType = "|S10"

    def __init__(self):

        vertexNo = 1000000
        self.edges = np.zeros(1500000, dtype = {"names":["ID", "from_to", "from_to_original", "cost", "inspection", "spot"], 
                                                   'formats':[self.strType, '2int', '2'+self.strType, "double", "3bool", "2int", "2int"]})
        self.edges["ID"] = np.arange(self.edges.size)
        self.edges["from_to_original"][:vertexNo, 0] = np.arange(vertexNo)
        self.edges["from_to_original"][vertexNo:, 0] = np.random.randint(0, vertexNo, self.edges.size-vertexNo)
        self.edges["from_to_original"][:,1] = np.random.randint(0, vertexNo, self.edges.size)

        vertexIDs = np.unique(self.edges["from_to_original"])
        self.vertices = np.zeros(vertexIDs.size, {"names":["ID", "type", "lakeID"], 
                                                  'formats':[self.strType, 'int', self.strType]})

    def singleSourceShortestPaths(self, sourceIndex):

        vertexData = np.zeros(self.vertices.size, dtype={"names":["predecessor", "edge", "cost"], 
                                         'formats':['int', "2int", "double"]})

        queue = np.zeros((self.vertices.size, 2), dtype=np.double)

        #Crucual line!! Commetning this decreases memory usage by 7GB in the parallel part
        queue[:,0] = np.arange(self.vertices.size)

        queue = heapdict(queue)

        print("self in singleSourceShortestPaths", asizeof.asized(self))
        print("queue in singleSourceShortestPaths", asizeof.asized(queue))
        print("vertexData in singleSourceShortestPaths", asizeof.asized(vertexData))

        # do stuff (in my real program Dijkstra's algorithm would follow)
        # I inserted this lines as an ugly version for 'wait()' to
        # give me enough time to measure the memory consumption in 'top'
        for i in range(10000000000):
            pass

        return vertexData

    def determineFlowInformation(self):
        print("self in determineFlowInformation", asizeof.asized(self))
        f = lambda i: self.singleSourceShortestPaths(i)
        self.parmap(f, range(30))

    def parmap(self, f, argList):
        """
        Executes f(arg) for arg in argList in parallel
        returns a list of the results in the same order as the 
        arguments, invalid results (None) are ignored
        """
        self.__make_np_arrays_sharable()
        with sharedmem.MapReduce() as pool:
            results, to_do_list = zip(*pool.map(f, argList))
        return results

    def __make_np_arrays_sharable(self):
        """
        Replaces all numpy array object variables, 
        which should have the same  
        behaviour / properties as the numpy array
        """
        varDict = self.__dict__
        for key, var in varDict.items():
            if type(var) is np.ndarray:
                varDict[key] = sharedmem.copy(var)

if __name__ == '__main__':
    network = RoadNetwork()

    print(asizeof.asized(network, detail=1))
    for key, var in network.__dict__.items():
        print(key, asizeof.asized(var))

    network.determineFlowInformation()
</code></pre>