Altering different python objects in parallel processes, respectively
<p><strong>In a Nutshell</strong></p>

<p>I want to change complex python objects concurrently, whereby each object is processed by a single process only. How can I do this (most efficiently)? Would implementing some kind of pickling support help? Would that be efficient?</p>

<p><strong>Full Problem</strong></p>

<p>I have a python data structure <code>ArrayDict</code> that basically consists of a <code>numpy</code> array and a dictionary and maps arbitrary indices to rows in the array. In my case, all keys are integers.</p>

<pre><code>a = ArrayDict()

a[1234] = 12.5
a[10] = 3

print(a[1234])                               #12.5
print(a[10])                                 # 3.0

print(a[1234] == a.array[a.indexDict[1234]]) #true
</code></pre>

<p>Now I have multiple such <code>ArrayDict</code>s and want to fill them in <code>myMethod(arrayDict, params)</code>. Since <code>myMethod</code> is expensive, I want to run it in parallel. Note that <code>myMethod</code> may add many rows to <code>arrayDict</code>. Every process alters its own <code>ArrayDict</code>. I do not need concurrent access to the <code>ArrayDict</code>s.</p>

<p>In <code>myMethod</code>, I change entries in the <code>arrayDict</code> (that is, I change the internal <code>numpy</code> array), I add entries to the <code>arrayDict</code> (that is, I add another index to the dictionary and write a new value in the internal array). Eventually, I would like to be able to exchange <code>arrayDict</code>'s internal <code>numpy</code> array when it becomes too small. This does not happen often and I could perform this action in the non-parallel part of my program, if no better solution exists. My own attempts were not successful even without the array exchange.</p>

<p>I have spent days researching on shared memory and python's <a href="https://docs.python.org/3.6/library/multiprocessing.html" rel="nofollow noreferrer">multiprocessing</a> module. Since I will finally be working on linux, the task seemed to be rather simple: the system call <code>fork()</code> allows to work with copies of the arguments efficiently. My thought was then to change each <code>ArrayDict</code> in its own process, return the changed version of the object, and overwrite the original object. To save memory and save the work for copying, I used in addition <a href="https://pypi.python.org/pypi/sharedmem" rel="nofollow noreferrer">sharedmem</a> arrays to store the data in <code>ArrayDict</code>. I am aware that the dictionary must still be copied.</p>

<pre><code>from sharedmem import sharedmem
import numpy as np

n = ...                   # length of the data array
myData = np.empty(n, dtype=object)
myData[:] = [ArrayDict() for _ in range(n)]
done = False

while not done:
    consideredData = ...  # numpy boolean array of length
                          # n with True at the index of
                          # considered data
    args = ...            # numpy array containing arguments
                          # for myMethod

    with sharedmem.MapReduce() as pool:
        results = pool.map(myMethod, 
                           list(zip(myData[considered], 
                                    args[considered])),
                           star=True)
        myData[considered] = results

    done = ...             # depends on what happens in
                           # myMethod
</code></pre>

<p>What I get is a segmentation fault error. I was able to circumvent this error by creating deepcopies of the <code>ArrayDict</code>s to <code>myMethod</code> and saving them into <code>myData</code>. I do not really understand why this is necessary, and copying my (potentially very large) arrays frequently (the while loop takes long) is not what seems to be efficient to me. However, at least it worked to a certain extent. Nevertheless, my program has some buggy behaviour at the 3rd iteration due to the shared memory. Therefore, I think that my way is not optimal.</p>

<p>I read <a href="https://stackoverflow.com/questions/5549190/is-shared-readonly-data-copied-to-different-processes-for-python-multiprocessing/5550156#5550156">here</a> and <a href="https://stackoverflow.com/questions/15976937/making-my-numpy-array-shared-across-processes">here</a> that it is possible to save aribtrary numpy arrays on the shared memory using <code>multiprocessing.Array</code>. However, I would still need to share the whole <code>ArrayDict</code>, which includes in particular a dictionary, which in turn is not pickable. </p>

<p>How could I achieve my goals in an efficient way? Would it be possible (and efficient) to make my object pickable somehow? </p>

<p>All solutions must run with python 3 and full numpy/scipy support on 64bit Linux.</p>

<p><strong>Edit</strong></p>

<p>I found <a href="https://stackoverflow.com/questions/3671666/sharing-a-complex-object-between-python-processes">here</a> that it is somehow possible to share arbitrary objects using Multiprocessing "Manager" classes and user-defined proxy classes. Will this be efficient? I would like to exploit that I do not need concurrent access to the objects, even though they are not handled in the main process. Would it be an option to create a manager for each object that I want to process? (I might still have some misconceptions about how mangers work.)</p>