2D array to represent a huge python dict, COOrdinate like solution to save memory
<p>I try to update a dict_with_tuples_key with the data from an array:</p>

<pre><code>myarray = np.array([[0, 0],  # 0, 1
                    [0, 1],
                    [1, 1],  # 1, 2
                    [1, 2],  # 1, 3
                    [2, 2],
                    [1, 3]]
) # a lot of this with shape~(10e6, 2)

dict_with_tuples_key = {(0, 1): 1,
                        (3, 7): 1} # ~10e6 keys 
</code></pre>

<p>Using an array to store the dict values, (thanks to @MSeifert) we get this:</p>

<pre><code>def convert_dict_to_darray(dict_with_tuples_key, myarray):
    idx_max_array = np.max(myarray, axis=0)
    idx_max_dict  = np.max(dict_with_tuples_key.keys(), axis=0)
    lens = np.max([list(idx_max_array), list(idx_max_dict)], axis=0)
    xlen, ylen = lens[0] + 1, lens[1] + 1
    darray = np.zeros((xlen, ylen)) # Empty array to hold all indexes in myarray
    for key, value in dict_with_tuples_key.items():
        darray[key] = value
    return darray

@njit
def update_darray(darray, myarray):
    elements = myarray.shape[0]
    for i in range(elements):
        darray[myarray[i][0]][myarray[i][1]] += 1
    return darray

def darray_to_dict(darray):
    updated_dict = {}
    keys = zip(*map(list, np.nonzero(darray)))
    for x, y in keys:
        updated_dict[(x, y)] = darray[x, y]
    return updated_dict

darray = convert_dict_to_darray(dict_with_tuples_key, myarray)
darray = update_darray(darray, myarray)
</code></pre>

<p>I get the exact result needed:</p>

<pre><code># print darray_to_dict(darray)
# {(0, 1): 2.0,
#  (0, 0): 1.0,
#  (1, 1): 1.0,
#  (2, 2): 1.0,
#  (1, 2): 1.0,
#  (1, 3): 1.0,
#  (3, 7): 1.0, }
</code></pre>

<p>For small matrix it work quit well, @njit work on it so it's very fast,
but... </p>

<p>the creation of the huge empty <code>darray = np.zeros((xlen, ylen))</code> <strong>does not fit on memory</strong>. How can we avoid to assign a very sparse array, and only store non null values like sparse matrix in COOrdinate format ?</p>