Saving with h5py arrays of different sizes
<p>I am trying to store about 3000 numpy arrays using HDF5 data format. Arrays vary in length from 5306 to 121999 np.float64 </p>

<p>I am getting
<code>Object dtype dtype('O') has no native HDF5 equivalent</code>
error since due to the irregular nature of the data numpy uses the general object class.</p>

<p>My idea was to pad all the arrays to 121999 length and storing the sizes in another dataset.</p>

<p>However this seems quite inefficient in space, is there a better way? </p>

<p>EDIT: To clarify, I want to store 3126 arrays of <code>dtype = np.float64</code>. I have them stored in a <code>list</code>and when h5py does the routine it converts to an array of <code>dtype = object</code> because they are different lengths. To illustrate it:</p>

<pre><code>a = np.array([0.1,0.2,0.3],dtype=np.float64)
b = np.array([0.1,0.2,0.3,0.4,0.5],dtype=np.float64)
c = np.array([0.1,0.2],dtype=np.float64)

arrs = np.array([a,b,c]) # This is performed inside the h5py call
print(arrs.dtype)
&gt;&gt;&gt; object
print(arrs[0].dtype)
&gt;&gt;&gt; float64
</code></pre>