How to fit two keras ImageDataGenerators for sets of images
<p>I am looking for a solution or an example for the following task:</p>

<p>I have sets of images of the <strong>same objects taken from different angles</strong>.
I would like to build a deep-CNN with keras, which can take <strong>sets</strong> of two images, perform data augmentation on each image separately, and feed them together into a connected model.</p>

<p><em>more detailed explanation:</em></p>

<p>The images are stored in a HDF5 file, with the following shapes:</p>

<pre><code>data['Xp'] # shape=(3000, 224, 224, 3) #RGB images
data['Xs'] # shape=(3000, 224, 224, 3) #RGB images
data['Y']  # shape=(3000, 9) #categorical data.
</code></pre>

<p>Now, I want a generator that can:</p>

<ol>
<li>scramble the indexes of the data sets.   </li>
<li>Then take images and</li>
<li>categories from  the data augment the images from <code>X1_train</code>, <code>X2_train</code></li>
<li>separately feed it into a netwrok with the following structure:</li>
</ol>

<p>The code...</p>

<pre><code>from keras.layers import Flatten, Dense, Input, Dropout, Convolution2D, MaxPooling2D, Merge


img_input = Input(shape=input_shape)
x = Convolution2D(64, 3, 3, activation='relu', border_mode='same', name='block1_conv1')(img_input)
x = Convolution2D(64, 3, 3, activation='relu', border_mode='same', name='block1_conv2')(x)
x = MaxPooling2D((2, 2), strides=(2, 2), name='block1_pool')(x)

# ... more network definition here ....

model1 = Model(img_input, x)
model2 = Model(img_input, x)

merged = Merge([model1, model2], mode='concat')

final_model = Sequential()
final_model.add(merged)
final_model.add(Dense(9, activation='softmax'))
</code></pre>

<p>I created the following generator that yields the expected data to feed the 
<code>fit_generator</code> of the model...</p>

<pre><code>def aug_train_iterator(Xp, Xs, Y, database_file=database_file, is_binary=True):
    from itertools import izip
    from keras.preprocessing.image import ImageDataGenerator

    seed = 7 #make sure that two iterators give same tomato each time...

    ig = ImageDataGenerator(dim_ordering='tf', rotation_range=90,
                                               width_shift_range=0.05,
                                               height_shift_range=0.05,
                                               zoom_range=0.05,
                                               fill_mode='constant',
                                               cval=0.0,
                                               horizontal_flip=True,
                                               rescale=1./255)


    for batch in izip(ig.flow(Xp,Y,  seed=seed), ig.flow(Xs, seed=seed)):
        for i in range(len(batch[0][0])):
            x1 = batch[0][0][i].reshape(1,224, 224, 3)
            x2 = batch[1][i].reshape(1, 224, 224, 3)
            y = batch[0][1][i].reshape(1,2)
            yield ([x1, x2], y)
</code></pre>

<p>Now, when I try to fit the model...</p>

<pre><code>gen = aug_train_iterator(Xp, Xs, Y)
final_model.fit_generator(gen, 1000, 20)
</code></pre>

<p>It actually running for a few images... and then raise error about ~ 15 images:</p>

<pre><code>Epoch 1/20
  15/1000 [..............................] - ETA: 606s - loss: 0.7001 - acc: 0.4000

Exception in thread Thread-44:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 404, in data_generator_task
    generator_output = next(generator)
  File "&lt;ipython-input-134-f128a127c7ce&gt;", line 35, in aug_train_iterator
    for batch in izip(ig.flow(Xp,Y,  seed=seed), ig.flow(Xs, seed=seed)):
  File "/usr/local/lib/python2.7/dist-packages/keras/preprocessing/image.py", line 495, in next
    x = self.X[j]
  File "/usr/lib/python2.7/dist-packages/h5py/_hl/dataset.py", line 367, in __getitem__
    if self._local.astype is not None:
AttributeError: 'thread._local' object has no attribute 'astype'
</code></pre>

<p>what is the problem?</p>