How to load a matrix to change the attention layer in seqToseq demo? - Paddle
<p>While attempting to replicate the section 3.1 in <a href="https://arxiv.org/abs/1606.02006">Incorporating Discrete Translation Lexicons into Neural MT</a> in <code>paddle-paddle</code></p>

<p>I tried to have a static matrix that I'll need to load into the <code>seqToseq</code> training pipeline, e.g.:</p>

<pre><code>&gt;&gt;&gt; import numpy as np
&gt;&gt;&gt; x = np.random.rand(3,2)  
&gt;&gt;&gt; x
array([[ 0.64077103,  0.03278357],
       [ 0.47133411,  0.16309775],
       [ 0.63986919,  0.07130613]])
# where there is 3 target words and 2 source words, 
# and each cell in the matrix represents some co-occurrence probabilities.
</code></pre>

<p>With the <code>seqToseq_net</code> demo, this matrix would need to be multiplied to the attention layer output in the <a href="https://github.com/baidu/Paddle/blob/master/demo/seqToseq/seqToseq_net.py#L106"><code>gru_decoder_with_attention</code></a>. The original demo:</p>

<pre><code>def gru_decoder_with_attention(enc_vec, enc_proj, current_word):
    decoder_mem = memory(name='gru_decoder',
                         size=decoder_size,
                         boot_layer=decoder_boot)

    # This attention context layer would have been 
    # a vector of size |src_vocab| x 1
    context = simple_attention(encoded_sequence=enc_vec,
                               encoded_proj=enc_proj,
                               decoder_state=decoder_mem, )

    with mixed_layer(size=decoder_size * 3) as decoder_inputs:
        decoder_inputs += full_matrix_projection(input=context)
        decoder_inputs += full_matrix_projection(input=current_word)

    gru_step = gru_step_layer(name='gru_decoder',
                              input=decoder_inputs,
                              output_mem=decoder_mem,
                              size=decoder_size)

    with mixed_layer(size=target_dict_dim,
                     bias_attr=True,
                     act=SoftmaxActivation()) as out:
        out += full_matrix_projection(input=gru_step)
    return out
</code></pre>

<p>The goal is to affect the attention layer by multiplying it with the static matrix:</p>

<pre><code>def gru_decoder_with_attention(enc_vec, enc_proj, current_word):
    decoder_mem = memory(name='gru_decoder',
                         size=decoder_size,
                         boot_layer=decoder_boot)

    # This attention context layer would have been 
    # of size |src_vocab| x 1
    context = simple_attention(encoded_sequence=enc_vec,
                               encoded_proj=enc_proj,
                               decoder_state=decoder_mem, )

    # This static matrix layer, x, would have been
    # of size |trg_vocab| x |src_vocab|
    static_matrix = some_sort_of_layer(x)

    # This should yield a vector of size
    # |trg_vocab| x 1
    static_matrix_multiply_context = some_sort_of_operation_layer( static_matrix, context)

    with mixed_layer(size=decoder_size * 3) as decoder_inputs:
        # 
        decoder_inputs += full_matrix_projection(input= static_matrix_multiply_context)
        decoder_inputs += full_matrix_projection(input=current_word)
</code></pre>

<p>I've tried looking through the code in <a href="https://github.com/baidu/Paddle/tree/master/python/paddle/trainer_config_helpers"><code>Paddle/python/trainer_config_helps</code></a> and <a href="http://www.paddlepaddle.org/doc/demo/index.html">walked-through all the demo code</a> and I've also asked on <a href="https://gitter.im/PaddlePaddle/Deep_Learning">PaddlePaddle's gitter</a>. But I can't find how can I load a customized static matrix that doesn't need to be updated in the training process and interact with one of Paddle's layer.</p>

<p><strong>How to load a matrix to change the attention layer in seqToseq demo?</strong></p>

<p><strong>What should <code>some_sort_of_layer</code> and <code>some_sort_of_operation_layer</code> be in the above example?</strong></p>