Triple-Encoders: Representations That Fire Together, Wire Together
Justus-Jonas Erker Florian Mai Nils Reimers Gerasimos Spanakis Iryna Gurevych
Search-based dialog models typically re-encode the dialog history at every
turn, incurring high cost. Curved Contrastive Learning, a representation
learning method that encodes relative distances between utterances into the
embedding space via a bi-encoder, has recently shown promising results for
dialog modeling at far superior efficiency. While high efficiency is achieved
through independently encoding utterances, this ignores the importance of
contextualization. To overcome this issue, this study introduces
triple-encoders, which efficiently compute distributed utterance mixtures from
these independently encoded utterances through a novel hebbian inspired
co-occurrence learning objective without using any weights. Empirically, we
find that triple-encoders lead to a substantial improvement over bi-encoders,
and even to better zero-shot generalization than single-vector representation
models without requiring re-encoding. Our code/model is publicly available.