Contrastive Learning and Mixture of Experts Enables Precise Vector
  Embeddings
Rohan Kapur Logan Hallee Arjun Patel Bohdan Khomtchouk
The advancement of transformer neural networks has significantly elevated the
capabilities of sentence similarity models, particularly in creating effective
vector representations of natural language inputs. However, these models face
notable challenges in domain-specific contexts, especially in highly
specialized scientific sub-fields. Traditional methods often struggle in this
regime, either overgeneralizing similarities within a niche or being overly
sensitive to minor differences, resulting in inaccurate text classification and
subpar vector representation. In an era where retrieval augmentation and search
are increasingly crucial, precise and concise numerical representations are
essential. In this paper, we target this issue by assembling niche datasets
using co-citations as a similarity metric, focusing on biomedical domains. We
employ two key strategies for fine-tuning state-of-the-art models: 1.
Domain-specific Fine-Tuning, which tailors pretrained models to a single
domain, and 2. Universal Applicability with Mixture of Experts (MoE), adapting
pretrained models with enforced routing for multiple domains simultaneously.
Our training approach emphasizes the use of abstracts for faster training,
incorporating Multiple Negative Rankings loss for efficient contrastive
learning. Notably, our MoE variants, equipped with $N$ experts, achieve the
efficacy of $N$ individual models, heralding a new era of versatile,
One-Size-Fits-All transformer networks for various tasks. This methodology
marks significant advancements in scientific text classification metrics and
holds promise for enhancing vector database search and compilation.