Dimensionality Reduction in Sentence Transformer Vector Databases with
  Fast Fourier Transform
Vitaly Bulgakov Alec Segal
Dimensionality reduction in vector databases is pivotal for streamlining AI
data management, enabling efficient storage, faster computation, and improved
model performance. This paper explores the benefits of reducing vector database
dimensions, with a focus on computational efficiency and overcoming the curse
of dimensionality. We introduce a novel application of Fast Fourier Transform
(FFT) to dimensionality reduction, a method previously underexploited in this
context. By demonstrating its utility across various AI domains, including
Retrieval-Augmented Generation (RAG) models and image processing, this
FFT-based approach promises to improve data retrieval processes and enhance the
efficiency and scalability of AI solutions. The incorporation of FFT may not
only optimize operations in real-time processing and recommendation systems but
also extend to advanced image processing techniques, where dimensionality
reduction can significantly improve performance and analysis efficiency. This
paper advocates for the broader adoption of FFT in vector database management,
marking a significant stride towards addressing the challenges of data volume
and complexity in AI research and applications. Unlike many existing
approaches, we directly handle the embedding vectors produced by the model
after processing a test input.