Using text embedding models and vector databases as text classifiers
  with the example of medical data
Rishabh Goel
The advent of Large Language Models (LLMs) is promising and has found
application in numerous fields, but as it often is with the medical field, the
bar is typically quite high [5]. In tandem with LLMs, vector embedding models
and vector databases provide a robust way of expressing numerous modes of data
that are easily digestible by typical machine learning models. Along with the
ease of adding information, knowledge, and data to these vector databases, they
provide a compelling reason to apply them in numerous fields where the task of
retrieving information is typically done by humans. Researchers at Google have
developed a clear alternative model, Med-PaLM [6] specifically designed to
match a clinician's level of accuracy when it comes to medical knowledge. When
training classifiers, and developing models, it is imperative to maintain
factuality and reduce bias [4]. Here, we explore the use of vector databases
and embedding models as a means of encoding, and classifying text with the
example and application in the field of medicine. We show the robustness of
these tools depends heavily on the sparsity of the data presented, and even
with low amounts of data in the vector database itself, the vector database
does a good job at classifying data [9]. Using various LLMs to generate the
medical data, we also understand the limitations of the medical knowledge of
these models and encourage further expert medical review of our testing data.
By using vector databases to classify a clinician's notes on a patient
presented with a certain ailment, we understand the limitations of such
methods, but also the promise of their prospective use and with continued
testing and experimentation, hope to explore a unique use case of vector
databases and embedding models.