PAPERCLIP: Associating Astronomical Observations and Natural Language
  with Multi-Modal Models
Siddharth Mishra-Sharma Yiding Song Jesse Thaler
We present PAPERCLIP (Proposal Abstracts Provide an Effective Representation
for Contrastive Language-Image Pre-training), a method which associates
astronomical observations imaged by telescopes with natural language using a
neural network model. The model is fine-tuned from a pre-trained Contrastive
Language-Image Pre-training (CLIP) model using successful observing proposal
abstracts and corresponding downstream observations, with the abstracts
optionally summarized via guided generation using large language models (LLMs).
Using observations from the Hubble Space Telescope (HST) as an example, we show
that the fine-tuned model embodies a meaningful joint representation between
observations and natural language through tests targeting image retrieval
(i.e., finding the most relevant observations using natural language queries)
and description retrieval (i.e., querying for astrophysical object classes and
use cases most relevant to a given observation). Our study demonstrates the
potential for using generalist foundation models rather than task-specific
models for interacting with astronomical data by leveraging text as an
interface.