xVal: A Continuous Number Encoding for Large Language Models
Siavash Golkar Mariel Pettee Michael Eickenberg Alberto Bietti Miles Cranmer Geraud Krawezik Francois Lanusse Michael McCabe Ruben Ohana Liam Parker Bruno Régaldo-Saint Blancard Tiberiu Tesileanu Kyunghyun Cho Shirley Ho
Large Language Models have not yet been broadly adapted for the analysis of
scientific datasets due in part to the unique difficulties of tokenizing
numbers. We propose xVal, a numerical encoding scheme that represents any real
number using just a single token. xVal represents a given real number by
scaling a dedicated embedding vector by the number value. Combined with a
modified number-inference approach, this strategy renders the model end-to-end
continuous when considered as a map from the numbers of the input string to
those of the output string. This leads to an inductive bias that is generally
more suitable for applications in scientific domains. We empirically evaluate
our proposal on a number of synthetic and real-world datasets. Compared with
existing number encoding schemes, we find that xVal is more token-efficient and
demonstrates improved generalization.