Online Foundation Model Selection in Robotics
Po-han Li Oyku Selin Toprak Aditya Narayanan Ufuk Topcu Sandeep Chinchali
Foundation models have recently expanded into robotics after excelling in
computer vision and natural language processing. The models are accessible in
two ways: open-source or paid, closed-source options. Users with access to both
face a problem when deciding between effective yet costly closed-source models
and free but less powerful open-source alternatives. We call it the model
selection problem. Existing supervised-learning methods are impractical due to
the high cost of collecting extensive training data from closed-source models.
Hence, we focus on the online learning setting where algorithms learn while
collecting data, eliminating the need for large pre-collected datasets. We thus
formulate a user-centric online model selection problem and propose a novel
solution that combines an open-source encoder to output context and an online
learning algorithm that processes this context. The encoder distills vast data
distributions into low-dimensional features, i.e., the context, without
additional training. The online learning algorithm aims to maximize a composite
reward that includes model performance, execution time, and costs based on the
context extracted from the data. It results in an improved trade-off between
selecting open-source and closed-source models compared to non-contextual
methods, as validated by our theoretical analysis. Experiments across
language-based robotic tasks such as Waymo Open Dataset, ALFRED, and Open
X-Embodiment demonstrate real-world applications of the solution. The results
show that the solution significantly improves the task success rate by up to
14%.