The Costly Dilemma: Generalization, Evaluation and Cost-Optimal
  Deployment of Large Language Models
Abi Aryan Aakash Kumar Nain Andrew McMahon Lucas Augusto Meyer Harpreet Singh Sahota
When deploying machine learning models in production for any
product/application, there are three properties that are commonly desired.
First, the models should be generalizable, in that we can extend it to further
use cases as our knowledge of the domain area develops. Second they should be
evaluable, so that there are clear metrics for performance and the calculation
of those metrics in production settings are feasible. Finally, the deployment
should be cost-optimal as far as possible. In this paper we propose that these
three objectives (i.e. generalization, evaluation and cost-optimality) can
often be relatively orthogonal and that for large language models, despite
their performance over conventional NLP models, enterprises need to carefully
assess all the three factors before making substantial investments in this
technology. We propose a framework for generalization, evaluation and
cost-modeling specifically tailored to large language models, offering insights
into the intricacies of development, deployment and management for these large
language models.