An Empirical Investigation into Benchmarking Model Multiplicity for
  Trustworthy Machine Learning: A Case Study on Image Classification
Prakhar Ganesh
Deep learning models have proven to be highly successful. Yet, their
over-parameterization gives rise to model multiplicity, a phenomenon in which
multiple models achieve similar performance but exhibit distinct underlying
behaviours. This multiplicity presents a significant challenge and necessitates
additional specifications in model selection to prevent unexpected failures
during deployment. While prior studies have examined these concerns, they focus
on individual metrics in isolation, making it difficult to obtain a
comprehensive view of multiplicity in trustworthy machine learning. Our work
stands out by offering a one-stop empirical benchmark of multiplicity across
various dimensions of model design and its impact on a diverse set of
trustworthy metrics. In this work, we establish a consistent language for
studying model multiplicity by translating several trustworthy metrics into
accuracy under appropriate interventions. We also develop a framework, which we
call multiplicity sheets, to benchmark multiplicity in various scenarios. We
demonstrate the advantages of our setup through a case study in image
classification and provide actionable insights into the impact and trends of
different hyperparameters on model multiplicity. Finally, we show that
multiplicity persists in deep learning models even after enforcing additional
specifications during model selection, highlighting the severity of
over-parameterization. The concerns of under-specification thus remain, and we
seek to promote a more comprehensive discussion of multiplicity in trustworthy
machine learning.