Images in Discrete Choice Modeling: Addressing Data Isomorphism in
  Multi-Modality Inputs
Brian Sifringer Alexandre Alahi
This paper explores the intersection of Discrete Choice Modeling (DCM) and
machine learning, focusing on the integration of image data into DCM's utility
functions and its impact on model interpretability. We investigate the
consequences of embedding high-dimensional image data that shares isomorphic
information with traditional tabular inputs within a DCM framework. Our study
reveals that neural network (NN) components learn and replicate tabular
variable representations from images when co-occurrences exist, thereby
compromising the interpretability of DCM parameters. We propose and benchmark
two methodologies to address this challenge: architectural design adjustments
to segregate redundant information, and isomorphic information mitigation
through source information masking and inpainting. Our experiments, conducted
on a semi-synthetic dataset, demonstrate that while architectural modifications
prove inconclusive, direct mitigation at the data source shows to be a more
effective strategy in maintaining the integrity of DCM's interpretable
parameters. The paper concludes with insights into the applicability of our
findings in real-world settings and discusses the implications for future
research in hybrid modeling that combines complex data modalities. Full control
of tabular and image data congruence is attained by using the MIT moral machine
dataset, and both inputs are merged into a choice model by deploying the
Learning Multinomial Logit (L-MNL) framework.