Augmenting Holistic Review in University Admission using Natural
  Language Processing for Essays and Recommendation Letters
Jinsook Lee Bradon Thymes Joyce Zhou Thorsten Joachims Rene F. Kizilcec
University admission at many highly selective institutions uses a holistic
review process, where all aspects of the application, including protected
attributes (e.g., race, gender), grades, essays, and recommendation letters are
considered, to compose an excellent and diverse class. In this study, we
empirically evaluate how influential protected attributes are for predicting
admission decisions using a machine learning (ML) model, and in how far textual
information (e.g., personal essay, teacher recommendation) may substitute for
the loss of protected attributes in the model. Using data from 14,915
applicants to an undergraduate admission office at a selective U.S. institution
in the 2022-2023 cycle, we find that the exclusion of protected attributes from
the ML model leads to substantially reduced admission-prediction performance.
The inclusion of textual information via both a TF-IDF representation and a
Latent Dirichlet allocation (LDA) model partially restores model performance,
but does not appear to provide a full substitute for admitting a similarly
diverse class. In particular, while the text helps with gender diversity, the
proportion of URM applicants is severely impacted by the exclusion of protected
attributes, and the inclusion of new attributes generated from the textual
information does not recover this performance loss.