Augmenting Holistic Review in University Admission using Natural Language Processing for Essays and Recommendation Letters
This research addresses the challenge of maintaining diversity in university admissions when using automated systems, but it is incremental as it builds on existing methods without introducing new paradigms.
The study tackled the problem of how protected attributes influence university admission decisions by evaluating a machine learning model's performance with and without these attributes, finding that excluding them reduced prediction accuracy and that textual information from essays and recommendation letters only partially restored performance, particularly failing to maintain diversity among underrepresented minority applicants.
University admission at many highly selective institutions uses a holistic review process, where all aspects of the application, including protected attributes (e.g., race, gender), grades, essays, and recommendation letters are considered, to compose an excellent and diverse class. In this study, we empirically evaluate how influential protected attributes are for predicting admission decisions using a machine learning (ML) model, and in how far textual information (e.g., personal essay, teacher recommendation) may substitute for the loss of protected attributes in the model. Using data from 14,915 applicants to an undergraduate admission office at a selective U.S. institution in the 2022-2023 cycle, we find that the exclusion of protected attributes from the ML model leads to substantially reduced admission-prediction performance. The inclusion of textual information via both a TF-IDF representation and a Latent Dirichlet allocation (LDA) model partially restores model performance, but does not appear to provide a full substitute for admitting a similarly diverse class. In particular, while the text helps with gender diversity, the proportion of URM applicants is severely impacted by the exclusion of protected attributes, and the inclusion of new attributes generated from the textual information does not recover this performance loss.