Weighted probability models for high-stakes admission decisions
Abstract
Selection models used in high-stakes admissions decisions are imperfect with regard to fairness and psychometric reliability, yet these models are often used in high-stakes decisions such as admissions to universities, special programs and funding opportunities. Data were simulated for a variety of cases differing in the reliability of the predictor, the reliability of the criterion and the correlation between the two. In each case I compared four different selection models: a deterministic model, a weighted probability model, a weighted probability model with a minimum threshold, and a random selection model. The receiver operating characteristic curves (ROC), area under the curve (AUC), Cohen’s d and mean difference curves for each selection rule, as well as relative improvement calculations, were compared to evaluate the models. The ROC curves show the tradeoff between the sensitivity and specificity from each of the four rules that were tested (i.e. deterministic, probabilistic rules with unequal probabilities and with, or without, a minimal threshold, and random rule) at five different levels of success. In all conditions, the deterministic rule performed the best in all success ratios, followed by the probabilistic with the minimal threshold, followed by the probabilistic without the minimal threshold rule, and lastly followed by the random selection rule. The results indicated that the rules perform better for low success ratios than higher success ratios as evidenced by the larger AUC for lower success ratios. Additionally, as the selection ratio increases, the probabilistic rule with unequal probabilities and a minimal threshold approaches the performance of the deterministic rule. When the success ratio is low (10% to 30%), the rules are more accurate than when the success ratio is high (70% to90%). As there are more qualified candidates, the ROC curves approach the chance line which indicates that the four rules lead to the same decisions as a random selection, because most candidates are qualified for selection. The standardized difference between accepted and rejected (Cohen’s d) of the deterministic and probabilistic rule with a minimal threshold increase as more candidates are qualified, but the performance of the probabilistic rule with unequal probabilities and the random rule remains relatively stable. In this thesis, we present a simulation study of four different selection models: One is a deterministic model, one is a random selection model and two are probability models, one of which has a minimal admission threshold. We examined the quality of these four rules by comparing each of their receiver operating characteristic curves (ROC), area under the curve (AUC), Cohen’s d, mean difference curves, as well as relative improvement calculations. The models were compared under a variety of situations with different correlations between the predictor and the criterion, and the reliabilities.
Subject Area
Statistics|Quantitative psychology
Recommended Citation
Song, Victoria, "Weighted probability models for high-stakes admission decisions" (2016). ETD Collection for Fordham University. AAI10016486.
https://research.library.fordham.edu/dissertations/AAI10016486
