Developing models and applying Bayesian decision theory for standard setting in college placement
Abstract
The goal of this study was to develop four statistical models and to apply Bayesian decision theory for standard setting in college placement and classification. Model 1 is a true score model that assumes multivariate normality. Model 2 is an observed score model and applies fully conditional specification multiple imputation (FCS-MI) to correct for range restriction and missing data. Model 3 is a true score model that explicitly includes maturity/practice and learning/coaching factors. Gain scores are modeled based on previous research on repeat test taking, stochastic learning models, Rescorla-Wagner model, and hierarchical linear models for longitudinal data. Model 4 applies FCS-MI to correct for range restriction for repeated measures. It has been found from Models 1 and 2 that optimal cut scores are related to threshold parameter for passing criterion variable in each course and classification accuracy is related to criterion-related validity. In Model 2, an informative prior is needed. However, FCS-MI with an informative prior still under- corrects correlations. In both Models 1 and 2, the curvature of accuracy index/Bayes' risk function to be optimized plays an important role in locating cut scores. If the curves is flat then cut scores have larger sampling variability. Moreover, even small RMSEs from correcting correlations in Model 2 by FCS-MI combined with flat curvature result in large bias for cut scores. This study found that Model 3 follows expected patterns such as total maturity and practice is negatively related to initial ability, and students with lower initial ability tend to have higher gain scores than students with higher ability since there is more room left for them to improve. Classification accuracy for Model 3 is high, but there are no natural cut scores for the three-category classification. Higher speed of learning in Model 3 lowers the smaller cut scores for the three-category classification. Parameter recovery for Model 4 is very good. It is possible to apply Model 4 if raw data are available. These four models are neither competing or nested models, but they are developed for different placement settings.
Subject Area
Quantitative psychology
Recommended Citation
Sakworawich, Arnond, "Developing models and applying Bayesian decision theory for standard setting in college placement" (2013). ETD Collection for Fordham University. AAI3598859.
https://research.library.fordham.edu/dissertations/AAI3598859
