Estimation and psychometric analysis of component profile scores via multivariate generalizability theory
Abstract
Component Universe Score Profile analysis (CUSP) is introduced in this paper as a psychometric alternative to multivariate profile analysis. The theoretical foundations of CUSP analysis are reviewed, which include multivariate generalizability theory and constrained principal components analysis. Because CUSP is a combination of generalizability theory and principal components analysis, CUSP was analyzed for accuracy and precision for small sample sizes via a simulation analysis. The stability and accuracy of CUSP analysis was also assessed using a large sample of examinee data (n=5,000) from the College Board Advanced Placement (AP) Statistics subject test. The simulation showed that CUSP reliability estimates and coordinate estimates are generally unbiased when sample sizes were larger than n=100. Subtest covariance structure caused significant bias in reliability and coordinate estimates when the covariance matrix was compound symmetric. Coordinate standard error estimates were significantly biased when sample sizes were very small (n<100). The AP data analysis, like the simulation, showed that profile estimates were consistent and stable for larger samples, but profile scores were inconsistent for the small sample condition (n=50). CUSP external analysis illustrated the use of an external variable (gender) to define and predict AP Statistics profiles. The estimated external gender profile was not useful for classifying examinees by gender, achieving accuracy approximately equal to chance assignment.
Subject Area
Educational tests & measurements|Behavioral psychology|Quantitative psychology
Recommended Citation
Grochowalski, Joseph H, "Estimation and psychometric analysis of component profile scores via multivariate generalizability theory" (2015). ETD Collection for Fordham University. AAI10013383.
https://research.library.fordham.edu/dissertations/AAI10013383
