Do performance standards reflect conceptions of competence? The relationship between implicit theories of competence and standard-setting judgments
Abstract
The role of implicit theories in standard-setting judgments was the focus of this study. In the first phase, behavioral descriptors of a minimally competent police recruit were gathered from 31 experts (experienced police) and 25 nonexperts (public sector managers) and used to construct an implicit theory ratings questionnaire. In the second phase, 196 experts and 195 nonexperts were assigned to experimental and control groups and completed the questionnaire in pretest and posttest conditions. Between pretest and posttest administrations, experimental groups also completed a reasoning ability test used in police recruitment and participated in an Angoff standard-setting exercise to determine a cutoff score for the test. Data from both phases revealed shared and unique conceptions of minimal competence held by experts and nonexperts. Exploratory factor analyses and reliability analyses of the implicit theory questionnaire on half-samples of the expert and nonexpert groups (calibration samples) suggested a four-factor structure of the implicit theories held by experts comprising (i) general policing skills; (ii) self-discipline; (iii) civic-mindedness; and (iv) policing values factors and a three-factor structure for nonexperts comprising (i) investigative skills; (ii) personal qualities; and (iii) job requirements. These hypothetical structures were then tested on the remaining half-samples (validation samples) using confirmatory factor analyses. however, the results were inconclusive. A comparison of the factor structures across expert and nonexpert groups using a LISREL hierarchical multi-sample approach revealed that the groups did not share the four-factor structure proposed for experts. Cutoff scores were also compared with experts setting higher standards on the reasoning ability test than nonexperts $(M\sb{\rm experts}=50.27$; $M\sb{\rm nonexp.}=45.53$; $t(189)=3.53,\ p<.01).$ Multiple and bivariate regressions of standards on factor scores revealed that, overall, the implicit theory factors accounted for a significant proportion of variance in the standards set $(R=.34,\ p<.05$ for experts; $R=.33,\ p<.05$ for nonexperts). Finally, a comparison of pretest and posttest factor structures showed that, contrary to expectations, the implicit theories of both groups were relatively unaffected by participation in the standard-setting exercise. Theoretical and statistical limitations of the study together with recommendations for future research were also presented.
Subject Area
Occupational psychology|Psychological tests
Recommended Citation
West, Julie Ann, "Do performance standards reflect conceptions of competence? The relationship between implicit theories of competence and standard-setting judgments" (1998). ETD Collection for Fordham University. AAI9825852.
https://research.library.fordham.edu/dissertations/AAI9825852
