Developing and Validating a Method of Coherence-Based Judgment Aggregation
Abstract
Subjective probabilities are individual judgments made to estimate the probability of an uncertain event, based on the available information at the time of making the judgment (e.g., “What are the chances the average sea level rise will exceed 5 meters in 2100?”). As subjective probabilities are often used to make judgments under uncertainty and can inform decision-making, they are essential for forecasting, or the prediction of future events. Forecasting is often evaluated by two dualistic qualities, identified by Hammond (1996): correspondence and coherence. Correspondence is the extent to which judgments conform to reality and are accurate; coherence is the extent to which the judgments follow logical and probabilistic axioms. Incoherent judgments are those that violate logical and probabilistic axioms in systematic ways, which may result in biases such as the conjunction fallacy and failure to update probabilities by Bayes’ rule. Some researchers have proffered that these incoherent judgments can be partially explained as a statistical artifact. Coherentizing judgments using statistics increases accuracy, suggesting coherence as a key to improving judgments. In this line of thinking, other researchers have attributed biases to random errors in recall, suggesting that the ability to make coherent judgments may be subject to inter-individual variation. Recent approaches to combining many judgments suggest that optimal performance might be achieved by a careful reliance of the “select crowd”, or judges who are far more accurate than the average (Mannes, Soll, & Larrick, 2014). A global forecasting tournament (e.g., Mellers et al., 2014) finds that those who are highly accurate also tend to score higher on coherence measures, furthering the argument for coherence as an individual difference. Despite coherence being a pivotal component of forecasting accuracy, there exists no unified measure of the construct. In this dissertation, I devised and psychometrically validated a measure for coherence (CFS; Coherence Forecasting Scale). I collected, synthesized the operationalizations of coherence in the literature, and wrote items designed to capture the construct. I applied the new method of Automatic Item Generation (AIG) in designing this coherence measure, which designs multiple forms measuring the same construct with predictable psychometric properties to minimize item exposure. To examine whether the forms could be interchangeable, I applied a novel technique that shuffles items from the two forms, producing a set of 2p forms and its associated scores, where p is the number of items in each form. Summary statistics showed low variance in scores, suggesting the items from the two forms could be substituted with each other with minimal differences in scores. After developing and psychometrically validating the CFS, I used the CFS scores as a new weighting scheme for judgment aggregation in two samples. In the first sample, participants from Amazon Mechanical Turk who had previously completed a series of events were invited to complete the Coherence Forecasting Scale along with other establishedpsychological scales. In the second sample, participants from an online forecasting platform, Good Judgment Open, were invited to complete a similar survey. I compared accuracy of coherence-weighted judgments with other various methods extant in the literature, which fall broadly into two categories: statistical methods, which do not rely on individual differences, and behavioral methods, which use performance from the coherence and numeracy measuresto determine weights. In both studies, across all behavioral methods, a form of weighting that selected only a subset of highly coherent forecasters consistently yielded the most accurate aggregated forecasts. However, across all methods, a statistical method yielded best performance, though these methods generally exhibited higher variation than thecoherence-based methods, which exhibited relatively stable performance even when varying the size of the crowd.
Subject Area
Psychology|Applied Mathematics|Quantitative psychology
Recommended Citation
Ho, Emily H, "Developing and Validating a Method of Coherence-Based Judgment Aggregation" (2021). ETD Collection for Fordham University. AAI28085857.
https://research.library.fordham.edu/dissertations/AAI28085857
