(Friedman, Baker, Mellers, Tetlock, & Zeckhauser, 2018)
and those distinctions help them more reliably distinguish
between 60/40 and 40/60 bets, even 55/45 and 45/55 bets.
Granularity in assessments of uncertainty pays off in accuracy.
Practitioners of cost–benefit analysis can readily compute
the net value to society of having productions systems—like
prediction polls and markets—that generate better probability estimates for policymakers (Sunstein, 2018). But setting
up production systems that hold up under real-world pressures will be a huge challenge that will require drawing on
many areas of behavioral and social science. It will not be
enough just to order analysts to use numbers. Analysts will
need to feel that it is psychologically safe to do what
“superforecasters” did—and focus, laser-like, on accuracy.
That means analysts will need to feel that they can make
mistakes of either under- or overestimation of threats and
opportunities and resist shading their estimates to please
powerful factions that would prefer one answer over the
other (Edmondson & Lei, 2014). None of that will be easy.
Organizations that speak truth to power are hard to sustain.
Abramowitz, A. (1988). An improved model for predicting presidential
election outcomes. Political Science & Politics, 21, 843– 847.ht tp://dx
Akerlof, G., & Schiller, R. (2010). Animal spirits: How human psychology
drives the economy and why it matters for global capitalism. Princeton,
NJ: Princeton University Press.
Allport, G. W. (1954). The nature of prejudice. Cambridge, MA: Perseus.
Armstrong, S. (2001). Principles of forecasting: A handbook for researchers and practitioners. New York, NY: Kluwer Academic Publishers.
Art, R., & Jervis, R. (Eds.). (2016). International politics: Enduring
concepts and contemporary issues (10th ed.). Boston, MA: Pearson.
Atanasov, P., Rescober, P., Stone, E., Swift, S., Servan-Schreiber, E.,
Tetlock, P., . . . Mellers, B. (2017). Distilling the wisdom of crowds:
Prediction markets versus prediction polls. Management Science, 63,
Baron, J. (2008). Thinking and deciding (4th ed.). New York, NY: Cambridge University Press.
Baron, J., Unger, L., Mellers, B., & Tetlock, P. (2014). Two reasons to
make aggregated probability forecasts more extreme. Decision Analysis,
11, 133–145. http://dx.doi.org/10.1287/deca.2014.0293
Bo, E. Y., Budescu, D. V., Lewis, C., Tetlock, P., & Mellers, B. (2017). An
IRT forecasting model: Linking proper scoring rules to item response
theory. Journal of Judgment and Decision Making, 12, 90–103.
Brier, G. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78, 1–3.
Budescu, D. V., & Chen, E. (2014). Identifying expertise to extract the
wisdom of crowds. Management Science, 61, 249– 286.
Cain, S. (2013). Quiet: The power of introverts in a world that can’t stop
talking. New York, NY: Penguin.
Camerer, C. (2003). Behavioral game theory: Experiments in strategic
interaction. Princeton, NJ: Princeton University Press.
Chang, W., Chen, E., Mellers, B., & Tetlock, P. (2016). Developing expert
political judgment: The impact of training and practice on judgmental
accuracy in geopolitical forecasting tournaments. Journal of Judgment
and Decision Making, 11, 509 –526.
Chen, E., Budescu, D., Lakshmikanth, S., Mellers, B., & Tetlock, P.
(2016). Validating the contribution-weighted model: Robustness and
cost-benefit analyses. Decision Analysis, 13, 1–25. http://dx.doi.org/10
Cowgill, B., & Zitzewitz, E. (2015). Corporate prediction markets: Evidence from Google, Ford, and Firm X. The Review of Economic Studies,
82, 1309 –1341. http://dx.doi.org/10.1093/restud/rdv014
Cross, D., Scott, D., Ramos, J., Mellers, B., & Tetlock, P. (2018). Robust
forecast aggregation: Fourier L2E regression. Journal of Forecasting,
37, 259–268. http://dx.doi.org/10.1002/for.2489
Dana, J., Atanasov, P., Tetlock, P., & Mellers, B. (in press). Are markets
more accurate than polls? The surprising informational value of “just
asking,”. Journal of Judgment and Decision Making.
DellaVigna, S. (2009). Psychology and economics: Evidence from the
field. Journal of Economic Literature, 47, 315–372. http://dx.doi.org/10
Edmondson, A., & Lei, Z. (2014). Psychological safety: The history,
renaissance, and future of an interpersonal conflict. Annual Review of
Organizational Psychology and Organizational Behavior, 1, 23–43.
Erev, I., & Roth, A. (1998). Predicting how people play games: Reinforcement learning in experimental games with unique, mixed strategy equilibria. The American Economic Review, 88, 848–881.
Fair, R. (1978). The effect of economic events on votes for the president.
The Review of Economics and Statistics, 60, 159 –173. http://dx.doi.org/
Fama, E. (1970). Efficient capital markets: A review of theory and empirical work. The Journal of Finance, 25, 383–417. http://dx.doi.org/10
Fehr, E., & Rangel, A. (2011). Neuroeconomic foundations of economic
choice—Recent advances. The Journal of Economic Perspectives, 25,
Friedman, J., Baker, J., Mellers, B., Tetlock, P., & Zeckhauser, R. (2018).
The value of precision in probability assessment: Evidence from a
large-scale geopolitical forecasting tournament. International Studies
Quarterly, 62, 410–422.
Gaertner, S., Dovidio, J., Banker, D., Houlette, M., Johnson, K., & McG-lynn, E. (2000). Reducing intergroup conflict: From superordinate goals
to decategorization, recategorization, and mutual differentiation. Group
Dynamics, 4, 98 –114. http://dx.doi.org/10.1037/1089-2622.214.171.124
Gneiting, T., & Raftery, A. (2007). Strictly proper scoring rules, prediction,
and estimation. Journal of the American Statistical Association, 102,
Goldstein, G., Hartman, R., Comstock, E., & Baumgarten, T. (2018).
Assessing the accuracy of geopolitical forecasts from the U.S. intelligence community’s prediction market. Manuscript under review.
Graefe, A., & Armstrong, S. (2011). Comparing face-to-face meetings,
nominal groups, Delphi and prediction markets on an estimation task.
International Journal of Forecasting, 27, 183–195. http://dx.doi.org/10
Gürkaynak, R. S., & Wolfers, J. (2006). Macroeconomic derivatives: An
initial analysis of market-based macro forecasts, uncertainty, and risk
[Working Paper 11929]. Cambridge, MA: National Bureau of Economic
Hogarth, R. (1981). Beyond discrete biases: Functional and dysfunctional
aspects of judgmental heuristics. Psychological Bulletin, 90, 197–217.
Kahneman, D. (2011). Thinking, fast and slow. New York, NY: Farrar,
Strauss, and Giroux.
Kahneman, D., & Tversky, A. (1979). Prospect theory: An analysis of
decision under risk. Econometrica, 47, 263– 291. http://dx.doi.org/10
Katz, N. (2001). Sports teams as a model for workplace teams: Lessons and
liabilities. The Academy of Management Executive, 15, 56 – 67.
299 BETTER PROBABILITY ESTIMATES FOR POLICY MAKERS