Project Seeks the Wisdom—and Foresight—of the Crowd

Imagine you’ve got a bunch of highly specific questions about the future bubbling around in your brain—not about jobs or relationships but more like: How much will a ton of aluminum cost on June 1? Will Republika Srpska secede from Bosnia-Herzegovina by next April? And which semiconductor company will rake in the most revenue this year?

Because you can’t rest until you know, do you seek out top experts on metal commodities, the former Yugoslavia and the microelectronics market? If that’s your strategy, brace for some suspect answers. It turns out that knowing everything about a topic and being able to predict what will happen next are two different things.

“If you want detailed technical explanations, it’s clear you need experts for that, but when you’re looking for forecasting ability, the evidence is pretty strong that expertise is not a good predictor of being right,” said Adam Russell, chief scientist at the University of Maryland’s Applied Research Laboratory for Intelligence and Security (ARLIS).

To help get around that limitation of expertise, Russell is leading the Integrated Forecasts and Estimates of Risks (INFER) project at ARLIS that started in January with twin goals: provide forecasts about the future of U.S. technology competitiveness—particularly in the increasingly crucial artificial intelligence area—to people and agencies in the federal government; and perhaps more profoundly, to explore how to derive such answers from the aggregated input of a large group of smart, engaged people who generally have little training in the topic at hand.

The platform for the crowdsourced forecasting experiment is ARLIS’ INFER website, where all the questions above were posted in recent months, and where anyone is welcome to sign up and begin offering predictions and insights. (As of Wednesday, just over half of 82 forecasts predicted aluminum prices would drop to somewhere between $3,000 and $3,500 a ton in June, while 94% of forecasts said no to a Serbian breakaway in Bosnia, and Samsung was far ahead of Intel in revenue leader predictions.)

It’s not about flinging wild guesses into an internet void, Russell said. Every question finally has a verifiable answer, although some have longer timeframes than others (and indeed, how far into the future forecasts are possible is also a question the INFER are researchers are asking).

“One of the fun things about forecasting is you find out whether you were right or wrong,” he said, unlike the real world of policy and statecraft where precise, actionable feedback is rare. The platform even features a leaderboard where top forecasters, all with anonymous usernames, vie for supremacy in accuracy rankings. INFER also offers paid “pro” status to talented forecasters, who can pull in a few hundred dollars a month.

It’s not about the money, however, said pro forecaster Nicole Catanzarite Ph.D. ’21, who graduated from the neuroscience and cognitive science program. She enjoys the mental challenge of applying her cognitive science expertise to a wide range of questions while developing a new cognitive skill of forecasting.

“Many people on the platform in its early stages had educational backgrounds in public policy, and I brought something a little different with insight from the cognitive side,” she said.

The work to research the questions thoroughly enough to provide useful answers, Catanzarite said, helped her stretch her graduate school experience beyond the intense “bench science” focus of her education, and complemented additional professional opportunities as she increasingly became fascinated by public policy. For example, she now serves in a leadership role with the Washington, D.C., chapter of the Society for Neuroscience.

Anyone with a bit of research skill—or the urge to dig deeper on subjects that interest them—can find their place on the INFER platform, said Vanessa Pineda, director of professional services at Cultivate Labs, a forecasting-oriented tech firm that’s collaborating with ARLIS on the project. The company, which built and runs the web platform for ARLIS, has also worked with major companies and governments around the world to implement crowd forecasting efforts.

Participants don’t have to answer every question—just ones that appeal to them—and there’s background material to help get started, she said.

“Forecasting is a different way of looking at things, and a skill you can develop,” she said. “I’ve done it myself, and been in the position of not being very good—but then you get better and surprise yourself, and you see yourself rising on the leaderboard.”

Although the actual government recipients of INFER forecasts can vary depending on their specific interests, the questions and answers are all high-priority for someone, somewhere, said Adam Siegel, co-founder and CEO of Cultivate Labs.

“If you think of the ways you can exercise your civic duty as a citizen of the United States—you can vote, you can comment on proposed rules, file amicus briefs in court cases,” Siegel said. “But how often do you get to help the government ‘skate to where the puck is going’ and see the future, and help base their policy on that developing picture?”