Yes, I heard from one first-rate forecaster that he thinks AI forecasters are especially weak in predicting big disruptive changes to the world.

Hard to study this, obviously!

Right. What's really surprising is how much better the best are. Human superforecasters, and prediction markets are surprisingly accurate too.

We could live in a world where things are much more chaotic, and the best humans (or AIs) would only be slightly better than chance. Evidently the world we live in is pretty darn predictable.

As someone who started working on AI forecasting 3 years ago, I can confidently say that most people did not expect AI to beat Tetlock's superforecasters, Metaculus pros, or prediction markets as quickly as it did.

The Economist has actually published other human forecasts many times, e.g. Metaculus or Good Judgment forecasts. They do year-end forecasts too.

Whether they draw on AI or other humans seems immaterial to the quality of their reporting.

I know this is tongue-in-cheek, but I think your idea could actually work, but not in financial markets. (The "keynesian beauty contest" of trying to predict what others think been played out to death there.)

You could train a model to anticipating scientific trends. Or policy trends. Others will definitely use mainline LLMs to make decisions there, so they may be more predictable now!

I spent a summer in PNG in 2010, on the island of Karkar. It was wild.

One of my most formative memories was finding out that few of the people who live there, even those who are literate, had books and some DVDs, were high school students, even had occasional access to computers with internet, knew that humanity had landed on the moon!

There must be a deeper read on why Google can rapidly ship better small models while being delayed months on the bigger model.

What's the simplest explanation?

Even though these predictions turned out mostly wrong, we should not castigate people for publicly forecasting! That is virtuous, and more people should do it.

Thank you Ed!

What are we to infer from no release of gemini-3.5-pro, but frequent releases of smaller flash models (presumably from the same large pre-training run?)

Google is in direct meetings with Scott Bessent and Howard Lutnick and is prudently keeping dangerous frontier models out of the hands of customers until reasonable precautions can be taken.

Not forecasting though. You can't goodhart predicting real-world events