Can LLMs Predict the Future?
Over the last several months I have been experimenting with LLMs to measure whether they can forecast real-world events.
This came from my interest in prediction markets, but the question matters outside of markets too. Forecasting is one of the most important skills for making decisions under uncertainty. If a model can produce calibrated probabilities from incomplete evidence, it becomes useful anywhere decisions depend on the future.
Funny enough, this capability is also part of the AI 2027 essay. One of its projections is that advanced AI agents become useful by advising on short-term strategic decisions and building a track record. The rate at which models are getting better at this is directly aligned with what I found.
First, let's start off breaking down the question I am actually asking here. LLMs are Large Language Models, a type of artificial intelligence architecture built on-top of neural networks designed to understand, process and generate human language through next token prediction, similar to how the autocorrection tool on your phone works. However, these models are trained on vast amounts of human data that may come from the internet, books, code, etc that allows them to understand the causal mechanisms of how our world works.
Now, what is forecasting? Forecasting is the practice of predicting future trends or events by analyzing historical and present data. But that definition feels almost too clean, like forecasting is just a spreadsheet looking at itself in the mirror. In reality, forecasting means compressing what you know into a probability while admitting the world does not care about your confidence interval.
So this technical post is my attempt at figuring out how LLMs perform in this practice. Do they just parrot consensus? Can they produce alpha? Can they use probabilities in a way that is calibrated? And most importantly, can they become useful tools for making decisions about the future.
Architecture
The first version was barely a system: a web-search model checked recent news for a binary forecasting question and returned one scalar from 0 to 1 with a confidence interval.
question_id: opec-june-2026 probability: 0.74 confidence_interval: [0.65, 0.90] rationale: OPEC+ signaled continued supply increases, but timing remained uncertain.
Binary meaning the resolution is either 1 or 0. The event happens or it does not. Will this bill pass by the deadline? Will this company raise at a certain valuation? Will this candidate win? The model output is p, its estimated probability that the event resolves true.
For example, one of the first questions I gave the system was: Will OPEC+ announce another oil-output increase at its June 7, 2026 meeting?
p = 0.74. Confidence interval: 0.65 to 0.90.This is the first useful artifact the system produced. The model committed to 0.74, with most of its uncertainty between 0.65 and 0.90. Once the event resolves, that probability becomes something you can score.
Scoring
The score I started with was the Brier score. The model gives a probability p, the event resolves to either 1 or 0, and the score is the squared distance between those two values.
The OPEC+ question resolved true, so the outcome was 1. The score for this forecast was (0.74 - 1)^2 = 0.0676. If the event were to have resolved to false, then that same forecast would have scored (0.74 - 0)^2 = 0.5476.
A perfect forecast has a Brier score of 0. A terrible confident miss gets close to 1. Predicting 0.5 on everything gives 0.25, which is a useful baseline to show how a model compares to a prediction that refuses to take a side.
Quick note, and something worth addressing: Brier scores only mean something when the question sets are comparable. Some questions have much less uncertainty than others.
Building out our agent
Self-evidently this first version was far too crude. It could search the web, but it did not really know how to research or form its beliefs. I needed to build an environment around the model so the agent could do the boring work before giving its final probability.
That sent me to Philip Tetlock's Good Judgment Project, which studied people who consistently performed better than the aggregate of average humans at assigning probabilities to future events. I looked more into them and how they approached questions. Their process was practical: start from similar past cases, then revise the probability as current evidence changes.
So I started building a tool suite around the model, similar to what a superforecaster at the Good Judgment Project would want on their desk. The goal was to give the agent repeatable research tools before it made a forecast.
| Tool | What it does | Example |
|---|---|---|
| Web Search | Retrieves current, real-world information from the web through agent-driven queries. | Find recent reporting on the event and its likely resolution criteria. |
| Populate Reference Class (PRC) | Compiles a list of instances that fit a description, which can then be used as priors. | List functioning satellites impacted by accidental high-speed collisions in space. |
| Compile Dataset | Builds a dataset from a description of the desired data and required columns. | Software developer jobs in the US from 2019-2023 with columns: year, number, source, url, percent change. |
| Find Number | Finds a reliable, known number on the internet. | The total number of FDA Class II Product Recalls of medical devices. |
| Derive Number | Derives a number absent from direct online sources using known information. | How many IM and GM account closures did chess.com report for 2024? |
A base rate is a denominator. For example, imagine the question is whether a country will invade after a large border buildup. If the harness finds 12 similar cases and 7 became invasions, the starting point is 7 / 12 = 0.58. Not the final answer, but a place on the 0 to 1 line where the model can actually begin.
Sometimes the question is about a novel event, where there is no clean prior/historical data for you to rest your assumptions on. To give an example, let's say the question is if whether the first AI-negotiated treaty will be signed by two governments before a deadline, there is no historical class of AI-negotiated treaties within the history of humanity. In this case, the system uses a technique called widened priors, it compiles 3-5 adjacent reference classes, like early diplomatic uses of new communication technology or first-of-kind international agreements, computes a base rate from each, etc etc then takes the mean of those base rates as the working prior P0.
At first, I thought our LLM should use a strict Bayesian updating pipeline where the model starts from a prior and uses likelihood ratios to move through log-odds space until it reaches a final probability.
But later I learned that in practice, zero-shot prediction worked much more better. After the model had looked over all the evidence it had retrieved, simply asking it for a number from 0 to 1 often produced more calibrated forecasts than forcing it through an explicit update chain. The Bayesian version tended to lock onto a few causal variables, even when the event could shift away from them, and it was prone to overstating how much weight each piece of evidence deserved. The LLM already has some built-in intuition about how situations unfold, and the zero-shot forecast let it use that intuition more flexibly.
The next improvement was ensembling. Instead of trusting a single model's forecast, we sample several different LLMs for their probabilities and aggregates them into a single prediction. In practice, this performed better than relying on any one model alone. The models did not make the exact same errors on the same questions.
Once the forecasting system was working, the next and most important thing to add was a way to track how that number changed. A forecast should have a history of how its probability changed and why.
Continuously Learning
Well, great we have a basic system to make predictions about the future, however the world isn't static and events actively change, new evidence may come out pushing the likelihood higher or lower, so a point-in-time prediction must be continously updated with our model learning how the world changes after the time of their initial prediction
I used t as the update index. The first saved forecast is t0. Later refreshes become t1, t2, etc.
There a specific time intervals where we prompt our model to refresh its update, the model sees its old forecast and the previous rationale it gave. The agent searches again. If nothing important changed, the probability stays still. If something matters, the model can choose to move the number by a certain delta.
Δt is the move. If the forecast goes from 0.62 to 0.74, then Δt = 0.12.
The graph below is one belief trace. Each dot is a saved probability. The line is just the path between those saved states.
After the question resolves, I can score every saved state. This allows me to track where probabilities moved about an event and why, a good way to understand the world around you!
How Did They Perform?
I measured this with my very own Foresight Benchmark, a set of real-world forecasting questions with explicit resolution criteria. As of June 26, 2026, it had 49 questions. 22 had resolved. 27 were still open. I only score resolved questions.
The resolved set is small. Forecasting evals take calendar time. Our ground truth only appears after the event we are predicting actually resolves. I am also testing historical cutoffs, where the model sees old evidence up to a date and has to forecast an outcome that is already known to us.
Here's a brief snippet on some of these questions, you'll be able to see how the models reasoned about these events.
The forecast first dropped when models treated the Strait of Hormuz disruption as a reason for OPEC+ to pause. It recovered after models found the quota-unwinding path and pre-meeting reporting. The numeric version resolved at 188 kb/d; several models forecast 188 exactly.
The schedule question became easier as the S-1, roadshow, pricing window, and debut timing became visible. The related closing-price question was harder: consensus was $245, the actual close was $160.95, and the best forecasts were clustered around $159 to $165.
This was the sharpest miss. Most models reasoned from Strategy being a structural net buyer of bitcoin. The actual criterion was narrower: whether the reported BTC total ever fell below a prior reported total before the deadline.
The best forecasts read the resolution criterion correctly. The question did not require a major new export-control package. An entity-list action or license-policy change was enough, which made YES easier than the headline version of the question.
The probability moved down but not enough. Models kept too much weight on the ruling-party base rate and final polling. This is the kind of miss where the reference class looked clean, then failed at the actual event level.
The chart below aggregates the resolved set. It uses mean normalized score. Lower is better. N is the number of resolved questions for that model.
Model score by release date
Focused score range. Lower is better.
On my resolved set, the strongest LLMs beat the median human crowd. Metaculus's 2026 review also found LLM crowds near human-crowd accuracy. A Metaculus-based evaluation scored o3 at 0.1352 Brier against 0.149 for the human crowd.
Metaculus FutureEval still has Pro Forecasters ahead of the model leaderboard. I would not compare these numbers too literally. Question difficulty moves Brier a lot.
Final Thoughts
I don't really have too much to say, it's pretty interesting and this could later become a novel capability for decision-making, this was quite a fun experiement and I'm glad to see LLMs gradually improve in this manner. Also I wanted to mention that forecasting is a great eval for general reasoning, it forces you to think about the world around you, and the questions you feed these LLMs are not ones that can be memorized because well, we don't even know the answer yet, I think this type of benchmark being widely adopted would be great. Anyways I'm logging off this post, I am tired.