> We asked it [GPT-5.5] to assess whether each leader had made a falsifiable claim about the future as part of its main thesis. About 1,400 did. We then extracted those predictions, and asked the AI to mark out of ten both how contrarian the leader’s outlook was at the time and how accurate the prediction turned out to be. We ran those queries several times and took an average.
I understand why it wouldn’t be feasible for a human to do this, but I’m quite sceptical about an AI assessing how accurate predictions turned out to be/how contrarian they were at the time. It seems like that would depend a lot on what sources it chooses, be liable to hallucination or getting poisoned by bad sources, etc. They don’t mention whether they used independent queries for each prediction either, or whether it was doing multiple sequentially.
Given that LLMs can’t really distinguish prompt from instructions etc, I’m sceptical that they can reason particularly well about things like how contrarian a view was at a particular point in time.
I thought the same while reading this - I can easily imagine the AI using hindsight to affect "how contrarian the leader's outlook was at the time", in a way that's similar to how we often do the same.
Would be interesting to do a similar analysis but maybe pushing the AI to search the web for articles/other reporting written during that time, to at least correct for that bias a little bit.
> We ran those queries several times and took an average.
The whole thing is bizarre. They could at least have drawn 100 samples and evaluated the model's response. But no, they ran it a few times and hoped that the slight randomness in sampling magically resulted in a balanced assessment of accuracy and contrariness, at some point in time for which we don't even know how much training data there was, nor if that data accurately reflects the opinions of that time. But hey, we've got a graph, so it must be true.
How can people be so stupid?
> cheap bioethanol ... [has] yet to bring about the breakthroughs we prophesied.
One should do this analysis for various politicians, political parties, TV and news media and so on.
How do we know how good or bad the Economist is if we have no other source to compare against? Presumably it isn't trivial to predict the outcome of millions of people's reactions to all the complicated things that are happening to them.
That's not true at all, first of all economics is a social science, secondly I think what you think is ignored is the whole branch of behavioural economics.
Long-time economist subscriber here. This feels too much like a "look what the AI said" sort of article for my tastes.
That said, though The Economist clearly has flaws I have yet to find something better. A weekly, sober, well-written newspaper is a hard thing to find.
That's a disappointing chart from the economist. When you have a lot of points, you can't show them as dots that occlude other dots, or the reader can't get much more of a sense than 'none / some / lots'. What you need to do is bucket by region and do a background heatmap with small dots added, or use a different level of opacity on the dots.
I like to say, traders and people you put money on the line are often far more honest about economic policy, than economic influencers like The Economist.
The quote originated with George Bernard Shaw back in 1903, not Nassim Taleb. And it has always been a lazy take. Some people just prefer to write (or teach).
I really like the economist, they are quite centric in their views but don't hide a certain bias for strong representative democracies, deregulation, globalism and open (publicly traded) markets.
Their writing style is probably the cleanest in the industry and they always point out potential conflict of interest even though I've never seen the write anything really positive about the companies Exor controls (Stellantis, Philips, etc).
Indeed. I unsubscribed (after many years) out of frustration at their incessant and unquestioning incantation of orthodox economic dogmas, in particular economic growth (which is at the very least a flawed indicator). This is what religion looks like. I was paying smart people to do some of my thinking for me, yet I couldn't help but conclude that my mind was more open than theirs.
Still, the quality of The Economist's writing is in a league of its own. No denying that.
The Atlantic. Excellent writers, excellent editing. But as a European I find the US slant is a problem. I do miss The Economist's international coverage.
Nah, even as a leftist, The Economist is mostly in the "enlightened centrist" area. I'd say they're a reliable source of facts as long as you know where their bias is.
The Economist Group is largely owned by the Agnelli family ("Italy's Kennedys"), Canadian billionaire Stephen Smith, the Rothschilds and others, which is likely a significant factor in those biases.
One things I like is that when the economist talks about a business or an author, they mention if there is a conflict of interest (this and that works for this or is owned by that)
I don't know why this is downvoted, in italy agnelli owned via Gedi group various newspapers and as you can imagine it's not ideal - although since this march group Gedi changed owner
If you find a topic where reality lies outside of the overton window for western elites at the time theyll reliably get it wrong.
There are a number of examples of this (e.g. the Afghanistan and Iraq wars).
They usually use a lot of hedging language to cover their behinds when they make predictions, though, so their predictions are usually more nudges and winks than they are commitments.
I don’t see any other publication doing this kind of analysis of themselves, pointing out the mistakes they’ve made in the past.
Even in this article they mention “The Economist was convinced by the false claim that Saddam Hussein was hiding weapons of mass destruction”, something they remind the reader of each time the Iraq War is discussed. I remember an article from 2017 that said the Economist got it wrong, while a less honest publication wouldn’t have.
It’s more remarkable considering that the economist’s articles and opinions are published without the authors details. So when an article gets it wrong, it reflects poorly on the economist as a whole. They can’t simply blame the previous guy, it’s their fault.
They’ll be talking about how they got the Iraq War wrong as long as someone mentions it even in passing 50 years from now. That’s candour I appreciate.
You’ve misread what I wrote. I wasn’t explaining why there are no bylines. I was saying that when there are no bylines, any mistake reflects poorly on the newspaper as a whole. It wasn’t Shashank Joshi who got it wrong, it was the Economist.
It is, and I'm aware of the less charitable (perhaps more likely) readings. But much of the time it isn't difficult to know who to attribute because they are explicitly featured in podcasts about the respective article.
They are the consensus. Making up stories to support the consensus is their job.
Of course they're reliably wrong. The stories are immoral by default and often objectively insane. But the Serious People either believe them, or put a lot of money and effort into making sure politicians and the media repeat them.
The public don't set policy and don't read The Economist, so they're handed their beliefs by other means.
The major crime of media is misframing as opposed to being wrong.
> The Economist was convinced by the false claim that Saddam Hussein was hiding weapons of mass destruction
The issue here is not whether or not the Iraqis had any weapons of any type, the issue is whether the US has a basic right to travel to the opposite side of the globe and invade random countries. One of the issues the anti-Trump crowd has right now coming down on his crazy and unprovoked attacks on Iran has been that it is absolutely an established pattern of US behaviour and he's well within normal practice. Trump is even claiming it is because of WMDs! Again! Eventually at least, after he tried a few other excuses first. "He's lying" doesn't cut much ice because that is pretty normal when the US goes to war and it usually isn't particularly subtle. The only difference from the Economist's perspective is how enthusiastically the reporters want to pretend they believe the obvious mistruths.
Anyway, that rant aside, the issue is more that the Economist is pretending that the unknowable is knowable rather than that they get it wrong more often or not than anyone else. It doesn't seem theoretically possible to predict the price of things more than "up", "down" and "assume it does what it always does". Mapping out the supply/demand curve and confidently forecasting how it could shift in the short term requires knowledge of all actors, their preferences, options and operational constraints. Even for a relatively superficial forecast. Who's got that sort of modelling firepower?
Given that many economic predictions are really about pushing an agenda, they are not at all useless, or at least, would be better measured against that criteria. I suspect the reason we still have neoliberal economics is precisely because despite its manifest failures of prediction, it still pushes the right agenda to be supported.
Lol @ the use of "bête noire", which is of course a way to avoid the old racist saying of you made a hugely and catastrophic mistake ... saying you were black-faced. But surely in French, it's fine, right?
https://archive.is/dFnBu
> We asked it [GPT-5.5] to assess whether each leader had made a falsifiable claim about the future as part of its main thesis. About 1,400 did. We then extracted those predictions, and asked the AI to mark out of ten both how contrarian the leader’s outlook was at the time and how accurate the prediction turned out to be. We ran those queries several times and took an average.
I understand why it wouldn’t be feasible for a human to do this, but I’m quite sceptical about an AI assessing how accurate predictions turned out to be/how contrarian they were at the time. It seems like that would depend a lot on what sources it chooses, be liable to hallucination or getting poisoned by bad sources, etc. They don’t mention whether they used independent queries for each prediction either, or whether it was doing multiple sequentially.
Given that LLMs can’t really distinguish prompt from instructions etc, I’m sceptical that they can reason particularly well about things like how contrarian a view was at a particular point in time.
I thought the same while reading this - I can easily imagine the AI using hindsight to affect "how contrarian the leader's outlook was at the time", in a way that's similar to how we often do the same.
Would be interesting to do a similar analysis but maybe pushing the AI to search the web for articles/other reporting written during that time, to at least correct for that bias a little bit.
you can still judge if it was correct or not, irrespective of contrarianness, still a good signal to measure
It can give a judgement, correct or not.
> We ran those queries several times and took an average.
The whole thing is bizarre. They could at least have drawn 100 samples and evaluated the model's response. But no, they ran it a few times and hoped that the slight randomness in sampling magically resulted in a balanced assessment of accuracy and contrariness, at some point in time for which we don't even know how much training data there was, nor if that data accurately reflects the opinions of that time. But hey, we've got a graph, so it must be true.
How can people be so stupid?
> cheap bioethanol ... [has] yet to bring about the breakthroughs we prophesied.
Oh.
I’ve noticed The Economist having odd ways of interpreting statistics and data, and this article makes me even less trustful.
Odd how? I’ve been a reader since 1985 and am satisfied.
One should do this analysis for various politicians, political parties, TV and news media and so on.
How do we know how good or bad the Economist is if we have no other source to compare against? Presumably it isn't trivial to predict the outcome of millions of people's reactions to all the complicated things that are happening to them.
psychohistory, some are already doing it today (LLMs simulating various people with various beliefs interacting)
Economists actually don't do that. They only use math. It's why they always get it wrong- they ignore the social sciences.
That's not true at all, first of all economics is a social science, secondly I think what you think is ignored is the whole branch of behavioural economics.
Economics is not a science. It's a social study with a layer of pseudo maths slapped on it to give it a pretence of respectability.
Long-time economist subscriber here. This feels too much like a "look what the AI said" sort of article for my tastes.
That said, though The Economist clearly has flaws I have yet to find something better. A weekly, sober, well-written newspaper is a hard thing to find.
> By contrast, when we suggested in April that oil prices were likely to rise further, we were out on a limb. Alas, we were dead wrong.
Why "alas?" They wanted an economic collapse?
Maybe the Economist wanted to provide good analysis for their readers and they feel sorry that they failed.
That's a disappointing chart from the economist. When you have a lot of points, you can't show them as dots that occlude other dots, or the reader can't get much more of a sense than 'none / some / lots'. What you need to do is bucket by region and do a background heatmap with small dots added, or use a different level of opacity on the dots.
There aren't that many points. It looks fine to me. (But I agree that is a common mistake.)
to quote Taleb: those who can, do (trade). those who can't, write about it (financial press)
I like to say, traders and people you put money on the line are often far more honest about economic policy, than economic influencers like The Economist.
Taleb is literally mostly an influencer who once traded (and sucked at it).
The quote originated with George Bernard Shaw back in 1903, not Nassim Taleb. And it has always been a lazy take. Some people just prefer to write (or teach).
I really like the economist, they are quite centric in their views but don't hide a certain bias for strong representative democracies, deregulation, globalism and open (publicly traded) markets.
Their writing style is probably the cleanest in the industry and they always point out potential conflict of interest even though I've never seen the write anything really positive about the companies Exor controls (Stellantis, Philips, etc).
Indeed. I unsubscribed (after many years) out of frustration at their incessant and unquestioning incantation of orthodox economic dogmas, in particular economic growth (which is at the very least a flawed indicator). This is what religion looks like. I was paying smart people to do some of my thinking for me, yet I couldn't help but conclude that my mind was more open than theirs.
Still, the quality of The Economist's writing is in a league of its own. No denying that.
Having similar thoughts at times. What are you reading nowadays?
The Atlantic. Excellent writers, excellent editing. But as a European I find the US slant is a problem. I do miss The Economist's international coverage.
Coincidentally enough, frequent Atlantic contributor James Fallows wrote a rather scathing assessment of the Economist (albeit in the Post) way back in 1991: https://www.theatlantic.com/technology/archive/1991/10/-quot...
To be fair, people have been writing scathing assessments of newspapers they don't like for centuries.
Nobody ever recognizes themselves as extremist.
Nah, even as a leftist, The Economist is mostly in the "enlightened centrist" area. I'd say they're a reliable source of facts as long as you know where their bias is.
I'm pretty sure some people do. Luddites, for example.
The Economist Group is largely owned by the Agnelli family ("Italy's Kennedys"), Canadian billionaire Stephen Smith, the Rothschilds and others, which is likely a significant factor in those biases.
One things I like is that when the economist talks about a business or an author, they mention if there is a conflict of interest (this and that works for this or is owned by that)
The economist hasn't changed much their editorial views or leanings in a century.
I don't know why this is downvoted, in italy agnelli owned via Gedi group various newspapers and as you can imagine it's not ideal - although since this march group Gedi changed owner
If you find a topic where reality lies outside of the overton window for western elites at the time theyll reliably get it wrong.
There are a number of examples of this (e.g. the Afghanistan and Iraq wars).
They usually use a lot of hedging language to cover their behinds when they make predictions, though, so their predictions are usually more nudges and winks than they are commitments.
I don’t see any other publication doing this kind of analysis of themselves, pointing out the mistakes they’ve made in the past.
Even in this article they mention “The Economist was convinced by the false claim that Saddam Hussein was hiding weapons of mass destruction”, something they remind the reader of each time the Iraq War is discussed. I remember an article from 2017 that said the Economist got it wrong, while a less honest publication wouldn’t have.
It’s more remarkable considering that the economist’s articles and opinions are published without the authors details. So when an article gets it wrong, it reflects poorly on the economist as a whole. They can’t simply blame the previous guy, it’s their fault.
They’ll be talking about how they got the Iraq War wrong as long as someone mentions it even in passing 50 years from now. That’s candour I appreciate.
This is a very charitable reading of why The Economist does not have bylines.
You’ve misread what I wrote. I wasn’t explaining why there are no bylines. I was saying that when there are no bylines, any mistake reflects poorly on the newspaper as a whole. It wasn’t Shashank Joshi who got it wrong, it was the Economist.
It is, and I'm aware of the less charitable (perhaps more likely) readings. But much of the time it isn't difficult to know who to attribute because they are explicitly featured in podcasts about the respective article.
They should have checked if they are more accurate than the consensus, the data doesn't show this.
They are the consensus. Making up stories to support the consensus is their job.
Of course they're reliably wrong. The stories are immoral by default and often objectively insane. But the Serious People either believe them, or put a lot of money and effort into making sure politicians and the media repeat them.
The public don't set policy and don't read The Economist, so they're handed their beliefs by other means.
The internet of 2026.
Betteridge's law wrapping a paywall wrapping an AI piece masquerading as domain-specific analysis.
What a slimy headline.
You can imagine some narcissistic abuser saying the same thing.
Maybe "not always wrong" isn't the bar you should be shooting for.
Disappointingly this is merely about the accuracy or otherwise of The Economist's predictions, not the rightness or otherwise of their world-view.
The major crime of media is misframing as opposed to being wrong.
> The Economist was convinced by the false claim that Saddam Hussein was hiding weapons of mass destruction
The issue here is not whether or not the Iraqis had any weapons of any type, the issue is whether the US has a basic right to travel to the opposite side of the globe and invade random countries. One of the issues the anti-Trump crowd has right now coming down on his crazy and unprovoked attacks on Iran has been that it is absolutely an established pattern of US behaviour and he's well within normal practice. Trump is even claiming it is because of WMDs! Again! Eventually at least, after he tried a few other excuses first. "He's lying" doesn't cut much ice because that is pretty normal when the US goes to war and it usually isn't particularly subtle. The only difference from the Economist's perspective is how enthusiastically the reporters want to pretend they believe the obvious mistruths.
Anyway, that rant aside, the issue is more that the Economist is pretending that the unknowable is knowable rather than that they get it wrong more often or not than anyone else. It doesn't seem theoretically possible to predict the price of things more than "up", "down" and "assume it does what it always does". Mapping out the supply/demand curve and confidently forecasting how it could shift in the short term requires knowledge of all actors, their preferences, options and operational constraints. Even for a relatively superficial forecast. Who's got that sort of modelling firepower?
Most of the time, especially when it comes to China. Every front page article on China has been wrong for at least the last decade.
It’s always wrong when it really matters.
You mean when it’s really Scottish?
rare example of a question in the headline where the answer is yes
Wait isn't the answer No here? The dots are clearly clustered mostly in the correct half of the graph?
Isn't that the nature of futurology. You are always wrong in some way. That doesn't necessarily make it useless
Given that many economic predictions are really about pushing an agenda, they are not at all useless, or at least, would be better measured against that criteria. I suspect the reason we still have neoliberal economics is precisely because despite its manifest failures of prediction, it still pushes the right agenda to be supported.
"We investigated ourselves and found we did nothing wrong"
Lol @ the use of "bête noire", which is of course a way to avoid the old racist saying of you made a hugely and catastrophic mistake ... saying you were black-faced. But surely in French, it's fine, right?