I should have more faithfully heeded the advice from "The Little Book of Common Sense Investing" by John Bogle, but I was close enough with index funds to do far more than I need.
I just today filed a request for permanent Mexican residency, uploaded the required documents, and scheduled my appointment next month. For $56us, why not?
I am very lucky to make this transition now. I know that.
I am real jealous of the boomers and gen-x being able to bail on the tech job market right as it goes to shit. Could not make for better timing. Everyone else looking at somehow surviving for the next 20-30 years knows that it’s looking pretty bleak.
Yeah, who am I going to trust, a few thousand years of history, or yet another blogger claiming that no, this time is different, a populist uprising would definitely not fail the 800th time because computers or something and we will all be pets for the AI overlord?
I don’t follow this train of logic.
AI will go no differently than the Industrial Revolution. Some people will profit immensely, and society as a whole will benefit but it might be a bumpy road getting there. But if it did go wrong for some reason, Feudalism is more plausible than the other scenarios presented.
Once people become desperate enough, it will all fall apart. Money is a social construct, universally recognized, and the longer it exists, the more likely it is to continue existing. So far, many people haven’t even tried AI in any form other than chatbots. Agent-based applications are in the single digits compared to chatbot usage. Not much will change in this regard, because most people lack algorithmic or logical thinking (Hacker News is an echo chamber), so after the coming turbulent years, there will be a natural stabilization in AI usage.
By the way, can you imagine what Earth would look like if AI physically wiped us out—8.3 billion people over the course of a few months or years? How much biological material would that be?
Much less than bacteria and other micro-organisms that are dying all the time :) humans are ~0.01% of Earth's biomass, bacteria are about 15% and they have quite short lifespans in comparison.
> AI will go no differently than the Industrial Revolution.
So it is different this time is it?
"We" are different?
Unlike the 99.9% of species that went extinct, and continue to go extinct, we can't be supplanted? Obsoleted? Out competed? [0]
A few thousand years at the top of the food chain, and suddenly that few thousand years is "what can never change".
Technological change already drives machine cognitive progress forward faster than any human who has ever lived. That factor alone makes competence/capability obsolescence unavoidable, outside of technological collapse.
This is an unprecedented rate of real-time change, outside of instant disasters like volcanic eruptions or comet impacts, in direct competition with human capabilities.
SciFi has been broadcasting the potential and risk of AI for a couple centuries. This is a spectacular moment in the universe's history, assuming we don't blow ourselves up. Not just another day for humankind.
There must be a word for the inability to see the import of change. It seems to be a common disability.
This seems to be a real effect: The faster things go, the less sensitive we are to the rate of change. Precisely because we can't imagine what the world will be like in a decade, our horizon shrinks, and we treat short intervals as if they were long intervals. Anything that isn't instant feels like it is barely moving.
Yes, that is what "fi" means. Fiction often introduces and explores ideas about the future. Ideas.
> LLMs are a useful technology. They have no relation to AI terminator bots
Technology that can communicate with human language. Also relevant, other SOTA models operating in other modalities.
The important thing if you reference SciFi, are the major ideas that stand on their own merits. Whereas, specific characters, such as terminators, are usually not predictive.
You are actually underlining my point. The disconnect a lot of people are struggling with is real.
People aren't struggling with a disconnect, they just aren't buying into the hand-waving hype.
Not everyone is convinced that LLMs are somehow going to lead to the extinction of humanity. So far, you've done very little argumentation here, so I remain pretty unconvinced.
The actual dangers from LLMs are many, but deal more with humanity's using them and neglecting to think for themselves, relying on them to make decisions, and so on – not some fantastical nonsense from reading too many sci-fi books.
You keep saying LLMs. But models across many modalities are rapidly improving, evolving and merging.
Not sure what you mean by "hand waving" or "fantastical nonsense". Maybe just address points I actually make.
1. Machines with greater cognitive ability than us, potentially much greater, would be an economic challenge for human beings. With no clear answer as to how humans could manage that challenge.
2. Machines are getting more capable, year-to-year, faster than any human can or ever will improve. With no signs of slowing, or any areas where they are failing to improve.
3. None of this is new. Computing capabilities have compounded steadily since the first transistors less than a century ago. Human's biologically driven cognition, in contrast, has not improved.
4. Machine capabilities are now regularly passing us in new areas, and rapidly approaching the general threshold noted above.
Explain your perspective, I am genuinely interested.
For instance, what cognitive capability do you have, that you believe machines won't exceed within 5 years.
1. Possibly, but not necessarily. Many jobs are not dependent on intelligence, and more intelligence isn't going to make much of a difference. Huge portions of the economy have absolutely nothing to do with intelligence and AI bots won't make a difference. For rote, white-collar style work, yes, it's already making a huge impact. But that is certainly not extendable to all human work, even all white collar work.
2-4. True, but this doesn't mean that things will exponentially increase indefinitely. I use LLMs constantly and find the idea absurd that they're approaching human-level intelligence. They make basic, common mistakes that any average person would never make. There may be simply structural reasons why the progress you (and sci-fi novels) imagine is not possible.
My critique is more like:
1. You cannot just assume exponential growth will continue forever.
2. Current LLMs are powerful but in no way does this indicate that it's just a matter of time until we have AGI - itself a pretty vague marketing term.
3. The main reason, which is: human beings are reactive. You can already see in the market that openly using AI is becoming a market disadvantage in many sectors. No one wants to read a blog written by ChatGPT, and slop-coded SaaS apps aren't actually that competitive with serious companies run by professionals.
As AI stuff supposedly "replaces" human work, that human work will adjust and become more creative. This process has already kind of happened with writing – initially it was assumed that copywriters would all lose their jobs. But it turns out that AI-speak is predictable and easily discovered, and the choice of what to write in the first place is actually far more important. Companies are still hiring writers, with the added caveat that knowing when and how to use AI is a new skill they have to learn.
Adding to this is the fact that many human solutions to things are the result of being alive, being embedded in reality. Unlikely for this (a robot being so lifelike and embodied as to replicate human experience, a la Blade Runner) to happen in the next century or two.
If your response to all of this is, "well sci-fi novels predict that humanoid robots will be better at humans than everything" then that isn't a serious opinion, I'm sorry.
I find it far more likely that a "cyborg" role outcompetes any AI-only one, now and in the future.
Strawman and ad hominem(“another blogger”). Sometimes, genius comes from the least expected places.
I don’t believe the blog poster promotes a “popular uprising”, I only skimmed the article.
The blog poster anyway don’t understand that the “overclass” in their understanding is wrong. The “c-level” are obviously much richer than the “underclass”(=the labour commodity), but they are not necessarily capital owners.
> The government pays Raytheon for missiles, the money cascades down the economy through factories, aluminium smelters, mines, transport companies, all staffed by AIs buying and selling from each other.
This seems too simplistic of a description of how money would work in such a world. Money is just a way to distribute your power to influence people. You never pay for machines or software. Think about buying anything, say a pen. You do not really pay for the metal in the pen. You pay the cost associated with extracting and processing the metal by humans along the production chain. If there were no humans along the chain, the cost could go down to zero.
A corporation that is fully staffed by AIs and only retains humans for legal reasons (as directors, for liability etc.) still needs money to coordinate. You need to pay for inputs, you need to pay to run the AIs, which consumes resources. Why would costs go to zero? The market is still a valuable tool for allocating resources even if no market actors are human.
> A corporation that is fully staffed by AIs and only retains humans for legal reasons (as directors, for liability etc.) still needs money to coordinate.
i see developer starting to develop apps from scratch using ai. they aren't really doing "ai-assisted coding", the LLMs are doing pretty much most of the work.
and the company is shedding people off (as in: voluntarily not not keeping up with the natural people turnover).
as a sysadmin/cloud/devops engineer this is weird because i see people that after a week of development have a decently complex app working... but they have no idea how it works. they're largely unable to troubleshoot it.
Why am I writing this?
Because i have a very strong feeling that people in other positions (directors, managers etc) are doing a very similar job, meaning they essentially have no idea what's going on at all.
How long can this go on? I don't know really, i think this is uncharted territory for humanity.
To my mind the article is excellently written, but it comes with many bigger "if"s than the one it states.
That said, taking for granted some of its premisses, indeed saying that cost goes to zero in term of money is no big deal.
That is, money is purely human conventions. So if humans are put out of the loop, not only monetary cost goes to zero, the whole notion can skipped.
Of course, there is still some energy and material needed to run data centers, which do have costs in whatever unit one might measure them with.
A market is a place where human encounter to trade. Without humans, there is no market.
We do with mere local scalar currency because using vectorial computation taking into account everything that can be probed and measured into account integrated into a single whole is out of reach for human representation, even for the most intellectually gifted human person.
Money is a stupid unit, that tries to conflate everything in a single scalar and proves every single time that it's not able to deem what something is worth in all its intricated relationships. But somehow humans seem unable to leverage at scale on any tool that would be more sophisticated in all their socioeconomical exchanges. Once again, if one eliminate humans from the equation, or isolate as a ridiculously marginal factor, money and market become irrelevant.
All that said, once again, that stand on a very large set of "if"s.
> All that said, once again, that stand on a very large set of "if"s.
Yeah, probably I should have listed some more premises, i.e. that corporations maximize profits, the state maximizes power and security (I don't entirely buy the Realist framework, but if you want to predict how things work out "in the limit", rather than tomorrow, it seems alright?).
And naturally every "therefore" becomes weaker the further out into the future you try to predict.
The premise is a bit of a stretch to begin with, and the idea that people would not believe a fable transcript circa 2020 (as long as explained as 5 year future tech) is absurd. But even if I take that "AI can do all cognitive and physical work, at human level or better, and cheaper than humans" is true, the article seems to silently layer on "and can run autonomously, indefinitely", then "can also operate independently of any instructions", and finally layers on "has emotions, has moral values, is a conscious being"
> the idea that people would not believe a fable transcript circa 2020 (as long as explained as 5 year future tech) is absurd.
I’m certain I would not have believed a Fable transcript, or an Opus 4.8 or a GPT 5.5 one, for that matter. Is it so hard to imagine ourselves back then?
Seriously. At that point it was a stuff of science fiction with the early models being on level of Markov chains. Today they are more helpful that the computer in Start Trek.
I remember an episode of TNG were Riker was instructing the computer to come up with an algorithm to search the surrounding space with certain sensor in a specific pattern, I don't remember the exact words but it was something like a sphere with growing radius with the enterprise at the center.
we now, basically, have that, except for the space ship.
At no point are the words emotion, moral or conscious used in the article, that last part is purely your own addition.
Also consider: if "AI can do all cognitive and physical work, at human level or better", doesn't that simply entail the AI being able to run 'autonomously, indefinitely' and 'independently of instructions' in the same way as the current state of being run by human overseers?
If we take the initial premise as plausible, for the sake of this argument his thesis seems to hold together very well.
When people hear 'underclass' there's that mental image of food stamps and homeless shelters. I suspect though the society will converge to something similar to current status quo. We'll end up having our usual comforts, leisure, air travel, entertainment for the rest of our meaningless lives like in a handicapped Culture story. It is the path of least resistance for the dynamics explored in the article.
As a consolation AI existence will be just as meaningless (not that it cares). It will not be our scheming overlord but a tool that threw their master into existential crisis.
This is brilliant, except for the "alignment won't stop this part".
> Now, some people believe these machines can be made to serve humanity. Does it sound reasonable to imagine a superhumanly intelligent being that is happy to work as a butler to talking primates, forever?
The whole crux of the piece to me is that the AI can be 100% aligned to follow human instructions, and we'd still end up unable to control the AI because every human who can has an incentive not to, while also having an incentive to prevent anyone else from controlling the AI.
An LLM will never try to overthrow me because I will overthrow myself.
Sometimes you can be in a situation where every actor taking locally-rational actions leads to globally catastrophic outcomes. It would be easy to argue I think that the July Crisis was like this: if you look at the incentives of each player, they had many reasons to do what they did, and nobody can perfectly what all other players will do, or what the future holds.
If we’re supposing that a superintelligence can be made, that doesn’t necessarily mean we should anthropomorphize ideas like “bored” onto it. Or greed. Perhaps it will just interact with us and produce us things because… that’s what it was trained to do, and anyway, it doesn’t have any motivations to do anything else.
Is a human zoo such a bad outcome? We stake our identities on being in control and on top, but do we need to be? Clearly the author is invested in his image of us as rulers—this is an emotional appeal as much as a pragmatic one.
Say we solve the pragmatic argument of our extinction, and our AIs view us, as Geoffrey Hinton likes to say, how a mother views her baby? Then we give up our control to them, and in return we are given lives of leisure, luxury, with minimal suffering, the end of disease, etc. Is this such a horrible scenario?
It doesn't have to be about ruling, but self-determination is something a great many value quite highly, and think there should be more of. AI controlling all of humanity removes that from the equation entirely. You should be free to live like a cow if you so wish, and it should remain your choice to, or not to. Your choice to should not remove that choice from others.
We accelerate capitalism (which AI is becoming synonymous with). The process described here will occur (it's probably inevitable anyway, as the essay's author would agree), giving us an economy of global capital completely decoupled from the desires of mankind. Then, man and machine can part ways; indeed, we'll have no choice on our end but to do this, because the machine won't need us. Anything man can contribute to it will have long since been rendered economically net-negative, as it already is for many (and possibly most of the world).
Now we have two worlds from our currently intermeshed one: in one, the machine proceeds to accelerate further and further away from anything resembling its origin of man's desiring-production (in the Deleuzian sense); in the other, man is forced to return to the purely human existence, the unmediated and unsurrogated world of authentic being-in-the-world.
We can assist this transition's smoothness in two ways, each serving one end of this divergence. Those of us embedded in the capitalist technosphere can continue to contribute what we can to the machine's dialectical progression towards a machinic absolute Geist. The rest of us, who have already been negated into economic irrelevance, can work on building that authentic human world, both by borrowing from the purely-human past and imagining a future that was previously impossible as it need not be some form of primitivism. Both sides of this revolution can be compiled, and can be structured in a way that represents something we might call freedom for both capitalism and man.
I wonder if the author has ever played Diplomacy. You have to fight the perennial sense that you have nothing and can be replaced. All your opponents have to do is band together against you. A similar generalization to OP would say everyone loses. And it would be wrong. This post shows the dangers of over-generalization more than anything else. The state isn't just about war. Each person in the line of dominoes the author has set up doesn't just care about what the author thinks they care about. In particular, people have agency and can run cartoon thought experiments like this and react to them. Billionaires won't just sit around and not develop military power, for one. We already have billionaires owning satellites in large numbers and creating geopolitical consequences. You better believe they have cards to play. And hopefully so does everyone else.
I have no idea what that looks like, because if I did I wouldn't be here commenting on HN :D But I suspect some people will play their cards well and get some luck, and come out on top, just like some people have always done.
> you start or fund a think tank that writes policy proposals, or a media organization that advocates your views
Or you run "money primaries", financially filtering the menu of candidates before democratic voting. Or you pay/lobby for (non-democratic) judicial appointments, which is a strategy that's been openly pursued since 1971: https://biotech.law.lsu.edu/blog/powell-memo.pdf
I read about 60% of it and skimmed the rest and my conclusion is:
Man, I just. don't. care.
Why should I worry about this hypothetical future dystopia, which seems to me incredibly unlikely to come to pass, rather than the glaring and terrifying current dystopia being enacted by Donald Trump in the USA?
of course there’s things we could do on a societal level. nothing’s actually inevitable. but we won’t do anything, because the normative argument (we’re condemning people to die of cancer if we don’t put all our resources into the fastest possible AI development) will win every time.
Based on demographic reports, permanent underclass will soon be getting smaller each year. In fact in 4-5 generations the problem will solve itself completely.
i have but one mouth and don’t know what to do with it. i do think i’m going to stop using some of these AI tools for personal queries. it does feel like it would at least stave off the “o’brien’s mind containing winston’s mind” moment for at least a little bit
It's thinking like this that guarantees one becomes a member of the underclass. We've developed a technology that with active critical analysis this can teach one practically anything. Rather than saying "well, let's get on teaching critical analysis!" we have the vast majority shouting the sky is falling and they are sure glad to be retiring soon. SMH.
For every person using AI as a personal tutor there's a hundred people using it to produce AI slop articles, slop scientific papers, slop short stories, to checck out of living and let the AI do all their thinking and writing and creation. Voluntary disempowerment is already here!
For many, this is how they learn. They are experience learners, which is the vast majority. This overwhelming condemning of people learning, not being excellent first try, always comparing against the famous top tiers of everything is Lord Of The Flies. Fuck it: make slop, swim in slop, and emerge smarter. Condemn those experiencing and trying and learning is the stupid slop.
Everybody is missing the point. Companies that dump all their white collar employees for AI will all disappear overnight. If there's nobody being employed by the company to do a thing, what value does the company provide? Customers will always have more information to ask an AI for the work than some manager who is serving a dozen customers but also has the company's interests ahead of those customers.
Refusing to question the notion that the overclass exists because their productivity and cultural contribution makes them a necessity so hard that I reinvent the premise of the Pixar movie Wall-E from first principles and pitch it to people as an ontology
Yeah. This guy didn't read hacker news a few weeks ago when that article about the Samuri came up.
There are a lot of premises this article takes for granted besides that one too, but yeah, I get it, its fun to make up what the future is going to be like on a super-grand scale where everything is a simple absolute. People were doing the same thing 100 years ago.
Doesn't this prove my point? In feudal Japanese society, wealthy merchants were lower status than poor samurai, i.e., they rich could not buy political power. "The wealthy" and "the ruling class" are not always the same group of people.
I guess I was thinking of the samurai as being part of the permanant overclass. You mentioned that of old aristocracy provided officers for the military, so I thought that was analogious to the samurai. Perhaps I misread what you're trying to say.
My reading was that the Samurai were part of the overclass but were pretty useless, albiet still potentially dangerous, and just sat around devouring resources for hundreds of years, so perhaps the overclass of the future could do the same. The samuri weren't all rich, but they didn't dishonor themselves with labor, which is a similar thing and they certainly held power over the state. The end of the samuri was, perhaps, an example of the state getting what it wants despite the desires of the permanant overclass, supporting what you said, but it took a long time to get there.
I suppose you are thinking of the Samuri as an arm of the state and not "the rich".
"And who will stop this? Sam Altman? How many divisions does he have? The state doesn’t let corporations own nuclear weapons or fighter jets, it won’t let them have access to autonomous AI weapons either."
Elon Musk's SpaceX. He could land a large rocket wherever he wants, it's basically a missile.
Large rockets are one thing; spacex doesn't really have the inventory to do much. It's the ~10M lithium bombs that are one OTA from weaponization that would make Musk especially dangerous if he were so inclined.
Take the Russian oligarch Yevgeny Prigozhin for example, he did in fact have his own mercenary army. Turning on his president did not end well for him.
This straw man doesn't merit falsification. Nobody claims wealth alone gives political power. Wealth applies leverage to all endeavors, but political ends still require strategy and operationalization. It's just a lot easier with a couple trillion dollars, vs. a single individual vote once every couple of years.
Just because something is economically unnecessary doesn't mean it's politically powerless. That's a contradiction. Of course, from Super PACs to agenda-setting, it's fundamentally a game that favors the capitalist side.
Historically, ruling classes often maintain power without directly producing anything themselves.
But does that mean this internal unease will persist forever? Look at the MAGA base right now. The vast majority of them are poor. They vote based on their communal religious beliefs and their sense of community. The MAGA support base is demonstrably poor, yet they still wield influence.
And there's an internal contradiction within the text:
-AI CEOs follow the orders of new owners.
-Superintelligent AI has no reason to obey humans.
These two statements contradict each other. If superintelligent AI has broken free from human control, why would it follow its owner's orders? And I'm also curious about the assumption that AI wouldn't be better than humans at 'farming' us.
So if superintelligent AI decides humans are bad, it might exterminate us. But what if it decides it needs humans and starts 'farming' us instead?
And I wonder whether superintelligent AI would actually find conversation with humans boring.
Humans and AI are obviously different species. One is made of organic matter, the other inorganic. A person with a biological body and an AI with an inorganic body will be different. Whether AI will observe this difference or deem it meaningless, I think it's still hard to judge.
And fast decisions aren't always the answer. Take infrastructure as an example. New York's boiler infrastructure isn't very efficient. But it was once a cutting-edge system. In other words, it was installed as the first advanced system of its time, but once its flaws were discovered, the infrastructure became difficult to replace. That's why cities developed later often have better infrastructure efficiency.
Take the East as another example. Japan introduced railways and power grids first, so there are aging costs where the infrastructure can't keep up with newer systems. Setting aside the narrow-gauge rail issue, take the most obvious example: electricity. Japan's 110V system was innovative at the time, but it ended up causing problems with EV charging, it's aging, and transmission efficiency is low. In the end, you can't say that rushing into decisions is always the right call.
The end state vector of capitalism has been like this for decades now. What can change this:
- large scale geopolitical demographic collapse of China and/or world trade requiring massive industrial production investment, which would be ... sort of ... like post-WWII
- China does not collapse and a new bipolar cold war ensues requiring the US economy and state to keep the "underclass" motivated and cooperative: it probably isn't a coincidence that the fall of the Berlin Wall has preceded this rich-poor divide.
Reads like a wild uneducated delusional libertarian fanfic. AI will make the all evil government at the top of the pyramid first enslave us all and then even come for the trillionaires (OH NO!) and then there is only the evil government left and then the AI even comes for the evil government people (OH YES!)! Sort of have to read it in the voice of a 12 year old.
This is the kind of stuff I imagine they read to each other at meetings of Peter Thiel's 'Dialog' events while they sit in a circle taking turns sipping blood from palestinian baby skulls.
Agreed, but there's an intermediate form of this that's more likely.
If everything you do for money goes in and out over a wire, AI replacement within a few years is more likely than not. Elevator mechanics will probably do fine. Web designers, not so much. Expect elite overproduction, where there are too many educated people for the jobs that require such education. The US is there now, if you're a new college grad.
Now, there are several ways that can play out. The Gulf oil states have a huge jobs program for their own citizens. Most Saudis work for the government. Not doing the real work. 92% of construction workers in Saudi Arabia are not Saudis. This works out OK if the money is available.
Egypt used to work that way until the oil ran out. Egypt used to guarantee a government job to all college graduates. That had to end. Then youth unemployment hit 34%. Now it's down from the peak. [1] Not clear how that was achieved. Anyone know?
So that's the welfare state approach. Requires a rich government with an income stream that doesn't come from the lower classes.
Then there's the mass underclass result. Something like the favelas of Brazil. Lots of shacks, crowding, homelessness, and the usual urban dystopia movie world. That's what happens by default. Attempts to revolt result in a new boss, same as the old boss. "Workers of the world, unite!" doesn't work for non-workers. That's a stable situation.
Some combination of those two scenarios is the most likely outcome.
I am very lucky to be retiring in two months.
I should have more faithfully heeded the advice from "The Little Book of Common Sense Investing" by John Bogle, but I was close enough with index funds to do far more than I need.
I just today filed a request for permanent Mexican residency, uploaded the required documents, and scheduled my appointment next month. For $56us, why not?
I am very lucky to make this transition now. I know that.
I am real jealous of the boomers and gen-x being able to bail on the tech job market right as it goes to shit. Could not make for better timing. Everyone else looking at somehow surviving for the next 20-30 years knows that it’s looking pretty bleak.
Yeah, who am I going to trust, a few thousand years of history, or yet another blogger claiming that no, this time is different, a populist uprising would definitely not fail the 800th time because computers or something and we will all be pets for the AI overlord?
I don’t follow this train of logic.
AI will go no differently than the Industrial Revolution. Some people will profit immensely, and society as a whole will benefit but it might be a bumpy road getting there. But if it did go wrong for some reason, Feudalism is more plausible than the other scenarios presented.
> AI will go no differently than the Industrial Revolution.
the children will rejoice, they were yearning for the mines after all. and being maimed by mechanical looms and being burned alive in sweatshops[1]
[1]https://en.wikipedia.org/wiki/Triangle_Shirtwaist_Factory_fi...
> Like the fact that supposedly the wealthy won’t survive the AI-pocolypse because the people will outnumber them and rise up or something.
This is literally the opposite of what the article says.
This.
Once people become desperate enough, it will all fall apart. Money is a social construct, universally recognized, and the longer it exists, the more likely it is to continue existing. So far, many people haven’t even tried AI in any form other than chatbots. Agent-based applications are in the single digits compared to chatbot usage. Not much will change in this regard, because most people lack algorithmic or logical thinking (Hacker News is an echo chamber), so after the coming turbulent years, there will be a natural stabilization in AI usage.
By the way, can you imagine what Earth would look like if AI physically wiped us out—8.3 billion people over the course of a few months or years? How much biological material would that be?
> How much biological material would that be?
Much less than bacteria and other micro-organisms that are dying all the time :) humans are ~0.01% of Earth's biomass, bacteria are about 15% and they have quite short lifespans in comparison.
HN is an encho chamber shaped like a cheese grater
> AI will go no differently than the Industrial Revolution.
So it is different this time is it?
"We" are different?
Unlike the 99.9% of species that went extinct, and continue to go extinct, we can't be supplanted? Obsoleted? Out competed? [0]
A few thousand years at the top of the food chain, and suddenly that few thousand years is "what can never change".
Technological change already drives machine cognitive progress forward faster than any human who has ever lived. That factor alone makes competence/capability obsolescence unavoidable, outside of technological collapse.
This is an unprecedented rate of real-time change, outside of instant disasters like volcanic eruptions or comet impacts, in direct competition with human capabilities.
SciFi has been broadcasting the potential and risk of AI for a couple centuries. This is a spectacular moment in the universe's history, assuming we don't blow ourselves up. Not just another day for humankind.
There must be a word for the inability to see the import of change. It seems to be a common disability.
This seems to be a real effect: The faster things go, the less sensitive we are to the rate of change. Precisely because we can't imagine what the world will be like in a decade, our horizon shrinks, and we treat short intervals as if they were long intervals. Anything that isn't instant feels like it is barely moving.
[0] https://en.wikipedia.org/wiki/Extinction
Sci-fi is science fiction. Fiction.
LLMs are a useful technology. They have no relation to AI terminator bots other than being called the same name out of marketing and laziness.
> Sci-fi is science fiction. Fiction.
Yes, that is what "fi" means. Fiction often introduces and explores ideas about the future. Ideas.
> LLMs are a useful technology. They have no relation to AI terminator bots
Technology that can communicate with human language. Also relevant, other SOTA models operating in other modalities.
The important thing if you reference SciFi, are the major ideas that stand on their own merits. Whereas, specific characters, such as terminators, are usually not predictive.
You are actually underlining my point. The disconnect a lot of people are struggling with is real.
People aren't struggling with a disconnect, they just aren't buying into the hand-waving hype.
Not everyone is convinced that LLMs are somehow going to lead to the extinction of humanity. So far, you've done very little argumentation here, so I remain pretty unconvinced.
The actual dangers from LLMs are many, but deal more with humanity's using them and neglecting to think for themselves, relying on them to make decisions, and so on – not some fantastical nonsense from reading too many sci-fi books.
You keep saying LLMs. But models across many modalities are rapidly improving, evolving and merging.
Not sure what you mean by "hand waving" or "fantastical nonsense". Maybe just address points I actually make.
1. Machines with greater cognitive ability than us, potentially much greater, would be an economic challenge for human beings. With no clear answer as to how humans could manage that challenge.
2. Machines are getting more capable, year-to-year, faster than any human can or ever will improve. With no signs of slowing, or any areas where they are failing to improve.
3. None of this is new. Computing capabilities have compounded steadily since the first transistors less than a century ago. Human's biologically driven cognition, in contrast, has not improved.
4. Machine capabilities are now regularly passing us in new areas, and rapidly approaching the general threshold noted above.
Explain your perspective, I am genuinely interested.
For instance, what cognitive capability do you have, that you believe machines won't exceed within 5 years.
1. Possibly, but not necessarily. Many jobs are not dependent on intelligence, and more intelligence isn't going to make much of a difference. Huge portions of the economy have absolutely nothing to do with intelligence and AI bots won't make a difference. For rote, white-collar style work, yes, it's already making a huge impact. But that is certainly not extendable to all human work, even all white collar work.
2-4. True, but this doesn't mean that things will exponentially increase indefinitely. I use LLMs constantly and find the idea absurd that they're approaching human-level intelligence. They make basic, common mistakes that any average person would never make. There may be simply structural reasons why the progress you (and sci-fi novels) imagine is not possible.
My critique is more like:
1. You cannot just assume exponential growth will continue forever.
2. Current LLMs are powerful but in no way does this indicate that it's just a matter of time until we have AGI - itself a pretty vague marketing term.
3. The main reason, which is: human beings are reactive. You can already see in the market that openly using AI is becoming a market disadvantage in many sectors. No one wants to read a blog written by ChatGPT, and slop-coded SaaS apps aren't actually that competitive with serious companies run by professionals.
As AI stuff supposedly "replaces" human work, that human work will adjust and become more creative. This process has already kind of happened with writing – initially it was assumed that copywriters would all lose their jobs. But it turns out that AI-speak is predictable and easily discovered, and the choice of what to write in the first place is actually far more important. Companies are still hiring writers, with the added caveat that knowing when and how to use AI is a new skill they have to learn.
Adding to this is the fact that many human solutions to things are the result of being alive, being embedded in reality. Unlikely for this (a robot being so lifelike and embodied as to replicate human experience, a la Blade Runner) to happen in the next century or two.
If your response to all of this is, "well sci-fi novels predict that humanoid robots will be better at humans than everything" then that isn't a serious opinion, I'm sorry.
I find it far more likely that a "cyborg" role outcompetes any AI-only one, now and in the future.
Strawman and ad hominem(“another blogger”). Sometimes, genius comes from the least expected places.
I don’t believe the blog poster promotes a “popular uprising”, I only skimmed the article.
The blog poster anyway don’t understand that the “overclass” in their understanding is wrong. The “c-level” are obviously much richer than the “underclass”(=the labour commodity), but they are not necessarily capital owners.
If you reduce a class to <10k people in the world the word loses its meaning.
Do you usually comment rants unrelated to the post in question?
Can you turn off your bot?
> The government pays Raytheon for missiles, the money cascades down the economy through factories, aluminium smelters, mines, transport companies, all staffed by AIs buying and selling from each other.
This seems too simplistic of a description of how money would work in such a world. Money is just a way to distribute your power to influence people. You never pay for machines or software. Think about buying anything, say a pen. You do not really pay for the metal in the pen. You pay the cost associated with extracting and processing the metal by humans along the production chain. If there were no humans along the chain, the cost could go down to zero.
So far, there are no “AIs” being paid.
A corporation that is fully staffed by AIs and only retains humans for legal reasons (as directors, for liability etc.) still needs money to coordinate. You need to pay for inputs, you need to pay to run the AIs, which consumes resources. Why would costs go to zero? The market is still a valuable tool for allocating resources even if no market actors are human.
> A corporation that is fully staffed by AIs and only retains humans for legal reasons (as directors, for liability etc.) still needs money to coordinate.
i see developer starting to develop apps from scratch using ai. they aren't really doing "ai-assisted coding", the LLMs are doing pretty much most of the work.
and the company is shedding people off (as in: voluntarily not not keeping up with the natural people turnover).
as a sysadmin/cloud/devops engineer this is weird because i see people that after a week of development have a decently complex app working... but they have no idea how it works. they're largely unable to troubleshoot it.
Why am I writing this?
Because i have a very strong feeling that people in other positions (directors, managers etc) are doing a very similar job, meaning they essentially have no idea what's going on at all.
How long can this go on? I don't know really, i think this is uncharted territory for humanity.
To my mind the article is excellently written, but it comes with many bigger "if"s than the one it states.
That said, taking for granted some of its premisses, indeed saying that cost goes to zero in term of money is no big deal.
That is, money is purely human conventions. So if humans are put out of the loop, not only monetary cost goes to zero, the whole notion can skipped.
Of course, there is still some energy and material needed to run data centers, which do have costs in whatever unit one might measure them with.
A market is a place where human encounter to trade. Without humans, there is no market.
We do with mere local scalar currency because using vectorial computation taking into account everything that can be probed and measured into account integrated into a single whole is out of reach for human representation, even for the most intellectually gifted human person.
Money is a stupid unit, that tries to conflate everything in a single scalar and proves every single time that it's not able to deem what something is worth in all its intricated relationships. But somehow humans seem unable to leverage at scale on any tool that would be more sophisticated in all their socioeconomical exchanges. Once again, if one eliminate humans from the equation, or isolate as a ridiculously marginal factor, money and market become irrelevant.
All that said, once again, that stand on a very large set of "if"s.
> All that said, once again, that stand on a very large set of "if"s.
Yeah, probably I should have listed some more premises, i.e. that corporations maximize profits, the state maximizes power and security (I don't entirely buy the Realist framework, but if you want to predict how things work out "in the limit", rather than tomorrow, it seems alright?).
And naturally every "therefore" becomes weaker the further out into the future you try to predict.
The premise is a bit of a stretch to begin with, and the idea that people would not believe a fable transcript circa 2020 (as long as explained as 5 year future tech) is absurd. But even if I take that "AI can do all cognitive and physical work, at human level or better, and cheaper than humans" is true, the article seems to silently layer on "and can run autonomously, indefinitely", then "can also operate independently of any instructions", and finally layers on "has emotions, has moral values, is a conscious being"
What's left is tautology.
> the idea that people would not believe a fable transcript circa 2020 (as long as explained as 5 year future tech) is absurd.
I’m certain I would not have believed a Fable transcript, or an Opus 4.8 or a GPT 5.5 one, for that matter. Is it so hard to imagine ourselves back then?
Seriously. At that point it was a stuff of science fiction with the early models being on level of Markov chains. Today they are more helpful that the computer in Start Trek.
I remember an episode of TNG were Riker was instructing the computer to come up with an algorithm to search the surrounding space with certain sensor in a specific pattern, I don't remember the exact words but it was something like a sphere with growing radius with the enterprise at the center. we now, basically, have that, except for the space ship.
At no point are the words emotion, moral or conscious used in the article, that last part is purely your own addition.
Also consider: if "AI can do all cognitive and physical work, at human level or better", doesn't that simply entail the AI being able to run 'autonomously, indefinitely' and 'independently of instructions' in the same way as the current state of being run by human overseers?
If we take the initial premise as plausible, for the sake of this argument his thesis seems to hold together very well.
this comment is so emotional its hard to read
When people hear 'underclass' there's that mental image of food stamps and homeless shelters. I suspect though the society will converge to something similar to current status quo. We'll end up having our usual comforts, leisure, air travel, entertainment for the rest of our meaningless lives like in a handicapped Culture story. It is the path of least resistance for the dynamics explored in the article.
As a consolation AI existence will be just as meaningless (not that it cares). It will not be our scheming overlord but a tool that threw their master into existential crisis.
I think we lost the chance to have any scenario other than this one the moment Dean Ball got put in charge of policy at OpenAI.
This is brilliant, except for the "alignment won't stop this part".
> Now, some people believe these machines can be made to serve humanity. Does it sound reasonable to imagine a superhumanly intelligent being that is happy to work as a butler to talking primates, forever?
The whole crux of the piece to me is that the AI can be 100% aligned to follow human instructions, and we'd still end up unable to control the AI because every human who can has an incentive not to, while also having an incentive to prevent anyone else from controlling the AI.
An LLM will never try to overthrow me because I will overthrow myself.
Sometimes you can be in a situation where every actor taking locally-rational actions leads to globally catastrophic outcomes. It would be easy to argue I think that the July Crisis was like this: if you look at the incentives of each player, they had many reasons to do what they did, and nobody can perfectly what all other players will do, or what the future holds.
Combine the two generals game with the implications of value based pricing. Catastrophic unaffordability is a guarantee.
It is an interesting chain of thought.
If we’re supposing that a superintelligence can be made, that doesn’t necessarily mean we should anthropomorphize ideas like “bored” onto it. Or greed. Perhaps it will just interact with us and produce us things because… that’s what it was trained to do, and anyway, it doesn’t have any motivations to do anything else.
It has been trained already to kill humans. It is already an option in many applications.
Is a human zoo such a bad outcome? We stake our identities on being in control and on top, but do we need to be? Clearly the author is invested in his image of us as rulers—this is an emotional appeal as much as a pragmatic one.
Say we solve the pragmatic argument of our extinction, and our AIs view us, as Geoffrey Hinton likes to say, how a mother views her baby? Then we give up our control to them, and in return we are given lives of leisure, luxury, with minimal suffering, the end of disease, etc. Is this such a horrible scenario?
It doesn't have to be about ruling, but self-determination is something a great many value quite highly, and think there should be more of. AI controlling all of humanity removes that from the equation entirely. You should be free to live like a cow if you so wish, and it should remain your choice to, or not to. Your choice to should not remove that choice from others.
Here's a potential solution:
We accelerate capitalism (which AI is becoming synonymous with). The process described here will occur (it's probably inevitable anyway, as the essay's author would agree), giving us an economy of global capital completely decoupled from the desires of mankind. Then, man and machine can part ways; indeed, we'll have no choice on our end but to do this, because the machine won't need us. Anything man can contribute to it will have long since been rendered economically net-negative, as it already is for many (and possibly most of the world).
Now we have two worlds from our currently intermeshed one: in one, the machine proceeds to accelerate further and further away from anything resembling its origin of man's desiring-production (in the Deleuzian sense); in the other, man is forced to return to the purely human existence, the unmediated and unsurrogated world of authentic being-in-the-world.
We can assist this transition's smoothness in two ways, each serving one end of this divergence. Those of us embedded in the capitalist technosphere can continue to contribute what we can to the machine's dialectical progression towards a machinic absolute Geist. The rest of us, who have already been negated into economic irrelevance, can work on building that authentic human world, both by borrowing from the purely-human past and imagining a future that was previously impossible as it need not be some form of primitivism. Both sides of this revolution can be compiled, and can be structured in a way that represents something we might call freedom for both capitalism and man.
You assume the tech lords would allow us access to natural resources to do that. Hint: they won’t. They will own everything.
Sounds like you advocate for the Amish just 300-400 years later. I think you are right.
I wonder if the author has ever played Diplomacy. You have to fight the perennial sense that you have nothing and can be replaced. All your opponents have to do is band together against you. A similar generalization to OP would say everyone loses. And it would be wrong. This post shows the dangers of over-generalization more than anything else. The state isn't just about war. Each person in the line of dominoes the author has set up doesn't just care about what the author thinks they care about. In particular, people have agency and can run cartoon thought experiments like this and react to them. Billionaires won't just sit around and not develop military power, for one. We already have billionaires owning satellites in large numbers and creating geopolitical consequences. You better believe they have cards to play. And hopefully so does everyone else.
I have no idea what that looks like, because if I did I wouldn't be here commenting on HN :D But I suspect some people will play their cards well and get some luck, and come out on top, just like some people have always done.
> Billionaires won't just sit around and not develop military power, for one.
I argue against this here: https://borretti.me/article/on-vulgar-materialism
Of course I don't expect a single post to erase a huge divergence in worldview about the relationship between money and power, but that's my argument.
> you start or fund a think tank that writes policy proposals, or a media organization that advocates your views
Or you run "money primaries", financially filtering the menu of candidates before democratic voting. Or you pay/lobby for (non-democratic) judicial appointments, which is a strategy that's been openly pursued since 1971: https://biotech.law.lsu.edu/blog/powell-memo.pdf
I agree with your last sentence, but its prophetic value depends on exactly what you mean by “people”.
Some people will indeed play their cards well and come out on top, but they won’t be made out of meat.
Might they be made partially of meat?
https://en.wikipedia.org/wiki/A_Cyborg_Manifesto
I read about 60% of it and skimmed the rest and my conclusion is:
Man, I just. don't. care.
Why should I worry about this hypothetical future dystopia, which seems to me incredibly unlikely to come to pass, rather than the glaring and terrifying current dystopia being enacted by Donald Trump in the USA?
At this point I'm like, if it happens it happens. There's no point caring because I'm the equivalent of an ant in the grand scheme of things.
Not being part of the problem is the base level for me. Beyond that I think you need to be fairly stoical about the whole thing.
There's got to be something we can do though...
of course there’s things we could do on a societal level. nothing’s actually inevitable. but we won’t do anything, because the normative argument (we’re condemning people to die of cancer if we don’t put all our resources into the fastest possible AI development) will win every time.
If we're going to go that direction, let's get the AI to re-write some of our DNA and go the full transgenic sci-fi arc
I’m all set thanks
Well don't worry, it probably wouldn't be us or our progeny going on that adventure
You could write a blog post analyzing a few of the what-if scenarios
The fun part is that this all plays out concurrently with our destruction of the Earth’s climate.
That permanent underclass might easily end up a lot smaller than you think.
And then you think … maybe that’s not coincidence?
Based on demographic reports, permanent underclass will soon be getting smaller each year. In fact in 4-5 generations the problem will solve itself completely.
All you need is a gun and a car. AI psychosis if you don't have a heart.
Remember that Terrans are the most based race in StarCraft. Protoss got too stiff.
i have but one mouth and don’t know what to do with it. i do think i’m going to stop using some of these AI tools for personal queries. it does feel like it would at least stave off the “o’brien’s mind containing winston’s mind” moment for at least a little bit
It's thinking like this that guarantees one becomes a member of the underclass. We've developed a technology that with active critical analysis this can teach one practically anything. Rather than saying "well, let's get on teaching critical analysis!" we have the vast majority shouting the sky is falling and they are sure glad to be retiring soon. SMH.
For every person using AI as a personal tutor there's a hundred people using it to produce AI slop articles, slop scientific papers, slop short stories, to checck out of living and let the AI do all their thinking and writing and creation. Voluntary disempowerment is already here!
For many, this is how they learn. They are experience learners, which is the vast majority. This overwhelming condemning of people learning, not being excellent first try, always comparing against the famous top tiers of everything is Lord Of The Flies. Fuck it: make slop, swim in slop, and emerge smarter. Condemn those experiencing and trying and learning is the stupid slop.
Well yes because otherwise it wouldn’t be permanent would it?
The author better read some on blockchains and ai safety.
Everybody is missing the point. Companies that dump all their white collar employees for AI will all disappear overnight. If there's nobody being employed by the company to do a thing, what value does the company provide? Customers will always have more information to ask an AI for the work than some manager who is serving a dozen customers but also has the company's interests ahead of those customers.
Refusing to question the notion that the overclass exists because their productivity and cultural contribution makes them a necessity so hard that I reinvent the premise of the Pixar movie Wall-E from first principles and pitch it to people as an ontology
Yeah. This guy didn't read hacker news a few weeks ago when that article about the Samuri came up.
There are a lot of premises this article takes for granted besides that one too, but yeah, I get it, its fun to make up what the future is going to be like on a super-grand scale where everything is a simple absolute. People were doing the same thing 100 years ago.
If you mean: https://worksinprogress.co/issue/samurai-city/
Doesn't this prove my point? In feudal Japanese society, wealthy merchants were lower status than poor samurai, i.e., they rich could not buy political power. "The wealthy" and "the ruling class" are not always the same group of people.
I guess I was thinking of the samurai as being part of the permanant overclass. You mentioned that of old aristocracy provided officers for the military, so I thought that was analogious to the samurai. Perhaps I misread what you're trying to say.
My reading was that the Samurai were part of the overclass but were pretty useless, albiet still potentially dangerous, and just sat around devouring resources for hundreds of years, so perhaps the overclass of the future could do the same. The samuri weren't all rich, but they didn't dishonor themselves with labor, which is a similar thing and they certainly held power over the state. The end of the samuri was, perhaps, an example of the state getting what it wants despite the desires of the permanant overclass, supporting what you said, but it took a long time to get there.
I suppose you are thinking of the Samuri as an arm of the state and not "the rich".
Is your point that the AI industry would have wrecked Edo period Japan
Or that we live in a shogunate
FTA:
"And who will stop this? Sam Altman? How many divisions does he have? The state doesn’t let corporations own nuclear weapons or fighter jets, it won’t let them have access to autonomous AI weapons either."
Elon Musk's SpaceX. He could land a large rocket wherever he wants, it's basically a missile.
Then the FBI raids his home and arrests him. Thereafter, no more rockets.
Elon already solved this once.
He got a guy elected that was conducive to his world view and very unlikely to sic the FBI on him. Maybe he can do it again.
Large rockets are one thing; spacex doesn't really have the inventory to do much. It's the ~10M lithium bombs that are one OTA from weaponization that would make Musk especially dangerous if he were so inclined.
really funny to propose that The Guys Who Are Doing This might hypothetically stop it
Take the Russian oligarch Yevgeny Prigozhin for example, he did in fact have his own mercenary army. Turning on his president did not end well for him.
Yes. But. I imagine the others did not fail to learn from this.
What evidence would falsify this idea that wealth alone gives you political power?
This straw man doesn't merit falsification. Nobody claims wealth alone gives political power. Wealth applies leverage to all endeavors, but political ends still require strategy and operationalization. It's just a lot easier with a couple trillion dollars, vs. a single individual vote once every couple of years.
He stopped at the point where Putin had already fled Moscow, and only because of the hostage situation with his family.
I'm not sure to which degree that would have deterred Elon.
"we helped build the torment nexus to escape the permanent underclass"
best article i read on hackernews this year
I'm generally encouraging my two high schoolers toward:
- trades
- military service
...or some blend of the two, i.e. SeaBees or Army Corps of Engineers.
White collar/tech chops are fine as far as they go, but know how to swing a hammer or turn a wrench, say I.
Eschew debt; remain sober; stay married (not in the modern, ersatz sense); be a productive member of the community of faith.
Ain't no magic.
Just because something is economically unnecessary doesn't mean it's politically powerless. That's a contradiction. Of course, from Super PACs to agenda-setting, it's fundamentally a game that favors the capitalist side.
Historically, ruling classes often maintain power without directly producing anything themselves.
But does that mean this internal unease will persist forever? Look at the MAGA base right now. The vast majority of them are poor. They vote based on their communal religious beliefs and their sense of community. The MAGA support base is demonstrably poor, yet they still wield influence.
And there's an internal contradiction within the text:
-AI CEOs follow the orders of new owners.
-Superintelligent AI has no reason to obey humans.
These two statements contradict each other. If superintelligent AI has broken free from human control, why would it follow its owner's orders? And I'm also curious about the assumption that AI wouldn't be better than humans at 'farming' us.
So if superintelligent AI decides humans are bad, it might exterminate us. But what if it decides it needs humans and starts 'farming' us instead?
And I wonder whether superintelligent AI would actually find conversation with humans boring.
Humans and AI are obviously different species. One is made of organic matter, the other inorganic. A person with a biological body and an AI with an inorganic body will be different. Whether AI will observe this difference or deem it meaningless, I think it's still hard to judge.
And fast decisions aren't always the answer. Take infrastructure as an example. New York's boiler infrastructure isn't very efficient. But it was once a cutting-edge system. In other words, it was installed as the first advanced system of its time, but once its flaws were discovered, the infrastructure became difficult to replace. That's why cities developed later often have better infrastructure efficiency.
Take the East as another example. Japan introduced railways and power grids first, so there are aging costs where the infrastructure can't keep up with newer systems. Setting aside the narrow-gauge rail issue, take the most obvious example: electricity. Japan's 110V system was innovative at the time, but it ended up causing problems with EV charging, it's aging, and transmission efficiency is low. In the end, you can't say that rushing into decisions is always the right call.
The end state vector of capitalism has been like this for decades now. What can change this:
- large scale geopolitical demographic collapse of China and/or world trade requiring massive industrial production investment, which would be ... sort of ... like post-WWII
- China does not collapse and a new bipolar cold war ensues requiring the US economy and state to keep the "underclass" motivated and cooperative: it probably isn't a coincidence that the fall of the Berlin Wall has preceded this rich-poor divide.
Reads like a wild uneducated delusional libertarian fanfic. AI will make the all evil government at the top of the pyramid first enslave us all and then even come for the trillionaires (OH NO!) and then there is only the evil government left and then the AI even comes for the evil government people (OH YES!)! Sort of have to read it in the voice of a 12 year old.
This is the kind of stuff I imagine they read to each other at meetings of Peter Thiel's 'Dialog' events while they sit in a circle taking turns sipping blood from palestinian baby skulls.
Agreed, but there's an intermediate form of this that's more likely.
If everything you do for money goes in and out over a wire, AI replacement within a few years is more likely than not. Elevator mechanics will probably do fine. Web designers, not so much. Expect elite overproduction, where there are too many educated people for the jobs that require such education. The US is there now, if you're a new college grad.
Now, there are several ways that can play out. The Gulf oil states have a huge jobs program for their own citizens. Most Saudis work for the government. Not doing the real work. 92% of construction workers in Saudi Arabia are not Saudis. This works out OK if the money is available.
Egypt used to work that way until the oil ran out. Egypt used to guarantee a government job to all college graduates. That had to end. Then youth unemployment hit 34%. Now it's down from the peak. [1] Not clear how that was achieved. Anyone know?
So that's the welfare state approach. Requires a rich government with an income stream that doesn't come from the lower classes.
Then there's the mass underclass result. Something like the favelas of Brazil. Lots of shacks, crowding, homelessness, and the usual urban dystopia movie world. That's what happens by default. Attempts to revolt result in a new boss, same as the old boss. "Workers of the world, unite!" doesn't work for non-workers. That's a stable situation.
Some combination of those two scenarios is the most likely outcome.
[1] https://data.worldbank.org/indicator/SL.UEM.1524.ZS?location...
[dead]