> it’s that we may stop producing people who know enough to notice when AI is confidently wrong.
The running-joke is that a LinkedIn-lunatic AI booster, with a Nano Banana-generated profile-pic, will immediately slide into the chat to tell you that that this is already a solved problem: just spin up another agent to do the work to verify the first agent. Token-cost-be-damned. And we laugh and downvote them to oblivion and carry on with our day.
But today I had some exposure to a SotA agentic team coding loop thingie which had been running almost hands-off for a few weeks on a (pretty serious) Win32+Direct3D-to-Emscripten+WebGL porting project - and I'm genuinely spooked at how well it all works; I mention this example because all the agents' processes involved a decently rigorous verification step: any time any agent confidently asserts something then it has to provide associated evidence, such as a unit test report, or build artefact, or external citation, and the system will spawn a new agent (perhaps using a different backing LLM) to verify the claim. I know a unit-test pass/fail isn't quite the same thing as, say, a medical AI agent confidently wrong about me having/not-having terminal spleen cancer, but the capability for a team-of-agents to be self-checking is definitely there.
----
Also, the past 3 years of AI/LLM/etc developments have taught me to never cling to any shortcoming or weakness they have because plenty of them do seem to have been solved or mitigated, either directly or indirectly.
It does happen. But this is why the “ralph loop” works: over dozens (or hundreds) of iterations, eventually every regression, implicit assumption, fake passing test, is noticed and fixed by another agent. The code slowly but continuously converges to a better state. I’m surprised myself, but haven’t seen it fall into chaos or degradation so far.
How is the token-cost-be-damned part in the latter example?
I do find that both porting and translation projects have a much higher signal given the ease of mapping to tokens, when there is a proven working source to refer to - the source itself provides the validation. In a new project, you don’t have that validation.
> How is the token-cost-be-damned part in the latter example?
It's there, but...
1. The project owner figured out a way to minimize token usage for agent claim verification tasks.
2. Verification agents used older and much cheaper models, including local models for the most trivial things.
3. They could afford it anyway; but I think it's an inevitability that the token-cost for a task will approach some limit for some quality threshold - concurrent with the dollar-cost-per-token shrinking over time as better hardware comes out.
> In a new project, you don’t have that validation.
I'm still trying to understand that part of the project's history, actually. Obviously the HTML5+WebGL+Emscripten+Etc entrypoint was a "new" project; one of the first things they did was build their own means of verification, I just don't know how that part worked-out in practice (besides the agents dogpiling in on TODO.md).
AIs should be forced to show their work. Every tool they use, every program they generate and run in the background, and every logical inference. They should be forced to produce Lean or Rocq proofs or execution traces for all the computation they use. For facts, they should be able to produce sources. For any abstract reasoning, they should be able to break it down into explainable steps.
Then, on top of that, they should be able to explain any of that, at any level of detail, whether talking to an expert or a layperson.
That's an implementation preference, and you can have all of that today if you write your own harness. Vibe coding a cat meme generator doesn't need a Lean proof.
AIs aren't people. I can demand whatever I want from them.
The entire premise of building machines that can provide intelligence is that they can be arbitraryly rigorous and do far more work then is reasonable or practical to demand of a human.
> AIs should be forced to show their work. Every tool they use, every program they generate and run in the background, and every logical inference.
And they should be 100% deterministic: same input (including a random seed / "temperature" if one wants), same output.
On a not totally dissimilar note I like the projects saying: "AI PRs aren't accepted but if you do something worthwhile, give us your prompts instead".
Extrapolating to other fields, we can say this development is the worst what one can do to mankind and society: pushing for AI but at the same time decimating the human capital that helped produce and understand it. What is the point of producing countless mathematical proofs or code that one cannot verify nor understand? It’s like having a machine to produce endless things that no one needs or will use. Just because the machine can bring profits for the (initial) investors?! That is the rottenest form of capitalism imaginable.
It sounds ‘the singularity is near’ not so much because AI is reaching escape velocity forward, but because we’re systematically pushing humans back beyond the threshold where computers are legible?
You both seem to be using a different definition of "singularity" from the one I'm familiar with. I've always understood it to mean a rapid feedback loop in which AI creates successive, increasingly capable generations of AI outside human control, rather than simply a level of technological advancement that would be incomprehensible to someone
The reason the term "singularity" was used when that term was coined was in analogy to a black hole, which has a point on its radius called the event horizon where the gravitational pull is strong enough to stop light from escaping, thereby making it impossible to see beyond that horizon. "Technological singularity" itself here refers to the technological event which causes this, like a black hole is sometimes explained as a gravitational singularity
This is important to the framing of the idea: "the technological singularity" refers to a point in the future past which further developments can't be meaningfully foreseen or maybe even understood. This is often associated with particular paths people think will cause this, but not defined by those. As such, the prior comments are making the claim that you can get there not just by accelerating technological progress, but also by reducing the ability of everyone to comprehend what's happening, which I agree is unfortunately a plausible outcome in the current world
The singularity is when an AI can improve itself, which among the results include humans no longer being able to understand its code and exponential improvements so we can't catch up either.
It requires a level of self-driven intent LLMs aren't capable of on their own.
I'm not exactly sure how you would go about grading mathematical proficiency. I went through calculus two and discrete mathematics, but I'm sure that there are things I have forgotten now even though I would be considered familiar with most leading edge AI technology. If I'm being honest, I'm not sure I could pass the final exams I took to get my CS degree right now.
“The product of mathematics is clarity and understanding. Not theorems, by themselves. [Their importance is not just in their specific statements], but their role in challenging our understanding, presenting challenges that led to mathematical developments that increased our understanding.
The world does not suffer from an oversupply of clarity and understanding (to put it mildly)… In short, mathematics only exists in a living community of mathematicians that spreads understanding and breaths life into ideas both old and new. The real satisfaction from mathematics is in learning from others and sharing with others. All of us have clear understanding of a few things and murky concepts of many more. There is no way to run out of ideas in need of clarification. The question of who is the first person to ever set foot on some square meter of land is really secondary. Revolutionary change does matter, but revolutions are few, and they are not self-sustaining --- they depend very heavily on the community of mathematicians.”
Curious to see if that can map to what's happening in the software industry/community.
> The product of software engineering (or computer science) is clarity and understanding. Not programs, by themselves. Their importance is not just in their specific statements (lines of code in a specific language), but their role in challenging our understanding, presenting challenges that led to computational (?) developments that increased our understanding.
> ..In short, software only exists in a living community of developers that spreads understanding and breaths life into ideas both old and new. The real satisfaction from computers is in learning from others and sharing with others.
That seems to work. What about other areas of human activity that are currently being consumed by automation and "AI"? Like writing, the arts, or the sciences.
Indeed, both Thurston's quote about mathematics, and Naur's programming as theory building, are classics that are relevant now more than ever. A download link for the latter: https://gwern.net/doc/cs/algorithm/1985-naur.pdf (PDF)
I see many people reacting with fear towards IA and many of them do not feel the same level of danger in other places which are clear to me (and I think that what they fear most is the unknown, as it has to be bad, for sure). I like the quote of the other message from Whitehead !!
I think the differentiation going forward is going to be if you're a person that understands things or if you're a person that delegates your understanding. I don't think there's going to be a lot of economic or social value to prompting an AI to do things you don't understand. First, anyone can do it, so why should someone give you money or credibility for that? Second, as fewer and fewer people bother to learn hard things, the scarcity of that knowledge will increase its value.
I guess even if I end up being wrong about that, I'd rather enrich my life and grow as a person by continuing to learn and do hard things rather than become an annoying cheerleader dependent on unreliable tools.
Uh, unless you're living with your well-off parents in perpetuity or you already have FU money... good luck with that. I'm pretty sure my landlord isn't going to accept "money doesn't matter anymore because of AI!"
Also do you really think billionaires are frothing at the mouth for this because it's going to make money less important?
I think the billionaire folks are pathological or opportunists, nothing really more.
As for your landlord, I doubt that will be a problem either.
You cannot make 40% - 80% of the country structurally unemployable with AI and eventually robots and keep this same economic system.
It will be weird, it’s not going to be hugely fun, but I’m betting that we either get massive deflation, huge changes in land use policy, revolution, or a combination of them all.
Personally I’m building a shack in the woods and building my own business. We will see how it turns out, but I doubt money as we know it survives this.
Edit to be clear:
I think it’s going to work out and probably be pretty cool after we deal with obvious problems that are coming up.
Things get sticky fast the longer you think on this problem and the fact leadership has always held little interest in equal distribution of the planet's spare resources vs crony corruption and nepotism. Hard to imagine a future that doesn't look like mad max outside the billionaire compounds protected by loitering robots and drones along with space based technology immune to any rebellion.
That "we" is suspicious, I suspect it doesn't adequately cover the whole civilization in question.
Sure, me and my neighbors waiting for the train don't need to build a sundial, we can tell the time "without thinking" by glancing at our ticking pocketwatches... but other people spent absolutely enormous amounts of brainpower thinking about meshing gear, friction in bearings, metal alloys that don't distort too much with temperature changes, and how to make accurate tiny screws, and...
“It is the first step in sociological wisdom, to recognize that the major advances in civilization are processes which all but wreck the societies in which they occur:—like unto an arrow in the hand of a child. The art of free society consists first in the maintenance of the symbolic code; and secondly in fearlessness of revision, to secure that the code serves those purposes which satisfy an enlightened reason. Those societies which cannot combine reverence to their symbols with freedom of revision, must ultimately decay either from anarchy, or from the slow atrophy of a life stifled by useless shadows.” A. N. Whitehead
Yeah how many of us know how to build an ICE engine or shoes of any meaningful quality
We set upon end of human craftsmanship decades ago
Math is probably the easiest to reclaim given its right in front our faces going about daily life. The syntax of math is not that important; real world quantification the syntax is meant to represent will still exist. Our biochemistry implicitly operates on senses of enough food and water, etc.
Such measures are so embedded in the daily routines we live an intuition will always exist
No one is born knowing how to make a computer as we know them today. A cup half filled is obvious
What worries me isn’t AI replacing experts, it’s that we may stop producing people who know enough to notice when AI is confidently wrong.
> it’s that we may stop producing people who know enough to notice when AI is confidently wrong.
The running-joke is that a LinkedIn-lunatic AI booster, with a Nano Banana-generated profile-pic, will immediately slide into the chat to tell you that that this is already a solved problem: just spin up another agent to do the work to verify the first agent. Token-cost-be-damned. And we laugh and downvote them to oblivion and carry on with our day.
But today I had some exposure to a SotA agentic team coding loop thingie which had been running almost hands-off for a few weeks on a (pretty serious) Win32+Direct3D-to-Emscripten+WebGL porting project - and I'm genuinely spooked at how well it all works; I mention this example because all the agents' processes involved a decently rigorous verification step: any time any agent confidently asserts something then it has to provide associated evidence, such as a unit test report, or build artefact, or external citation, and the system will spawn a new agent (perhaps using a different backing LLM) to verify the claim. I know a unit-test pass/fail isn't quite the same thing as, say, a medical AI agent confidently wrong about me having/not-having terminal spleen cancer, but the capability for a team-of-agents to be self-checking is definitely there.
----
Also, the past 3 years of AI/LLM/etc developments have taught me to never cling to any shortcoming or weakness they have because plenty of them do seem to have been solved or mitigated, either directly or indirectly.
> any time any agent confidently asserts something then it has to provide associated evidence
And this is enforced by... another LLM? Seems like it would work alright until something is asserted implicitly and not categorized as an assertion.
It does happen. But this is why the “ralph loop” works: over dozens (or hundreds) of iterations, eventually every regression, implicit assumption, fake passing test, is noticed and fixed by another agent. The code slowly but continuously converges to a better state. I’m surprised myself, but haven’t seen it fall into chaos or degradation so far.
How is the token-cost-be-damned part in the latter example?
I do find that both porting and translation projects have a much higher signal given the ease of mapping to tokens, when there is a proven working source to refer to - the source itself provides the validation. In a new project, you don’t have that validation.
> How is the token-cost-be-damned part in the latter example?
It's there, but...
1. The project owner figured out a way to minimize token usage for agent claim verification tasks.
2. Verification agents used older and much cheaper models, including local models for the most trivial things.
3. They could afford it anyway; but I think it's an inevitability that the token-cost for a task will approach some limit for some quality threshold - concurrent with the dollar-cost-per-token shrinking over time as better hardware comes out.
> In a new project, you don’t have that validation.
I'm still trying to understand that part of the project's history, actually. Obviously the HTML5+WebGL+Emscripten+Etc entrypoint was a "new" project; one of the first things they did was build their own means of verification, I just don't know how that part worked-out in practice (besides the agents dogpiling in on TODO.md).
It's agents all the way down~!
"Determined" -> "Probabilistic"
AIs should be forced to show their work. Every tool they use, every program they generate and run in the background, and every logical inference. They should be forced to produce Lean or Rocq proofs or execution traces for all the computation they use. For facts, they should be able to produce sources. For any abstract reasoning, they should be able to break it down into explainable steps.
Then, on top of that, they should be able to explain any of that, at any level of detail, whether talking to an expert or a layperson.
That's an implementation preference, and you can have all of that today if you write your own harness. Vibe coding a cat meme generator doesn't need a Lean proof.
Conversely, https://www.lesswrong.com/posts/fzeoYhKoYPR3tDYFT/beware-iso...
AIs aren't people. I can demand whatever I want from them.
The entire premise of building machines that can provide intelligence is that they can be arbitraryly rigorous and do far more work then is reasonable or practical to demand of a human.
> AIs should be forced to show their work. Every tool they use, every program they generate and run in the background, and every logical inference.
And they should be 100% deterministic: same input (including a random seed / "temperature" if one wants), same output.
On a not totally dissimilar note I like the projects saying: "AI PRs aren't accepted but if you do something worthwhile, give us your prompts instead".
why?
Extrapolating to other fields, we can say this development is the worst what one can do to mankind and society: pushing for AI but at the same time decimating the human capital that helped produce and understand it. What is the point of producing countless mathematical proofs or code that one cannot verify nor understand? It’s like having a machine to produce endless things that no one needs or will use. Just because the machine can bring profits for the (initial) investors?! That is the rottenest form of capitalism imaginable.
It sounds ‘the singularity is near’ not so much because AI is reaching escape velocity forward, but because we’re systematically pushing humans back beyond the threshold where computers are legible?
I think about this a lot. From the POV of a medieval peasant, we likely reached the singularity a long time ago.
You both seem to be using a different definition of "singularity" from the one I'm familiar with. I've always understood it to mean a rapid feedback loop in which AI creates successive, increasingly capable generations of AI outside human control, rather than simply a level of technological advancement that would be incomprehensible to someone
The reason the term "singularity" was used when that term was coined was in analogy to a black hole, which has a point on its radius called the event horizon where the gravitational pull is strong enough to stop light from escaping, thereby making it impossible to see beyond that horizon. "Technological singularity" itself here refers to the technological event which causes this, like a black hole is sometimes explained as a gravitational singularity
This is important to the framing of the idea: "the technological singularity" refers to a point in the future past which further developments can't be meaningfully foreseen or maybe even understood. This is often associated with particular paths people think will cause this, but not defined by those. As such, the prior comments are making the claim that you can get there not just by accelerating technological progress, but also by reducing the ability of everyone to comprehend what's happening, which I agree is unfortunately a plausible outcome in the current world
The singularity is when an AI can improve itself, which among the results include humans no longer being able to understand its code and exponential improvements so we can't catch up either.
It requires a level of self-driven intent LLMs aren't capable of on their own.
I'm not exactly sure how you would go about grading mathematical proficiency. I went through calculus two and discrete mathematics, but I'm sure that there are things I have forgotten now even though I would be considered familiar with most leading edge AI technology. If I'm being honest, I'm not sure I could pass the final exams I took to get my CS degree right now.
See also Bill Thurston’s classic Math Overflow answer to a student wondering where they fit compared to a Gauss or Euler:
https://mathoverflow.net/questions/43690/whats-a-mathematici...
“The product of mathematics is clarity and understanding. Not theorems, by themselves. [Their importance is not just in their specific statements], but their role in challenging our understanding, presenting challenges that led to mathematical developments that increased our understanding.
The world does not suffer from an oversupply of clarity and understanding (to put it mildly)… In short, mathematics only exists in a living community of mathematicians that spreads understanding and breaths life into ideas both old and new. The real satisfaction from mathematics is in learning from others and sharing with others. All of us have clear understanding of a few things and murky concepts of many more. There is no way to run out of ideas in need of clarification. The question of who is the first person to ever set foot on some square meter of land is really secondary. Revolutionary change does matter, but revolutions are few, and they are not self-sustaining --- they depend very heavily on the community of mathematicians.”
Curious to see if that can map to what's happening in the software industry/community.
> The product of software engineering (or computer science) is clarity and understanding. Not programs, by themselves. Their importance is not just in their specific statements (lines of code in a specific language), but their role in challenging our understanding, presenting challenges that led to computational (?) developments that increased our understanding.
> ..In short, software only exists in a living community of developers that spreads understanding and breaths life into ideas both old and new. The real satisfaction from computers is in learning from others and sharing with others.
That seems to work. What about other areas of human activity that are currently being consumed by automation and "AI"? Like writing, the arts, or the sciences.
Peter Naur wrote an essay about exactly that https://www.sciencedirect.com/science/article/abs/pii/016560...
Indeed, both Thurston's quote about mathematics, and Naur's programming as theory building, are classics that are relevant now more than ever. A download link for the latter: https://gwern.net/doc/cs/algorithm/1985-naur.pdf (PDF)
> "the essay makes the case for treating mathematical capacity as a strategic asset on a par with semiconductor capability."
In other words, the mathematicians want more funding.
And the Chinese are still going to kick our ass.
Ancaps are in for a major cope and seethe when they are forced to admit that the communists won the cold war.
I see many people reacting with fear towards IA and many of them do not feel the same level of danger in other places which are clear to me (and I think that what they fear most is the unknown, as it has to be bad, for sure). I like the quote of the other message from Whitehead !!
Let's enjoy the ride. It might be last one!
Learning and understanding is enjoying the ride.
I think the differentiation going forward is going to be if you're a person that understands things or if you're a person that delegates your understanding. I don't think there's going to be a lot of economic or social value to prompting an AI to do things you don't understand. First, anyone can do it, so why should someone give you money or credibility for that? Second, as fewer and fewer people bother to learn hard things, the scarcity of that knowledge will increase its value.
I guess even if I end up being wrong about that, I'd rather enrich my life and grow as a person by continuing to learn and do hard things rather than become an annoying cheerleader dependent on unreliable tools.
lol, there aren’t going to be people giving money to other people in the long run.
“Money is the sign of a poor civilization” or something similar that a Culture GCU would say.
Uh, unless you're living with your well-off parents in perpetuity or you already have FU money... good luck with that. I'm pretty sure my landlord isn't going to accept "money doesn't matter anymore because of AI!"
Also do you really think billionaires are frothing at the mouth for this because it's going to make money less important?
I think the billionaire folks are pathological or opportunists, nothing really more.
As for your landlord, I doubt that will be a problem either.
You cannot make 40% - 80% of the country structurally unemployable with AI and eventually robots and keep this same economic system.
It will be weird, it’s not going to be hugely fun, but I’m betting that we either get massive deflation, huge changes in land use policy, revolution, or a combination of them all.
Personally I’m building a shack in the woods and building my own business. We will see how it turns out, but I doubt money as we know it survives this.
Edit to be clear:
I think it’s going to work out and probably be pretty cool after we deal with obvious problems that are coming up.
Things get sticky fast the longer you think on this problem and the fact leadership has always held little interest in equal distribution of the planet's spare resources vs crony corruption and nepotism. Hard to imagine a future that doesn't look like mad max outside the billionaire compounds protected by loitering robots and drones along with space based technology immune to any rebellion.
"Civilization advances by extending the number of important operations which we can perform without thinking about them." - A. N. Whitehead
Yes, now we can do thinking without thinking. Good job.
Computers can not think & that is why they are useful. Thinking computers would become very problematic very quickly.
but people thinking that computers and specifically AI are thinking and therefore their answers are thought through is equally problematic i think.
That "we" is suspicious, I suspect it doesn't adequately cover the whole civilization in question.
Sure, me and my neighbors waiting for the train don't need to build a sundial, we can tell the time "without thinking" by glancing at our ticking pocketwatches... but other people spent absolutely enormous amounts of brainpower thinking about meshing gear, friction in bearings, metal alloys that don't distort too much with temperature changes, and how to make accurate tiny screws, and...
“It is the first step in sociological wisdom, to recognize that the major advances in civilization are processes which all but wreck the societies in which they occur:—like unto an arrow in the hand of a child. The art of free society consists first in the maintenance of the symbolic code; and secondly in fearlessness of revision, to secure that the code serves those purposes which satisfy an enlightened reason. Those societies which cannot combine reverence to their symbols with freedom of revision, must ultimately decay either from anarchy, or from the slow atrophy of a life stifled by useless shadows.” A. N. Whitehead
Key word: perform, i.e. execute. Scale indeed comes by performing more things per unit time, things we understand, on execution engines we understand.
Yeah how many of us know how to build an ICE engine or shoes of any meaningful quality
We set upon end of human craftsmanship decades ago
Math is probably the easiest to reclaim given its right in front our faces going about daily life. The syntax of math is not that important; real world quantification the syntax is meant to represent will still exist. Our biochemistry implicitly operates on senses of enough food and water, etc.
Such measures are so embedded in the daily routines we live an intuition will always exist
No one is born knowing how to make a computer as we know them today. A cup half filled is obvious