The most successful applications like coding are not the result of pure LLM/generative modeling. They come from closing the loop with an agentic harness. The generate-test-selectively refine loop is the core modality of scientific work. An LLM + RL with Verifiable Rewards + feedback from compiler/terminal runs mimics this process to a great extend.
This is Fisher/Box feedback loop (https://www-sop.inria.fr/members/Ian.Jermyn/philosophy/writi...) implemented on a modern computational system. LLM is just a component. I wish Sutton had commented on this fuller picture of what we have now instead of commenting just on the LLM/Backprop side of things. I am honestly curious of whether such a loop can at least partially automate discovery.
There are more elements to discovery though. It is still not clear where the initial working model/hypothesis comes from or how the updates are selected (unless it is just parameter induction). I recently read about Hanson's Patterns of Discovery which aims in that direction. I have still not read it, but I am curious if it has any mechanistic clues.
> There are more elements to discovery though. It is still not clear where the initial working model/hypothesis comes from or how the updates are selected
That is a problem in RL, so we usually do supervised training first, teach it to imitate some trajectories, then do RL to refine the model. RL alone has a huge problem because it might be hard to reach a reward, hence hard to learn the task by pure reinforcement. Humans also combine supervision (learn from books) with search (solving problems) to break the discovery problem. For example, a human with no initial instruction in math would not produce great results no matter how smart they are. The bootstrap was exploration paid for in the past.
SFT + RL connection to model/hypothesis search is insightful. Brute force / scalable search is where Sutton's Bitter Lesson also points to. Once your search domain is small compared to your search budget, that makes a lot of sense.
If I get your meaning right, SFT creates the right inductive bias so that the RL search + reward guidance does the trick.
For novel discovery, the question might then be whether the inductive bias builds a strong enough prison so no new discovery is possible by RL or if the search can escape the boundaries set by SFT given enough randomization and the right reward function.
I know that RL is usually not performed at inference time, but in-context learning mechanisms might be developed by RL to discover at test time. Edit: I would love to hear if that actually happens or not, like new induction heads (https://transformer-circuits.pub/2022/in-context-learning-an...) forming during RL. I really have no idea.
the role of evolution is always a confounding factor as well and all the various analogies to how it maps onto AI research are always not quite satisfactory.
Completely agree on the importance of the harness.
The problem I see is the same problem Evolutionary Algorithms had: you can generate potential solutions until you run out of cash, but you still need to evalulate those solutions. You need a fitness function, and that means you need to at least know the general shape of the solution. If anyone knows of any work towards more open-ended fitness functions, I'd love to read it.
Just some speculation, but, I think humans have on the one hand a lot of degrees of freedom in behaviors and thoughts they can do, but at the same time all that freedom is reigned in by our biological needs, like preserving the integrity of our body, but also preserve the integrity of our minds. But this extends further to preserving our surroundings (for our safety, a changing environment brings uncertainty), but also of people we care about and even entire societies that we have. And preserving our future selves through prediction of future environments.
So all that is to say, I'm not sure it is even theoretically possible to create a single algorithm to do open ended search and evaluation. Biology has billions of years of evolution and accumulation, whereas a simple algorithm in a computer, even if smart and connected to the real world, has no such accumulation.
I think humans hit the perfect sweet spot where we have the simplicity of the self preservation instinct, but we have the complexity of the cortex and lots of degrees of freedom because of it, plus on top of that we have a lot of accumulated degrees of freedom in the society and technology and knowledge that have we, which has been built up for thousands of years, all of which we can't just create an algorithm to encapsulate without going through the actual evolution.
And just to make it explicit - a large percentage of what humans think derives from an instinct to preserve the self, the mind, the future and the environment, even if it is very abstract at times. Not absolutely all, but I think a good chunk. And the complexity and degrees of freedom comes from that we have so many neurons in the brain, and a complex body with hands and whatever else that allows a lot of behaviors, as well as a complex environment that is constantly challenging us.
> If anyone knows of any work towards more open-ended fitness functions, I'd love to read it.
There is research in open-ended learning, see "Why Greatness Cannot Be Planned" by Kenneth O. Stanley. The core idea is that in open-ended scenarios you don't know what action was good except in hindsight because your path is deceptive. So the idea is to replace fitness with novelty search which provides more stepping stones towards the goal.
Seems to a layperson like myself that in Math they're using Lean and in programming contexts they're using compilers, such that the models themselves tend towards embedding that determinism "intuitively".
Yes it seems most anti-LLM researchers take issue with LLMs on fundamental math/architecture based properties, but seem to miss all the engineering going on around the model to make it useful.
Those mathematical shortcomings very well might mean they arent a path to true AGI, but that honestly seems fairly irrelevant at this point tbh.
> The generate-test-selectively refine loop is the core modality of scientific work.
Can you expand on this? Because when I think of it, it calls to mind p-hacking and selective publishing more than anything.
Like, when you try prompts over and over and over until you get code that finally works. At which point you stop and pronounce the dubious claim, "AI is amazing, look what it does!"
Most importantly, the reinforcement loop is used during training. I don't agree with Sutton's original hypothesis, but it holds even less after reinforcement learning.
RLVR still does not expand beyond the base distribution though, it only mode-seeks within it.
i.e, evaluation, retention yes. variation or "planning" no.
That is not to say you cannot use LLMs. Alpha evolve does exactly that. It uses an external simple evolutionary planner. The overarching point he's making is that our planner is still "dumb" and we need to work on it.
When you iteratively guide an LLM in claude code, you are the external planner. That also works.
> That is, I would say that creativity requires that the new things generated be Evaluated. Without evaluation, and retention of the best, there is nothing created. The novelty flickers into existence but, if its value is unrecognized, it flickers away and is lost.
I really like the way he frames this here. I think a lot of people in the twitter comments (and maybe a few here) aren't reading past the introduction. He isn't saying AI systems are incapable of creativity and discovery. He is claiming generative AI without a harness is not capable of creativity and discovery. There needs to be some other system that "recognizes the value" of the novel idea and remembers it. He gives examples of where this value recognition step is automated and thus by his definition achieve creativity and discovery in a fully automated system.
I agree in general. But making this claim with a narrowed definition of generative AI that doesn't match what is being used in practice is setting up a strawman.
Unless I'm missing something, this argument seems to apply only to the original pretraining era (eg GPT 1-4). The post-training and reinforcement learning paradigms are clearly doing variation, evaluation and selective retention no?
RLVR still does not expand beyond the base distribution though, it only mode-seeks within it.
i.e, evaluation, retention yes. variation or "planning" no.
That is not to say you cannot use LLMs. Alpha evolve does exactly that. It uses an external simple evolutionary planner though. The overarching point he's making is that our planner is still "dumb" and we need to work on it.
When you iteratively guide an LLM in claude code, you are the external planner. That also works.
> RLVR still does not expand beyond the base distribution though, it only mode-seeks within it.
Seems clearly false. Pretraining finds the mean/mode of the data distribution. RL can easily generate many samples around that mode, evaluate them on an external source of truth (eg compile the code and run it) and then selectively train on the good samples. This clearly can go beyond the initial data distribution.
The model’s distribution will certainly change from the base model’s output distribution during reinforcement learning, shifting toward outputs that score well on an external evaluation. This is very different from mode-seeking. Am I missing something?
Mode-seeking is describing the way in which the distribution changes. RL is capable of picking out slightly lower probability trajectories and moving them toward the top of the distribution. However, exploration is fundamentally limited by the base policy itself. If a trajectory has near-zero probability under the original model, RLVR is unlikely to discover it because it must first be sampled before it can be rewarded. External search/planning methods such as MCTS or evolutionary search are useful precisely because they can explore candidate trajectories beyond what the policy would ordinarily generate. This is also not theoretical, GRPO style methods are shown to mostly improve `maj@k` and `pass@1` evals while not so much `pass@k` especially for high k, meaning it mostly sharpening the top of the distribution.
I'm not saying this makes it useless - it clearly helps for math and coding tasks. But the ceiling exists and that's what the original tweet was referring to. Alpha evolve also shows what lies beyond the ceiling, altho their planner was rudimentary.
Sure, but I'd say that moving desirable trajectories from very low probability to high probability is characteristic of genuine human learning and discovery.
Technically, quantum gravity, a bestselling novel, or a yet undiscovered proof of the Riemann Hypothesis is "in my distribution", but when we are talking about a long chain of unlikely token completions (with multiplicative probabilities), whether that trajectory lives in the tail of the distribution vs. in the mode makes all the difference.
Would you agree that it is a matter of degrees rather than a qualitative distinction? There seems to be a broad misconception in Sutton and others that output quality cannot exceed that of the base internet distribution; my point is that RL allows you to easily produce an output distribution that is better than whatever data you trained on according to some evaluation criteria. There are no clear theoretical limits on how much better it can get, rather there are many people asserting guesses that there is an upper bound and it lives below "human creativity." I just haven't seen any solid theoretical argument, and the empirical evidence has so far shown continual improvement.
Also, I would be keen to look at any sources you have of pass@k not improving much during GRPO.
I said slightly lower, I meant it. It's virtually impossible to sample a trajectory that is really really low probability (say, by smoothening the distribution before sampling) without incurring crazy amounts of noise. And only when you sample it, can you reward it and do the update.
Again, no one is saying models can't improve beyond the internet i.e data distribution! They clearly can. The claim is that RL without real exploration cannot exceed the base models distribution, which by virtue of SGD _does_ generalize.
And also, it doesn't mean it's not useful. Improving sample efficiency and making something that happens 1 in 15 times happen 1 in 1.2 times is insanely useful and is what has enabled the kind of coding agents we have today.
Sutton, especially, I doubt has a misconception about this :)
> pass@k
Yeah, AFK now. But it's a well researched thing. You can look for more, but here's one off the top of my head: https://openreview.net/forum?id=4OsgYD7em5 The original deepseek paper also had the result, i.e the paper that first got famous for using grpo as a method that works for LLMs. A side result in one of these papers I forget which one, is that the base model converges in performance with the RLd one at high k.
Thanks, I appreciate the discussion. The paper you sent is interesting. I agree it looks like for moderate values of K (on the order of 100-1000), RL models actually look a little worse at pass@k than their base models.
So perhaps the right framing of your/Sutton's claim is: RL can upweight low-probability (p) but correct outputs, but there is a limit to how small p can be, and it is on the order of 1 in a 100 or 1 in a 1000. Implicitly there must be some crossover point where you would call this discovery/creativity if it works for sufficiently small p right? Eg if RL can upweight a correct but 1 in a trillion output to 1 in 5, that's got to count as discovery given that all possible sequences are technically "in the distribution"?
In practice, it does seem like that kind of progress is happening. For example with the recent Erdos solution [0], I would wager that GPT 4's hit rate on this would have been functionally 0 (certainly less than 1 in a thousand). Curious to hear whether you'd still say this is mode-seeking within a base distribution, or if not then what is the right explanation if not iterative RL.
I'd also highlight that the paper you linked with the pass@k equivalence doesn't technically address the question of how small p can be before RL upweighting breaks down - all of the example problems were easy enough that the base model had decent hit rate with 128 tries.
I'm absolutely uninterested in the semantic discussions of what is a real discovery, what is creativity, what is intelligence, etc. I simply don't care. If it's useful great use it. If it's not great don't.
> How small p can be
All that depends on your sampling procedure. If you intentionally smooth the distribution out you can sample the smallest thing, but you pay for it with noise. Taken to an extreme, this is the monkeys typing on the keyboard argument.
It's a mathematical fact that RL cannot improve things it doesn't sample. In any learned distribution you pay a heavy cost by sampling far away from the mode. Most RL algos sample rollouts maybe with some smoothing but that's it. This is why external planners are necessary in order to sample something effectively un-sampleable in the base distribution. Simple example: tool use!
Sutton and everyone are simply calling for a focus on improving these external planners in the same way, as they also enable much better "continual" learning and so on.
> Erdos solution
The RL was what enabled such a huge trajectory to ever become efficiently sampleable in our lifetimes probably. You can do many useful things like this and more purely with the base model distribution.
In fact. Doing RL on user chats and so on especially from pair coding sessions are improving these models coding abilities by a lot making them even more reliable for SWE. In this regard, mode-seeking is a win.
> All sequences are technically in distribution
If it was truly improving 1 in
million things systemically, then you wouldn't see base getting the same results given many samples. Albeit they are not erdos problems.
Could it be that at 1T scale, and for difficult problems specifically, grpo somehow filters through the noise and picks out the 1 in trillion? Extremely unlikely (you have your expected rollouts required to sample that, and then you have your sparse reward signal and no credit assignment on top of that...). But of course, only 2 companies in the world can do experiments with it, so there could be some unknown effect the rest of the world has not seen. Barring that, no.
The transcript does seem to overlook post-training steps like Reinforcement Learning with Verifiable Rewards (RLVR) (but I'll certainly won't claim that Rich Sutton is unaware of such things; RLVR has a very narrow set of evaluation approaches).
I wonder if this is a precursor to Keen Tech leaning into David Silver's Ineffable Intelligence approach.
This was exactly what I was thinking of. RLVR is the secret sauce behind o3 and its many successors.
Its the secret sauce behind why the current models are so great at coding and soon to be unbeatable at math.
LLMs can pose many questions and if they are easily verifiable, fine tune very heavily. A lot of the world models discussion will inevitable lean into simulations as verification.
I'll admit that I miss having access to the ChatGPT 4.5 "absolutely gigantic model" with enough tuning to make it sane and useful. The RLVR models are superb for actual tasks in those RLVR domains, but that fine tuned view of the world as a verifiable problem to solve makes them feel worse for touchy feely stuff. Even for medical consultation and diagnosis, RLVR model's urge to reach a conclusion often is a liability.
It's RLVR tuned, but not to the ChatGPT level of brain damage, and it's still backed by a fuck off huge pool of model weights - which matters for what you call "touchy feely stuff".
There seems to be a problem with how he poses the problems alphaGo and these GAI models face.
AlphaGO is given a hard evaluation externally. It did not itself come up with it.
When GAI models are given an external hard evaluation, they can also succeed in many different domains (that is one of the remarkable features, succeeding in many domains) ranging from simple programming tasks to frontier mathematics (disproving conjectures recently) to writing more optimized kernel code than before.
And there is plenty of RL especially in these fields where the solution may be extremely complex but eval is rather less complex. And even the discovery and the "evolution-like" trace-selection is also happening.
For this reason it seems strange to compare it to AlphaGO as alphago is given a hard eval independent of itself, from an external source (humans) in a narrow domain. If GAI is given such, it can also show some remarkable results.
But what I find more strange is that innovation and moving forward in many many many cases does not require truly novel ideas but instead a high-quality execution of layering different methods, tactics, ideas on top of each other. Because in many domains our collective knowledge is incredibly sparse and complex, something being able to recombine tools, models, ideas in a high quality way (as he mentions being selective) I think is extraordinarily powerful.
And in such cases, with a finite exploration horizon (time, resource available) with 1% "good choices" vs 3% "good choices" are worlds apart, incomparable.
Most importantly: none of the above is about intelligence, it's barren solution-farming to important, valuable problems we have. Most of the AGI and intelligence-related debate seems to miss out on this simple fact. (Insert the usual stuff like a plane being unable to fly like a bird or a submarine not swimming is totally irrelevant to it being useful).
And then a final point: do we really think this thing is incapable of doing better on average on problems we average people face in our lifetime?
What should we think, how should we define human intelligence when we give out degrees in science or medicine for 60-70% exam results on problems considered to be generic in the field?
You are right in that. Propulsion is different in both cases. The point is, until you have not recreated the function, difficult to say which part is important in other implementations (~Marr levels).
I don't quite follow his point. Is it: a) that we need a new foundational algorithm that integrates a goal (one with "taste") directly into the training step, or b) that we need to point trained models towards goals as they iterate?
If it's a), he doesn't propose such an algorithm, and I don't know how you'd do it at such a low level because how do you quantify abstract goals? Did he suggest such an algorithm and I misread? If it's b), that already exists, see AlphaEvolve or any number of things he said. Or, to be a bit of a smart-ass, just type /goal and let it rip ...
I also think he's just categorically wrong that LLMs cannot do good and novel things. And if it can, then you could just say "well that's not novel, that's derivative". A simple example, if I make up a programming language with an LLM and it works well for my purposes, then is that not novel and good? I mean, is any language other than FORTRAN not novel?
Everything is derivative and you can put an LLM in a loop to evaluate LLMs trying things. I must be misunderstanding because he's too smart to be this wrong.
LLMs possess the map but are unable to discern fertile from barren ground. For instance: how does Anthropic's new model generate promising 'medications'? Because, beyond the knowledge embedded within the model, it has assimilated AlphaFold's reasoning paradigm. By itself, Claude would be incapable of engineering a protein analysis method
I think a lot of deep learning is compositional generalization. Models learn reusable pieces (abstractions, styles, procedures, constraints, etc) and recombine them in ways that may never have appeared as a whole in the training data. So even if the ingredients come from past data the final composition can still be novel in a meaningful sense
I don’t think ML can’t be creative or make discoveries. I think creativity and discovery are, ultimately, simultaneously thinking about the right seemingly-disparate concepts (whereas algorithmic thinking is more obviously related concepts). If not an LLM, some other model can generate random ideas, rank them, then output the best.
But I think humans are better at it, while ML is better at algorithmic thinking. “Better” being more efficient and something we more enjoy doing; we can also more accurately rank what subjectively appeals to humans (i.e. taste), especially ourselves.
I think ML should be optimized for tasks that require more generalization than programming, but are still mostly logic. Like software development, translation, and tools for art and discovery.
One has to be very specific when throwing around words like "creative" when talking about A.I
Can A.I create art. Well it can create something that's pleasing to our senses but art is ultimately about conveying human feelings and emotions. Even as humans, understanding art is not universal. "feelings and emotions" and therefore art, can be deeply tied to a particular groups shared beliefs and experiences.
Can it be creative in non-subjective fields such as math or sciences. Einstein derived GR from his creative thought experiments. If an A.I poped out GR's field equations simply by testing different mathematical frameworks that resolve the issues discovered by experiments, is that creative? Perhaps but certainly not in the same way.
You can create art with everything from sticks and mud to glass and air. Of course you can make art with AI.
Now if the question is, can a machine make art, well ultimately someone needed to turn the machine on and design the machine to make art, so arguably that person/people are the ones making the art.
Historically, every question of "is x art" ends up having the answer "yes". I don't know why people fall for the same thing over and over.
> art is ultimately about conveying human feelings and emotions
you made a small error, art is mostly about generating an emotion in the viewer/listener/.... not about transmitting an emotion of the creator
the Wikipedia page on art starts with:
> Art is a diverse range of cultural activity centered around works utilizing creative or imaginative talents, which are expected to evoke a worthwhile experience
Current models are trained on image pastiche and style remixing. But there's no reason you couldn't add an Artistic Director layer which has been trained on emotional and cultural signifiers and to direct the pastiche and remixing.
The practical problem is that models have very limited prompt adherence. The level of detail you can specify in scene design is very crude. So you can get the slop effect where there's a lot of in-fill pastiche detail, but you could never create something like this, where all of the incidental objects are specifically included to enforce the message.
It's basically the professional version of the "Draw me a pelican on a bicycle" problem.
There are situations where you want that level of creative control, and current image generators don't get close to it.
And without it you can't get to the meta-creativity level where you're creating a new aesthetic that's a cultural landmark - which is what the famous artists did, and still do.
>But there's no reason you couldn't add an Artistic Director layer which has been trained on emotional and cultural signifiers and to direct the pastiche and remixing.
I gave this approach a shot over the first few months of this year[1] (although my director didn't have any custom training). The results were interesting, but I'd not call them "art", since they're low-quality derivative pieces. With reasoning traces enabled, you can see that there's not much intent going on. Though they do attempt to include "incidental objects" to reinforce meaning, like in this jungle scene[2].
Recent image models are advancing rapidly at prompt adherence specifically, and being able to iterate on the same image is propelling them even further. Images 2.0 being the poster child of this "agentic iterative image composition" approach.
Images 2.0 isn't anywhere close to the kind of detail control I'm talking about.
It's the opposite of a skill issue. No image generator is anywhere near the ballpark of pro-level manual Photoshop or Illustrator editing for individual elements in an image.
If you don't understand this, try precisely kerning the text in a generated book cover to handle letter combinations like A and V.
This is one of the big problems with GenAI. You can do new things with it, but it's crude Dunning Kruger good-enough-if-you-don't-ask-for-more creativity.
The pros can see what most people can't, and the flaws and missing features are frustrating and obvious creatively, not just in terms of production values.
We went from "AI can't generate text that isn't at least 20% typos and it always looks like shit" to "some letter combinations aren't kerned to perfection sometimes and adjusting that with prompts is hard". In a couple of generations.
You might be missing that those rare humans were sitting on tons of failed or somewhat useful discoveries made by more “mediocre” humans that history forgot.
I'm trying to keep an open mind and understand what the author is trying to say because he is credentialed.
His main point is that discoveries involve
1. Variation,
2. Evaluation, and
3. Selective retention.
He makes a jump saying AI is only capable of 1) and humans are capable of 1) 2) and 3). I don't know what makes humans special enough that they can do 2) and 3)?
In fact, the more you think of this it is kind of strange - in science humans can only do "evaluation" because they have access to the real world. They can evaluate a new drug because they can do it on people so it is not some inherent limitation of AI but rather access to physical realm.
Finally I want to ask a specific thing: how do you mathematically falsify what this person is saying? How can you formally prove that - no AI can not "evaluate"? I ask because I make AI evaluate a lot of people's claims and it works for me.
He's saying that pre-training an LLM alone can't do it, but if you run an LLM in a loop with tools (like any coding agent) then it can. Also, the technique his group came up with should be used more:
> This is the weakness of deep learning that is alleviated with a new algorithm that my group presented in Nature a couple of years ago. Our “continual backpropagation” made one small change: every so often a less-used neuron would be re-initialized to small random weights. This allows the variation to continue and plasticity to be retained.
Sorry this makes no sense — humans also use tools to evaluate their discoveries.
Kant said something like this: knowledge can’t be obtained by pure thinking, it needs interaction with the world.
This is obvious to me so why is the author making a claim that LLMs can make knowledge without access to environment but purely through thinking in aether
He actually says the areas in which AI has had the novel successes are those which can be evaluated (like coding or Go). Not that it can’t happen at all.
That’s my point, he says ai does well where evaluation is neurosymbolically closed.
But so do humans? How do humans make discoveries without having formal ways to evaluate? In my pharma drug example, humans could evaluate only because they had access to the physical realm.
I can’t think of an example of humans evaluating a discovery in a way that LLMs can’t. can you?
I don't think there is any "humans are metaphysically superior to LLMs" subtext to this talk, it's just a technical/educational observation.
Access to some forms of evaluation and selective retention is inherent to humans and it's not inherent to LLMS. But it can be somehow bolted on and that's when they work best. It makes sense that more focus on those principles can yield better AI. I think the retention part is the real limitation of LLMs, because it's limited to stuffing things in context window.
> Access to some forms of evaluation and selective retention is inherent to humans and it's not inherent to LLMS
I'm not sure I understood - what forms of evaluation is inherent to humans? If you don't give humans tools or access to the physical world, how can they evaluate?
So technically the only reason AI can't do discovery is access to physical word. When you give AI and humans access, they both do discoveries - that is the clean summary of the author's position.
Its not too interesting.. we already know that giving AI access to compilers and tools make them better.
Humm maybe. But a plain model sampling outputs obviously isn't doing discovery in the AlphaGo sense. But once you put the model in a loop with tests, feedback, tools or even a human picking the good result, it starts to get much closer to the process he's describing.
It's funny to me that one would rate their own takes as "new and possibly controversial". Whatever comes next is read under that light of an author that thinks this about their own thoughts.
And the core point is not even true. They can definitely output novel things that are good - less so but they can and they do. Plenty of examples.
> Thus, the trajectory is either novel or good—based on randomness or based on data—but never both at the same time.
This assumes no possible unexplored path yields good results, or said another way, that none of the random results can be good, which is not true. The whole text seems to try to prove a point decided a-priori rather than make a case based on reality.
Yes, the guy with a PhD in Machine Intelligence, co-author of Reinforcement Learning: An Introduction, which is universally considered the bible of the field, recipient of the AAAI fellowship award and the Turing Award, and the inventor of Temporal Difference Learning doesn't know what he's talking about.
Sure, but does that mean he's right all the time about all things, including everything in his own field?
He is saying no generative AI is going to produce output that is both good and novel because it is always derivative. And then adds a generative AI (Claude Code) into his list of AI that have produced output that he feels is good and novel, invalidating what he is arguing.
"...no matter how many instances of white swans we may have observed, this does not justify the conclusion that all swans are white."
If you read it he says that CC has additional aspects beyond ordinary GAI, namely the ability to verify. That aspect is necessary for GAI to be good and novel.
Although personally I think code doesn’t actually need to be very novel so it’s actually the best example.
“When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.”
I don't completely disagree but its worth noting how new a lot of the empirical evidence in favour of LLMs are, so its not impossible to be a tad ignorant of the present
You know it’s occurred to me recently that there really is no value in this age to any kind of professionalism or seniority
If you don’t agree with somebody, nothing else matters
It’s like people (you as an example) have taken the concept of experts and fucked it up so bad that simultaneously everybody thinks they’re an expert while also dismissing everybody else who claims they are an expert
It’s like the whole concept is entirely poisoned. Worse everyone is smugly pointing at the Wikipedia for “appeal to authority.”
Nothing new I suppose, Socrates after all was driven to suicide by the madness of his society accusing him of impiety.
Surprisingly enough, Turing Award winner and father of reinforcement learning Richard Sutton knows perfectly well what he's talking about. The whole talk is about the need to have the ability to test novel outputs against reality and iterate to find ones that are good. This is exactly what Claude Code, the agent framework, adds to Claude, the LLM, to allow it to find novel coding solutions that actually work.
> When we ask for a fiction or novelty, the AI can give it to us because its processing is in part stochastic. Every decision can go multiple ways and will go different ways and produce a different trajectory every time. The trajectory can be random—and thus novel—or it can be based on the training data—and thus “good” because the training data is good, sourced from people or reality. Thus, the trajectory is either novel or good—based on randomness or based on data—but never both at the same time.
This doesn't seem true? You can be both random and based on training data.
I think they meant more "it can be extrapolated or interpolated" or "it can be high variance and 'creative' or it can be low variance and 'reliable/correct/likely'". If you want to see something new, the model will need to step off the manifold. But the manifold is where you've learned the "correct" solutions live.
I think that creativity / intelligence / humour is indissociable from "making errors".
A joke is just an "error" - your brain predicted something, and the butt of the joke goes in another direction, and it's the mismatch that makes it funny.
The same goes with creativity, and intelligence.
The problem is that, by design, while trying to make machines "reliable", we make it impossible for them to be intelligent and creative
In addition to the importance of making "mistakes", I would say "surprise" is a big element in creativity and humor. Perhaps these are related concepts, because a mistake is surprising. It's an unexpected departure from the normal routine and habit of behavior or thinking.
The term "surprisal" is used in information theory:
> For a given probability space, the measurement of rarer events are intuitively more "surprising", and yield more information content than more "common" events.
Can a machine surprise us? Given enough complexity, I think so. They can produce unpredictable results, even novelty, something we've never seen before. But does that mean a machine can be creative? Or funny? Maybe there's a threshold of acceptance, where eventually its output will become surprising enough that we might as well call it creative.
He seems to be saying that Claude Code can make discoveries. Does anyone think novel discoveries can be made from systems created by supervised learning only, and attempting to do so?
> Claude-Code, which have brought true advances in ... programming. ... these systems have found things that are both novel and good.
I think its worth emphasizing that his argument isn't completely against generative ai, but rather its environment. Although I don't see why it would be impossible for something like an LLM to learn some sort of self-play within its context window
I think there is a rich seam of work for computers to help humans spot when there are anomalies in the hypotheses humans hold that can help humans form the corrections to hypothesis that fit our aesthetic sensibilities
I think the variation, evaluation, and selection idea is a good, if not the only, way do do creative work.
I don't think I would attribute anything in that process that I would consider an AI to be incapable of.
The characterisation of variation like this would seem to rest on the same 'random but directed' crutch that some free will arguments rest upon.
There is no random but directed of course, there is random and there is caused, and there are things that use both as components, but the random remains wholly random, and the caused remains entirely deterministic.
I think there is a good case to say that, in many fields, AI is better than humans at evaluation.
To find avenues to consider, I'm not entirely convinced that human innovation is more than a heuristic that appears more chaotic by virtue of a inconsistent and opaque formulation.
Many aspects of ideas com from noting how some two things are different and then considering that axis of difference when applied to another thing.
The possibilities thrown up by this extremely simple method are vast enough to require multiple layers of evaluation, most could be dismissed out of hand by a quick 'This is nonsense' check that I suspect people do so often and at a rate that it wouldn't even rise to the level of consciousness.
I enjoyed reading this at the start, the language is very... inspiring. By the end, I was disappointed. I don't disagree with what they're saying, but the opening style and statements made me expect some more specific or groundbreaking conclusions.
The point seems to be that generative AI just generates stuff, and that real discovery requires variation, evaluation and selective retention.
The call to arms seems based on the assumption that people only every talk about generative AI as discovery machines themselves. I think it's pretty widely accepted that's not the case by everyone apart from cliche out-of-touch CEOs.
But the talk makes me realise that generative AI are incredible tools to do the discovery cycle with, and this is what I imagine professionally successful AI users are doing: variation, evaluation and selective retention of their inputs and outputs to generative AI.
"So that is my call to arms. If we want the full power of AI scientists, then we should share the goals with them so they can create, evaluate, discover, and in these ways fully participate in achieving the goals. Let’s be bold! Let’s fully automate Creativity and Discovery!"
Should we automate exercise and play as well? How about learning?
The machine didn't have a soul, so we donated ours.
I understand the skepticism. I am worried about the implications of AI as well. The deeper issue at stake is that the depth of scientific knowledge has been increasing for a very long time. Now you get to have a PhD in esoteric subproblems and that slows down research especially if the discoveries require depth in multiple subdomains. Socially and economically training people in every combinatorial combination of subfields at the required depth may not be possible. I am especially interested in two problems to be resolved and do not care if an AI scientist performed the discovery. It will be humbling, but totally worth it:
- Fusion (a clean sustainable form): Without this I think we are heading in a very wrong direction, whether it is conflict or climate change does not matter. Everyone is aware of this and instinctively afraid of the implied loss of quality+quantity of life.
- Cure for Cancer: It is a world wonder even in Civ. I and for good reason. As a father of a teenager, every time I hear a story of someone losing a parent/child I cringe. We have to accept this as a reality of life until a proper/generic cure is found that eliminates the most common offenders.
I am skeptical that we will have AGI anytime soon and I think the social aspects will help balance the technical developments even it becomes a reality (Three laws, A Butlerian uprising, you name it).
Chess bots can beat grandmasters, but I have a friend who takes his son to tournaments. Humans are still playing chess, kids in the same tournament with grand masters. We have to have faith in the humanity, or all else will not matter.
And I will definitely keep playing Factorio even if AGI comes to pass ;-)
Commercial fusion energy is almost certainly a pipe dream. We already have good alternatives. Short of some unpredictable major breakthrough in physics on the level of something like cold fusion, the current activity in the fusion space will basically just serve to demonstrate definitively that it's not viable.
We also have competitive energy wasting (bitcoin), that will translate into competitive water deleting which will greatly accelerate how quickly we burn through all of it.
Eh, it seems increasingly clear that no matter the fusion breakthroughs to come, it's going to be complicated and big and therefore expensive compared to renewables. By the time we figure it out it'll be a nice breakthrough but will end up on the shelf.
They do say "live a little", and it strikes me now that perhaps I've completely misinterpreted that phrase. For some reason I've always thought it to be a call to live a lot, or at least more.
But now I see that my interpretation is almost opposite of what the words mean... Huh...
Most discovery takes a ton of iteration and repetition. That's a lot less fun than the hedonism of sitting on a beach drinking a mahi thai.
Same with learning, humans historically where generalists without that deep of knowledge when compared to one another. Now we study a quarter of our short lives just to get to the point where we can specialize for the rest of our lives. This situation doesn't seem exactly tenable as complexity increases in the future.
I don't know what the future looks like, but I can tell you that entropy and complexity will only increase.
Another famous dude dumping his thoughts on HN who is gulping it up like an addict.
Add this to the long list of names like Terence Tao, and others who seem to be intellectually incontinent lately in the sense that one cannot navigate this space anymore without encountering their thoughts
Creativity is personal taste and the taste which captures the most is mainstream.
I tell jokes and the group of friends get them, for family they don't get them anymore.
I do not like 'basic jokes' and despite that, german television is full of it.
Most things in our world are more of a challange of finding the answers and not 'creating' the answers.
Math: the answer is already there, you only need to find it an fast space of posibilities. This is perfect for LLMs.
Creativity: a LLM can iterate over things a lot faster than a person. So we can iterate over this space too. We can also get feedback from people, tiktok, instagram.
Can you please make your substantive points thoughtfully and without snark? I'm sure there is one here, but it's hard to make out what it is, and in any case the poison does more harm than the information does good.
I'd link to the HN guidelines here but I'm on my phone!
The ad hoc heuristics are the domain knowledge baked into the model by human experts, like features, architecture and loss function.
"Evaluation" means environments or datasets, the model is supposed to discover its representations from scaled up experience. That was the bitter lesson - more data and compute beat heuristics.
Stochastic: describes a process, model, or system that involves inherent randomness, meaning its future states or outcomes cannot be predicted with absolute certainty but can be described using probability distributions.
That reminds me of a quote by Freeman Dyson on Stephen Wolfram's work.
> "There's a tradition of scientists approaching senility to come up with grand, improbable theories. Wolfram is unusual in that he's doing this in his forties."
I always felt it was an unfair dismissal of someone's life's work. Maybe it was true but it didn't enrich the discussion or our understanding. I suppose it means even a respected thinker can be guilty of shallow dismissals and saying hurtful things in public about others.
It's similar to a "thought-terminating cliché", in that it just reinforces an existing opinion without adding anything, making us think deeper, or furthering the conversation.
The most successful applications like coding are not the result of pure LLM/generative modeling. They come from closing the loop with an agentic harness. The generate-test-selectively refine loop is the core modality of scientific work. An LLM + RL with Verifiable Rewards + feedback from compiler/terminal runs mimics this process to a great extend.
This is Fisher/Box feedback loop (https://www-sop.inria.fr/members/Ian.Jermyn/philosophy/writi...) implemented on a modern computational system. LLM is just a component. I wish Sutton had commented on this fuller picture of what we have now instead of commenting just on the LLM/Backprop side of things. I am honestly curious of whether such a loop can at least partially automate discovery.
There are more elements to discovery though. It is still not clear where the initial working model/hypothesis comes from or how the updates are selected (unless it is just parameter induction). I recently read about Hanson's Patterns of Discovery which aims in that direction. I have still not read it, but I am curious if it has any mechanistic clues.
> There are more elements to discovery though. It is still not clear where the initial working model/hypothesis comes from or how the updates are selected
That is a problem in RL, so we usually do supervised training first, teach it to imitate some trajectories, then do RL to refine the model. RL alone has a huge problem because it might be hard to reach a reward, hence hard to learn the task by pure reinforcement. Humans also combine supervision (learn from books) with search (solving problems) to break the discovery problem. For example, a human with no initial instruction in math would not produce great results no matter how smart they are. The bootstrap was exploration paid for in the past.
SFT + RL connection to model/hypothesis search is insightful. Brute force / scalable search is where Sutton's Bitter Lesson also points to. Once your search domain is small compared to your search budget, that makes a lot of sense.
If I get your meaning right, SFT creates the right inductive bias so that the RL search + reward guidance does the trick.
For novel discovery, the question might then be whether the inductive bias builds a strong enough prison so no new discovery is possible by RL or if the search can escape the boundaries set by SFT given enough randomization and the right reward function.
I know that RL is usually not performed at inference time, but in-context learning mechanisms might be developed by RL to discover at test time. Edit: I would love to hear if that actually happens or not, like new induction heads (https://transformer-circuits.pub/2022/in-context-learning-an...) forming during RL. I really have no idea.
the role of evolution is always a confounding factor as well and all the various analogies to how it maps onto AI research are always not quite satisfactory.
Completely agree on the importance of the harness.
The problem I see is the same problem Evolutionary Algorithms had: you can generate potential solutions until you run out of cash, but you still need to evalulate those solutions. You need a fitness function, and that means you need to at least know the general shape of the solution. If anyone knows of any work towards more open-ended fitness functions, I'd love to read it.
Just some speculation, but, I think humans have on the one hand a lot of degrees of freedom in behaviors and thoughts they can do, but at the same time all that freedom is reigned in by our biological needs, like preserving the integrity of our body, but also preserve the integrity of our minds. But this extends further to preserving our surroundings (for our safety, a changing environment brings uncertainty), but also of people we care about and even entire societies that we have. And preserving our future selves through prediction of future environments.
So all that is to say, I'm not sure it is even theoretically possible to create a single algorithm to do open ended search and evaluation. Biology has billions of years of evolution and accumulation, whereas a simple algorithm in a computer, even if smart and connected to the real world, has no such accumulation.
I think humans hit the perfect sweet spot where we have the simplicity of the self preservation instinct, but we have the complexity of the cortex and lots of degrees of freedom because of it, plus on top of that we have a lot of accumulated degrees of freedom in the society and technology and knowledge that have we, which has been built up for thousands of years, all of which we can't just create an algorithm to encapsulate without going through the actual evolution.
And just to make it explicit - a large percentage of what humans think derives from an instinct to preserve the self, the mind, the future and the environment, even if it is very abstract at times. Not absolutely all, but I think a good chunk. And the complexity and degrees of freedom comes from that we have so many neurons in the brain, and a complex body with hands and whatever else that allows a lot of behaviors, as well as a complex environment that is constantly challenging us.
> If anyone knows of any work towards more open-ended fitness functions, I'd love to read it.
There is research in open-ended learning, see "Why Greatness Cannot Be Planned" by Kenneth O. Stanley. The core idea is that in open-ended scenarios you don't know what action was good except in hindsight because your path is deceptive. So the idea is to replace fitness with novelty search which provides more stepping stones towards the goal.
The TRM architecture models both the problem and the solution at the same time. You might find it an interesting read.
https://arxiv.org/abs/2510.04871
Seems to a layperson like myself that in Math they're using Lean and in programming contexts they're using compilers, such that the models themselves tend towards embedding that determinism "intuitively".
Yes it seems most anti-LLM researchers take issue with LLMs on fundamental math/architecture based properties, but seem to miss all the engineering going on around the model to make it useful.
Those mathematical shortcomings very well might mean they arent a path to true AGI, but that honestly seems fairly irrelevant at this point tbh.
> The generate-test-selectively refine loop is the core modality of scientific work.
Can you expand on this? Because when I think of it, it calls to mind p-hacking and selective publishing more than anything.
Like, when you try prompts over and over and over until you get code that finally works. At which point you stop and pronounce the dubious claim, "AI is amazing, look what it does!"
Most importantly, the reinforcement loop is used during training. I don't agree with Sutton's original hypothesis, but it holds even less after reinforcement learning.
RLVR still does not expand beyond the base distribution though, it only mode-seeks within it.
i.e, evaluation, retention yes. variation or "planning" no.
That is not to say you cannot use LLMs. Alpha evolve does exactly that. It uses an external simple evolutionary planner. The overarching point he's making is that our planner is still "dumb" and we need to work on it.
When you iteratively guide an LLM in claude code, you are the external planner. That also works.
> That is, I would say that creativity requires that the new things generated be Evaluated. Without evaluation, and retention of the best, there is nothing created. The novelty flickers into existence but, if its value is unrecognized, it flickers away and is lost.
I really like the way he frames this here. I think a lot of people in the twitter comments (and maybe a few here) aren't reading past the introduction. He isn't saying AI systems are incapable of creativity and discovery. He is claiming generative AI without a harness is not capable of creativity and discovery. There needs to be some other system that "recognizes the value" of the novel idea and remembers it. He gives examples of where this value recognition step is automated and thus by his definition achieve creativity and discovery in a fully automated system.
I agree in general. But making this claim with a narrowed definition of generative AI that doesn't match what is being used in practice is setting up a strawman.
Unless I'm missing something, this argument seems to apply only to the original pretraining era (eg GPT 1-4). The post-training and reinforcement learning paradigms are clearly doing variation, evaluation and selective retention no?
RLVR still does not expand beyond the base distribution though, it only mode-seeks within it.
i.e, evaluation, retention yes. variation or "planning" no.
That is not to say you cannot use LLMs. Alpha evolve does exactly that. It uses an external simple evolutionary planner though. The overarching point he's making is that our planner is still "dumb" and we need to work on it.
When you iteratively guide an LLM in claude code, you are the external planner. That also works.
> RLVR still does not expand beyond the base distribution though, it only mode-seeks within it.
Seems clearly false. Pretraining finds the mean/mode of the data distribution. RL can easily generate many samples around that mode, evaluate them on an external source of truth (eg compile the code and run it) and then selectively train on the good samples. This clearly can go beyond the initial data distribution.
by base distribution, I meant the base model's output distribution
The model’s distribution will certainly change from the base model’s output distribution during reinforcement learning, shifting toward outputs that score well on an external evaluation. This is very different from mode-seeking. Am I missing something?
Mode-seeking is describing the way in which the distribution changes. RL is capable of picking out slightly lower probability trajectories and moving them toward the top of the distribution. However, exploration is fundamentally limited by the base policy itself. If a trajectory has near-zero probability under the original model, RLVR is unlikely to discover it because it must first be sampled before it can be rewarded. External search/planning methods such as MCTS or evolutionary search are useful precisely because they can explore candidate trajectories beyond what the policy would ordinarily generate. This is also not theoretical, GRPO style methods are shown to mostly improve `maj@k` and `pass@1` evals while not so much `pass@k` especially for high k, meaning it mostly sharpening the top of the distribution.
I'm not saying this makes it useless - it clearly helps for math and coding tasks. But the ceiling exists and that's what the original tweet was referring to. Alpha evolve also shows what lies beyond the ceiling, altho their planner was rudimentary.
Sure, but I'd say that moving desirable trajectories from very low probability to high probability is characteristic of genuine human learning and discovery. Technically, quantum gravity, a bestselling novel, or a yet undiscovered proof of the Riemann Hypothesis is "in my distribution", but when we are talking about a long chain of unlikely token completions (with multiplicative probabilities), whether that trajectory lives in the tail of the distribution vs. in the mode makes all the difference.
Would you agree that it is a matter of degrees rather than a qualitative distinction? There seems to be a broad misconception in Sutton and others that output quality cannot exceed that of the base internet distribution; my point is that RL allows you to easily produce an output distribution that is better than whatever data you trained on according to some evaluation criteria. There are no clear theoretical limits on how much better it can get, rather there are many people asserting guesses that there is an upper bound and it lives below "human creativity." I just haven't seen any solid theoretical argument, and the empirical evidence has so far shown continual improvement.
Also, I would be keen to look at any sources you have of pass@k not improving much during GRPO.
I said slightly lower, I meant it. It's virtually impossible to sample a trajectory that is really really low probability (say, by smoothening the distribution before sampling) without incurring crazy amounts of noise. And only when you sample it, can you reward it and do the update.
Again, no one is saying models can't improve beyond the internet i.e data distribution! They clearly can. The claim is that RL without real exploration cannot exceed the base models distribution, which by virtue of SGD _does_ generalize.
And also, it doesn't mean it's not useful. Improving sample efficiency and making something that happens 1 in 15 times happen 1 in 1.2 times is insanely useful and is what has enabled the kind of coding agents we have today.
Sutton, especially, I doubt has a misconception about this :)
> pass@k
Yeah, AFK now. But it's a well researched thing. You can look for more, but here's one off the top of my head: https://openreview.net/forum?id=4OsgYD7em5 The original deepseek paper also had the result, i.e the paper that first got famous for using grpo as a method that works for LLMs. A side result in one of these papers I forget which one, is that the base model converges in performance with the RLd one at high k.
Thanks, I appreciate the discussion. The paper you sent is interesting. I agree it looks like for moderate values of K (on the order of 100-1000), RL models actually look a little worse at pass@k than their base models.
So perhaps the right framing of your/Sutton's claim is: RL can upweight low-probability (p) but correct outputs, but there is a limit to how small p can be, and it is on the order of 1 in a 100 or 1 in a 1000. Implicitly there must be some crossover point where you would call this discovery/creativity if it works for sufficiently small p right? Eg if RL can upweight a correct but 1 in a trillion output to 1 in 5, that's got to count as discovery given that all possible sequences are technically "in the distribution"?
In practice, it does seem like that kind of progress is happening. For example with the recent Erdos solution [0], I would wager that GPT 4's hit rate on this would have been functionally 0 (certainly less than 1 in a thousand). Curious to hear whether you'd still say this is mode-seeking within a base distribution, or if not then what is the right explanation if not iterative RL.
I'd also highlight that the paper you linked with the pass@k equivalence doesn't technically address the question of how small p can be before RL upweighting breaks down - all of the example problems were easy enough that the base model had decent hit rate with 128 tries.
[0] https://openai.com/index/model-disproves-discrete-geometry-c...
Pass@128 is a lot. They were not easy.
> Discovery / creativity
I'm absolutely uninterested in the semantic discussions of what is a real discovery, what is creativity, what is intelligence, etc. I simply don't care. If it's useful great use it. If it's not great don't.
> How small p can be
All that depends on your sampling procedure. If you intentionally smooth the distribution out you can sample the smallest thing, but you pay for it with noise. Taken to an extreme, this is the monkeys typing on the keyboard argument.
It's a mathematical fact that RL cannot improve things it doesn't sample. In any learned distribution you pay a heavy cost by sampling far away from the mode. Most RL algos sample rollouts maybe with some smoothing but that's it. This is why external planners are necessary in order to sample something effectively un-sampleable in the base distribution. Simple example: tool use!
Sutton and everyone are simply calling for a focus on improving these external planners in the same way, as they also enable much better "continual" learning and so on.
> Erdos solution
The RL was what enabled such a huge trajectory to ever become efficiently sampleable in our lifetimes probably. You can do many useful things like this and more purely with the base model distribution.
In fact. Doing RL on user chats and so on especially from pair coding sessions are improving these models coding abilities by a lot making them even more reliable for SWE. In this regard, mode-seeking is a win.
> All sequences are technically in distribution
If it was truly improving 1 in million things systemically, then you wouldn't see base getting the same results given many samples. Albeit they are not erdos problems.
Could it be that at 1T scale, and for difficult problems specifically, grpo somehow filters through the noise and picks out the 1 in trillion? Extremely unlikely (you have your expected rollouts required to sample that, and then you have your sparse reward signal and no credit assignment on top of that...). But of course, only 2 companies in the world can do experiments with it, so there could be some unknown effect the rest of the world has not seen. Barring that, no.
The transcript does seem to overlook post-training steps like Reinforcement Learning with Verifiable Rewards (RLVR) (but I'll certainly won't claim that Rich Sutton is unaware of such things; RLVR has a very narrow set of evaluation approaches).
I wonder if this is a precursor to Keen Tech leaning into David Silver's Ineffable Intelligence approach.
This was exactly what I was thinking of. RLVR is the secret sauce behind o3 and its many successors.
Its the secret sauce behind why the current models are so great at coding and soon to be unbeatable at math.
LLMs can pose many questions and if they are easily verifiable, fine tune very heavily. A lot of the world models discussion will inevitable lean into simulations as verification.
I'll admit that I miss having access to the ChatGPT 4.5 "absolutely gigantic model" with enough tuning to make it sane and useful. The RLVR models are superb for actual tasks in those RLVR domains, but that fine tuned view of the world as a verifiable problem to solve makes them feel worse for touchy feely stuff. Even for medical consultation and diagnosis, RLVR model's urge to reach a conclusion often is a liability.
Fable 5/Mythos 5 is the next "big chungus LLM".
It's RLVR tuned, but not to the ChatGPT level of brain damage, and it's still backed by a fuck off huge pool of model weights - which matters for what you call "touchy feely stuff".
There seems to be a problem with how he poses the problems alphaGo and these GAI models face.
AlphaGO is given a hard evaluation externally. It did not itself come up with it.
When GAI models are given an external hard evaluation, they can also succeed in many different domains (that is one of the remarkable features, succeeding in many domains) ranging from simple programming tasks to frontier mathematics (disproving conjectures recently) to writing more optimized kernel code than before.
And there is plenty of RL especially in these fields where the solution may be extremely complex but eval is rather less complex. And even the discovery and the "evolution-like" trace-selection is also happening.
For this reason it seems strange to compare it to AlphaGO as alphago is given a hard eval independent of itself, from an external source (humans) in a narrow domain. If GAI is given such, it can also show some remarkable results.
But what I find more strange is that innovation and moving forward in many many many cases does not require truly novel ideas but instead a high-quality execution of layering different methods, tactics, ideas on top of each other. Because in many domains our collective knowledge is incredibly sparse and complex, something being able to recombine tools, models, ideas in a high quality way (as he mentions being selective) I think is extraordinarily powerful. And in such cases, with a finite exploration horizon (time, resource available) with 1% "good choices" vs 3% "good choices" are worlds apart, incomparable.
Most importantly: none of the above is about intelligence, it's barren solution-farming to important, valuable problems we have. Most of the AGI and intelligence-related debate seems to miss out on this simple fact. (Insert the usual stuff like a plane being unable to fly like a bird or a submarine not swimming is totally irrelevant to it being useful).
And then a final point: do we really think this thing is incapable of doing better on average on problems we average people face in our lifetime? What should we think, how should we define human intelligence when we give out degrees in science or medicine for 60-70% exam results on problems considered to be generic in the field?
>> (Insert the usual stuff like a plane being unable to fly like a bird or a submarine not swimming is totally irrelevant to it being useful).
Just a brief reminder that planes have wings with airfoils just like birds and submarines have air tanks just like fish have swimming bladders.
Some birds fly without flapping their wings much, too, e.g. albatrosses.
You are right in that. Propulsion is different in both cases. The point is, until you have not recreated the function, difficult to say which part is important in other implementations (~Marr levels).
Leonardo tried to create flapping wings.
I don't quite follow his point. Is it: a) that we need a new foundational algorithm that integrates a goal (one with "taste") directly into the training step, or b) that we need to point trained models towards goals as they iterate?
If it's a), he doesn't propose such an algorithm, and I don't know how you'd do it at such a low level because how do you quantify abstract goals? Did he suggest such an algorithm and I misread? If it's b), that already exists, see AlphaEvolve or any number of things he said. Or, to be a bit of a smart-ass, just type /goal and let it rip ...
I also think he's just categorically wrong that LLMs cannot do good and novel things. And if it can, then you could just say "well that's not novel, that's derivative". A simple example, if I make up a programming language with an LLM and it works well for my purposes, then is that not novel and good? I mean, is any language other than FORTRAN not novel?
Everything is derivative and you can put an LLM in a loop to evaluate LLMs trying things. I must be misunderstanding because he's too smart to be this wrong.
No, I think I he’s saying that we have that, and we should use it more.
AlphaGo uses discovery when it evaluates potential moves and iterates.
Claude Code uses discovery when it generates a script and the evaluates whether it works or not.
He’s saying we need to allow ai systems to do the evaluation and iteration themselves for science and engineering the same way we do for code.
Basically, harness engineering for engineering.
but this has to happen during training no?
Idk one of his yt video presentations was saying we're entering a "designer" age of the universe
https://youtu.be/ThFq87Rp21s?si=SrKj72_X8bjnB6ED
Around 35min mark
LLMs possess the map but are unable to discern fertile from barren ground. For instance: how does Anthropic's new model generate promising 'medications'? Because, beyond the knowledge embedded within the model, it has assimilated AlphaFold's reasoning paradigm. By itself, Claude would be incapable of engineering a protein analysis method
I think a lot of deep learning is compositional generalization. Models learn reusable pieces (abstractions, styles, procedures, constraints, etc) and recombine them in ways that may never have appeared as a whole in the training data. So even if the ingredients come from past data the final composition can still be novel in a meaningful sense
I don’t think ML can’t be creative or make discoveries. I think creativity and discovery are, ultimately, simultaneously thinking about the right seemingly-disparate concepts (whereas algorithmic thinking is more obviously related concepts). If not an LLM, some other model can generate random ideas, rank them, then output the best.
But I think humans are better at it, while ML is better at algorithmic thinking. “Better” being more efficient and something we more enjoy doing; we can also more accurately rank what subjectively appeals to humans (i.e. taste), especially ourselves.
I think ML should be optimized for tasks that require more generalization than programming, but are still mostly logic. Like software development, translation, and tools for art and discovery.
One has to be very specific when throwing around words like "creative" when talking about A.I
Can A.I create art. Well it can create something that's pleasing to our senses but art is ultimately about conveying human feelings and emotions. Even as humans, understanding art is not universal. "feelings and emotions" and therefore art, can be deeply tied to a particular groups shared beliefs and experiences.
Can it be creative in non-subjective fields such as math or sciences. Einstein derived GR from his creative thought experiments. If an A.I poped out GR's field equations simply by testing different mathematical frameworks that resolve the issues discovered by experiments, is that creative? Perhaps but certainly not in the same way.
You can create art with everything from sticks and mud to glass and air. Of course you can make art with AI.
Now if the question is, can a machine make art, well ultimately someone needed to turn the machine on and design the machine to make art, so arguably that person/people are the ones making the art.
Historically, every question of "is x art" ends up having the answer "yes". I don't know why people fall for the same thing over and over.
His definition of creativity is closed-loop, i.e. the produced concept has to be novel for the system itself, not for the side observer.
> art is ultimately about conveying human feelings and emotions
you made a small error, art is mostly about generating an emotion in the viewer/listener/.... not about transmitting an emotion of the creator
the Wikipedia page on art starts with:
> Art is a diverse range of cultural activity centered around works utilizing creative or imaginative talents, which are expected to evoke a worthwhile experience
https://en.wikipedia.org/wiki/Art
so AIs can do art, because they are only required to generate an emotional response in the receiver
No art is a means of expression - the creator uses an object as a vehicle to capture and distribute an expression.
Also lol @ using Wikipedia as some source of absolute truth.
yeah, a bet a random HN-er definition of Art is much more reliable than Wikipedia
[flagged]
[dead]
Using your provided definition, you come to a conclusion completely ignoring the "utilizing creative or imaginative talents" part.
Current models are trained on image pastiche and style remixing. But there's no reason you couldn't add an Artistic Director layer which has been trained on emotional and cultural signifiers and to direct the pastiche and remixing.
The practical problem is that models have very limited prompt adherence. The level of detail you can specify in scene design is very crude. So you can get the slop effect where there's a lot of in-fill pastiche detail, but you could never create something like this, where all of the incidental objects are specifically included to enforce the message.
https://en.wikipedia.org/wiki/The_Awakening_Conscience
It's basically the professional version of the "Draw me a pelican on a bicycle" problem.
There are situations where you want that level of creative control, and current image generators don't get close to it.
And without it you can't get to the meta-creativity level where you're creating a new aesthetic that's a cultural landmark - which is what the famous artists did, and still do.
>But there's no reason you couldn't add an Artistic Director layer which has been trained on emotional and cultural signifiers and to direct the pastiche and remixing.
I gave this approach a shot over the first few months of this year[1] (although my director didn't have any custom training). The results were interesting, but I'd not call them "art", since they're low-quality derivative pieces. With reasoning traces enabled, you can see that there's not much intent going on. Though they do attempt to include "incidental objects" to reinforce meaning, like in this jungle scene[2].
[1] https://news.ycombinator.com/item?id=48105385
[2]https://www.liamlaverty.com/paint-by-language-model/inspect/...
Sounds like a skill issue?
Recent image models are advancing rapidly at prompt adherence specifically, and being able to iterate on the same image is propelling them even further. Images 2.0 being the poster child of this "agentic iterative image composition" approach.
Images 2.0 isn't anywhere close to the kind of detail control I'm talking about.
It's the opposite of a skill issue. No image generator is anywhere near the ballpark of pro-level manual Photoshop or Illustrator editing for individual elements in an image.
If you don't understand this, try precisely kerning the text in a generated book cover to handle letter combinations like A and V.
This is one of the big problems with GenAI. You can do new things with it, but it's crude Dunning Kruger good-enough-if-you-don't-ask-for-more creativity.
The pros can see what most people can't, and the flaws and missing features are frustrating and obvious creatively, not just in terms of production values.
I fail to see anything other than a skill issue.
We went from "AI can't generate text that isn't at least 20% typos and it always looks like shit" to "some letter combinations aren't kerned to perfection sometimes and adjusting that with prompts is hard". In a couple of generations.
Contemporary humans downvote art because it makes them uncomfortable.
They just want dopamine. No thinking because that hurts.
It’s ok, LLMs are useful as they are today. Even if they can never can come up with the next generation of math, physics etc.
Even for humans the brains who managed a step change in thinking are so rare that we literally know them by name.
You might be missing that those rare humans were sitting on tons of failed or somewhat useful discoveries made by more “mediocre” humans that history forgot.
I'm trying to keep an open mind and understand what the author is trying to say because he is credentialed.
His main point is that discoveries involve
1. Variation,
2. Evaluation, and
3. Selective retention.
He makes a jump saying AI is only capable of 1) and humans are capable of 1) 2) and 3). I don't know what makes humans special enough that they can do 2) and 3)?
In fact, the more you think of this it is kind of strange - in science humans can only do "evaluation" because they have access to the real world. They can evaluate a new drug because they can do it on people so it is not some inherent limitation of AI but rather access to physical realm.
Finally I want to ask a specific thing: how do you mathematically falsify what this person is saying? How can you formally prove that - no AI can not "evaluate"? I ask because I make AI evaluate a lot of people's claims and it works for me.
He's saying that pre-training an LLM alone can't do it, but if you run an LLM in a loop with tools (like any coding agent) then it can. Also, the technique his group came up with should be used more:
> This is the weakness of deep learning that is alleviated with a new algorithm that my group presented in Nature a couple of years ago. Our “continual backpropagation” made one small change: every so often a less-used neuron would be re-initialized to small random weights. This allows the variation to continue and plasticity to be retained.
Here's the paper: https://www.nature.com/articles/s41586-024-07711-7
It has a fair number of citations, but I haven't looked into how much it's used.
Sorry this makes no sense — humans also use tools to evaluate their discoveries.
Kant said something like this: knowledge can’t be obtained by pure thinking, it needs interaction with the world.
This is obvious to me so why is the author making a claim that LLMs can make knowledge without access to environment but purely through thinking in aether
[dead]
He actually says the areas in which AI has had the novel successes are those which can be evaluated (like coding or Go). Not that it can’t happen at all.
That’s my point, he says ai does well where evaluation is neurosymbolically closed.
But so do humans? How do humans make discoveries without having formal ways to evaluate? In my pharma drug example, humans could evaluate only because they had access to the physical realm.
I can’t think of an example of humans evaluating a discovery in a way that LLMs can’t. can you?
I don't think there is any "humans are metaphysically superior to LLMs" subtext to this talk, it's just a technical/educational observation.
Access to some forms of evaluation and selective retention is inherent to humans and it's not inherent to LLMS. But it can be somehow bolted on and that's when they work best. It makes sense that more focus on those principles can yield better AI. I think the retention part is the real limitation of LLMs, because it's limited to stuffing things in context window.
> Access to some forms of evaluation and selective retention is inherent to humans and it's not inherent to LLMS
I'm not sure I understood - what forms of evaluation is inherent to humans? If you don't give humans tools or access to the physical world, how can they evaluate?
There's no such thing as a human without access to the physical world.
So technically the only reason AI can't do discovery is access to physical word. When you give AI and humans access, they both do discoveries - that is the clean summary of the author's position.
Its not too interesting.. we already know that giving AI access to compilers and tools make them better.
[dead]
Humm maybe. But a plain model sampling outputs obviously isn't doing discovery in the AlphaGo sense. But once you put the model in a loop with tests, feedback, tools or even a human picking the good result, it starts to get much closer to the process he's describing.
It's funny to me that one would rate their own takes as "new and possibly controversial". Whatever comes next is read under that light of an author that thinks this about their own thoughts.
And the core point is not even true. They can definitely output novel things that are good - less so but they can and they do. Plenty of examples.
> Thus, the trajectory is either novel or good—based on randomness or based on data—but never both at the same time.
This assumes no possible unexplored path yields good results, or said another way, that none of the random results can be good, which is not true. The whole text seems to try to prove a point decided a-priori rather than make a case based on reality.
"We have many AI systems which can give us more. ... and Claude-Code, which have brought true advances in science, mathematics, and programming."
That contradiction kind of says he doesn't know what he's talking about.
Yes, the guy with a PhD in Machine Intelligence, co-author of Reinforcement Learning: An Introduction, which is universally considered the bible of the field, recipient of the AAAI fellowship award and the Turing Award, and the inventor of Temporal Difference Learning doesn't know what he's talking about.
Sure, but does that mean he's right all the time about all things, including everything in his own field?
He is saying no generative AI is going to produce output that is both good and novel because it is always derivative. And then adds a generative AI (Claude Code) into his list of AI that have produced output that he feels is good and novel, invalidating what he is arguing.
"...no matter how many instances of white swans we may have observed, this does not justify the conclusion that all swans are white."
If you read it he says that CC has additional aspects beyond ordinary GAI, namely the ability to verify. That aspect is necessary for GAI to be good and novel.
Although personally I think code doesn’t actually need to be very novel so it’s actually the best example.
“When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.”
https://en.wikipedia.org/wiki/Clarke%27s_three_laws
I don't completely disagree but its worth noting how new a lot of the empirical evidence in favour of LLMs are, so its not impossible to be a tad ignorant of the present
You know it’s occurred to me recently that there really is no value in this age to any kind of professionalism or seniority
If you don’t agree with somebody, nothing else matters
It’s like people (you as an example) have taken the concept of experts and fucked it up so bad that simultaneously everybody thinks they’re an expert while also dismissing everybody else who claims they are an expert
It’s like the whole concept is entirely poisoned. Worse everyone is smugly pointing at the Wikipedia for “appeal to authority.”
Nothing new I suppose, Socrates after all was driven to suicide by the madness of his society accusing him of impiety.
Surprisingly enough, Turing Award winner and father of reinforcement learning Richard Sutton knows perfectly well what he's talking about. The whole talk is about the need to have the ability to test novel outputs against reality and iterate to find ones that are good. This is exactly what Claude Code, the agent framework, adds to Claude, the LLM, to allow it to find novel coding solutions that actually work.
> When we ask for a fiction or novelty, the AI can give it to us because its processing is in part stochastic. Every decision can go multiple ways and will go different ways and produce a different trajectory every time. The trajectory can be random—and thus novel—or it can be based on the training data—and thus “good” because the training data is good, sourced from people or reality. Thus, the trajectory is either novel or good—based on randomness or based on data—but never both at the same time.
This doesn't seem true? You can be both random and based on training data.
I think they meant more "it can be extrapolated or interpolated" or "it can be high variance and 'creative' or it can be low variance and 'reliable/correct/likely'". If you want to see something new, the model will need to step off the manifold. But the manifold is where you've learned the "correct" solutions live.
Only to a limited extent.
AI reminds me a lot https://en.wikipedia.org/wiki/TRIZ , it is like a good machine implementation of it.
I think that creativity / intelligence / humour is indissociable from "making errors".
A joke is just an "error" - your brain predicted something, and the butt of the joke goes in another direction, and it's the mismatch that makes it funny.
The same goes with creativity, and intelligence.
The problem is that, by design, while trying to make machines "reliable", we make it impossible for them to be intelligent and creative
In addition to the importance of making "mistakes", I would say "surprise" is a big element in creativity and humor. Perhaps these are related concepts, because a mistake is surprising. It's an unexpected departure from the normal routine and habit of behavior or thinking.
The term "surprisal" is used in information theory:
> For a given probability space, the measurement of rarer events are intuitively more "surprising", and yield more information content than more "common" events.
Can a machine surprise us? Given enough complexity, I think so. They can produce unpredictable results, even novelty, something we've never seen before. But does that mean a machine can be creative? Or funny? Maybe there's a threshold of acceptance, where eventually its output will become surprising enough that we might as well call it creative.
Alt link: <https://xcancel.com/RichardSSutton/status/206121608774494665...>
(Currently returning 502 "Bad Gateway" for me, but should be restored at some point.)
The world will not be satisfied until we have read, and discussed every half-famous person’s opinion on AI.
Still about ten million discussions to go.
I mean even I know who Sutton is [1]. He is one of the reinforcement learning pioneers.
[1] https://en.wikipedia.org/wiki/Richard_S._Sutton
He is a Turing award winner
He seems to be saying that Claude Code can make discoveries. Does anyone think novel discoveries can be made from systems created by supervised learning only, and attempting to do so?
> Claude-Code, which have brought true advances in ... programming. ... these systems have found things that are both novel and good.
https://www.forbes.com/sites/anishasircar/2026/04/17/ai-solv...
I think its worth emphasizing that his argument isn't completely against generative ai, but rather its environment. Although I don't see why it would be impossible for something like an LLM to learn some sort of self-play within its context window
I think there is a rich seam of work for computers to help humans spot when there are anomalies in the hypotheses humans hold that can help humans form the corrections to hypothesis that fit our aesthetic sensibilities
I think the variation, evaluation, and selection idea is a good, if not the only, way do do creative work.
I don't think I would attribute anything in that process that I would consider an AI to be incapable of.
The characterisation of variation like this would seem to rest on the same 'random but directed' crutch that some free will arguments rest upon.
There is no random but directed of course, there is random and there is caused, and there are things that use both as components, but the random remains wholly random, and the caused remains entirely deterministic.
I think there is a good case to say that, in many fields, AI is better than humans at evaluation.
To find avenues to consider, I'm not entirely convinced that human innovation is more than a heuristic that appears more chaotic by virtue of a inconsistent and opaque formulation.
Many aspects of ideas com from noting how some two things are different and then considering that axis of difference when applied to another thing.
The possibilities thrown up by this extremely simple method are vast enough to require multiple layers of evaluation, most could be dismissed out of hand by a quick 'This is nonsense' check that I suspect people do so often and at a rate that it wouldn't even rise to the level of consciousness.
I enjoyed reading this at the start, the language is very... inspiring. By the end, I was disappointed. I don't disagree with what they're saying, but the opening style and statements made me expect some more specific or groundbreaking conclusions.
The point seems to be that generative AI just generates stuff, and that real discovery requires variation, evaluation and selective retention.
The call to arms seems based on the assumption that people only every talk about generative AI as discovery machines themselves. I think it's pretty widely accepted that's not the case by everyone apart from cliche out-of-touch CEOs.
But the talk makes me realise that generative AI are incredible tools to do the discovery cycle with, and this is what I imagine professionally successful AI users are doing: variation, evaluation and selective retention of their inputs and outputs to generative AI.
Richard talks about AI been either novel or good
Then it shifts to discovery.
These seems related but not exactly the same thing.
Creativity = variation + evaluation + selection. It's not bad, though every example he gives has a built-in scoring function haha.
Best thing about nerds is watching them try and build frameworks and formulas for the creative act. Like a metronome trying to compose a symphony.
yes, when you drive these systems you can get novel output, depends on the effort you put in
This reset my thoughts
why no one talks about claude shannon deep understanding of information theory and the consequences for compression and intelligence
Can't tell if you are spending time in the wrong circles or just shitposting. Here's one: https://youtu.be/l6DKRf-fAAM?si=6u54mry0TLgS868a
"So that is my call to arms. If we want the full power of AI scientists, then we should share the goals with them so they can create, evaluate, discover, and in these ways fully participate in achieving the goals. Let’s be bold! Let’s fully automate Creativity and Discovery!"
Should we automate exercise and play as well? How about learning?
The machine didn't have a soul, so we donated ours.
Eureka! My AI found it!
I understand the skepticism. I am worried about the implications of AI as well. The deeper issue at stake is that the depth of scientific knowledge has been increasing for a very long time. Now you get to have a PhD in esoteric subproblems and that slows down research especially if the discoveries require depth in multiple subdomains. Socially and economically training people in every combinatorial combination of subfields at the required depth may not be possible. I am especially interested in two problems to be resolved and do not care if an AI scientist performed the discovery. It will be humbling, but totally worth it:
- Fusion (a clean sustainable form): Without this I think we are heading in a very wrong direction, whether it is conflict or climate change does not matter. Everyone is aware of this and instinctively afraid of the implied loss of quality+quantity of life.
- Cure for Cancer: It is a world wonder even in Civ. I and for good reason. As a father of a teenager, every time I hear a story of someone losing a parent/child I cringe. We have to accept this as a reality of life until a proper/generic cure is found that eliminates the most common offenders.
I am skeptical that we will have AGI anytime soon and I think the social aspects will help balance the technical developments even it becomes a reality (Three laws, A Butlerian uprising, you name it).
Chess bots can beat grandmasters, but I have a friend who takes his son to tournaments. Humans are still playing chess, kids in the same tournament with grand masters. We have to have faith in the humanity, or all else will not matter.
And I will definitely keep playing Factorio even if AGI comes to pass ;-)
Commercial fusion energy is almost certainly a pipe dream. We already have good alternatives. Short of some unpredictable major breakthrough in physics on the level of something like cold fusion, the current activity in the fusion space will basically just serve to demonstrate definitively that it's not viable.
And then there's France, which gets about 70% of its electricity from nuclear energy.
Not sure what you’re trying to say. France gets 0% of its energy from fusion, and the same is true of every country on Earth.
Look up fission vs fusion.
Note that all fusion energy is unsustainable as it irreversibly turns water into helium.
We do have a LOT of water though. And surplus helium would be nice.
We also have competitive energy wasting (bitcoin), that will translate into competitive water deleting which will greatly accelerate how quickly we burn through all of it.
Eventually we'll burn up the visible universe, but I think we have some time before then.
Eh, it seems increasingly clear that no matter the fusion breakthroughs to come, it's going to be complicated and big and therefore expensive compared to renewables. By the time we figure it out it'll be a nice breakthrough but will end up on the shelf.
[dead]
[dead]
It seems human beings spend a significant portion of their lives trying to figure out how to live as little as possible.
They do say "live a little", and it strikes me now that perhaps I've completely misinterpreted that phrase. For some reason I've always thought it to be a call to live a lot, or at least more.
But now I see that my interpretation is almost opposite of what the words mean... Huh...
It seems like human societies are run by elites who convince the masses to live as little as possible so that the elites can fully enjoy themselves.
Like how during Covid the beaches were closed while the billionaires partied on yachts.
Most discovery takes a ton of iteration and repetition. That's a lot less fun than the hedonism of sitting on a beach drinking a mahi thai.
Same with learning, humans historically where generalists without that deep of knowledge when compared to one another. Now we study a quarter of our short lives just to get to the point where we can specialize for the rest of our lives. This situation doesn't seem exactly tenable as complexity increases in the future.
I don't know what the future looks like, but I can tell you that entropy and complexity will only increase.
[dead]
> Should we automate exercise and play as well? How about learning?
Obviously. Exercise, play and learning is in large part what makes us smart. If we want AI to be smart we should definitely automate those.
But that's not what you had in mind. To address that I can only say that you are allowed to still do the automated things manually.
Another famous dude dumping his thoughts on HN who is gulping it up like an addict.
Add this to the long list of names like Terence Tao, and others who seem to be intellectually incontinent lately in the sense that one cannot navigate this space anymore without encountering their thoughts
Creativity is personal taste and the taste which captures the most is mainstream.
I tell jokes and the group of friends get them, for family they don't get them anymore.
I do not like 'basic jokes' and despite that, german television is full of it.
Most things in our world are more of a challange of finding the answers and not 'creating' the answers.
Math: the answer is already there, you only need to find it an fast space of posibilities. This is perfect for LLMs.
Creativity: a LLM can iterate over things a lot faster than a person. So we can iterate over this space too. We can also get feedback from people, tiktok, instagram.
Hmm, so the guy who said more data and compute will outperform any adhoc heuristics has shared a three step adhoc heuristic?
Can you please make your substantive points thoughtfully and without snark? I'm sure there is one here, but it's hard to make out what it is, and in any case the poison does more harm than the information does good.
I'd link to the HN guidelines here but I'm on my phone!
Assuming OP meant "the bitter lesson" https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson...
I assume so as well but there isn't enough information in the comment to make a substantive point about it.
The ad hoc heuristics are the domain knowledge baked into the model by human experts, like features, architecture and loss function.
"Evaluation" means environments or datasets, the model is supposed to discover its representations from scaled up experience. That was the bitter lesson - more data and compute beat heuristics.
[dead]
[dead]
[dead]
[dead]
[flagged]
"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something."
https://news.ycombinator.com/newsguidelines.html
(For others like me who need to look it up.)
Stochastic: describes a process, model, or system that involves inherent randomness, meaning its future states or outcomes cannot be predicted with absolute certainty but can be described using probability distributions.
[flagged]
"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something."
https://news.ycombinator.com/newsguidelines.html
That reminds me of a quote by Freeman Dyson on Stephen Wolfram's work.
> "There's a tradition of scientists approaching senility to come up with grand, improbable theories. Wolfram is unusual in that he's doing this in his forties."
I always felt it was an unfair dismissal of someone's life's work. Maybe it was true but it didn't enrich the discussion or our understanding. I suppose it means even a respected thinker can be guilty of shallow dismissals and saying hurtful things in public about others.
It's similar to a "thought-terminating cliché", in that it just reinforces an existing opinion without adding anything, making us think deeper, or furthering the conversation.