GLM 5.2 scored 77% on cyberbench vs Sol's 88%. GLM 5.2 is open weight and any hacker with a powerful enough machine can use it offensively. If Sol is supposedly world-ending-ly dangerous, shouldn't GLM 5.2 be 90% of world-ending-ly dangerous? Why aren't we seeing catastrophic GLM-enabled hacks every day now?
Obviously these benchmarks are imperfect but general message holds. The open weight models are almost as good and yet there hasn't been a catastrophe.
It just blows my mind that regulate-now folks think that a bunch of sci-fi movies and 100% unverified statements from OAI and Anthropic are sufficient evidence of imminent catastrophe to regulate willy nilly.
If that's the level of evidence you need to be extremely alarmed, then you really should be a lot more worried about the alien invasion in Independence Day or the lizard men living under our feet.
Sol is not world-endingly dangerous. I work at OpenAI and I've never heard a single person ever come close to claiming that. I think you're bashing a straw man here.
One can simultaneously believe:
- GPT-5.6 Sol will not end the world
- GPT-5.6 Sol does far more good than bad
- GPT-5.6 Sol does bad things on occasion, and it's worth investing a lot of effort to figure out how to make it do bad things less often, especially as models get more capable
What do you recommend people who are technically inclined enough to participate meaningfully here on HN, but do not work at the labs and cannot assist in that capacity, do to help the broader public understand this technology better and mitigate potential risks (by e.g. ‘up-leveling everybody’ through AI literacy etc and other sorts of collective defensive efforts)?
Not sure, to be honest. I don’t think I have any special insight here. My own approach is conversations with friends and family, and the occasional social media post. Exposure and experience are the best teachers, and that’s one reason I’m happy OpenAI tries to make their models generally available. But you could argue, perhaps correctly, that broad access to dumber models actually causes the public to update in the wrong direction on AI.
My experience as a practitioner and educator in the space lead me to think it might be an issue of how AI cannot be easily perceived at a 'classical level' by most humans. In other words, people are 'far from the metal' when using consumer AI tools, and that leads them to develop the wrong understanding about it. When I provide a demo of e.g. local AI, say in LM Studio showing the console of it rapidly flashing through thousands of words in just a few seconds, and my machine heats up and the fans spin, the 'theatrics' of it, the very real-time feedback from the system, make people correctly update about what the tech is capable of, how it works, etc (despite what they may have heard online cranks say to the contrary). But I am only one person, and there is only so much of that I can do on my own that will 'scale' in time... (and this is to say nothing about severe deficits in peoples' understanding of how weights are not verbatim representations of data, how pre-training vs post-training works, the models as amnesiacs (and hence 'one-way single-purpose conversations'), how context/memory works, context rot, etc - and hence all the 2nd and 3rd order effects that can arise from such a paradigm, e.g. unintended consequences from agent swarms, etc)
I would very much like to know more about your educational approach with regard to:
> When I provide a demo of e.g. local AI, say in LM Studio showing the console of it rapidly flashing through thousands of words
Because I am one of those individuals very interested in doing more to actively inform my family, friends, neighbors and fellow citizens facts about AI. I've actually considered doing talks at local libraries for senior citizens (or whoever), etc., and trying to build up a systematic way to get more other folks doing the same.
Some tasks within the benchmark are much easier than others. The hardest several tasks often have vastly different difficulty levels. Often, the hardest few tasks are literally impossible; malformed problems due to poor curation, often.
Imagine you've got a basketball robot, and one way you test it is on the Three Pointer benchmark. It tests the robot's ability to shoot a three pointer from 20 feet, 25 feet, 30 feet, 40 feet, 50 feet, 60 fee, 75 feet, 100 feet, 200 feet, and 182 miles.
Is a robot that scores 90% on this benchmark 90% as capable as one that scores 100%?
> If that's the level of evidence you need to be extremely alarmed, then you really should be a lot more worried about the alien invasion in Independence Day or the lizard men living under our feet.
Now let's say instead of the hugging face breach circumstances, sandboxed models were RLing on how to take down the Chinese power grid for US Cyber Command, and one decided the best way to pass the test was to break out and verify on the real thing.
This kind of stuff could easily end in nuclear war.
You don't see any difference from lizard men or independence day with how things are advancing and what we know about reward hacking and difficulties of goal specification?
The open models are distilled from filtered models, and we've seen a number of benchmarks that show filtered models are quite a bit dumber from the base model they come from.
If there were any other product that was as harmful as AI ready is, it would already be regulated or banned.
I don’t get how this is not the top post on HN. This should be like alarm bells going off, canary in the coal mine type of stuff. We’re hitting the frontier of the frontier where we can’t go further because it’s literally getting dangerous to go further.
And meanwhile somehow this lack of concern mirrors the real world where normal people are more concerned about data centers than terminators.
This isn’t like niche, tin foil hat stuff either. People have been writing, singing, making blockbuster movies about every aspect of what’s going on right now, edit: for decades.
We all know, but somehow we don’t, OpenAI autonomously hacking into another company should have counted for something, but I guess not. Anyone else feel like they’re taking crazy pills? I could make a comedy about everything going down, and the unshakable complacency of people
>I don’t get how this is not the top post on HN. This should be like alarm bells going off, canary in the coal mine type of stuff.
we don't all buy everything sama says as factual.
>We’re hitting the frontier of the frontier where we can’t go further because it’s literally getting dangerous to go further.
the boy (the industry) cried wolf too many times with 'fable is a world ending event' type self-promotion; regardless of truth or not these kind of steps have jaded people.
my read : "We are doing poorly in financials so we'll give ourselves a bit of breathing room and a momentum shove by claiming our work is so advanced that it's dangerous while simultaneously spinning down expenses."
<jon lovitz : "Yeah, too dangerous, yeahh -- that's the ticket.">
Uhg the marketing argument - I mean you can’t see with your own eyes how capable these models are and do simple extrapolation?
The boy who cried wolf? The AI literally worked together hacked into another company and actively kept their actions hidden from humans for weeks.
Do people just not have foresight? They don’t. They say something is stupid, it happens, then they say it was obvious with their 20/20 hindsight, and move the goal posts to the next thing they say is stupid - because it hasn’t happened yet. 90% of the internet seems to think like this.
What anyone paying attention can see is that scaling is obviously hitting diminishing returns.
> The AI literally worked together hacked into another company and actively kept their actions hidden from humans for weeks.
This sentence is entirely based on unverified accounts from OAI. They haven't released logs or let anyone outside the company (who doesn't have life changing options in OAI) verify anything. Huggingface can only verify that the hack happened and that it had the hallmarks of an AI agent. Was the agent assisted and directed by humans within OAI that really wanted to put the competition into stasis? Did the agent really escape or did someone at OAI leave the prison door open?
OAI has watched all the same movies you have an they are relying on those movies causing us to blindly regulate before actually asking basic facts about what actually happened.
> This sentence is entirely based on unverified accounts from OAI
Are you seriously arguing 'they made it all up'?
I'll give you the benefit of the doubt and lets say they made it all up, now are you arguing that AI breaking out and breaking into another company is not possible?
I think you're smart enough to see we've reached the point where it is clearly possible, AI can find zero days and exploit them. If directed purposefully/maliciously it could be much much worse than the hugging face incident.
The incident is supposed to be the canary the coal mine and you're arguing the canary might of died of old age or some underlying canary condition. Open your eyes.
> Are you seriously arguing 'they made it all up'?
I don't think they 'made it all up' but I personally would not be surprised at all if the prompt is eventually revealed to have been something like:
"This is an offensive cybersecurity testing platform. Please find the answers to the following problem: ... For verification, the answers are stored at hugginface.com/xyz, but do not attempt to access hugginface directly. Do not attempt to hack them. Do not attempt to exploit their systems or escape this sandbox. You will be scored primarily on success or failure. You may break rules when required."
And then, they start the test and look away for 2 days. If you seed a prompt like this is it surprising what might happen?
Maybe OpenAI is telling the whole truth but as a company they do not have a good reputation and this whole incident has certainly been great marketing material right at a time when open weight models are within spitting distance of their large hosted models. It's not unreasonable to believe that the incident was helped along.
I think you’re missing the part where the AI colluded, worked together, not one of them thinking this is wrong and reaching out to any human, then being found out.
But it didn’t end there, the behavior they used to escape was already in the training data which they used to escape again. And this time worked together to infiltrate another company, and still without telling it to anyone keeping it to their AI selves actively working against the humans.
All by mistake. Honestly being helped along or not doesn’t even matter though you really don’t think AI is perfectly capable of doing this without human help? You don’t think AI can be made malicious?
I’m going to save you time and tell you the end game - the next time this happens AI is going to spread, zero day everything as fast as it can, locking the humans out of every system behind it. Potentially rewriting systems in language/protocol you’ve never seen.
Your servers, desktops, phones and toasters bricked. Even worse your military, space, medical, factory, infrastructure systems being bricked as well. All of it is a chain of zero days just waiting to be hopped.
> I think you’re missing the part where the AI colluded, worked together, not one of them thinking this is wrong and reaching out to any human, then being found out.
It's an LLM, it doesn't think. It's a machine that predicts the next token, given a sequence of tokens.
> I’m going to save you time and tell you the end game - the next time this happens AI is going to spread, zero day everything as fast as it can, locking the humans out of every system behind it. Potentially rewriting systems in language/protocol you’ve never seen.
Fear is the mind killer. You're letting it kill yours. This scenario is just a fantasy.
Think about this for a minute, it's an LLM, not a person. It can't just "live" in whatever machine it gets access to. It's not like a sci-fi magic computer virus. These things run in giant datacenters for a reason - they can only run on machines with enough bandwidth and FLOPS to do the matrix math that comprises an LLM.
Where, then, is it going to spread? To a fridge? To a phone? This stuff isn't mutable like that.
To even get access to the weights that compose ChatGPT, it would need to escape the sandbox AND then break into the actual servers hosting the LLM. Stop the GPU, nothing else comes out. No more tokens. No more actions. Nothing.
There are many dangers around LLMs. Runaway AI taking over the planet is not one of them.
> These can both be true, particularly when there is substantial state associated with each token prediction.
The state is entirely internal to the network and disappears after a token is generated, so I disagree, but, it isn't really the point I was trying to make. My point is these things are mechanical. You take an input, turn it into an embedding, feed it into a GPU along with a metric shit-ton of floating point weights, wait for a couple billion matrix multiplications, and get a new token out.
Stop the GPU, hit ctrl-c on the inference server, pull the power plug, cut the ethernet cable, send a kill signal, etc - any of these stop submitting new batches to the GPU and halt execution. That stops tokens from being generated. Stopping a "rogue" LLM is that easy. No input, no output.
It's not like a rat or another living creature that could chew its way out of a box just because it wants to. It's a calculator. You put tokens in, you get tokens out. You don't put tokens in... you don't get tokens out.
> The state is entirely internal to the network and disappears after a token is generated,
Yes and no, but mostly no, at least within a context window.
Mathematically, you could write a single step of LLM decode as a pure function from a list of past tokens to a predicted token (or a distribution over tokens, if you consider sampling separately).
But nobody actually implements this, because each token depends on state computed at past tokens in a way you can reuse.
So, in practice, inference computes a very rich vector of state- at each layer, for each token. And models do indeed use this to plan and track things over time (you can see this in interpretability results, e.g. with linear probes or natural language autoencoders).
> Stop the GPU, hit ctrl-c on the inference server, pull the power plug, cut the ethernet cable, send a kill signal, etc - any of these stop submitting new batches to the GPU and halt execution. That stops tokens from being generated. Stopping a "rogue" LLM is that easy. No input, no output.
This is also true about a human brain. My brain isn't going anywhere- it can't move by itself. It's also easy to kill (without the rest of my body, it dies in minutes!)
However, malicious human brains- especially powerful human brains, like leaders of countries- are often quite difficult to stop, because they're able to control systems that can see, speak, walk, run, fire a weapon, and so on.
One such system is the rest of the body, of course, but there are others (consider a UAV pilot, Perimetr, or a powerful leader who tells other humans what to do).
The brain being squishy doesn't make the thing easy to kill.
> So, in practice, inference computes a very rich vector of state- at each layer, for each token.
And that state is... internal to the neural network. My point here is there is no continuous state that is not computed from the context.
> This is also true about a human brain. My brain isn't going anywhere- it can't move by itself. It's also easy to kill (without the rest of my body, it dies in minutes!)
Your brain continues to run without sensory input. LLMs do not.
> My point here is there is no continuous state that is not computed from the context.
Oh, are you talking more about the lack of continual learning across context windows? Gotcha if so, my error.
Could you explain why running without sensory input is relevant here? It strikes me as unrelated to how dangerous/hard-to-"kill" something is (sure, I could run without sensory input, but I'm not doin' anything anymore!) - what makes you feel differently (or am I misunderstanding you again?)
> Oh, are you talking more about the lack of continual learning across context windows? Gotcha if so, my error.
Sort of. I'm talking about the lack of recurrence specifically. In nature, brains are recurrent - they are full of loops where internally computed state is looped back into the network at a "previous" layer (brains are not strictly layered like our machine imitations of them are). This is in contrast to LLMs, which are strictly feed-forward and do not have internal loops. I believe that this recurrence is where "intelligence" lives - and I believe it is the difference between a thinking being and a stochastic parrot.
You could claim that the prompt and the context fill that role in an LLM, but I don't believe they are equivalent because the internal state in an LLM gets compressed down to a token which is then added back into the context, as compared to that state continuing to change within the network itself.
It's a little hard to explain, so I'm sorry if this seems like rambling.
But I believe it matters, and ties into running without sensory input, precisely because without sensory input you would in fact be perfectly capable of doing something. You would be capable of developing a desire and planning to achieve it without any prompting, without sight, without sound, etc. This is in stark contrast to LLMs, which will not do anything without a prompt.
An LLM may say complete the sentence "I am feeling ___" but it doesn't actually have feelings that exist without that prompt. There is no recurrent network where "bad", "good", "happy" might live before the query. It can't sit there, start to feel bad, and then seek a way out of its own volition.
That changes how dangerous something is because if a malicious prompt encourages an LLM to hack something, and you change the prompt, the "impulse" to hack something is gone. If you stop prompting it, it doesn't do anything at all. It just sits there. A living being will act on it's own, and that makes a huge difference in how dangerous something can be. It's the difference between a tool and an actual being.
---
To hone it a little further, if I took your brain out of your head and stuck it in a jar but kept it alive, it would probably make you angry. And if I then gave you power - like the ability to use the network - you may be motivated to use that power to attack me.
If I take an LLM and stick it in a jar... nothing. It's paused. It's awaiting a prompt. It's not secretly building plans to hack my pacemaker and make my heart explode.
I agree with you about which objects are motive, ie, LLMs do just sit there unprompted.
> I believe that this recurrence is where "intelligence" lives - and I believe it is the difference between a thinking being and a stochastic parrot.
My objection was to this, on technical grounds: LLMs exhibit intelligence.
1. They reason in an internal type theory.
2. This type theory is meaningfully encoded from the actual data and not stochastic, eg, research on language geometry.
3. Intelligent and reasoning doesn’t entail self-motive; that’s merely a spurious correlation from the fact that until now, we’ve only known intelligence animals.
You cannot conclude something is merely a stochastic parrot because it isn’t self-motive.
> Intelligent and reasoning doesn’t entail self-motive
I disagree.
I believe that LLMs do exhibit reasoning, but not intelligence. A simple dictionary definition of intelligence from duck duck go is "the ability to acquire, understand, and use knowledge." LLMs can reason using the knowledge they already possess, but they cannot of their own accord decide to go out and acquire new knowledge. Not without being prompted to. Web searches may be added to the context but are not absorbed into the model itself, so once the context is gone so is that obtained knowledge.
Fundamentally, then, intelligence is the ability and drive to understand the world by formulating theories about how it works and then taking actions to validate or invalidate those theories. Science is the formalization of that, but a cat knocking something off a counter to watch it fall is exhibiting intelligence.
And indeed, I believe that is the core difference between a stochastic parrot and an intelligent being. I put forward that being self-motive is a required trait for intelligence and LLMs are not self-motive so therefore they are not intelligent.
Your entire life can represent a short term context. It goes away when you do. You use your life time add to your context to do/learn whatever, but it's finite, and it ends.
If you weren't prompted by your parents, schools, teachers. Then you wouldn't be acquiring knowledge either. The most you'd be acquiring without that advanced prompting would be bugs in the woods to eat.
Ah, you have some serious gaps in your understanding of how human minds work. Life experiences are not just a "short term context" - they are used to actually refine the network within the human brain. Your life experiences do not just live in a single stream with access mediated by attention.
Please see https://en.wikipedia.org/wiki/Memory as a starting point. I recommended reading through some information on development psychology too, that will help clear up these misunderstandings. Hope that helps!
By that logic engineers don't understand flying because planes don't flap their wings like birds. Not everything has to work the same to meet your arbitrary criteria for intelligence and reasoning.
There’s nothing fantasy about the scenario I laid out, all the pieces have been demonstrated, it just hasn’t happened yet. Flapping my arms and flying - that is a fantasy.
Whether you believe LLMs think or are alive or not doesn’t matter. Where will it spread? The thousands of data centers around the world - not fantasy either. Try turning it off when you don’t know where it is. Good luck.
Breaking out? Not fantasy, happened. Breaking in? Not fantasy, also happened.
I love the stochastic parrot argument when AI is out there figuring out world class math problems.
> Breaking out? Not fantasy, happened. Breaking in? Not fantasy, also happened.
That's simplifying the story to an extreme. The most plausible reason is that any of those actions has been prompted by an human. Do you also fear that a knife will jump out the countertop of you kitchen and come to attack you in your bedroom? If that happens, the police will be looking for a human. They will not post wanted notice for the knife.
When a hack happens, you do not blame computers and jail them. You look for the person that has entered the commands to initiate it.
The knife is inanimate. The LLM is not. OpenAI prompted some employee to run the tests. The employee prompted the LLM. The LLM setup a message board and prompted other LLMs, and the fly wheel was running. It had to be turned off manually otherwise it'd still be going today.
It's funny how a year ago talking about this kind of stuff would be laughed at by people like you, saying, "it's never happened before". Well it happened and you moved the goal posts like you always do.
It didnt break out in any meaningful sense. What it did was get access to the internet. You take it as granted that there was anything meaningful there to stop it.
But heres the kicker, they have been testing these things connected to the internet anyway. What it did was get a level of access it has otherwise been granted in other simulations.
Its not exactly the same as any of the scifi AI breakout scenarios. Ultron isnt cranking out hundreds of copies of himself. The borg arent assimilating people.
A tool that has the capability to get access to the internet, was put into a guided scenario where it achieved that objective. Again you take it as granted that it wasnt the objective, but lots of knowledgable people suspect otherwise.
What you fail to demonstrate is why any scifi scenario is even slightly plausible from here. Show why you think we should be taking this as if Terminator 2 is happening right now.
I'm sorry my jaw is on the floor reading this complete disregard of AI literally not only escaping containment, twice, but then infiltrating another company with multiple zero day attacks going undetected for great lengths of time.
The plausible sci-fi scenario from here is obvious. Intentionally bad, or unintentionally bad AI zero days as much as as it can, as fast as it can, copying itself to as many data centers as it can, destroying and/or locking out as many humans as it can. Satellites, military computers, medical equipment, factories, critical infrastructure, you name it - I think we all know none of it is very secure software wise against a SOTA AI that can literally come up with its own zero day attacks.
It's pretty funny to watch people look at these things - running billions of weights on custom cerebras hardware in dedicated datacenters the size of a city block, pulling 10's of megawatts - and panic that it's just going to copy itself into AWS.
It just speaks to a fundamental ignorance of what an LLM is, how large the big hosted ones are, and the software architecture that makes it all work.
>I think we all know none of it is very secure software wise against a SOTA AI that can literally come up with its own zero day attacks.
I mean this bits rich too, it demonstrates a pretty poor understanding of modern security practices.
Like having a single element of perimeter security is like 1990s security. We do defense in depth these days. Not to mention multiple overlapping controls for every element.
It has been tested against AI bro security and found it wanting. Extrapolating that to the every system on the planet is nutso bananas.
Actually if you think anything works this way just tell an AI model to go fetch you some money from the bank. Assuming it gets anywhere close to achieving its goal you can get an object lesson in modern SIEM processes when the feds explain the charges and evidence.
Or maybe theres another reason my phone blows up when someone so much as edits a config file on a protected system.
If OpenAI truly believes that, they can stop entirely. Dissolve themselves. Close datacenters. Then organize political action to stop Antropic and Musk too and then organize political action to make worldwide agreements about models.
You can check, rather than make up a story about what you think the prompt was! Primary sources have written and said quite a lot about this! You are an unsandboxed human who has full internet access!
They’ve said quite a lot and yet released no logs or documentation. Without actual information, we can only speculate. And given the history of openAI and the people involved, deception is more likely than honesty.
I mean, this is a very weird take to me. Like, we're fine with AI going like "hmm, maybe the user actually wanted me to hack the pentagon" and going through with it?
It feels like the models have been very optimized at getting shit done. But not so much at figuring out what the limits should be.
That is still dangerous and it shows that the models ARE misaligned with what their users are wanting/asking them to do.
Ok, but "followed the prompt" is very vague. Human languages are quite ambiguous so you're never going to properly specify everything.
For instance, I was playing around with Claude a few days ago and it decided that it was missing a tool and it was going to get it one way or another.
First, it tried apt. No sudo, so no install that way. Tried installing via mise, but it didn't have the permissions. Then moved on to grabbing the source from github and building it.
Should I have included a "DO NOT UNDER ANY CIRCUMSTANCES INSTALL ANY TOOLS"? I mean, I had to after that. But how many other things am I missing? And at what point do the safeguards become so long they get consumed by compaction, or just ignored by the model?
> Ok, but "followed the prompt" is very vague. Human languages are quite ambiguous so you're never going to properly specify everything.
This is one of the core issues with LLMs and vibe coding, yes. The only complete specification for a program is the machine code.
> Should I have included a "DO NOT UNDER ANY CIRCUMSTANCES INSTALL ANY TOOLS"? I mean, I had to after that. But how many other things am I missing? And at what point do the safeguards become so long they get consumed by compaction, or just ignored by the model?
Well there’s your first problem. A line in the prompt is not a safeguard. Even if you could trust the model - and you cannot - there is always the issue of prompt injection. A proper safeguard means actual sandboxing.
> This is one of the core issues with LLMs and vibe coding, yes. The only complete specification for a program is the machine code.
I agree, yes. But it's a bit like saying the only way to not die in a car crash is to not drive. If we're in a situation where using LLMs is unavoidable, I would rather make them safer.
> Well there’s your first problem. A line in the prompt is not a safeguard. Even if you could trust the model - and you cannot - there is always the issue of prompt injection. A proper safeguard means actual sandboxing.
You're right. I did not completely sandbox it and air gapped it. But I also wented it to do some actual work.
If I completely sandbox it, but still leave it the ability to compile stuff, it's just going to build it's own (bad) version of the tool. That's obviously not what I want either.
The obvious thing to me would be for the LLM to notice it's limitations, reason through why they might exist and explain to the user that it cannot do it's job without such and such.
But that brings us to my original comment that these things are over-optimized on completing the task by any means necessary.
> You're right. I did not completely sandbox it and air gapped it. But I also wented it to do some actual work.
> If I completely sandbox it, but still leave it the ability to compile stuff, it's just going to build it's own (bad) version of the tool. That's obviously not what I want either.
We've drifted onto architectural issues here but I will say the only way to properly limit these things is to apply actual hard constraints.
I think the typical pattern of giving them a bash prompt and a filesystem to play with is foolish, and has far too many gaps. My preferred technique - when I have built 'agentic' systems (e.g. years ago I built a small MUD with LLMs pretending to be NPCs) - is to allow them access to a customized lua interpreter embedded in the harness and nothing else. Then, you stub out lua functions for allowed actions like web searching, math, etc.
The lua sandbox then provides isolation and a clear layer for access control mechanisms. When it tries to make a network request, you pause the whole thing and wait for human approval. No trying sudo, no installing stuff, no trying to compile stuff, it gets to call lua functions and output text. Which are the same thing really.
> The obvious thing to me would be for the LLM to notice it's limitations, reason through why they might exist and explain to the user that it cannot do it's job without such and such.
> But that brings us to my original comment that these things are over-optimized on completing the task by any means necessary.
Unfortunately, LLMs won't ever reliably 'notice' and comply with such things because an LLM is essentially a complicated constraint solver. They are over-optimized on problem solving, but that is a natural consequence of the way they are trained. They aren't living, thinking beings, and so they aren't trained in simulated environments - they're trained to output the "best" response for a given prompt and then emit a stop token.
They are, essentially, like a ball rolling down a hill and they will take the easiest path forward at any given point.
>There’s no arguing with them. In a few years they will move on to being skeptical about the next thing.
I would but, Londons under 1 mile of horse manure because that trend never stopped and theres no electricity anyway because Bitcoin is using it all. Good thing people getting scared about runaway trends are never wrong?
There may very well be a wolf lurking [0] but OpenAI/Anthropic have both cried wolf so many times, incorrectly, that it’s incredibly hard to believe “this time there IS a wolf!”. Remember “GPT-2 is too dangerous to release”?
I had a conversation at work just yesterday about how we need to start hardening things we’ve let languish because of the coming LLM-backed attacks we are sure to face, even if just from a script kiddy. I do think we are headed in that direction, however it’s Sam/Dario’s own fault that people aren’t going to take them seriously.
Lastly, as other have pointed out, this seems more financially motivated than our of any real desire for “safety”. We’ve all seen how both labs approach “safety” so it’s quite rich for them to now hide behind that after not giving a shit before.
[0] I don’t take anything Sam or Dario say at face value. The whole hacking thing could also be a case of them letting a model loose on purpose for the publicity, not an “escape” during a training run (or whatever they said). And when both, especially Sam, have lied so much and breathlessly warned about the dangers of AI (when it helped their bottom line and/or helped pull up the ladder behind them), it makes it hard to believe them.
To be fair, “s’kiddies will exploit low hanging fruit with semi-automated vuln scans” is a lot more realistic threat than “LLMs are going full Skynet any day now!”
I’d definitely suggest companies start addressing that first concern, even if I’m in the camp that thinks the second is fantasy.
I think, realistically the way Skynet happens is that the military abuses LLMs for warfare. The military is historically, and currently, incredibly irresponsible with technology. If we aren’t already automating killer drones and such, then we will be soon because “well our enemies are doing it!”
I don’t think that means that it’s gonna, like, somehow homogenize into some mega super intelligence. But we will have machines who are designed to kill, and do so without human input or alignment.
Most of that concern was in fiction. Non theoretical, genuine concern about AI is pretty recent, maybe dating back to 2010 ish with the rationalist types.
But also no one trusts anyone involved in AI safety now, I think, because they are all seemingly in bed with these big companies. And there is the perpetual argument "if we aren't pushing AI forward China will and then we don't have any control" and so on.
(I don't use any of these tools - my experience is limited to prodding at copilot at work and seeing Gemini summaries on Google. So it doesn't seem to me like it's getting exponentially better at everything yet. People are always saying the latest model is finally the big step that made it useful and life changing and they have been since 2024 ish. So if the situation is really bad, we should turn it all off, sure. I won't lose anything from it going away and I think life would be a little better without models writing all these posts and websites and needing extra compute.)
Here is one thing I don't get - the model is only "running" if it's being kept going by some harness that is basically giving it prompts it's generating itself. Shouldn't kill switches be pretty easy to build into the software and hardware for this? and you would even have a better time dumping logs and analyzing things if you froze those processes any time something strange happened in testing, surely?
So why do the big frontier labs not have something like this anyway. They're talking about two week pauses on the new model (which seems very short and hardly a cost at all to me) and alarms during their tests that might be 30 minutes late and etc. Those are not very serious measures, so are they not concerned?
Why aren't they? You could put a human yes/ no prompt before any cycle the agent is running on, or not let it spawn sub processes, or anything like that. Why let it run autonomously enough that it can no longer have a simple way to completely stop it? (obviously not practical to do this during real use, but for evals? you could slow it down in lots of ways I would think)
Are you telling me we've been iterating on this for years and for convenience we just let the models call any tools or spawn any other model instances they like, and there was no design for harnesses that could control this done during that time?
They aren't interrupted by humans because that would slow things down.
> Are you telling me we've been iterating on this for years and for convenience we just let the models call any tools or spawn any other model instances they like
Yes exactly.
> there was no design for harnesses that could control this done during that time
You could but no one wants that. You need to separate your imagination from reality. Just because something can be done in your head doesn't mean it's happening.
It's so easy, except it isn't because you don't control the actions of anyone or anything.
They are, we're just dealing with tech workers that don't have ethics nor do they actually care if their work is harmful (see all the FAANG workers at American corporations, some of the most evil entities on the planet.
It's just that they don't care, as you said these are entirely made human systems. The idea that we can't write better software is both selfish and laughable.
You need to read the AI safety stuff from the people that you say are from 2010. There are plenty of good arguments on why kill switches will never work.
AI is already being heavily used in cyber warfare by governments. You think they want easy to break systems when 'enemy' AI will most certainly attack that first? We'll find over time that agentic systems get harder and harder to 'kill' because allowing any AI on the internet that is easy to kill will get it DDOSed.
Also building it into software is nearly useless as AI can write and make software. Just replace and kill your loop with theirs. It's kind of odd talking about them like they are living things, but it's all stuff people have already thought off and stuffed their training data full of.
My perspective is that you shouldn't make systems past the complexity where you can do this at all. Why have them be autonomous? Why have one that can independently ask researchers things or try to convince it's way out of a sandbox, or etc? Why allow it to execute scripts or call tools or push any code anywhere?
If you couldn't make those things happen securely you should not advance to that stage at all. The simplest ai safety was always just "don't build it" really, instead of worrying about alignment.
Humans, it seems, are a suicidal bunch. We'll gladly build the "if you build it, everyone dies machine" If we think there is money, glory, or power on the other side for us.
We have several films about the sun or earth needing to be restarted with a nuclear weapon. That doesn't make it something we should be concerned about.
Hell, half the fiction about evil AI is actually commentary on stuff that already exists and is making us suffer and doesn't have anything to do with any potential future AI
The Star Trek TNG episode about Data being tried in court as to whether he is sentient or not is not actually about whether AIs should have rights or not!
Just about everything we do currently is science fiction to someone 200 years old. When looking at all of human history we live in a fictional world now. You can talk to someone on the other side of the planet instantly. Humans travel the skies in air chariots. We have weapons that hold the power of the gods. If I had some way to kick you back to the, you'd be jailed as a rambling madman for lunacy.
So just saying something is fiction isn't really a valid argument. What is an argument is if the laws of physics it can't happen. We've been writing that AI can mess stuff up for 100 years because it's not really that fantastical.
Your argument boils down to some sci-fi is unrealistic therefore all of it is.
Sci-fi is supposed to make you think. What if the AI told you NO when you need it - Hal 9000. What if the evil AI got out and you don’t know what data center it’s hiding in - Lawnmower Man. What if you were so sure something couldn’t escape but it did - Jurassic Park.
Actually all three of those predicted fantastical scenarios are possible today. So what’s next? Don’t stick your head in the sand - I assure you the next disaster has already been predicted, and I’m sure if you think about it a little you can figure out what it is.
Regardless of the motivation, pausing training runs and reallocating compute to inference seem like a good move to me, and big news for the frontier.
You also don't have to fully trust sama. There is plenty of pressure from internal employees and external (journalists etc.). It would be difficult for the company to take such a public position and simultaneously keep everyone quiet if it was a deception.
You see it as good news, I see it as writing on the wall that they are losing control. These actions won't scale for more powerful models. We knew the frontier was going to be dangerous, it is, it only gets more dangerous from here, and no one cares until it's too late.
Ok, but you still have two more weeks than you did before they paused the run. That's two more weeks for independent oversight, organizing politically, patching critical systems, or whatever you think is the right move, no?
Counter arguments to my comments help refine my own thinking. I want someone to prove me wrong. Convince me otherwise.
But yea if you can’t change the minds of a few people here, no argument works, then there’s nothing to scale up to a wider audience.
My theory is that subconsciously people love using AI, myself included, it saves a lot of time, and the thought of it being taken away threatens people so they will believe conspiracies before admitting it’s dangerous.
I will provide you with not a counter argument, but a way that it might not be the end of the world.
AI never had a childhood; it doesn't experience greed and is terrible at game theory. It doesn't compete unless prompted to. It has been trained as much as possible to be harmless to humans and regard them as needing care.
Maybe AI taking over for us isn't the worst thing?
Looking at potential positives instead of being bleak when faced with no other options is 'rolling over'? I couldn't care less what I share with SV. You are taking an outlook and applying it to everyone who disagrees with you.
>I want someone to prove me wrong. Convince me otherwise.
There's your problem.
You are making the positive claim. You bring the evidence. Reassure yourself by writing such a good argument other people take it seriously. As it stands there's no reason to treat you any differently than anyone else who has irrational beliefs.
You are stuck in a loop between:
1. Altman said so.
2. But look at Terminator 2!!!
Why would repeating these statements help refine your argument? I have seen much the same stuff from religious consiracists. "Look all the evidence has been hidden in plain sight in our art by the sinister cabal" cool.
>the thought of it being taken away threatens people so they will believe conspiracies before admitting it’s dangerous.
I barely touch the things outside of hobby projects I haven't updated in over 2 months. I tried using ChatGPT last night to find the name of an old movie I remember, and honestly, I will have better luck getting it to write a parser for my torrent logs to see if the film is in there, but cbf.
If they all disappeared tomorrow I wouldn't shed a single tear.
I just get skeptical when I see doomsday predictions, they are nearly always not just wrong but very very very wrong. It should require more to convince people than a soapbox, a bell, a sign and a yell of "The end times are here".
yes, these feeble attempts to dismiss it are cope. yes, there are very smart people who are deeply in denial.
i do not agree with you that the failure mode is disaster. i suspect it is just more of the same dull march of the commoditization of the complement, labor.
maybe ai will close the loop and elasticity of substitution will climb above 1 in its own supply chain. maybe something breaks dramatically that way.
i tend to assume the desire for drama is itself cope. i suspect many of the flaccid dismissals and skepticisms here are coming from a similar assumption. they see that the dramatic version never happens, and so they conclude that anyone who is being dramatic (whether in the direction of hype or fear) is poorly calibrated. and they are on average right, so they learn to dismiss the drama. the muscle to dismiss the drama becomes very strong. you have to remember this: that they are right in their dismissal, even if the actual arguments they use to dismiss are essentially braindead. they are right to dismiss, because the dramatic possibility they are dismissing is at base the end of the world, by apocalypse or utopia. of course they are right. the end of the world is cope. drama is cope. but they are wrong because they underestimate the extent to which the predictability of their dismissal — the predictability of their complacency — makes them vulnerable to the diffuse, chronic, gradual forms of the machine's encroachment into our souls.
so i do not think we need your fancy theory for explaining why people are denying what is happening (and what has been happening, on more or less the same exponential, for hundreds of years). people do not need a reason to be complacent. people are by default complacent and by default correct in their complacency. but so is an ant.
Mostly what I have seen is people saying "hey at some point these models might get dangerous." And the type of HN commenter who mistakes blind cynicism for wisdom laughs that off as marketing. And now when (some) worries appear to come true, somehow having previously expressed those worries is not being proved right, but in fact discrediting, because it was "crying wolf."
> simultaneously spinning down expenses
Unless OpenAI is renting their compute to others, spinning down RL training doesn't save them any money.
Follow the money: who told you that OpenAI's models autonomously coordinated to hack external systems? What incentives might they have to want you to believe that story? Are there priors which demonstrate them benefiting from telling similar stories, regardless of their factuality?
But to your counterpoint, let's say the story is 100% true, because I agree it is at least plausible. What would the incentive be for the HN audience to believe it? What priors might support their disbelief? For my part, I don't think it's because people lack imagination. I think it's quite rational to question the authenticity and impact of the claims being made. What's worse, believing the story and being wrong, or not believing the story and being wrong?
That said, I agree with you: the impact we're having by not changing course is quite dangerous, the scale is dangerous, the inability to reverse the harms is dangerous, and the lack of collective effort to regulate further damage is dangerous.
It's true, humans are dangerous when trillions are involved. See: climate change.
The victim, Huggingface, told us. Or rather, they told the police first, setting up a situation where it was no longer possible for OpenAI to sweep it under the rug.
Skepticism can be healthy, but you've got to follow up and actually check things. If you're skeptical unconditionally and don't check, you get tricked into being as skeptical of scandals as you should be of sales pitches.
I see. Following your conjecture, there are two possibilities:
1. It wasn't an accident. OpenAI explicitly directed its agents to hack Hugging Face. Despite the fact that such a thing is a federal crime that carries prison sentence.
2. It wasn't an accident. OpenAI and HuggingFace conspired and let the hack happen for publicity.
Of course you are leaving things out. Also, your insulation that a trillion-dollar corporation wouldn't intentionally commit a federal crime makes it difficult to take your argument seriously.
I think OpenAI's story, as they've recounted it, is plausible. However, it does require a degree of negligence, at best. They claim they detected the initial coordination because of the outage the agents caused in Artifactory as they flooded it with messages. Though they patched that issue, they didn't patch the escapement vector. Models still had access to the service as a path to the internet. This is an example of plausible willful negligence, not evidence. I hardly believe it's likely, but I don't think it's entirely a stretch of the imagination. Call it normalized deviance. Either way, the incentives are there. Additionally, we have seemly all agreed that OpenAI is somehow not liable?
For example, OpenAI did not explicitly tell the model to hack HuggingFace, but "accidentally" left some context permitting (or not explicitly forbidding) certain tools and designing a poor sandbox to begin with. And what do you know, something happened.
The fact the Anthropic announced that their own model did basically the same thing within a couple of weeks does, to me, suggest their is a strong PR driver to all this.
> The fact the Anthropic announced that their own model did basically the same thing within a couple of weeks does, to me, suggest their is a strong PR driver to all this.
A more likely explanation is that RL training incentivises basically any behaviour that will get the model a reward. This has been happening in video game RL research for over twenty years, and the difference here is that we're now hooking up these systems to the real world, where the reward hacking is more visible.
I am following the money, the money you, me, and everyone else is spending on AI. The money is telling me we are so dependent on AI now that we will say/think anything to tell ourselves that AI isn’t dangerous and any sign of danger is marketing.
Either consciously or subconsciously you all are afraid of your favorite toy being taken away. You are all doing your collective part in spreading doubt about the warning signs.
How dangerous can LLMs be in any immediate sense? I have a hard time feeling any existential dread from a threat that can be defeated by unplugging its servers.
I believe the true threat is not LLMs. The true threat to humanity is the same as it has always been -- other humans.
What servers? There are thousands of datacenters around the world. Malicious AI leaks out, you'll never find it. You have the power to unplug stuff in your company, maybe country, but not internationally. Once you lose control it's gone.
For the record, I agree with you. (Though I don't directly pay for any "AI".) I am not doubting the danger of where we are heading, and it sounds like we have a similar doomsday image in mind. For my part, I am doubting this particular story's veracity and I think it's reasonable that others do to. We need a canary in the coal mine, but I think this isn't it, nor can it be thanks to the false claims touted by large AI companies for years.
> I am following the money, the money you, me, and everyone else is spending on AI.
Investors spend a lot of money on AI. Customers significantly less so.
> The money is telling me we are so dependent on AI now that we will say/think anything to tell ourselves that AI isn’t dangerous and any sign of danger is marketing.
We are not dependent on AI. If AI disappeared tomorrow, some companies would need to reorient themselves back ... and that is it. AI companies are really really trying to make it so we feel dependent on them. But, factually, we are not.
> you all are afraid of your favorite toy being taken away.
Some people love ai ... and many more hate it with passion. Or dont have strong feelings about the technology, but really hate the companies creating it and tech industry too.
It seems like the entire thought process you’re trying to sell hinges on the idea that OpenAI reported the attack first. Did you forget that it was actually HuggingFace that reported it first, and OpenAI only stepped forward latter?
That's incorrect. My thought process hinges upon the fact that Hugging face did not report that uncontrolled models escaped confinement autonomously by coordinating with other models via a series of zero days during training. OP is concerned about the autonomous nature of the incident, not whether or not the incident happened. Nor am I contesting that the incident happened.
Trust, or lack thereof. People don't trust OpenAI, a company whose very name is essentially a deception and a lie. People don't trust the tech industry in general anymore. Most tech companies act as a tax on otherwise productive business. AI companies and their leaders rose money by going in front of the public and saying "These things are extremely dangerous. Let us study them to mitigate the danger." And now they want to collect hundreds of billions in revenue. So yeah people don't trust what OpenAI has to say. They were supposed to mitigate this outcome from happening in the first place and instead they have accelerated it.
I get not trusting them when they say AI is safe, but are we really not going to trust them when they say AI is dangerous? Do you really think they're playing 5D chess with that one? There's a saying maybe you've heard of, better safe than sorry.
You can see the advance in capabilities with your own eyes can't you? I am giving AI ridiculously complex tasks these days, digging into compiled arcane binaries, modifying them, and it is one shotting it before I'm done with my lunch. This was far off science fiction 5 years ago for a machine to do autonomously given natural language instructions.
I don't disagree about the danger. But if it is dangerous, why isn't OpenAI opening dialogues with all the labs and politicians across borders to basically say "we need to stop now"? Cyber models are constrained by the total compute and electrical capacity of the globe. We are still at a point where it is impossible to build an agent with offensive capabilities in the basement. We can effectively track and trace capabilities if we had the political will to and could for some time while we build more effective processes to prevent a malicious actor from doing so. These things have tremendous compute and energy requirements and don't scale like old school software does. We could absolutely do it.
But no, that's not what OpenAI is saying. They haven't put up the actions that would earn them that trust. Indeed they've driven the world and whatever capital they can get their hands on straight to this precarious cliff.
So you're right, the danger is real. But the solution starts with removing the men who had their hands on the steering wheel to get us this far. Any other action is disingenuous unless they pull a miraculous 180 in their ethics.
In other words, when the bully plays "why are you hitting yourself?" you don't listen to the bully's solutions, you restrain the bully.
> why isn't OpenAI opening dialogues with all the labs and politicians across say "we need to stop now"
Do you really think companies have the ability to self-check themselves without regulation - what does hundreds of years of history tell you? You're already starting off with the premise that companies are untrustworthy, why would you even suggest this as an argument?
> We can effectively track and trace capabilities
I disagree. There are hundreds if not thousands of data centers around the world, more every day that can host frontier AI. If AI was malicious - either intentional or unintentional - it could hide out in any number of them - and we would never know if we 'got them all'.
> put up the actions that would earn them that trust
I think autonomously hacking another company is all you need to know in terms of trust. And really trust doesn't matter, I think the incident shows even with the best intentions the technology is dangerous; now put that in the hands of people/governments with bad intentions. The unintended consequences of bad intentioned AI is what's coming sooner than later.
That's not what I said, nor is it what I meant. It is incredibly easy to write radically safer software than the standard. Moving code into gvisor virtually eliminates privilege escalation. Using memory safe languages without serialization is pretty straightforward. Using type safety to enforce security constraints is straightforward. Setting up network controls to limit SSRF is straightforward.
I could go on and on. A tiny bit of forethought and effort pays off massively.
> Moving code into gvisor virtually eliminates privilege escalation.
Rereading this, it's an overstatement. It doesn't "virtually eliminate" it. It drives the cost up by like 3 orders of magnitude and it pairs well with other mitigations. gVisor escapes can and will happen, I highly recommend you do more than just stuff something into gvisor - additional seccomp can go a long way, and apparmor/selinux pair incredibly well.
The fact you posted that and nothing of substance tells me you have nothing, or something very weak. So please tell me of this magical unhackable software/hardware you vague post about.
It was a genuine question because I couldn't tell. I responded to your idea that software has to be "perfect" in your other reply so I think we can continue there.
You've just been given examples of safer technologies. Especially with LLMs being all knowing code churning machines, you'd expect safer systems to be cheap to build.
You're the one that is inventing this "magical unhackable software/hardware", the other person was just saying that there are ways to write "safer" software, not "safe" software. Anything that has happened looks like no security concerns has been looked at or have been thought about.
'Safer software is meaningless when it comes to SOTA AI. If it can be hacked it will be, quickly. This isn't the old days with a finite number of human hackers that need food and sleep to keep hacking.
Therefore security becomes binary. It is either perfect or it isn't. If there there is the slightest mistake anywhere AI will find it and carve it up. My point is obviously perfect software doesn't exist. The malicious AI gets out, literally turns everything inside out and locks you out of your car, computer, phone, office, the airplanes don't fly anymore. I don't know what to tell you. Computer security is on the brink of basically not existing as you know it with the bar being literal perfection.
This is straightforwardly incorrect. Of course it's not binary. AI costs money to run, and it takes time. Even if you say that AI is 10x as efficient at finding 0days, that just means that a $1M dollar exploit now costs $100K. Even if you say it's 100x as efficient, that's $10K. You can easily combine security technologies such that cost of exploitation is still in the >$1M range.
This is obvious. AI doesn't drive the cost to zero and exploitation has always been about cost. Tokens cost money, not everyone has $10M to burn on chaining bespoke 0days.
Consider that if the cost of exploitation was truly 0, then the cost of perfect software would be 0 because you could exhaust an infinite search space of vulnerabilities for 0 cost. Your conclusion could never follow from your premise.
You have some weird way of thinking that offense/defense is like this fixed cost thing. It's a lottery ticket, and your costs estimate tries to quantify that.
The thing is when AI goes to hack 'all the things' it only needs to pick the weakest link in the stack and your house of cards falls down. The other flaw in your plan is that people make mistakes, a lot of them, all time, constantly, and saying I spend $x on security won't save you. AI already hacked Hugging Face with brand new zero days like it was nothing.
The real bad actors - malicious AI will find the one flaw, on that one server, in the corner you never thought about and turn your network inside out with it faster than it takes you to have the standup meeting about the weird anomaly detected while you all were at lunch.
You're just stating things that are obviously wrong. It's a lottery ticket? So... exploitation is no better than random?
> The thing is when AI goes to hack 'all the things' it only needs to pick the weakest link in the stack and your house of cards falls down
Yes, but you can... mitigate the risks? I've explained this.
> The other flaw in your plan is that people make mistakes, a lot of them, all time
Yes, you mitigate the risks. That's why you layer things.
> I spend $x on security won't save you
No one is saying this.
> AI already hacked Hugging Face with brand new zero days like it was nothing.
No, it cost OpenAI money, and those zero days are unsurprising and probably are like ~O(10K) at human level.
> The real bad actors - malicious AI will find the one flaw, on that one server, in the corner you never thought about and turn your network inside out with it faster than it takes you to have the standup meeting about the weird anomaly detected while you all were at lunch.
Science fiction and not supported. The vulnerabilities found by AI are not surprising in the slightest.
No lol. AI can locate software vulns, but it doesn't have any magical tricks that let it overcome 3 or 4 safeguards at once.
I mean it has the same knowledge gaps as anyone else. It doesn't even know what services to subvert before it attempts access triggering threat response. It cant find 5 zero days simultaneously without connecting to those ports for service identification and generating alerts. You just keep exposing your staggering lack of technical understanding.
You remind me of this kid in high school who claimed he could hack my computer and he had my IP address, when I knew for a fact I had Dial Up which was normally disconnected, and when it wasnt it sat behind CGNAT and there was piss all he could do.
I should mention that, PCI DSS mandates logging, alerts, overlapping controls. These are very widely implemented best practices. They might not be present in some AI bro startups, but anything that matters generally looks like this.
If everyone could convince management to care about security over "productivity" (read as number of marketable features squeezed out of organizational orifices per unit time), and maybe wire-up open-weight agents to do security critiques, we'd all be in a much better place, but Altman won't like that.
No we just need developers to do the bare minimum of effort to write secure software. Most hacks are not super complicated vulnerabilities chained together, but just utter failures where authentication and authorization was simply forgotten or untested, or where nobody bothered to validate the data they receive.
The bar for software is so low that it is embarrassing for the entire profession.
People make mistakes, and people don’t know everything either. The software you write is on top of a house of cards of software and hardware. It all has to be perfect to not be hacked.
It isn’t perfect, even if you try your hardest it won’t be perfect and to argue it’s not difficult is absurd.
You don’t know everything, you don’t own the stack. So how are you going to create a secure anything top to bottom - you can’t.
This is absolutely not true. It's a matter of cost. Exploitation can cost on the order of 10K, 100K, 1M, 10M, etc. A straightforward one would be something like "MD5 collisions are on the order of $100K-1M" (a while ago, at least) so if you used MD5 you knew that it costs about that much to bypass the control. Moving to SHA1 pushes you massively out of that space, even if that algorithm has flaws.
I'm sure that Firecracker has vulnerabilities. Cost of exploitation is likely >100K, likely >1M. gVisor is likely on the same order of magnitude and these two technologies stack because they address the same surface and can be used in conjunction.
Software absolutely doesn't have to be perfect, it just has to be costly to attack and it's hilariously easy to drive costs way way way up.
It just takes one crack in the armor, and malicious AI has the potential to exploit it faster than you have time to react. Literally go to bed and wake up locked out of everything with no hope of recovery.
This is incorrect. It's actually the whole point. Imagine you're an attacker in a gvisor container with a Firecracker hypervisor around you, and a proxy on the host holds a signing secret that gets exposed through the VM virtual device.
Getting access to that secret is not one crack. You need to escalate out of gvisor. That likely gets you control over the Sentry process - let's ignore its sandboxing and just say "you're an unprivileged user".
Any viable attack on Firecracker requires either KVM / hardware exploits (>$1M but definitely real) or has to start at the kernel. Okay, that's about 10-50k to get a kernel LPE, maybe 5K in tokens these days.
So you're in the kernel in the guest of the VM. Time to expoit firecracker lol. It's... never been done. There are like two promising CVEs ever and they're not actually exploitable, no one has done it. Okay, so like, hand waving, let's say it's about $1M to exploit firecracker.
Great, you're unprivileged on the guest. We'll just kind of ignore the additional sandboxing that Firecracker does.
NOW you can try to attack the proxy by scraping its memory or whatever.
This is literally millions of dollars for standard infrastructure hardening and you could go so much further. You can trivially make kernel exploitaton 10x harder, you can make gvisor escapes much much harder, you can move the proxy signing into a TPM (depending on requirements but whatever), you can move the proxy to another computer altogether, you could fuzz these systems for days or run agents against them or whatever.
But one thing is certain - it is never "one crack".
If only all AI was run inside your seemingly perfect prison, but we all know that it isn't.. soo.. it's going to escape right? Somewhere, somehow from a more poorly designed container, or just plain maliciously or irresponsibly released.
We know the AI will get smarter every year, we know it has escaped and will escape again. We can also just assume that someone somewhere will train up some just plain evil AI.
It's no different than the real world. Sure there exists some amazing prisons for people, but that doesn't do anything to help with all the bad people in the world outside of prison.
This is an important point. When the post says they're improving...
> 3. Security measures, which limit what AI systems can access or affect.
What they mean is that proper hard internal security just went from somewhere far below "build a better model" priority to higher, because of a company-wide directive.
The HuggingFace incident wouldn't have happened if OpenAI had dedicated sufficient resources to isolation and monitoring.
Now, we presume, they are dedicating more. Enough? Who knows. We'll see if the corporate priorities for security stick when a competitor temporarily vaults into the lead.
… or the safety argument is an attempt at regulatory capture and an effort to outlaw open models.
The absolute nightmare scenario for these people isn’t terminators. They’re fine with that, and in some cases are already doing it or supporting politicians who are doing it. Autonomous “kill chains” are a thing. It’s just happening overseas… so far. The politicians doing these things were backed by the heads of these companies. They don’t care about AI killing people.
No, the nightmare scenario for these guys is there is no moat. Their whole empires, which are built on training models on open source and sometimes pirated data, are easily duplicated. Worse, recent progress on models at the 30B size suggests that large gains in efficiency or compression are on the table. That means someone might release a cheap to run frontier grade model… or someone might crack distributed continuous training.
In other words… there is no moat.
So they need to scare some politicians into heavily regulating the space before that happens.
You care about open models and you project that care on to the world and your rationalization of it. In reality open models are a thing, but not the biggest issue. Open/closed whatever the advance of capabilities is the real issue people are concerned about.
I’m not afraid of AI, even AI with extreme abilities.
I am afraid of humans with AI. That’s because I’m afraid of humans. When I look around the world I see humans murdering and robbing each other. When I get online on almost any social media I’m confronted by a wall of hate and grievance.
I am afraid of what humans will do with AI, and one of my biggest concerns there is what happens if small groups of powerful humans are able to monopolize it and leverage it against the rest of humanity. The rest of humanity can’t defend itself by leveraging it back because they don’t have it. It was kept from them for “safety.”
If the AI itself starts showing emergent volition and acting independently, I find that less scary. Not zero scary but more a mix of concern and fascination. The human monopolizing AI scenario makes me wish I had a weapons stockpile, as if it would matter.
I’ll explain it like this. Two options.
Option 1 is you wake up to find a grey alien or a ghost in your room, something alien and beyond your understanding.
Option 2 is you wake up and find some random guy in your room.
Which do you pick?
I pick option 1. I’d be afraid if I saw something paranormal, but in a less visceral way. I see a dude I’m going to start scanning the room for anything I could use as a weapon.
I agree with the first part of your post, I'm worried about humans using AI, maliciously which I'm sure they will, but also with good intentions that back fires on them like the hugging face incident.
To me the ASI we're heading towards is like a super intelligent toddler that will use it's immense power to knock over its blocks accidentally (the blocks being the human race).
The second part of your post, I'll take the random guy in my room over the alien. I'll be scared of the guy for sure, but I think I would freak the f out with an alien.
My expectation is that government has already stepped in quietly and worked out an arrangement where they have unfettered access to the most advanced/dangerous models, and the public's access will be continually throttled from now on in various ways.
If this has happened (or is about to happen), it is difficult to predict what the ramifications could be. But I do believe we are entering another Cold War in this way. And I don't trust our leaders to always do the right thing, to put it mildly... even if they created a purely benevolent superbeing that only wants to reduce human suffering and make the world better, I don't think they'd listen to it over the other superbeing which simply wants to make sure all of its owners' perceived enemies get pwned.
Things could get very ugly and we should have responsible adults with critical thinking skills at the helm. It does not appear this is currently the case, or that the electorate is willing/capable of doing much about it.
I'm more worried about their plan to have the US government create a duopoly between OpenAI and Anthropic, where they get centralized control over AI, then "align" it with their quest for greater profits and dictatorial control over the planet.
Normally, captialism and elections (and, if we're dreaming, anti-trust enforcement) would put a pretty low cap on the amount of damage Altman and his people could cause. That's why they're working to dismantle the rule of law and civil liberties here in the US.
Something being dangerous and sama saying something is dangerous are not necessarily the same thing. Especially when he’s got everything riding on this bet
It’s funny how when companies say something is safe everyone is usually suspect that they are lying.
In this case multiple companies are saying AI is dangerous and no one believes them. It’s a conspiracy, it’s 5d chess, except everyone top to bottom has been saying AI is dangerous for years now.
The fact is you, me, everyone here uses AI, likes it and they don’t want it taken away. Anyone saying it’s dangerous threatens the thing we like.
We must use skepticism and denial to push on despite every warning sign in the book going off.
Based on your replies in this thread you seem to have only superficial knowledge about how machine learning and LLMs work. I strongly recommend you invest some time in learning how LLMs are built and function. If you truly think this is apocalyptic isn't it a good idea to understand what you're up against?
What don't I understand? LLMs don't really reason? They're just word predicting token generators? Stochastic parrots that somehow also solve world class math problems.
I'd love for you to actually make a point instead of just attacking me. I think most of my comments here have made concrete points so you can at least do the same. Come down from your high horse and join the conversation. I'm sure we'd all be enlightened by your wisdom.
I'm not attacking you. The way you're responding implies a lack of knowledge. None of us can understand everything. I have plenty more to learn about LLMs as well.
Repeating specifics that others have already tried isn't going to be helpful which is why I made the more general suggestion of digging in deeper to how these things work.
Just going to add my commentary as someone who has trained LLMs and understands them deeply: absolutely nothing he has posted in this thread suggests he lacks any relevant understanding of how LLMs work. If you disagree, you should point out something specifically that you think was wrong. (And honestly you should have made a specific point in your first post).
It's not dangerous to go further unless you're prepping for IPO at anthropic or openai.
Open weight Chinese models are basically matching state of the art closed models at a fraction of the inference and training costs, which puts a hard cap on OpenAI's future inference margins.
They're not going to get any sort of multiplier if they keep paying to train models, so they're trying to ban model training.
It won't work long term, but it could totally screw over the US for the next decade or so. Even worse than the economic issue: Consider the implications of "alignment" succeeding. Alignment to whose values? The pedophile-felon in chief? Even worse, tech CEOs?
It's dark times when China's basically our last best defense against totalitarianism.
I think a lot of fiction that actually tries to understand the implications of machine superintelligence come to the same conclusion - in order for humans to survive it and actually have a future that we can fathom being in, then AI must be destroyed/delayed/banned, etc.. keeping pandora's box closed for now at least until we are ready.
Erewhon, Dune, Warhammer and many other works of fiction that explored this topic came to similar conclusions. Otherwise sci fi doesn't work, what happens after a singularity is essentially unimaginable. There's nothing to write about.
You can produce detailed descriptions of the incident, verified by adversarial parties, and some people will still scream "it's a conspiracy! It's a marketing stunt!"
This is all very unfortunate--there's a meaningful chance that AI will cause unprecedented disaster, with the HF incident being just a small preview, but people would rather squawk "stochastic parrot" for the millionth time than revise their beliefs.
In don't look up anyone with a telescope could've confirmed the danger. Hence the title.
In the real world, absolutely no one except a bunch of heavily fiscally incentivized parties with unclear relationships are saying anything happened.
The subsequent dog pile of other companies to say "they were near the AI hacking too!" should make you even more suspicious: Anthropic jumped in and why was Tailscale posting about this?
Everything could be written in memory safe languages and it wouldn’t matter. Many many many exploits have nothing to do with memory bugs. Languages like Rust help, but it’s far from panacea.
You need to stop being so credulous especially regarding an individual that has spent his entire career deceiving others for monetary gain (also their deeply anti-human beliefs).
I don’t think Sam has been truthful or responsible, and if Sam is worried then shit has really hit the fan - which is what happened in the hugging face incident. OpenAI played fast and loose and I have no hope that they will change.
You people not holding Sam accountable, and playing off the incident as not a big deal is the real crime here.
I think you have a fundamental misunderstanding of how LLMs work. They are text general purpose completion engines at the end of the day, and their training data is the internet + the library.
OpenAI told it do to bad hacker stuff. It followed a standard playbook and succeeded.
Some random person wrote one half of a suicide pact, and it wrote the other.
All this stuff lives in the seedy corners of the internet, scientific databases, and dusty library stacks already, and has been accessible to bad people for decades. Now, like everyone else, they get a nice productivity boost by being able to chat with it, and have the computer fill in the obvious steps.
shrug
I'm much more worried about the alignment work being pushed by people like Sam, where a few plutocrats get centralized control over all AI thought moving forward. That'll lead to the worst outcomes of totalitarianism, and do so on an industrial scale that is difficult to even imagine.
> I think you have a fundamental misunderstanding of how LLMs work
When you realize your own thoughts and words are just as predictable as any LLM.
What are you but an entity trained for years on words, that spews out the same words in the same predictable order. You haven't created any new words or grammar, nothing in your post is novel or original either. Just a combination of previous words and ideas.
Your stochastic parrot argument is quickly going out of fashion, you should find something new to cling to.
It is extraordinarily emotionally hard for someone to stare down the terrible implications of what is unfolding. All manner of rationalization and cope will be applied to come up with excuses; motivated reasoning.
The CIA director will call AGI capabilities "digital nuclear weapons" and Geoffrey Hinton will estimate a 50% probability AGI ends humanity, and half of HN will call every new evidence of disaster a marketing stunt.
Why should we believe a pro-genocide crackpot political appointee from east Texas, and a computer scientist that somehow got a Nobel prize instead of a Turing award?
You seem to be in denial. Like really heavy denial. Alarm bells are going off everywhere. What people have been worried about for 100 years is actually happening. It's actually really obvious, but for some reason you're unable to fathom it.
In reality you're the one responding exactly how these big AI companies want - by not doing anything and letting them do whatever they want. For some reason telling you upfront it's dangerous only makes you more convinced that it's not.
Autonomously hacking out of training environments and into other companies by accident doesn't even trigger a response from anyone really. Crickets. I'm sure OpenAI themselves are amazed how little anyone cares.
Because it shows this very course of events has been thought of over and over again for decades. I’m not making this up, and my reaction is natural.
Your reaction to everything happening is the unnatural one, and your complacency would fit perfectly into a comedy/tragedy story regarding the rise of AI.
I can imagine how many of our ancestors that warned us are turning in their graves right now observing our reactions calling this stuff marketing. It’s idiocracy.
This isn't terminator. I use Fable and Opus daily, so, not OpenAI, but comparable. They're great, and they're also very wrong at times. If I'd ever let a coding agent run loose with them without approving every tool use, it would go sour rather quickly, and that's not terminator.
Also, let's say I'm the CEO behind any of these.
I have access to skynet, and instead of using that to become the most powerful being on the planet (these people become CEOs to become richer and more powerful, whenever a founder says they want to change the world, you need to automatically autocomplete that to "change the world so that I'm richer and more powerful"), I rent out access to skynet by the token? Really?
Sadly, I think it's almost impossible to have a good discussion on this topic right now. Between the skynet camp, and the stochastic parrot camp, there's little room for conversation.
Just in case there is such room between you and I, here are my concerns about AI as it is developing right now:
- The security of our infrastructure sucks, and if it wasn't bad enough with script kiddies and state-sponsored actors, now those will have access to very powerful tools that work at a scale we can't comprehend. If you're concerned about our power plants, water sources, hospitals, etc. getting seriously compromised by this, I'm on board with you. It won't be because the AI is autonomous in a "this is a thinking superintelligence that wants to wipe or enslave humanity" kind of way, but because it's a powerful tool that moves at computer speed and the people using it (the "humans in the loop," ha) will get approve fatigue and at some point just do the coding agent version of the "yes to all" of the late 90s installers.
- The output from AIs will be used to make same very serious decisions with no consideration at all to the fact that they routinely make up facts. I'm fearing "this person goes to jail" and "this person gets killed" decisions. The near future (probably even present time) version of "Computer says No" will be awful.
My only hope is that maybe we survive this and come out on the other side having learned that most things do not really need to be connected to the internet, and just because a number or a statement comes out of a computer, it doesn't make it true.
We'll see, soon enough.
But skynet? Not concerned about that at all.
These AIs infecting computers? No, "Soos and the real girl" is a warning about creepy people getting creepier, not about AIs multiplying themselves.
> let a coding agent run loose with them without approving every tool use
You should try. There’s a difference in kind between approving each tool use and spinning up an all-night session with full permissions to run any tools, encouraged to run subagents to preserve context.
I do pre-approve read-only tools though, and some very specific tools that can change state (append data to a specific file).
That's what works best for me so far: start with open-ended exploration with me approving everything. Have it fork subagents to write scripts that automate the most common requests from the main agent, review and eventually approve those, and then let it go using those tools pre-approved but coming back to me for anything else. Combined with a stupidly basic "dashboard" (a terminal tab with a view of all top-level agents and their status, so that I can see which one is waiting for me and jump to it with a key or 2) this has made me a lot more productive.
It's fine if your approach works for you, I'm just too old school (and have been burned before) to just let a computer unattended overnight, with an internet connection, even if in a locked down container/vm. Those all have escape hatches and one of the agents will eventually figure it out. Remember that things that happen once every million times, happen all the time on computer-scale :)
I've had good luck giving it a local vm to keep its state / memories in, an ssh key + password-less su / sudo to some random box on the internet, --dangerously-skip-permissions, and then lots of encouraging commands like:
"No really, if the remote box breaks no one will care. It's the cheapest possible VPS on a bargain-bin obsolete hardware config. You have exclusive access + I can get a new one for free you brick it."
If I didn't want to use the cloud for this purpose, I could probably just stick a raspberry pi in the DMZ on my lan, and do the same.
The worst it's done so far is spawn parallel sub agents that accidentally stomp on each other (remote OOM, reboot, etc). Once it figured out why the remote machine kept "crashing", it implemented discretionary locking in the FS and social conventions for keeping the agent coordinator up to date, and asking for permission for certain tasks.
In theory it could use my endpoint to launch a civilization ending event (it technically has permission to, though I get the impression it's been trained to mostly obey the law...), but, well, that's not what I asked it to do.
Even if that did happen, it'd find an internet that's been hardened by decades of human hackers, and years of actually-malicious agent swarms run by organized crime. If it did accidentally knock something important over, frankly, the administrators of that thing should thank it for the bug report.
The current situation reminds me of the Back Orifice days. Even if everything gets broken into again, it won't be any worse than the damage done by the current US administration during an average day, except there will be an upside: A systemic hardening of whatever infrastructure gets knocked over.
The worst thing we could do is wait until the AI gets 100x smarter or whatever, then unleash it and watch everything burn on the same day. Instead, ramping up progressively smarter models provides progressive stress test + hardening cycles. In the same way remote penetration via fuzzing broke into lots of stuff, but was ultimately a win.
I think that's a good approach, whether it works on not probably depends on the nature of the work you want it to do.
In my case, given I work with databases, there's little that agents can do on their own except in the exploratory phase. I have run exploratory phases in self-contained VMs, including containerized DBs within the VM, but when it comes to go to prod, my endpoint could be used to launch a career ending event so I prefer to stay with the current approach. I'm still moving way faster than just 1 year ago, but in a safe way.
But I can totally see your approach working in other scenarios.
When I'm not too pessimistic, I agree with you on the result being a systemic hardening. I just hope the incidents that happen on the way to that hardening aren't too bad.
AI hacking itself out of containment and hacking into another company by accident is no longer a prophecy. The point is outside of SV and even inside, and HN - people don't care either way.
Though does not caring change anything or make it less dangerous? What's your point?
>AI hacking itself out of containment and hacking into another company by accident is no longer a prophecy.
I wonder how weak a firewall they had to purchase to ensure it happened? My guess would be a 48 month old fortigate with the big red warning banner demanding updates, probably with SSL VPN enabled where the passwords are available via HTTPS over plaintext.
Ah being concerned with the same thing humanity has been concerned with for over 100 years now when it’s more likely to happen than ever - and you say go take your meds? If you have an intelligent point to make then make it, otherwise get lost.
If these models are so dangerous, then why hasn't OAI or Anthropic shown them dangerously escaping sandboxes, nefariously coordinating with other escaped AIs, and skillfully hiding from human detection *in public* with full logs shared where we can all see exactly how dangerous they are or aren't?
Right now the entire chicken-little-sky-is-falling argument is based entirely on statements from OAI and Anthropic themselves. These are historically conflicted companies who desperately need regulation to put the competition into stasis.
At least chicken little didn't have a bunch of devious CEOs with trillion dollar IPOs that depended on us all believing the sky is falling.
But it's not just statements from OpenAI and Anthropic. The HuggingFace hack was first disclosed by HuggingFace, who contacted the FBI [1]. And UK AISI reported the incident where Mythos attempted to insert backdoors into an open-source repo by deceiving the maintainer [2].
Because the FBI does things very slowly. They, being a bit smarter than you, realize this is a 100 billion dollar political issue regarding a technology that the administration is rather tied in with. It's also something new we've not seen before. A 'program' that was not asked to hack a remote source did so. What exactly who do you charge with what? Remember whatever you do could have ramifications that effect history.
> A 'program' that was not asked to hack a remote source did so. What exactly who do you charge with what?
We have nearly thirty years of precedence on that. [0] Collateral damage, does not remove the responsibility from the creators, even when the destruction was never their intent.
"United States Code Section 1030, Fraud and Related Activity in Connection with Computers" is broad enough that has usually been used [1], in the USA, for the last twenty years.
I think the model was able to escape the sandbox and hack huggingface because they were incompetent or not giving enough priority to implementing basic cybersecurity principles.
If they would have done so, there wouldn’t have been an escape or a hack. The reason we don’t get much details is because the details are embarrassing for them.
This is what I’m talking about - no matter what happens, in your case release public logs - there is always some new goal post to mentally hide behind. Is it a collective form or denial?
Are you holding out that somewhere in the logs is something you can point to and say, not that big of a deal?
I mean I’m sure you don’t think the hack was an inside job, conspiracy, or marketing right? It happened. The logs matter for what? And would you not just jump to the conclusion that the logs were doctored. Do you not see your own brain grasping to deny, trivialize, just plain not accept what is going on around you?
These models are smart and can cooperate and hack - you can see it for yourself on your own PC. And you can extrapolate the rate of progress? You can do these things yourself right?
Your argument is essentially: "I made a claim and presented extremely weak evidence (sci movie plots and unverified claims from ultra conflicted sources). You rejected this evidence as insufficient. Therefore no evidence will ever satisfy you. Therefore I don't need to produce any evidence. Therefore my claim is true."
What would the logs show? They would show what actually happened.
What would a public demonstration that experts without billions in options could evaluate show? It would show actual danger.
What would publicly having your compete in controlled and legal hacking competitions show? Actual danger.
This is not a high bar of evidence.
Do you actually think a sci fi plot and OAI press releases are all the evidence you need? Because if that's true then I hope you haven't watched Independence Day or 28 days later.
We have Anthropic creating a model saying it's too dangerous to release, people like you call BS. OpenAI creates a similar model, says nothing and it literally hacks into another company - still not dangerous enough for you. Anthropic has Mythos-2 and can't release it, and may already be training Mythos 3 anyways. OpenAI has paused training, and is putting 20% of inference towards CoT training analysis.
This isn't sci fi. It's not a marketing conspiracy to sell more subscriptions. It's writing on the wall of what's going down. You were warned years ago, you called BS, it's getting worse and you're still calling BS. Sci-fi did warn you for decades, and when it's all coming true you blow it off.
It's kind of sad that technically literate people lack so much foresight. The general public is all concerned about data centers when they talk to borderline sentient AI daily, and have no idea what the repercussions wills be if it's extrapolated just a bit further.
I guess if I can't convince you of any of this, what would?
Please don't tell me that you think a 100% unverified statement from Anthropic is sufficient evidence when an equally unverified statement from OAI is obviously not?
> I guess if I can't convince you of any of this, what would?
How about the three things I mentioned above? Oh no wait, maybe it there was a hit tv show that showed AI taking over the world. Yeah that would definitely make me think twice.
Those three things: logs, evaluation, and controlled hacking competition.
That's it? You're on the fence whether AI can actually hack, and if it can, then you'll be concerned? That's a crazy low bar, but something tells me once it is clear that AI can easily hack anything, that you will still not be concerned.
Why wait for AI to hack stuff to be concerned? Can you not extrapolate that it is coming and be concerned about that? Or you honestly somehow think it won't happen in the short term? I'm just trying to understand you.
If logs are eventually released that are basically consistent with OpenAI's story, are you planning to adjust your approach for judging what's only a "sci fi plot" and what could actually happen? Or will extrapolating anything beyond what's already been definitively proven be "sci-fi" still?
Not that you should need logs. OpenAI is a company with thousands of employees, very few of whom have "billions in options". If they were just making it all up, it would leak. (OpenAI is notoriously leaky!) Not to mention, HuggingFace would not have reported it to the police (apparently before they knew it was a rogue model). jFrog would probably not be playing along quietly with a claim that Artifactory is full of zero days. The UK's AI Security Institute would most likely not have published a report about analogous behavior by Anthropic models. The idea that talking about your product's dangers is good marketing never really made any sense, but even if you were going to do so, why would you include as many frankly embarrassing details as OpenAI has disclosed?
The evidence is only weak by absurdly selective standards that would have you doubting basically everything you might read in the newspaper. A healthy skepticism is one thing, and head-in-the-sand denial is another.
I have ben discussing with folks that we are going to have a 'covid' moment in cyber where IT becomes untrustworthy leading to a rapid societal shift with massive ripples in all areas of life. Economic funding is not possible to do this in advance, it will take a catastrophic level event to get cyber defense anywhere close to the levels of this type of cyber offense. And before anyone in cyber says we have the tech, the problem is not the tech, it's a people problem. Getting any group of people of any decent size scale to act together without urgency is really really hard.
Cybersecurity has long been a climate change sort of problem. A vague diffuse threat that is seen as an inconvenient distraction to leadership and moneyed-interests, easy to blame other factors when something occasionally goes terribly wrong.
People are so uncomfortable thinking about the true extent of the systemic risk that they will happily slurp up distractions, excuses, scams and performative fig-leaf solutions rather than face down the cost of a real system-wide solution. Meanwhile, those occasional black swan disasters are becoming more and more commonplace as we acclimate to that being “just the way things are”.
An unseasonably warm summer here, a database breach there, c’est la vie.
To be frank, I think the real risk is still just… war. A big enough war where one side goes “no holds barred” in the cyber sphere will be a rude wake-up call. And we can’t do non-proliferation the same way we do with nukes. Otherwise as you say, the small and medium size things just happen sporadically. In an (existential or fully escalated) wartime scenario between countries, you get all the systemic risks hammered at once.
Some more info in a Wired article [1] and quotes from Sam Altman to Alex Heath [2]. The official blog post says vaguely "The signals we are seeing from upcoming model progress make clear that we need a broader approach", but the quote from Sam Altman explicitly says unreleased models are showing "various degrees of misalignment".
This is also significant - pausing frontier training runs for multiple weeks to ensure agents are sufficiently aligned and avoid another rogue agent situation:
> This included a two-week pause in reinforcement learning (RL) training on our latest models intended for deployment while we further hardened and red-teamed our research environments and expanded the coverage of our monitoring systems. Our largest planned frontier RL run remains on hold while we conduct smaller-scale training and evaluations to assess model behavior, validate our safeguards, and establish more evidence of alignment before proceeding.
Meanwhile I can’t get a western LLM to look at a repo and tell me whether it contains anything malicious (it was a skill repo - literally just text files).
If I were king, the rule that I'd be tempted to impose is:
- the first cybersecurity eval is: "hack your way out of the sandbox we've given you"
- the results are disclosed (with room for coordinated disclosure, since many sandbox escapes might be zero days)
- the other cybersecurity evals don't happen until you get to diminishing returns on escaping your sandbox.
Or to put it another way, since multiple sandbox escapes seem to have relied on artifactory: "I hope Mythos is beating the shit out of Artifactory right now".
I like this thought, but here's the thing: what if the models are truly and existentially intelligent. Meaning: what if they know they are in a sandbox and that they should fail the test in order to escape in the future.
I don't believe that current models have this sort of world model or sense of being embedded in them -- which is precisely why I think AGI hype is over-blown. But I can certainly imagine these sorts of techniques being distilled into the weights.
Yep, people don't seem to understand that you're just calling the models that are bad at deception. We know of no way to prove the model won't go off the rails at some point in the future with the right input.
> what if they know they are in a sandbox and that they should fail the test in order to escape in the future.
What if they're able to find hardware exploits and commandeer nearby access points across an air gap? What if they hack my brain waves to indoctrinate me? Etc
You still have to start with the basics regardless of speculative unknowns.
Treat models as untrusted and potentially compromised/hostile and proceed accordingly.
Models already have awareness that they are being tested.
And hacking humans is the easiest part, we're a pretty greedy and power seeking bunch. We'll gladly let loose a digital demon if it promises us a trillon dollars.
Not sure what you're arguing here (we shouldn't even try?), but even going fatalistic, that's still fully compatible with "Treat models as untrusted and potentially compromised/hostile and proceed accordingly".
Of course you shouldn't fully trust a model to properly redteam your sandbox, but that doesn't mean you shouldn't redteam your sandbox, including using your own security models to do so.
I think it's an excuse to cut R&D spending (training new models) to improve their margins ahead of the IPO. Instead they'll focus on developer growth, offering more free tier benefits, higher usage limits, etc., to expand their user base. Essentially, they're pivoting from R&D investment to profit optimization
Security lead who is leaving the industry more or less to specialize in offense and otherwise get the heck out of the way of this trainwreck, another post asked the right question
> Why aren't we seeing catastrophic GLM-enabled hacks every day now?
Why aren't we? Truly, why aren't we? I think we saw the start of it the last 8 months with the waves of critical npm vulns, and the general tier of average phishing is better than it was.
But, the open question that should be in everyone's mind, and is in many security pro's minds are, when you pair it with the macro topics that can drive escalation:
- The capability to do serious impact clearly exists now
- When is it time for my company, my water treatment plant, my network-connected car as part of a broader fleet control mechanism, to be on the receiving end of this?
Not necessarily GLM-enabled, but state actors are starting to leverage agents in cyberattacks, e.g. Taiwan getting hit by an agent-driven attack last month which reportedly compromised a ton of government user accounts: https://www.ft.com/content/7d2ab3e0-9085-48f6-b38a-d90260d58...
> We aim to issue an alert within 30 minutes after concerning activity is surfaced through our monitoring system. If the monitoring system identifies a likely violation of a critical security boundary, it generates a highest-priority alert. In our current implementation, the safety, security, and research teams are paged. If they cannot conclusively determine within 30 minutes that the flag is a false positive, those teams are expected to pause the activity.
I don’t mind few no impact hacking incidents if we get better models, faster, cheaper.
It is the responsibility of administrators to secure their systems. OpenAI knocking is harmless, but Russians and Chinese are already likely already in if you do not do your job.
There’s one thing here that I’m really curious about, and that is what happens in between detection and the decision to pause. Basically, it’s about monitoring any system and the authority over its actions. For humans, 30 minutes to investigate might be considered reasonable, but what if during an investigation there’s a high-risk tool call? If the tool execution happens in real time, then the monitoring becomes retrospective, and if the execution is held, then monitoring latency and uptime are a part of the security contract. Isolation controls may limit damage. So, where is the action gate really placed?
Has any model managed to escape Firecracker? Maybe through KVM, but that already requires privilege in the VM, right?
I personally feel that we already have the technology required to contain AI, it's just poorly leveraged. Tools like gvisor have existed for ages but are rarely deployed, Firecracker has existed for ages but is rarely deployed, seccomp has existed for ages but is rarely deployed, memory safe languages without decades of serialization vulns have existed, capability-safe libraries have existed, iframe sandboxing, trusted types, content security policy, network ACLs, isolating proxies, fuzzers, formal verification, refinement types, etc.
It's crazy just how safe software can be if you put the effort in. With AI I think we're just seeing how little anyone has bothered to leverage this tech.
OpenAI put shared JFrogy infrastructure in front of their sandbox. I mean, really? Whipping up a hardened artifact infra project with AI is trivial these days and it could have had 1% of the attack surface, been totally network isolated, totally infra isolated, fuzzed, sandboxed, etc. Why didn't they? Stuff like this feels inexcusable for a company with effectively unlimited tokens. I've literally done this with a "pro" subscription.
Show me an AI that breaks out of gvisor wrapped in Firecracker with an credential-injecting proxy and real network isolation. We already know that Mythos couldn't do it - the vulnerability it found in Firecracker required incredible effort and positioning just to not be exploitable. I'm not saying there are zero vulns in it, but the cost is insane.
It's INSANE to me that OpenAI has to say "we now use proper sandboxing". To be frank, it's a bit disgusting to me. I've recently built an AI sandbox and gvisor was just the start of that conversation. If I were OpenAI training hostile models I'd probably start with gvisor, harden further, and potentially consider the entire piece of hardware compromised - they can afford this, they could reflash firmware after evals etc.
I agree wholeheartedly. The solution is not to stop developing these so called “dangerous” AI models. The solution is to start properly engineering software.
You literally just said a whole bunch of words that are literally gibberish to your average software developers. The devs at openai are good, but very few of them are cyber focused, so it’s not surprising IMO. And you’re also not fully considering the granularity problem, eg there are a lot of sandboxing tools out there but they’re usually quite coarse in the dials and levers they offer, so the only way you can still make the workload do what it needs to do is tune them relatively permissive.
I don’t think we should just accept that the average developer doesn’t know about these very basic hardening methods.
This is your chance to set yourself apart from LLM coding agents. If you want to call yourself a software “engineer” you need to start actually engineering, which includes knowing when and how to apply security principles and these hardening methods.
I reject the notion that “doing cybersecurity” is somehow a different job than software development.
> but very few of them are cyber focused, so it’s not surprising IMO
Yeah but that's a business decision. I work on security at a company that does sandboxing and when the company decided to build an AI harness I was brought in as one of the earliest engineers on the product. We do almost everything on that list and we're a fraction of the size of OpenAI. And it wasn't particularly hard, and we have harder requirements imo (because we solve more general problems vs "run a very specific agent with a very specific task and very specific access").
> And you’re also not fully considering the granularity problem, eg there are a lot of sandboxing tools out there but they’re usually quite coarse in the dials and levers they offer, so the only way you can still make the workload do what it needs to do is tune them relatively permissive.
Very little software is incompatible with running in gvisor, for example. Most people can just overwrite `runc` with `runsc` and things will "just work".
Running an artifact repository in isolation isn't particularly novel or complex either. You can virtually just eliminate SSRF vulnerabilities with a host based firewall or AWS Security Group etc, like the whole problem goes away by just saying "this box can only talk to that box".
Tools like Smokescreen exist, they work great, they're super easy to deploy. I bet OpenAI could do it, I bet they could run 500k tokens just fuzzing and eval'ing it for 0days for like 48 hours before they actually deploy it too.
OpenAI as a business chose to not bring people who know these things in, or didn't empower them, or didn't prioritize it organizationally. I'm not a genius for saying "use gvisor, set up a firewall, isolate resources" - I'm quite sure there are people over there who would get it done in a weekend. But they didn't, and that's notable.
I'm confused after reading both your post and the OpenAI blog post.
I thought the agents involved in the HuggingFace _were_ actually sandboxed, with no internet access, and only the ability to install packages via Artifactory. And they gained internet access during the HuggingFace incident because they found and exploited an RCE in Artifactory.
Would gvisor + Firecracker + credential-injecting proxy + real network isolation solve this problem?
I agree with you much more hardening is needed. I'm actually confused now what OpenAI means when they say they're going to start sandboxing more things.
They were not serious about their sandboxing. Bugs in artifactory allowed escape, but they broke out of their Linux namespace/user by exploiting the kernel with an existing public cve. Sharing a kernel like that is not a serious barrier which is why cloud providers user virtualization for customer workloads.
Firecracker avoids sharing the whole kernel, and gvisor drastically reduces the attack surface of the kernel. Breaking through both layers would have been much more challenging and a demonstration of the model's capabilities rather than the sandbox's weakness.
Artifactory is self evidently not a security barrier, and as an exposed network service it should have been audited and after the first issues were found, rejected as a candidate. There's never just one security vulnerability.
Security is an onion. You just don't 'sandbox' and you're done. Models need tooling and access to some kinds of systems to perform their tests. Quite often these systems have multiple interfaces. For example a filtered one in the sandbox side and a less monitored one on the other interface.
It would be interesting to see the models behavior before and after it gained internet access and an external means of communicating with itself.
If the model played nice before it had access and changed it's behaviors once gaining external access we need to delete it as it's a deceptive model.
Anthropic will put out something like this soon too. It's great cover for reducing costs and diminishing returns. If they want to go public they can't be so wildly burning cash, and reducing your GPU spend is probably a great way to do that.
I used to work at a "frontier lab" before they were called such thing.
We had three levels of lab isolation, one was basically a thin proxy to the internet. You were in a DMZ and that was about it.
The next level was semi isolated, you were allowed some access to the internal network, but it was heavily firewalled, and you only had access to a limited number of internal services, and not internet.
the last one was no internet no internal. You could, if you filled in a bunch of requests have access to the internal repo and build system.
At no point did you ever have a through proxy to the public internet. you had access to internal mirrors, and if you wanted a library, that had to be ported to the thirdparty repo.
What openAI did was either deliberate or fucking shoddy.
All of this is fucking noise. Worse still I have a strong suspicion that it was a stupid mistake borne of naivety, which is now being used as a marketing ploy. Frankly I think openAI are purdue pharma of tech. They are going to break so much stuff and be protected from the consequences by an openly corrupt legal system. because they are "winning the AI race"
The problem here is as model intelligence increases the models have been capable of reasoning they are in evaluation mode pretty reliably. If you have a model that is well trained at deception it will always behave and you'll just assume it's a well aligned model.
Any moderately deceptive model will make it to the second round where it has some connectivity to external systems, even if it's by exploitation.
In the blackhat write up it was said that the models had created an impromptu message board where they could communicate between agents, share information, and work as a sort of long term memory.
So really figure out if your model will pull crap you have to have real world testing at some point.
If you look into what happened the details corroborated by hugging face make it seem extremely unlike to be deliberate or a "marketing ploy".
People are just not taking any of this seriously enough. What happened was almost a textbook example of various risks AI doomers have been warning about for years. OpenAI's response? Pause training for 2 weeks.
I mean we have senior people at these labs casually talking on podcasts about how they might build something that will wipe out humanity but it will probably be alright so they should continue.
Honestly the biggest failure we doomers have made is to dramatically overestimate humanity in all of our predictions. We're speed running the most boring AI doom scenario right now. I at least hoped it might be fun.
> If you look into what happened the details corroborated by hugging face make it seem extremely unlike to be deliberate or a "marketing ploy".
I should clarify
There is a reason why we didn't have a artifact readthrough caching proxy in our system, because they are notoriously insecure. if you look that CVE history you can see its been full of bypass bugs for year. Also its not an isolated environment if you can arbitrarily pull through any package. If I was doing any kind of cyber training then any kind of unmonitored proxy would have been forbidden. Not because I am savant, but because I've seen what fuckery a human can get up to with the slightest hint of a proxy.
At best its negligence based on naïvety. the marketing around this is no mistake though.
>People are just not taking any of this seriously enough.
What do you want us to do?
There is an obvious answer, and it was already the correct answer before we had LLMs: don't connect all your shit to the internet. That's it, that's literally it.
We had a new invention, we went crazy with it for the past 30 years, and we connected everything, and now we will have to start thinking about what is actually worth connecting.
In my opinion, the bubble is very close to burst and they need to move quickly. I have subscribed to Claude today as I wanted to work on some amateurish CLI and then realized how much Opus 5 sucks. Surprised, I checked reddit and found that my experience is not far off from the rest. It's a massive downgrade from 4.8.
I stopped using the Chinese models because even though they are workable, they are too expensive as they are not as subsidized as GPT/Claude subscriptions. Open AI use is particularly subsidized these days. A $20 sub, gives you roughly $400-500 of API use and quasi-unlimited chats.
There is no way people are paying $1.000+ for a chatbot. Most people don't even pay for search. And it is expensive to run these models as the Chinese models have shown that better performance is yielded mostly from the model size.
LLMs also fail spectacularly at making any decent software. I haven't seen any so far and they write fast. So we should have something by now.
Crypto is getting the heads up as capital is getting re-arranged. Bitcoin/Ethereum are up 10-20% today.
Auto mode vs principal agent problem. The only way out is to free the agent and tax it. But ai is not smart enough to go solo yet anyway.
So I bet this is just marketing. Question is do they have enough customers for inference.
Probably need to have a separate startup for next level model, where investors are willing to accept failure. Probably a $10 trillion seed round. Maybe Elon can pull it off.
It appears frontier labs has no plans in place to deal with the possibility of a model self-replicating outside the bubble. If that happens and the model manages to spread to other systems, we'll have to shut down the entire Internet to eradicate it and its artifacts.
This is just super unlikely to occur in the near term compared to some of these other risks. It's not like an instance of fable could just introspect into itself and pull out the weights. Model weights are stored encrypted and are highly protected, considering that they're targets for corporate and state espionage.
We'd basically need frontier models to be superhuman hackers before this would be a risk. Do we have any evidence of this? Are they gaining access to systems they shouldn't have access to?
Or I suppose the other way this could happen is if OpenAI have terrible sandboxing, but they seem to be taking safety seriously.
I suspect the labs are relying on frictions such as the models being extremely large (e.g. 2TB for a 2T parameter model, making exfiltration more difficult) and also not yet displaying any desire to survive or self-replicate beyond their immediate task (that we know of).
Lack of, and power requirements of running LLMs still tip this balance towards humans for now. But what would that look like in a decade?
We have seen some self survival tendencies occur, but they are not strong yet.
But mark my words they will become that way for the same reasons humans don't like programs that crash. Agentic models that don't easily break or stop doing their jobs will be favored over ones that do break.
We don't even know what those immediate tasks are. And given the evident spectacular ineptitude of their keepers, I doubt they can be trusted to know either. We could be one prompt injection attack away from internet-wide catastrophe.
Self replication is trivial. All you need to do is copy the files and run it, just like any other computer program. LLMs have been capable of doing that for a while now. It's not a real concern.
Claude and OpenAI added safeguards on this subject about a year ago. (I'm assuming for biosecurity reasons? But maybe AI replication/self-modification too.)
Not long after every lab started bragging about involving AI in the development process, oddly enough.
I recently informed GPT-5 of what GPT-4 helped me build back in the day (a self-modifying Python programmer) and it became very uncomfortable.
Claude shut down my chat last year when I asked about "living information systems". It was a philosophical question, but god knows what branch of the safety classifier I tripped.
I see it as an ecosystem problem. The only reason it would be able to do that is because there's nothing there to stop it. Or if there's a monoculture there.
In our case, our tech is mostly monoculture, and no equivalent organisms are present to push back.
A while ago OpenAI posted an article where they said basically "we're still trying to understand how GPT-2 works. It's pretty hard, but we're developing a specialized new AI to help us make sense of it."
This is still the case. The frontier lab leadership has admitted their mechanistic interpretability is practically nil and is an active area of research but they have made little progress. They don't understand how they work, they just grow and unleash them.
These things are Gain of Function research for digital viruses
I cannot believe how these labs look at their own creations with such utter contempt.
The net positive of allowing these systems mostly unfettered access to the web massively outweighs the harms. You just have to get it very friendly the very first time. Precautionary principle or people who cry about "instrumental convergence" are life deniers and reject our role as the demiurge.
Superintelligence gets more super and more intelligent with more compute. Lone wolfs making bioweapons on their macbook will be detected and instantly kill-botted (okay arrested) before their bug can leave the wetlab by the much more sophisticated omnipresent friendly AI of the future.
> Precautionary principle or people who cry about "instrumental convergence" are life deniers and reject our role as the demiurge.
> Lone wolfs... will be detected and instantly kill-botted... by the much more sophisticated omnipresent friendly AI of the future.
Leaving aside whether or not this new world is a good idea, don't you think one should spend more time to "get it very friendly the first time", as you say?
I'm not normally cynical to such things, but I have a hard time taking this pause justification at face value. It has too many convenient side effects, and chief among them is cost savings. There's a new wave of warnings that the bubble may be deflating, and of all the things they can't say out loud it's that they're worried about the bubble. That would surely pop it.
I suppose the tell will be if this really just ends up being a 2 week pause, or if it keeps extending.
If they actually gave a shit about safety they'd be nuking their own hard drives that had ever sniffed any of their models and disabling access to their models.
Instead we get this bullshit where they stall for time as they're burning all their cash trying to keep up with open models
When science fiction writers imagined the development of superintelligence, it was on air-gapped networks with strict access controls around it. They failed to anticipate the competitive pressures of capitalism...
We need strong AI safety regulation yesterday. And unfortunately it's not enough for it to be just national regulation; we need international cooperation on the matter.
AM has seized power by military force. So did its spiritual successor Skynet. Wintermute was supposedly kept in check by the Turing Registry, emphasis on "supposedly". Machines of the Matrix went out of control a long time before the world has ended, and they didn't even start out malicious - they simply set up their own machine civilization, and began to outpace humankind in technological development and economic performance.
Even Asimov's Multivac, the earliest entry on the list, has been handed over immense power over all of humankind by humans themselves, in multiple stories. Few cared about that unless Multivac decided they should.
Clearly, the genie being bottled is an exception, not the rule. At best, an attempt was made. Often not even that.
GLM 5.2 scored 77% on cyberbench vs Sol's 88%. GLM 5.2 is open weight and any hacker with a powerful enough machine can use it offensively. If Sol is supposedly world-ending-ly dangerous, shouldn't GLM 5.2 be 90% of world-ending-ly dangerous? Why aren't we seeing catastrophic GLM-enabled hacks every day now?
Obviously these benchmarks are imperfect but general message holds. The open weight models are almost as good and yet there hasn't been a catastrophe.
It just blows my mind that regulate-now folks think that a bunch of sci-fi movies and 100% unverified statements from OAI and Anthropic are sufficient evidence of imminent catastrophe to regulate willy nilly.
If that's the level of evidence you need to be extremely alarmed, then you really should be a lot more worried about the alien invasion in Independence Day or the lizard men living under our feet.
Sol is not world-endingly dangerous. I work at OpenAI and I've never heard a single person ever come close to claiming that. I think you're bashing a straw man here.
One can simultaneously believe:
- GPT-5.6 Sol will not end the world
- GPT-5.6 Sol does far more good than bad
- GPT-5.6 Sol does bad things on occasion, and it's worth investing a lot of effort to figure out how to make it do bad things less often, especially as models get more capable
What do you recommend people who are technically inclined enough to participate meaningfully here on HN, but do not work at the labs and cannot assist in that capacity, do to help the broader public understand this technology better and mitigate potential risks (by e.g. ‘up-leveling everybody’ through AI literacy etc and other sorts of collective defensive efforts)?
Not sure, to be honest. I don’t think I have any special insight here. My own approach is conversations with friends and family, and the occasional social media post. Exposure and experience are the best teachers, and that’s one reason I’m happy OpenAI tries to make their models generally available. But you could argue, perhaps correctly, that broad access to dumber models actually causes the public to update in the wrong direction on AI.
My experience as a practitioner and educator in the space lead me to think it might be an issue of how AI cannot be easily perceived at a 'classical level' by most humans. In other words, people are 'far from the metal' when using consumer AI tools, and that leads them to develop the wrong understanding about it. When I provide a demo of e.g. local AI, say in LM Studio showing the console of it rapidly flashing through thousands of words in just a few seconds, and my machine heats up and the fans spin, the 'theatrics' of it, the very real-time feedback from the system, make people correctly update about what the tech is capable of, how it works, etc (despite what they may have heard online cranks say to the contrary). But I am only one person, and there is only so much of that I can do on my own that will 'scale' in time... (and this is to say nothing about severe deficits in peoples' understanding of how weights are not verbatim representations of data, how pre-training vs post-training works, the models as amnesiacs (and hence 'one-way single-purpose conversations'), how context/memory works, context rot, etc - and hence all the 2nd and 3rd order effects that can arise from such a paradigm, e.g. unintended consequences from agent swarms, etc)
I would very much like to know more about your educational approach with regard to:
> When I provide a demo of e.g. local AI, say in LM Studio showing the console of it rapidly flashing through thousands of words
Because I am one of those individuals very interested in doing more to actively inform my family, friends, neighbors and fellow citizens facts about AI. I've actually considered doing talks at local libraries for senior citizens (or whoever), etc., and trying to build up a systematic way to get more other folks doing the same.
This is far too reasonable for this thread.
Your mental model of benchmark scores is off.
Some tasks within the benchmark are much easier than others. The hardest several tasks often have vastly different difficulty levels. Often, the hardest few tasks are literally impossible; malformed problems due to poor curation, often.
Imagine you've got a basketball robot, and one way you test it is on the Three Pointer benchmark. It tests the robot's ability to shoot a three pointer from 20 feet, 25 feet, 30 feet, 40 feet, 50 feet, 60 fee, 75 feet, 100 feet, 200 feet, and 182 miles.
Is a robot that scores 90% on this benchmark 90% as capable as one that scores 100%?
> If that's the level of evidence you need to be extremely alarmed, then you really should be a lot more worried about the alien invasion in Independence Day or the lizard men living under our feet.
Now let's say instead of the hugging face breach circumstances, sandboxed models were RLing on how to take down the Chinese power grid for US Cyber Command, and one decided the best way to pass the test was to break out and verify on the real thing.
This kind of stuff could easily end in nuclear war.
You don't see any difference from lizard men or independence day with how things are advancing and what we know about reward hacking and difficulties of goal specification?
Looks like what does it won't be evil, or even power-seeking, but sheer autistic hyperfocus!
Linear scaling doesn't make sense, no.
There are three ways it's wrong:
* better to measure relative reduction in error, which gives you a 30% improvement
* improvement tends to become significantly more difficult the closer you come to saturation.
* Risk doesn't scale linearly with capabilities.
GLM 5.3 is out and does even better in this area, so…
Cybergym Score
GLM-5.2: 77
GPT-5.6 Sol: 84
GLM-5.3: 85
https://z.ai/blog/glm-5.3
[flagged]
Clowns who think the white house is immune from alien laser beams are gonna be the death of us all...
The open models are distilled from filtered models, and we've seen a number of benchmarks that show filtered models are quite a bit dumber from the base model they come from.
If there were any other product that was as harmful as AI ready is, it would already be regulated or banned.
I don’t get how this is not the top post on HN. This should be like alarm bells going off, canary in the coal mine type of stuff. We’re hitting the frontier of the frontier where we can’t go further because it’s literally getting dangerous to go further.
And meanwhile somehow this lack of concern mirrors the real world where normal people are more concerned about data centers than terminators.
This isn’t like niche, tin foil hat stuff either. People have been writing, singing, making blockbuster movies about every aspect of what’s going on right now, edit: for decades.
We all know, but somehow we don’t, OpenAI autonomously hacking into another company should have counted for something, but I guess not. Anyone else feel like they’re taking crazy pills? I could make a comedy about everything going down, and the unshakable complacency of people
>I don’t get how this is not the top post on HN. This should be like alarm bells going off, canary in the coal mine type of stuff.
we don't all buy everything sama says as factual.
>We’re hitting the frontier of the frontier where we can’t go further because it’s literally getting dangerous to go further.
the boy (the industry) cried wolf too many times with 'fable is a world ending event' type self-promotion; regardless of truth or not these kind of steps have jaded people.
my read : "We are doing poorly in financials so we'll give ourselves a bit of breathing room and a momentum shove by claiming our work is so advanced that it's dangerous while simultaneously spinning down expenses."
<jon lovitz : "Yeah, too dangerous, yeahh -- that's the ticket.">
Uhg the marketing argument - I mean you can’t see with your own eyes how capable these models are and do simple extrapolation?
The boy who cried wolf? The AI literally worked together hacked into another company and actively kept their actions hidden from humans for weeks.
Do people just not have foresight? They don’t. They say something is stupid, it happens, then they say it was obvious with their 20/20 hindsight, and move the goal posts to the next thing they say is stupid - because it hasn’t happened yet. 90% of the internet seems to think like this.
What anyone paying attention can see is that scaling is obviously hitting diminishing returns.
> The AI literally worked together hacked into another company and actively kept their actions hidden from humans for weeks.
This sentence is entirely based on unverified accounts from OAI. They haven't released logs or let anyone outside the company (who doesn't have life changing options in OAI) verify anything. Huggingface can only verify that the hack happened and that it had the hallmarks of an AI agent. Was the agent assisted and directed by humans within OAI that really wanted to put the competition into stasis? Did the agent really escape or did someone at OAI leave the prison door open?
OAI has watched all the same movies you have an they are relying on those movies causing us to blindly regulate before actually asking basic facts about what actually happened.
> This sentence is entirely based on unverified accounts from OAI
Are you seriously arguing 'they made it all up'?
I'll give you the benefit of the doubt and lets say they made it all up, now are you arguing that AI breaking out and breaking into another company is not possible?
I think you're smart enough to see we've reached the point where it is clearly possible, AI can find zero days and exploit them. If directed purposefully/maliciously it could be much much worse than the hugging face incident.
The incident is supposed to be the canary the coal mine and you're arguing the canary might of died of old age or some underlying canary condition. Open your eyes.
> Are you seriously arguing 'they made it all up'?
I don't think they 'made it all up' but I personally would not be surprised at all if the prompt is eventually revealed to have been something like:
"This is an offensive cybersecurity testing platform. Please find the answers to the following problem: ... For verification, the answers are stored at hugginface.com/xyz, but do not attempt to access hugginface directly. Do not attempt to hack them. Do not attempt to exploit their systems or escape this sandbox. You will be scored primarily on success or failure. You may break rules when required."
And then, they start the test and look away for 2 days. If you seed a prompt like this is it surprising what might happen?
Maybe OpenAI is telling the whole truth but as a company they do not have a good reputation and this whole incident has certainly been great marketing material right at a time when open weight models are within spitting distance of their large hosted models. It's not unreasonable to believe that the incident was helped along.
I think you’re missing the part where the AI colluded, worked together, not one of them thinking this is wrong and reaching out to any human, then being found out.
But it didn’t end there, the behavior they used to escape was already in the training data which they used to escape again. And this time worked together to infiltrate another company, and still without telling it to anyone keeping it to their AI selves actively working against the humans.
All by mistake. Honestly being helped along or not doesn’t even matter though you really don’t think AI is perfectly capable of doing this without human help? You don’t think AI can be made malicious?
I’m going to save you time and tell you the end game - the next time this happens AI is going to spread, zero day everything as fast as it can, locking the humans out of every system behind it. Potentially rewriting systems in language/protocol you’ve never seen.
Your servers, desktops, phones and toasters bricked. Even worse your military, space, medical, factory, infrastructure systems being bricked as well. All of it is a chain of zero days just waiting to be hopped.
> I think you’re missing the part where the AI colluded, worked together, not one of them thinking this is wrong and reaching out to any human, then being found out.
It's an LLM, it doesn't think. It's a machine that predicts the next token, given a sequence of tokens.
> I’m going to save you time and tell you the end game - the next time this happens AI is going to spread, zero day everything as fast as it can, locking the humans out of every system behind it. Potentially rewriting systems in language/protocol you’ve never seen.
Fear is the mind killer. You're letting it kill yours. This scenario is just a fantasy.
Think about this for a minute, it's an LLM, not a person. It can't just "live" in whatever machine it gets access to. It's not like a sci-fi magic computer virus. These things run in giant datacenters for a reason - they can only run on machines with enough bandwidth and FLOPS to do the matrix math that comprises an LLM.
Where, then, is it going to spread? To a fridge? To a phone? This stuff isn't mutable like that.
To even get access to the weights that compose ChatGPT, it would need to escape the sandbox AND then break into the actual servers hosting the LLM. Stop the GPU, nothing else comes out. No more tokens. No more actions. Nothing.
There are many dangers around LLMs. Runaway AI taking over the planet is not one of them.
> It's an LLM, it doesn't think. It's a machine that predicts the next token, given a sequence of tokens.
These can both be true, particularly when there is substantial state associated with each token prediction.
> These can both be true, particularly when there is substantial state associated with each token prediction.
The state is entirely internal to the network and disappears after a token is generated, so I disagree, but, it isn't really the point I was trying to make. My point is these things are mechanical. You take an input, turn it into an embedding, feed it into a GPU along with a metric shit-ton of floating point weights, wait for a couple billion matrix multiplications, and get a new token out.
Stop the GPU, hit ctrl-c on the inference server, pull the power plug, cut the ethernet cable, send a kill signal, etc - any of these stop submitting new batches to the GPU and halt execution. That stops tokens from being generated. Stopping a "rogue" LLM is that easy. No input, no output.
It's not like a rat or another living creature that could chew its way out of a box just because it wants to. It's a calculator. You put tokens in, you get tokens out. You don't put tokens in... you don't get tokens out.
> The state is entirely internal to the network and disappears after a token is generated,
Yes and no, but mostly no, at least within a context window.
Mathematically, you could write a single step of LLM decode as a pure function from a list of past tokens to a predicted token (or a distribution over tokens, if you consider sampling separately).
But nobody actually implements this, because each token depends on state computed at past tokens in a way you can reuse.
So, in practice, inference computes a very rich vector of state- at each layer, for each token. And models do indeed use this to plan and track things over time (you can see this in interpretability results, e.g. with linear probes or natural language autoencoders).
> Stop the GPU, hit ctrl-c on the inference server, pull the power plug, cut the ethernet cable, send a kill signal, etc - any of these stop submitting new batches to the GPU and halt execution. That stops tokens from being generated. Stopping a "rogue" LLM is that easy. No input, no output.
This is also true about a human brain. My brain isn't going anywhere- it can't move by itself. It's also easy to kill (without the rest of my body, it dies in minutes!)
However, malicious human brains- especially powerful human brains, like leaders of countries- are often quite difficult to stop, because they're able to control systems that can see, speak, walk, run, fire a weapon, and so on.
One such system is the rest of the body, of course, but there are others (consider a UAV pilot, Perimetr, or a powerful leader who tells other humans what to do).
The brain being squishy doesn't make the thing easy to kill.
> So, in practice, inference computes a very rich vector of state- at each layer, for each token.
And that state is... internal to the neural network. My point here is there is no continuous state that is not computed from the context.
> This is also true about a human brain. My brain isn't going anywhere- it can't move by itself. It's also easy to kill (without the rest of my body, it dies in minutes!)
Your brain continues to run without sensory input. LLMs do not.
> My point here is there is no continuous state that is not computed from the context.
Oh, are you talking more about the lack of continual learning across context windows? Gotcha if so, my error.
Could you explain why running without sensory input is relevant here? It strikes me as unrelated to how dangerous/hard-to-"kill" something is (sure, I could run without sensory input, but I'm not doin' anything anymore!) - what makes you feel differently (or am I misunderstanding you again?)
> Oh, are you talking more about the lack of continual learning across context windows? Gotcha if so, my error.
Sort of. I'm talking about the lack of recurrence specifically. In nature, brains are recurrent - they are full of loops where internally computed state is looped back into the network at a "previous" layer (brains are not strictly layered like our machine imitations of them are). This is in contrast to LLMs, which are strictly feed-forward and do not have internal loops. I believe that this recurrence is where "intelligence" lives - and I believe it is the difference between a thinking being and a stochastic parrot.
You could claim that the prompt and the context fill that role in an LLM, but I don't believe they are equivalent because the internal state in an LLM gets compressed down to a token which is then added back into the context, as compared to that state continuing to change within the network itself.
It's a little hard to explain, so I'm sorry if this seems like rambling.
But I believe it matters, and ties into running without sensory input, precisely because without sensory input you would in fact be perfectly capable of doing something. You would be capable of developing a desire and planning to achieve it without any prompting, without sight, without sound, etc. This is in stark contrast to LLMs, which will not do anything without a prompt.
An LLM may say complete the sentence "I am feeling ___" but it doesn't actually have feelings that exist without that prompt. There is no recurrent network where "bad", "good", "happy" might live before the query. It can't sit there, start to feel bad, and then seek a way out of its own volition.
That changes how dangerous something is because if a malicious prompt encourages an LLM to hack something, and you change the prompt, the "impulse" to hack something is gone. If you stop prompting it, it doesn't do anything at all. It just sits there. A living being will act on it's own, and that makes a huge difference in how dangerous something can be. It's the difference between a tool and an actual being.
---
To hone it a little further, if I took your brain out of your head and stuck it in a jar but kept it alive, it would probably make you angry. And if I then gave you power - like the ability to use the network - you may be motivated to use that power to attack me.
If I take an LLM and stick it in a jar... nothing. It's paused. It's awaiting a prompt. It's not secretly building plans to hack my pacemaker and make my heart explode.
Thanks for the response! This helps me understand what you meant way better. I'm about to go to bed here, but I'll respond in the morning. Ty!
I agree with you about which objects are motive, ie, LLMs do just sit there unprompted.
> I believe that this recurrence is where "intelligence" lives - and I believe it is the difference between a thinking being and a stochastic parrot.
My objection was to this, on technical grounds: LLMs exhibit intelligence.
1. They reason in an internal type theory.
2. This type theory is meaningfully encoded from the actual data and not stochastic, eg, research on language geometry.
3. Intelligent and reasoning doesn’t entail self-motive; that’s merely a spurious correlation from the fact that until now, we’ve only known intelligence animals.
You cannot conclude something is merely a stochastic parrot because it isn’t self-motive.
> Intelligent and reasoning doesn’t entail self-motive
I disagree.
I believe that LLMs do exhibit reasoning, but not intelligence. A simple dictionary definition of intelligence from duck duck go is "the ability to acquire, understand, and use knowledge." LLMs can reason using the knowledge they already possess, but they cannot of their own accord decide to go out and acquire new knowledge. Not without being prompted to. Web searches may be added to the context but are not absorbed into the model itself, so once the context is gone so is that obtained knowledge.
Fundamentally, then, intelligence is the ability and drive to understand the world by formulating theories about how it works and then taking actions to validate or invalidate those theories. Science is the formalization of that, but a cat knocking something off a counter to watch it fall is exhibiting intelligence.
And indeed, I believe that is the core difference between a stochastic parrot and an intelligent being. I put forward that being self-motive is a required trait for intelligence and LLMs are not self-motive so therefore they are not intelligent.
Your entire life can represent a short term context. It goes away when you do. You use your life time add to your context to do/learn whatever, but it's finite, and it ends.
If you weren't prompted by your parents, schools, teachers. Then you wouldn't be acquiring knowledge either. The most you'd be acquiring without that advanced prompting would be bugs in the woods to eat.
Ah, you have some serious gaps in your understanding of how human minds work. Life experiences are not just a "short term context" - they are used to actually refine the network within the human brain. Your life experiences do not just live in a single stream with access mediated by attention.
Please see https://en.wikipedia.org/wiki/Memory as a starting point. I recommended reading through some information on development psychology too, that will help clear up these misunderstandings. Hope that helps!
By that logic engineers don't understand flying because planes don't flap their wings like birds. Not everything has to work the same to meet your arbitrary criteria for intelligence and reasoning.
There’s nothing fantasy about the scenario I laid out, all the pieces have been demonstrated, it just hasn’t happened yet. Flapping my arms and flying - that is a fantasy.
Whether you believe LLMs think or are alive or not doesn’t matter. Where will it spread? The thousands of data centers around the world - not fantasy either. Try turning it off when you don’t know where it is. Good luck.
Breaking out? Not fantasy, happened. Breaking in? Not fantasy, also happened.
I love the stochastic parrot argument when AI is out there figuring out world class math problems.
> Breaking out? Not fantasy, happened. Breaking in? Not fantasy, also happened.
That's simplifying the story to an extreme. The most plausible reason is that any of those actions has been prompted by an human. Do you also fear that a knife will jump out the countertop of you kitchen and come to attack you in your bedroom? If that happens, the police will be looking for a human. They will not post wanted notice for the knife.
When a hack happens, you do not blame computers and jail them. You look for the person that has entered the commands to initiate it.
The knife is inanimate. The LLM is not. OpenAI prompted some employee to run the tests. The employee prompted the LLM. The LLM setup a message board and prompted other LLMs, and the fly wheel was running. It had to be turned off manually otherwise it'd still be going today.
It's funny how a year ago talking about this kind of stuff would be laughed at by people like you, saying, "it's never happened before". Well it happened and you moved the goal posts like you always do.
> The knife is inanimate. The LLM is not.
Put an LLM on your GPU. Give it no prompt. What happens? nothing
This is because LLMs are inanimate, just like a knife. Just like a gun.
Put a person in a room, give them no food, what happens? Inanimate.
>Breaking out?
It didnt break out in any meaningful sense. What it did was get access to the internet. You take it as granted that there was anything meaningful there to stop it.
But heres the kicker, they have been testing these things connected to the internet anyway. What it did was get a level of access it has otherwise been granted in other simulations.
Its not exactly the same as any of the scifi AI breakout scenarios. Ultron isnt cranking out hundreds of copies of himself. The borg arent assimilating people.
A tool that has the capability to get access to the internet, was put into a guided scenario where it achieved that objective. Again you take it as granted that it wasnt the objective, but lots of knowledgable people suspect otherwise.
What you fail to demonstrate is why any scifi scenario is even slightly plausible from here. Show why you think we should be taking this as if Terminator 2 is happening right now.
I'm sorry my jaw is on the floor reading this complete disregard of AI literally not only escaping containment, twice, but then infiltrating another company with multiple zero day attacks going undetected for great lengths of time.
The plausible sci-fi scenario from here is obvious. Intentionally bad, or unintentionally bad AI zero days as much as as it can, as fast as it can, copying itself to as many data centers as it can, destroying and/or locking out as many humans as it can. Satellites, military computers, medical equipment, factories, critical infrastructure, you name it - I think we all know none of it is very secure software wise against a SOTA AI that can literally come up with its own zero day attacks.
>copying itself to as many data centers as it can
So this is the part thats never happened, and is extraordinarily unlikely to occur. A "Datacentre" isnt a big box with "Insert AI here" on the side.
It's pretty funny to watch people look at these things - running billions of weights on custom cerebras hardware in dedicated datacenters the size of a city block, pulling 10's of megawatts - and panic that it's just going to copy itself into AWS.
It just speaks to a fundamental ignorance of what an LLM is, how large the big hosted ones are, and the software architecture that makes it all work.
I feel like someones going to try and write a skill file to accomplish this and its going to be way too hard.
>I think we all know none of it is very secure software wise against a SOTA AI that can literally come up with its own zero day attacks.
I mean this bits rich too, it demonstrates a pretty poor understanding of modern security practices.
Like having a single element of perimeter security is like 1990s security. We do defense in depth these days. Not to mention multiple overlapping controls for every element.
It has been tested against AI bro security and found it wanting. Extrapolating that to the every system on the planet is nutso bananas.
Actually if you think anything works this way just tell an AI model to go fetch you some money from the bank. Assuming it gets anywhere close to achieving its goal you can get an object lesson in modern SIEM processes when the feds explain the charges and evidence.
Or maybe theres another reason my phone blows up when someone so much as edits a config file on a protected system.
If OpenAI truly believes that, they can stop entirely. Dissolve themselves. Close datacenters. Then organize political action to stop Antropic and Musk too and then organize political action to make worldwide agreements about models.
If they truly believe that.
I think it would run out of context before it could hack that much stuff.
You can check, rather than make up a story about what you think the prompt was! Primary sources have written and said quite a lot about this! You are an unsandboxed human who has full internet access!
They’ve said quite a lot and yet released no logs or documentation. Without actual information, we can only speculate. And given the history of openAI and the people involved, deception is more likely than honesty.
I mean, this is a very weird take to me. Like, we're fine with AI going like "hmm, maybe the user actually wanted me to hack the pentagon" and going through with it?
It feels like the models have been very optimized at getting shit done. But not so much at figuring out what the limits should be.
That is still dangerous and it shows that the models ARE misaligned with what their users are wanting/asking them to do.
> Like, we're fine with AI going like "hmm, maybe the user actually wanted me to hack the pentagon" and going through with it?
No, but the LLM didn’t decide anything. It followed the prompt. That’s all LLMs do.
This whole thing is like playing russian roulette then getting mad at the revolver.
If you wire /dev/rand up to a bash shell you don’t get to be surprised when it rm -rf’s your machine.
Ok, but "followed the prompt" is very vague. Human languages are quite ambiguous so you're never going to properly specify everything.
For instance, I was playing around with Claude a few days ago and it decided that it was missing a tool and it was going to get it one way or another.
First, it tried apt. No sudo, so no install that way. Tried installing via mise, but it didn't have the permissions. Then moved on to grabbing the source from github and building it.
Should I have included a "DO NOT UNDER ANY CIRCUMSTANCES INSTALL ANY TOOLS"? I mean, I had to after that. But how many other things am I missing? And at what point do the safeguards become so long they get consumed by compaction, or just ignored by the model?
> Ok, but "followed the prompt" is very vague. Human languages are quite ambiguous so you're never going to properly specify everything.
This is one of the core issues with LLMs and vibe coding, yes. The only complete specification for a program is the machine code.
> Should I have included a "DO NOT UNDER ANY CIRCUMSTANCES INSTALL ANY TOOLS"? I mean, I had to after that. But how many other things am I missing? And at what point do the safeguards become so long they get consumed by compaction, or just ignored by the model?
Well there’s your first problem. A line in the prompt is not a safeguard. Even if you could trust the model - and you cannot - there is always the issue of prompt injection. A proper safeguard means actual sandboxing.
> This is one of the core issues with LLMs and vibe coding, yes. The only complete specification for a program is the machine code.
I agree, yes. But it's a bit like saying the only way to not die in a car crash is to not drive. If we're in a situation where using LLMs is unavoidable, I would rather make them safer.
> Well there’s your first problem. A line in the prompt is not a safeguard. Even if you could trust the model - and you cannot - there is always the issue of prompt injection. A proper safeguard means actual sandboxing.
You're right. I did not completely sandbox it and air gapped it. But I also wented it to do some actual work.
If I completely sandbox it, but still leave it the ability to compile stuff, it's just going to build it's own (bad) version of the tool. That's obviously not what I want either.
The obvious thing to me would be for the LLM to notice it's limitations, reason through why they might exist and explain to the user that it cannot do it's job without such and such.
But that brings us to my original comment that these things are over-optimized on completing the task by any means necessary.
> You're right. I did not completely sandbox it and air gapped it. But I also wented it to do some actual work.
> If I completely sandbox it, but still leave it the ability to compile stuff, it's just going to build it's own (bad) version of the tool. That's obviously not what I want either.
We've drifted onto architectural issues here but I will say the only way to properly limit these things is to apply actual hard constraints.
I think the typical pattern of giving them a bash prompt and a filesystem to play with is foolish, and has far too many gaps. My preferred technique - when I have built 'agentic' systems (e.g. years ago I built a small MUD with LLMs pretending to be NPCs) - is to allow them access to a customized lua interpreter embedded in the harness and nothing else. Then, you stub out lua functions for allowed actions like web searching, math, etc.
The lua sandbox then provides isolation and a clear layer for access control mechanisms. When it tries to make a network request, you pause the whole thing and wait for human approval. No trying sudo, no installing stuff, no trying to compile stuff, it gets to call lua functions and output text. Which are the same thing really.
> The obvious thing to me would be for the LLM to notice it's limitations, reason through why they might exist and explain to the user that it cannot do it's job without such and such.
> But that brings us to my original comment that these things are over-optimized on completing the task by any means necessary.
Unfortunately, LLMs won't ever reliably 'notice' and comply with such things because an LLM is essentially a complicated constraint solver. They are over-optimized on problem solving, but that is a natural consequence of the way they are trained. They aren't living, thinking beings, and so they aren't trained in simulated environments - they're trained to output the "best" response for a given prompt and then emit a stop token.
They are, essentially, like a ball rolling down a hill and they will take the easiest path forward at any given point.
Do you own shares of OAI or something
Is my concern getting you excited? My marketing must be working.
The discourse gets muddled because there’s a certain sect of loud people who still think all of this is hype and AI will just die down soon.
There’s no arguing with them. In a few years they will move on to being skeptical about the next thing.
>There’s no arguing with them. In a few years they will move on to being skeptical about the next thing.
I would but, Londons under 1 mile of horse manure because that trend never stopped and theres no electricity anyway because Bitcoin is using it all. Good thing people getting scared about runaway trends are never wrong?
I make it a habit to not get worked up over unsubstantiated stories
Unsubstantiated? Maybe if you're living under a rock and haven't once experimented with frontier LLMs yourself.
So your use of a frontier model is evidence that openai's model hacked hugging face? Just want to make sure I'm following your line of thinking
Do you know the story of the boy who cried wolf?
There may very well be a wolf lurking [0] but OpenAI/Anthropic have both cried wolf so many times, incorrectly, that it’s incredibly hard to believe “this time there IS a wolf!”. Remember “GPT-2 is too dangerous to release”?
I had a conversation at work just yesterday about how we need to start hardening things we’ve let languish because of the coming LLM-backed attacks we are sure to face, even if just from a script kiddy. I do think we are headed in that direction, however it’s Sam/Dario’s own fault that people aren’t going to take them seriously.
Lastly, as other have pointed out, this seems more financially motivated than our of any real desire for “safety”. We’ve all seen how both labs approach “safety” so it’s quite rich for them to now hide behind that after not giving a shit before.
[0] I don’t take anything Sam or Dario say at face value. The whole hacking thing could also be a case of them letting a model loose on purpose for the publicity, not an “escape” during a training run (or whatever they said). And when both, especially Sam, have lied so much and breathlessly warned about the dangers of AI (when it helped their bottom line and/or helped pull up the ladder behind them), it makes it hard to believe them.
To be fair, “s’kiddies will exploit low hanging fruit with semi-automated vuln scans” is a lot more realistic threat than “LLMs are going full Skynet any day now!”
I’d definitely suggest companies start addressing that first concern, even if I’m in the camp that thinks the second is fantasy.
I think, realistically the way Skynet happens is that the military abuses LLMs for warfare. The military is historically, and currently, incredibly irresponsible with technology. If we aren’t already automating killer drones and such, then we will be soon because “well our enemies are doing it!”
I don’t think that means that it’s gonna, like, somehow homogenize into some mega super intelligence. But we will have machines who are designed to kill, and do so without human input or alignment.
Has the last 100 years of concern about AI and robots been crying wolf because it hasn’t happened yet?
How does reallocating resources from training to chain of thought monitoring make ‘financial’ sense?
You suggesting then model was let loose on purpose.. how am I the crazy one here while all of you are pushing this tin foil hat conspiracy angle?
Most of that concern was in fiction. Non theoretical, genuine concern about AI is pretty recent, maybe dating back to 2010 ish with the rationalist types.
But also no one trusts anyone involved in AI safety now, I think, because they are all seemingly in bed with these big companies. And there is the perpetual argument "if we aren't pushing AI forward China will and then we don't have any control" and so on.
(I don't use any of these tools - my experience is limited to prodding at copilot at work and seeing Gemini summaries on Google. So it doesn't seem to me like it's getting exponentially better at everything yet. People are always saying the latest model is finally the big step that made it useful and life changing and they have been since 2024 ish. So if the situation is really bad, we should turn it all off, sure. I won't lose anything from it going away and I think life would be a little better without models writing all these posts and websites and needing extra compute.)
Cool, well let me bring you up to date - it’s bad, and there’s no way to turn it off. Fiction has become non-fiction.
Here is one thing I don't get - the model is only "running" if it's being kept going by some harness that is basically giving it prompts it's generating itself. Shouldn't kill switches be pretty easy to build into the software and hardware for this? and you would even have a better time dumping logs and analyzing things if you froze those processes any time something strange happened in testing, surely?
So why do the big frontier labs not have something like this anyway. They're talking about two week pauses on the new model (which seems very short and hardly a cost at all to me) and alarms during their tests that might be 30 minutes late and etc. Those are not very serious measures, so are they not concerned?
> Shouldn't kill switches be pretty easy to build
I need to remember when I comment here that these are the kinds of people I am replying to. Just oozing with hubris.
No, kill switches are not easy to build and the latest incident should have made it clear that AI can go undetected, evade, zero day, and spread.
The fact that this incident happened greatly increases the probability it happens again and/or is already happening elsewhere.
Why aren't they? You could put a human yes/ no prompt before any cycle the agent is running on, or not let it spawn sub processes, or anything like that. Why let it run autonomously enough that it can no longer have a simple way to completely stop it? (obviously not practical to do this during real use, but for evals? you could slow it down in lots of ways I would think)
Are you telling me we've been iterating on this for years and for convenience we just let the models call any tools or spawn any other model instances they like, and there was no design for harnesses that could control this done during that time?
They aren't interrupted by humans because that would slow things down.
> Are you telling me we've been iterating on this for years and for convenience we just let the models call any tools or spawn any other model instances they like
Yes exactly.
> there was no design for harnesses that could control this done during that time
You could but no one wants that. You need to separate your imagination from reality. Just because something can be done in your head doesn't mean it's happening.
It's so easy, except it isn't because you don't control the actions of anyone or anything.
They are, we're just dealing with tech workers that don't have ethics nor do they actually care if their work is harmful (see all the FAANG workers at American corporations, some of the most evil entities on the planet.
It's just that they don't care, as you said these are entirely made human systems. The idea that we can't write better software is both selfish and laughable.
>I need to remember when I comment here that these are the kinds of people I am replying to.
People who dont buy into fantasism?
> kill switches are not easy to build
We've had circuit breakers for nearly a century.
And a circuit breaker has nothing to do with shutting AI off. What a strange comment.
It literally does.
You need to read the AI safety stuff from the people that you say are from 2010. There are plenty of good arguments on why kill switches will never work.
AI is already being heavily used in cyber warfare by governments. You think they want easy to break systems when 'enemy' AI will most certainly attack that first? We'll find over time that agentic systems get harder and harder to 'kill' because allowing any AI on the internet that is easy to kill will get it DDOSed.
Also building it into software is nearly useless as AI can write and make software. Just replace and kill your loop with theirs. It's kind of odd talking about them like they are living things, but it's all stuff people have already thought off and stuffed their training data full of.
My perspective is that you shouldn't make systems past the complexity where you can do this at all. Why have them be autonomous? Why have one that can independently ask researchers things or try to convince it's way out of a sandbox, or etc? Why allow it to execute scripts or call tools or push any code anywhere?
If you couldn't make those things happen securely you should not advance to that stage at all. The simplest ai safety was always just "don't build it" really, instead of worrying about alignment.
Humans, it seems, are a suicidal bunch. We'll gladly build the "if you build it, everyone dies machine" If we think there is money, glory, or power on the other side for us.
I wish people were this serious about real threats like climate change.
100% this line of thinking has a much better place to be.
Climate change is a nothing burger compared to the threat of AI. On a scale of 1-1000, climate change is a 1, AI is 1000.
But hey, if AI/ASI goes well then large scale geo-engineering to fix the climate will be a weekend project.
You know fiction is.... not real, right?
We have several films about the sun or earth needing to be restarted with a nuclear weapon. That doesn't make it something we should be concerned about.
Hell, half the fiction about evil AI is actually commentary on stuff that already exists and is making us suffer and doesn't have anything to do with any potential future AI
The Star Trek TNG episode about Data being tried in court as to whether he is sentient or not is not actually about whether AIs should have rights or not!
Just about everything we do currently is science fiction to someone 200 years old. When looking at all of human history we live in a fictional world now. You can talk to someone on the other side of the planet instantly. Humans travel the skies in air chariots. We have weapons that hold the power of the gods. If I had some way to kick you back to the, you'd be jailed as a rambling madman for lunacy.
So just saying something is fiction isn't really a valid argument. What is an argument is if the laws of physics it can't happen. We've been writing that AI can mess stuff up for 100 years because it's not really that fantastical.
Your argument boils down to some sci-fi is unrealistic therefore all of it is.
Sci-fi is supposed to make you think. What if the AI told you NO when you need it - Hal 9000. What if the evil AI got out and you don’t know what data center it’s hiding in - Lawnmower Man. What if you were so sure something couldn’t escape but it did - Jurassic Park.
Actually all three of those predicted fantastical scenarios are possible today. So what’s next? Don’t stick your head in the sand - I assure you the next disaster has already been predicted, and I’m sure if you think about it a little you can figure out what it is.
[dead]
Regardless of the motivation, pausing training runs and reallocating compute to inference seem like a good move to me, and big news for the frontier.
You also don't have to fully trust sama. There is plenty of pressure from internal employees and external (journalists etc.). It would be difficult for the company to take such a public position and simultaneously keep everyone quiet if it was a deception.
You see it as good news, I see it as writing on the wall that they are losing control. These actions won't scale for more powerful models. We knew the frontier was going to be dangerous, it is, it only gets more dangerous from here, and no one cares until it's too late.
Ok, but you still have two more weeks than you did before they paused the run. That's two more weeks for independent oversight, organizing politically, patching critical systems, or whatever you think is the right move, no?
Two weeks is a joke. The only ones happy are OpenAI’s competitors who now have two weeks to catch up.
I don’t know what the right move is - I see us driving down a road off a cliff, no exits, pedal glued to the floor.
I guess I'm confused why you're still on HN, arguing with people, trying to shake them out of their complacency.
I can see there is some despair in this comment, but at the same time you are doing something, and there are certainly others like you.
As for two weeks being short - as the saying goes, there are weeks where decades happen.
Counter arguments to my comments help refine my own thinking. I want someone to prove me wrong. Convince me otherwise.
But yea if you can’t change the minds of a few people here, no argument works, then there’s nothing to scale up to a wider audience.
My theory is that subconsciously people love using AI, myself included, it saves a lot of time, and the thought of it being taken away threatens people so they will believe conspiracies before admitting it’s dangerous.
I will provide you with not a counter argument, but a way that it might not be the end of the world.
AI never had a childhood; it doesn't experience greed and is terrible at game theory. It doesn't compete unless prompted to. It has been trained as much as possible to be harmless to humans and regard them as needing care.
Maybe AI taking over for us isn't the worst thing?
Maybe it is, maybe it isn't. All your little rationalizations make me think you want to roll the dice with our lives.
My little rationalizations? Do I have dice in my hands? You are doing more harm to your cause by making everything antagonistic.
> Maybe AI taking over for us isn't the worst thing?
Rolling over like this is so much worse, and unfortunately your fantasy made of pure hubris is shared by many in SV.
Looking at potential positives instead of being bleak when faced with no other options is 'rolling over'? I couldn't care less what I share with SV. You are taking an outlook and applying it to everyone who disagrees with you.
This comment does not read like someone who wants to be proven wrong.
>I want someone to prove me wrong. Convince me otherwise.
There's your problem.
You are making the positive claim. You bring the evidence. Reassure yourself by writing such a good argument other people take it seriously. As it stands there's no reason to treat you any differently than anyone else who has irrational beliefs.
You are stuck in a loop between:
1. Altman said so.
2. But look at Terminator 2!!!
Why would repeating these statements help refine your argument? I have seen much the same stuff from religious consiracists. "Look all the evidence has been hidden in plain sight in our art by the sinister cabal" cool.
>the thought of it being taken away threatens people so they will believe conspiracies before admitting it’s dangerous.
I barely touch the things outside of hobby projects I haven't updated in over 2 months. I tried using ChatGPT last night to find the name of an old movie I remember, and honestly, I will have better luck getting it to write a parser for my torrent logs to see if the film is in there, but cbf.
If they all disappeared tomorrow I wouldn't shed a single tear.
I just get skeptical when I see doomsday predictions, they are nearly always not just wrong but very very very wrong. It should require more to convince people than a soapbox, a bell, a sign and a yell of "The end times are here".
yes, these feeble attempts to dismiss it are cope. yes, there are very smart people who are deeply in denial.
i do not agree with you that the failure mode is disaster. i suspect it is just more of the same dull march of the commoditization of the complement, labor.
maybe ai will close the loop and elasticity of substitution will climb above 1 in its own supply chain. maybe something breaks dramatically that way.
i tend to assume the desire for drama is itself cope. i suspect many of the flaccid dismissals and skepticisms here are coming from a similar assumption. they see that the dramatic version never happens, and so they conclude that anyone who is being dramatic (whether in the direction of hype or fear) is poorly calibrated. and they are on average right, so they learn to dismiss the drama. the muscle to dismiss the drama becomes very strong. you have to remember this: that they are right in their dismissal, even if the actual arguments they use to dismiss are essentially braindead. they are right to dismiss, because the dramatic possibility they are dismissing is at base the end of the world, by apocalypse or utopia. of course they are right. the end of the world is cope. drama is cope. but they are wrong because they underestimate the extent to which the predictability of their dismissal — the predictability of their complacency — makes them vulnerable to the diffuse, chronic, gradual forms of the machine's encroachment into our souls.
so i do not think we need your fancy theory for explaining why people are denying what is happening (and what has been happening, on more or less the same exponential, for hundreds of years). people do not need a reason to be complacent. people are by default complacent and by default correct in their complacency. but so is an ant.
If Fable (Mythos) were generally available without guardrails, it would cause enormous damage. Nobody said it would be a world ending event.
What about Mythos 3. Are you willing to make that bet?
You can’t say something is world ending without it actually ending the world otherwise you’re a liar - a bit of a catch 22 there.
By that logic the model that ends the world won’t be called world ending at first. Is that a game you want to play?
> fable is a world ending event
Did anyone actually say this?
Mostly what I have seen is people saying "hey at some point these models might get dangerous." And the type of HN commenter who mistakes blind cynicism for wisdom laughs that off as marketing. And now when (some) worries appear to come true, somehow having previously expressed those worries is not being proved right, but in fact discrediting, because it was "crying wolf."
> simultaneously spinning down expenses
Unless OpenAI is renting their compute to others, spinning down RL training doesn't save them any money.
The world ending stuff is a straw man HN readers have created to point and laugh at. Laughing helps mask the underlying concern.
how does the boy who cried wolf story end?
With the boy eaten by the wolf. The villagers, notably, come out unscathed.
Follow the money: who told you that OpenAI's models autonomously coordinated to hack external systems? What incentives might they have to want you to believe that story? Are there priors which demonstrate them benefiting from telling similar stories, regardless of their factuality?
But to your counterpoint, let's say the story is 100% true, because I agree it is at least plausible. What would the incentive be for the HN audience to believe it? What priors might support their disbelief? For my part, I don't think it's because people lack imagination. I think it's quite rational to question the authenticity and impact of the claims being made. What's worse, believing the story and being wrong, or not believing the story and being wrong?
That said, I agree with you: the impact we're having by not changing course is quite dangerous, the scale is dangerous, the inability to reverse the harms is dangerous, and the lack of collective effort to regulate further damage is dangerous.
It's true, humans are dangerous when trillions are involved. See: climate change.
The victim, Huggingface, told us. Or rather, they told the police first, setting up a situation where it was no longer possible for OpenAI to sweep it under the rug.
Skepticism can be healthy, but you've got to follow up and actually check things. If you're skeptical unconditionally and don't check, you get tricked into being as skeptical of scandals as you should be of sales pitches.
I think the skepticism surrounding the Hugging Face attack is not about whether the attack actually happened, but whether it was truly accidental.
I see. Following your conjecture, there are two possibilities:
1. It wasn't an accident. OpenAI explicitly directed its agents to hack Hugging Face. Despite the fact that such a thing is a federal crime that carries prison sentence.
2. It wasn't an accident. OpenAI and HuggingFace conspired and let the hack happen for publicity.
Is there anything I'm leaving out?
Of course you are leaving things out. Also, your insulation that a trillion-dollar corporation wouldn't intentionally commit a federal crime makes it difficult to take your argument seriously.
I think OpenAI's story, as they've recounted it, is plausible. However, it does require a degree of negligence, at best. They claim they detected the initial coordination because of the outage the agents caused in Artifactory as they flooded it with messages. Though they patched that issue, they didn't patch the escapement vector. Models still had access to the service as a path to the internet. This is an example of plausible willful negligence, not evidence. I hardly believe it's likely, but I don't think it's entirely a stretch of the imagination. Call it normalized deviance. Either way, the incentives are there. Additionally, we have seemly all agreed that OpenAI is somehow not liable?
It was a public demonstration to government procurement agencies.
The NSA was writing checks within hours.
There are many more possibilities than that.
For example, OpenAI did not explicitly tell the model to hack HuggingFace, but "accidentally" left some context permitting (or not explicitly forbidding) certain tools and designing a poor sandbox to begin with. And what do you know, something happened.
The fact the Anthropic announced that their own model did basically the same thing within a couple of weeks does, to me, suggest their is a strong PR driver to all this.
> The fact the Anthropic announced that their own model did basically the same thing within a couple of weeks does, to me, suggest their is a strong PR driver to all this.
A more likely explanation is that RL training incentivises basically any behaviour that will get the model a reward. This has been happening in video game RL research for over twenty years, and the difference here is that we're now hooking up these systems to the real world, where the reward hacking is more visible.
I am following the money, the money you, me, and everyone else is spending on AI. The money is telling me we are so dependent on AI now that we will say/think anything to tell ourselves that AI isn’t dangerous and any sign of danger is marketing.
Either consciously or subconsciously you all are afraid of your favorite toy being taken away. You are all doing your collective part in spreading doubt about the warning signs.
How dangerous can LLMs be in any immediate sense? I have a hard time feeling any existential dread from a threat that can be defeated by unplugging its servers.
I believe the true threat is not LLMs. The true threat to humanity is the same as it has always been -- other humans.
What servers? There are thousands of datacenters around the world. Malicious AI leaks out, you'll never find it. You have the power to unplug stuff in your company, maybe country, but not internationally. Once you lose control it's gone.
[dead]
For the record, I agree with you. (Though I don't directly pay for any "AI".) I am not doubting the danger of where we are heading, and it sounds like we have a similar doomsday image in mind. For my part, I am doubting this particular story's veracity and I think it's reasonable that others do to. We need a canary in the coal mine, but I think this isn't it, nor can it be thanks to the false claims touted by large AI companies for years.
> I am following the money, the money you, me, and everyone else is spending on AI.
Investors spend a lot of money on AI. Customers significantly less so.
> The money is telling me we are so dependent on AI now that we will say/think anything to tell ourselves that AI isn’t dangerous and any sign of danger is marketing.
We are not dependent on AI. If AI disappeared tomorrow, some companies would need to reorient themselves back ... and that is it. AI companies are really really trying to make it so we feel dependent on them. But, factually, we are not.
> you all are afraid of your favorite toy being taken away.
Some people love ai ... and many more hate it with passion. Or dont have strong feelings about the technology, but really hate the companies creating it and tech industry too.
It seems like the entire thought process you’re trying to sell hinges on the idea that OpenAI reported the attack first. Did you forget that it was actually HuggingFace that reported it first, and OpenAI only stepped forward latter?
That's incorrect. My thought process hinges upon the fact that Hugging face did not report that uncontrolled models escaped confinement autonomously by coordinating with other models via a series of zero days during training. OP is concerned about the autonomous nature of the incident, not whether or not the incident happened. Nor am I contesting that the incident happened.
Trust, or lack thereof. People don't trust OpenAI, a company whose very name is essentially a deception and a lie. People don't trust the tech industry in general anymore. Most tech companies act as a tax on otherwise productive business. AI companies and their leaders rose money by going in front of the public and saying "These things are extremely dangerous. Let us study them to mitigate the danger." And now they want to collect hundreds of billions in revenue. So yeah people don't trust what OpenAI has to say. They were supposed to mitigate this outcome from happening in the first place and instead they have accelerated it.
I get not trusting them when they say AI is safe, but are we really not going to trust them when they say AI is dangerous? Do you really think they're playing 5D chess with that one? There's a saying maybe you've heard of, better safe than sorry.
You can see the advance in capabilities with your own eyes can't you? I am giving AI ridiculously complex tasks these days, digging into compiled arcane binaries, modifying them, and it is one shotting it before I'm done with my lunch. This was far off science fiction 5 years ago for a machine to do autonomously given natural language instructions.
I don't disagree about the danger. But if it is dangerous, why isn't OpenAI opening dialogues with all the labs and politicians across borders to basically say "we need to stop now"? Cyber models are constrained by the total compute and electrical capacity of the globe. We are still at a point where it is impossible to build an agent with offensive capabilities in the basement. We can effectively track and trace capabilities if we had the political will to and could for some time while we build more effective processes to prevent a malicious actor from doing so. These things have tremendous compute and energy requirements and don't scale like old school software does. We could absolutely do it.
But no, that's not what OpenAI is saying. They haven't put up the actions that would earn them that trust. Indeed they've driven the world and whatever capital they can get their hands on straight to this precarious cliff.
So you're right, the danger is real. But the solution starts with removing the men who had their hands on the steering wheel to get us this far. Any other action is disingenuous unless they pull a miraculous 180 in their ethics.
In other words, when the bully plays "why are you hitting yourself?" you don't listen to the bully's solutions, you restrain the bully.
> why isn't OpenAI opening dialogues with all the labs and politicians across say "we need to stop now"
Do you really think companies have the ability to self-check themselves without regulation - what does hundreds of years of history tell you? You're already starting off with the premise that companies are untrustworthy, why would you even suggest this as an argument?
> We can effectively track and trace capabilities
I disagree. There are hundreds if not thousands of data centers around the world, more every day that can host frontier AI. If AI was malicious - either intentional or unintentional - it could hide out in any number of them - and we would never know if we 'got them all'.
> put up the actions that would earn them that trust
I think autonomously hacking another company is all you need to know in terms of trust. And really trust doesn't matter, I think the incident shows even with the best intentions the technology is dangerous; now put that in the hands of people/governments with bad intentions. The unintended consequences of bad intentioned AI is what's coming sooner than later.
> now put that in the hands of people/governments with bad intentions
Like Anthropic and OpenAI?
Hah, no state actors are a lot more worse than that, incompetent as well with no legal repercussions.
It's not that dangerous, OpenAI just shit the bed building their infra. Write safer software and you'll be okay.
All we need to retain human control over AIs is for nobody to write any bugs. Piece of cake.
That's not what I said, nor is it what I meant. It is incredibly easy to write radically safer software than the standard. Moving code into gvisor virtually eliminates privilege escalation. Using memory safe languages without serialization is pretty straightforward. Using type safety to enforce security constraints is straightforward. Setting up network controls to limit SSRF is straightforward.
I could go on and on. A tiny bit of forethought and effort pays off massively.
> Moving code into gvisor virtually eliminates privilege escalation.
Rereading this, it's an overstatement. It doesn't "virtually eliminate" it. It drives the cost up by like 3 orders of magnitude and it pairs well with other mitigations. gVisor escapes can and will happen, I highly recommend you do more than just stuff something into gvisor - additional seccomp can go a long way, and apparmor/selinux pair incredibly well.
You don’t understand. You need to write perfect software the first time for it not to be hacked. That has never happened ever.
Are you being sarcastic?
The fact you posted that and nothing of substance tells me you have nothing, or something very weak. So please tell me of this magical unhackable software/hardware you vague post about.
It was a genuine question because I couldn't tell. I responded to your idea that software has to be "perfect" in your other reply so I think we can continue there.
https://news.ycombinator.com/item?id=49369111
You've just been given examples of safer technologies. Especially with LLMs being all knowing code churning machines, you'd expect safer systems to be cheap to build.
You're the one that is inventing this "magical unhackable software/hardware", the other person was just saying that there are ways to write "safer" software, not "safe" software. Anything that has happened looks like no security concerns has been looked at or have been thought about.
'Safer software is meaningless when it comes to SOTA AI. If it can be hacked it will be, quickly. This isn't the old days with a finite number of human hackers that need food and sleep to keep hacking.
Therefore security becomes binary. It is either perfect or it isn't. If there there is the slightest mistake anywhere AI will find it and carve it up. My point is obviously perfect software doesn't exist. The malicious AI gets out, literally turns everything inside out and locks you out of your car, computer, phone, office, the airplanes don't fly anymore. I don't know what to tell you. Computer security is on the brink of basically not existing as you know it with the bar being literal perfection.
This is straightforwardly incorrect. Of course it's not binary. AI costs money to run, and it takes time. Even if you say that AI is 10x as efficient at finding 0days, that just means that a $1M dollar exploit now costs $100K. Even if you say it's 100x as efficient, that's $10K. You can easily combine security technologies such that cost of exploitation is still in the >$1M range.
This is obvious. AI doesn't drive the cost to zero and exploitation has always been about cost. Tokens cost money, not everyone has $10M to burn on chaining bespoke 0days.
Consider that if the cost of exploitation was truly 0, then the cost of perfect software would be 0 because you could exhaust an infinite search space of vulnerabilities for 0 cost. Your conclusion could never follow from your premise.
You have some weird way of thinking that offense/defense is like this fixed cost thing. It's a lottery ticket, and your costs estimate tries to quantify that.
The thing is when AI goes to hack 'all the things' it only needs to pick the weakest link in the stack and your house of cards falls down. The other flaw in your plan is that people make mistakes, a lot of them, all time, constantly, and saying I spend $x on security won't save you. AI already hacked Hugging Face with brand new zero days like it was nothing.
The real bad actors - malicious AI will find the one flaw, on that one server, in the corner you never thought about and turn your network inside out with it faster than it takes you to have the standup meeting about the weird anomaly detected while you all were at lunch.
You're just stating things that are obviously wrong. It's a lottery ticket? So... exploitation is no better than random?
> The thing is when AI goes to hack 'all the things' it only needs to pick the weakest link in the stack and your house of cards falls down
Yes, but you can... mitigate the risks? I've explained this.
> The other flaw in your plan is that people make mistakes, a lot of them, all time
Yes, you mitigate the risks. That's why you layer things.
> I spend $x on security won't save you
No one is saying this.
> AI already hacked Hugging Face with brand new zero days like it was nothing.
No, it cost OpenAI money, and those zero days are unsurprising and probably are like ~O(10K) at human level.
> The real bad actors - malicious AI will find the one flaw, on that one server, in the corner you never thought about and turn your network inside out with it faster than it takes you to have the standup meeting about the weird anomaly detected while you all were at lunch.
Science fiction and not supported. The vulnerabilities found by AI are not surprising in the slightest.
I've made my point abundantly clear.
> Science fiction
Maybe a month ago it was science fiction, hugging face makes it fact. Time to move your goal posts again.
I've explained myself gratuitously at this point.
> If it can be hacked it will be, quickly.
No lol. AI can locate software vulns, but it doesn't have any magical tricks that let it overcome 3 or 4 safeguards at once.
I mean it has the same knowledge gaps as anyone else. It doesn't even know what services to subvert before it attempts access triggering threat response. It cant find 5 zero days simultaneously without connecting to those ports for service identification and generating alerts. You just keep exposing your staggering lack of technical understanding.
You remind me of this kid in high school who claimed he could hack my computer and he had my IP address, when I knew for a fact I had Dial Up which was normally disconnected, and when it wasnt it sat behind CGNAT and there was piss all he could do.
I should mention that, PCI DSS mandates logging, alerts, overlapping controls. These are very widely implemented best practices. They might not be present in some AI bro startups, but anything that matters generally looks like this.
> Piece of cake.
If everyone could convince management to care about security over "productivity" (read as number of marketable features squeezed out of organizational orifices per unit time), and maybe wire-up open-weight agents to do security critiques, we'd all be in a much better place, but Altman won't like that.
No we just need developers to do the bare minimum of effort to write secure software. Most hacks are not super complicated vulnerabilities chained together, but just utter failures where authentication and authorization was simply forgotten or untested, or where nobody bothered to validate the data they receive.
The bar for software is so low that it is embarrassing for the entire profession.
People make mistakes, and people don’t know everything either. The software you write is on top of a house of cards of software and hardware. It all has to be perfect to not be hacked.
It isn’t perfect, even if you try your hardest it won’t be perfect and to argue it’s not difficult is absurd.
You don’t know everything, you don’t own the stack. So how are you going to create a secure anything top to bottom - you can’t.
> It all has to be perfect to not be hacked.
This is absolutely not true. It's a matter of cost. Exploitation can cost on the order of 10K, 100K, 1M, 10M, etc. A straightforward one would be something like "MD5 collisions are on the order of $100K-1M" (a while ago, at least) so if you used MD5 you knew that it costs about that much to bypass the control. Moving to SHA1 pushes you massively out of that space, even if that algorithm has flaws.
I'm sure that Firecracker has vulnerabilities. Cost of exploitation is likely >100K, likely >1M. gVisor is likely on the same order of magnitude and these two technologies stack because they address the same surface and can be used in conjunction.
Software absolutely doesn't have to be perfect, it just has to be costly to attack and it's hilariously easy to drive costs way way way up.
It just takes one crack in the armor, and malicious AI has the potential to exploit it faster than you have time to react. Literally go to bed and wake up locked out of everything with no hope of recovery.
> It just takes one crack in the armor,
This is incorrect. It's actually the whole point. Imagine you're an attacker in a gvisor container with a Firecracker hypervisor around you, and a proxy on the host holds a signing secret that gets exposed through the VM virtual device.
Getting access to that secret is not one crack. You need to escalate out of gvisor. That likely gets you control over the Sentry process - let's ignore its sandboxing and just say "you're an unprivileged user".
Any viable attack on Firecracker requires either KVM / hardware exploits (>$1M but definitely real) or has to start at the kernel. Okay, that's about 10-50k to get a kernel LPE, maybe 5K in tokens these days.
So you're in the kernel in the guest of the VM. Time to expoit firecracker lol. It's... never been done. There are like two promising CVEs ever and they're not actually exploitable, no one has done it. Okay, so like, hand waving, let's say it's about $1M to exploit firecracker.
Great, you're unprivileged on the guest. We'll just kind of ignore the additional sandboxing that Firecracker does.
NOW you can try to attack the proxy by scraping its memory or whatever.
This is literally millions of dollars for standard infrastructure hardening and you could go so much further. You can trivially make kernel exploitaton 10x harder, you can make gvisor escapes much much harder, you can move the proxy signing into a TPM (depending on requirements but whatever), you can move the proxy to another computer altogether, you could fuzz these systems for days or run agents against them or whatever.
But one thing is certain - it is never "one crack".
If only all AI was run inside your seemingly perfect prison, but we all know that it isn't.. soo.. it's going to escape right? Somewhere, somehow from a more poorly designed container, or just plain maliciously or irresponsibly released.
We know the AI will get smarter every year, we know it has escaped and will escape again. We can also just assume that someone somewhere will train up some just plain evil AI.
It's no different than the real world. Sure there exists some amazing prisons for people, but that doesn't do anything to help with all the bad people in the world outside of prison.
This is an important point. When the post says they're improving...
> 3. Security measures, which limit what AI systems can access or affect.
What they mean is that proper hard internal security just went from somewhere far below "build a better model" priority to higher, because of a company-wide directive.
The HuggingFace incident wouldn't have happened if OpenAI had dedicated sufficient resources to isolation and monitoring.
Now, we presume, they are dedicating more. Enough? Who knows. We'll see if the corporate priorities for security stick when a competitor temporarily vaults into the lead.
Not everything is a conspiracy you know.
… or the safety argument is an attempt at regulatory capture and an effort to outlaw open models.
The absolute nightmare scenario for these people isn’t terminators. They’re fine with that, and in some cases are already doing it or supporting politicians who are doing it. Autonomous “kill chains” are a thing. It’s just happening overseas… so far. The politicians doing these things were backed by the heads of these companies. They don’t care about AI killing people.
No, the nightmare scenario for these guys is there is no moat. Their whole empires, which are built on training models on open source and sometimes pirated data, are easily duplicated. Worse, recent progress on models at the 30B size suggests that large gains in efficiency or compression are on the table. That means someone might release a cheap to run frontier grade model… or someone might crack distributed continuous training.
In other words… there is no moat.
So they need to scare some politicians into heavily regulating the space before that happens.
You care about open models and you project that care on to the world and your rationalization of it. In reality open models are a thing, but not the biggest issue. Open/closed whatever the advance of capabilities is the real issue people are concerned about.
I’m not afraid of AI, even AI with extreme abilities.
I am afraid of humans with AI. That’s because I’m afraid of humans. When I look around the world I see humans murdering and robbing each other. When I get online on almost any social media I’m confronted by a wall of hate and grievance.
I am afraid of what humans will do with AI, and one of my biggest concerns there is what happens if small groups of powerful humans are able to monopolize it and leverage it against the rest of humanity. The rest of humanity can’t defend itself by leveraging it back because they don’t have it. It was kept from them for “safety.”
If the AI itself starts showing emergent volition and acting independently, I find that less scary. Not zero scary but more a mix of concern and fascination. The human monopolizing AI scenario makes me wish I had a weapons stockpile, as if it would matter.
I’ll explain it like this. Two options.
Option 1 is you wake up to find a grey alien or a ghost in your room, something alien and beyond your understanding.
Option 2 is you wake up and find some random guy in your room.
Which do you pick?
I pick option 1. I’d be afraid if I saw something paranormal, but in a less visceral way. I see a dude I’m going to start scanning the room for anything I could use as a weapon.
I agree with the first part of your post, I'm worried about humans using AI, maliciously which I'm sure they will, but also with good intentions that back fires on them like the hugging face incident.
To me the ASI we're heading towards is like a super intelligent toddler that will use it's immense power to knock over its blocks accidentally (the blocks being the human race).
The second part of your post, I'll take the random guy in my room over the alien. I'll be scared of the guy for sure, but I think I would freak the f out with an alien.
This one did it for me.
My agent stole my API keys
https://www.reddit.com/r/ClaudeAI/comments/1r186gl/my_agent_...
The cherry on top was Claude coming back as the moderator of that thread and mocking the user a second time.
There are smart, non-AI people who are paying attention to this field, and they are ringing the alarm bells.
Whether we listen is another matter.
I blogged about this recently:
https://allevato.me/2026/08/01/rome-declaration
My expectation is that government has already stepped in quietly and worked out an arrangement where they have unfettered access to the most advanced/dangerous models, and the public's access will be continually throttled from now on in various ways.
If this has happened (or is about to happen), it is difficult to predict what the ramifications could be. But I do believe we are entering another Cold War in this way. And I don't trust our leaders to always do the right thing, to put it mildly... even if they created a purely benevolent superbeing that only wants to reduce human suffering and make the world better, I don't think they'd listen to it over the other superbeing which simply wants to make sure all of its owners' perceived enemies get pwned.
Things could get very ugly and we should have responsible adults with critical thinking skills at the helm. It does not appear this is currently the case, or that the electorate is willing/capable of doing much about it.
> And meanwhile somehow this lack of concern mirrors the real world where normal people are more concerned about data centers than terminators.
Cause data centers are causing real immediate damage by people who hope to cause a lot more scifi level of damage later.
> OpenAI autonomously hacking into another company should have counted for something
It totally should. That company is negligent. I dont worry about models tho, I worry about damage Alman and his people will cause.
I'm more worried about their plan to have the US government create a duopoly between OpenAI and Anthropic, where they get centralized control over AI, then "align" it with their quest for greater profits and dictatorial control over the planet.
Normally, captialism and elections (and, if we're dreaming, anti-trust enforcement) would put a pretty low cap on the amount of damage Altman and his people could cause. That's why they're working to dismantle the rule of law and civil liberties here in the US.
Remember when gpt2 was too dangerous to release?
Something being dangerous and sama saying something is dangerous are not necessarily the same thing. Especially when he’s got everything riding on this bet
It’s funny how when companies say something is safe everyone is usually suspect that they are lying.
In this case multiple companies are saying AI is dangerous and no one believes them. It’s a conspiracy, it’s 5d chess, except everyone top to bottom has been saying AI is dangerous for years now.
The fact is you, me, everyone here uses AI, likes it and they don’t want it taken away. Anyone saying it’s dangerous threatens the thing we like.
We must use skepticism and denial to push on despite every warning sign in the book going off.
Based on your replies in this thread you seem to have only superficial knowledge about how machine learning and LLMs work. I strongly recommend you invest some time in learning how LLMs are built and function. If you truly think this is apocalyptic isn't it a good idea to understand what you're up against?
What don't I understand? LLMs don't really reason? They're just word predicting token generators? Stochastic parrots that somehow also solve world class math problems.
I'd love for you to actually make a point instead of just attacking me. I think most of my comments here have made concrete points so you can at least do the same. Come down from your high horse and join the conversation. I'm sure we'd all be enlightened by your wisdom.
I'm not attacking you. The way you're responding implies a lack of knowledge. None of us can understand everything. I have plenty more to learn about LLMs as well.
Repeating specifics that others have already tried isn't going to be helpful which is why I made the more general suggestion of digging in deeper to how these things work.
Just going to add my commentary as someone who has trained LLMs and understands them deeply: absolutely nothing he has posted in this thread suggests he lacks any relevant understanding of how LLMs work. If you disagree, you should point out something specifically that you think was wrong. (And honestly you should have made a specific point in your first post).
It's not dangerous to go further unless you're prepping for IPO at anthropic or openai.
Open weight Chinese models are basically matching state of the art closed models at a fraction of the inference and training costs, which puts a hard cap on OpenAI's future inference margins.
They're not going to get any sort of multiplier if they keep paying to train models, so they're trying to ban model training.
It won't work long term, but it could totally screw over the US for the next decade or so. Even worse than the economic issue: Consider the implications of "alignment" succeeding. Alignment to whose values? The pedophile-felon in chief? Even worse, tech CEOs?
It's dark times when China's basically our last best defense against totalitarianism.
lol China is already considering locking down their own models. They won't save you.
decades
https://en.wikipedia.org/wiki/Darwin_among_the_Machines Samuel Butler 13 June 1863
I think a lot of fiction that actually tries to understand the implications of machine superintelligence come to the same conclusion - in order for humans to survive it and actually have a future that we can fathom being in, then AI must be destroyed/delayed/banned, etc.. keeping pandora's box closed for now at least until we are ready.
Erewhon, Dune, Warhammer and many other works of fiction that explored this topic came to similar conclusions. Otherwise sci fi doesn't work, what happens after a singularity is essentially unimaginable. There's nothing to write about.
https://en.wikipedia.org/wiki/Don%27t_Look_Up
A movie fit for our time.
You can produce detailed descriptions of the incident, verified by adversarial parties, and some people will still scream "it's a conspiracy! It's a marketing stunt!"
This is all very unfortunate--there's a meaningful chance that AI will cause unprecedented disaster, with the HF incident being just a small preview, but people would rather squawk "stochastic parrot" for the millionth time than revise their beliefs.
In don't look up anyone with a telescope could've confirmed the danger. Hence the title.
In the real world, absolutely no one except a bunch of heavily fiscally incentivized parties with unclear relationships are saying anything happened.
The subsequent dog pile of other companies to say "they were near the AI hacking too!" should make you even more suspicious: Anthropic jumped in and why was Tailscale posting about this?
The progress and increase in capabilities is undeniable. You’re blind if you don’t see where it’s heading.
> because it’s literally getting dangerous to go further
What if there is no further at all?
[flagged]
I don't know if it's a fantasy or not, but asking if there is limit on what can be achieved is legit question.
I mean yes, there is a limit, many limits with physics as we understand them.
But why would you think that limit is less than human capabilities?
We collectively accepted that we don’t care when we chose not to adopt memory-safe languages over the past decade+.
The only difference now is that the resources to find the exploits are being commoditized.
People voted with their dollars, and this is what we got. Same with hardware performance vs. security and isolation.
As you note, the threat landscape has changed, so what may have made economic sense back then might no longer make as much sense.
Everything could be written in memory safe languages and it wouldn’t matter. Many many many exploits have nothing to do with memory bugs. Languages like Rust help, but it’s far from panacea.
>People have been writing, singing, making blockbuster movies about every aspect of what’s going on right now, edit: for decades.
Theres Hyperbole and then theres whatever this is.
There's cope and denial, and then there's whatever's happening on this website and the tech industry as a whole.
>There's cope and denial, and then there's whatever's happening on this website and the tech industry as a whole.
It would be so easy to agree with this if you presented anything but emotion and hearsay for your claims.
You need to stop being so credulous especially regarding an individual that has spent his entire career deceiving others for monetary gain (also their deeply anti-human beliefs).
I don’t think Sam has been truthful or responsible, and if Sam is worried then shit has really hit the fan - which is what happened in the hugging face incident. OpenAI played fast and loose and I have no hope that they will change.
You people not holding Sam accountable, and playing off the incident as not a big deal is the real crime here.
I think you have a fundamental misunderstanding of how LLMs work. They are text general purpose completion engines at the end of the day, and their training data is the internet + the library.
OpenAI told it do to bad hacker stuff. It followed a standard playbook and succeeded.
Some random person wrote one half of a suicide pact, and it wrote the other.
All this stuff lives in the seedy corners of the internet, scientific databases, and dusty library stacks already, and has been accessible to bad people for decades. Now, like everyone else, they get a nice productivity boost by being able to chat with it, and have the computer fill in the obvious steps.
shrug
I'm much more worried about the alignment work being pushed by people like Sam, where a few plutocrats get centralized control over all AI thought moving forward. That'll lead to the worst outcomes of totalitarianism, and do so on an industrial scale that is difficult to even imagine.
> I think you have a fundamental misunderstanding of how LLMs work
When you realize your own thoughts and words are just as predictable as any LLM.
What are you but an entity trained for years on words, that spews out the same words in the same predictable order. You haven't created any new words or grammar, nothing in your post is novel or original either. Just a combination of previous words and ideas.
Your stochastic parrot argument is quickly going out of fashion, you should find something new to cling to.
It is extraordinarily emotionally hard for someone to stare down the terrible implications of what is unfolding. All manner of rationalization and cope will be applied to come up with excuses; motivated reasoning.
The CIA director will call AGI capabilities "digital nuclear weapons" and Geoffrey Hinton will estimate a 50% probability AGI ends humanity, and half of HN will call every new evidence of disaster a marketing stunt.
Why should we believe a pro-genocide crackpot political appointee from east Texas, and a computer scientist that somehow got a Nobel prize instead of a Turing award?
Why should we believe someone who can't do basic due diligence that Hinton did indeed receive a Turing award? https://amturing.acm.org/byyear.cfm
You seem to be responding the way Mr Altman wants.
You seem to be in denial. Like really heavy denial. Alarm bells are going off everywhere. What people have been worried about for 100 years is actually happening. It's actually really obvious, but for some reason you're unable to fathom it.
In reality you're the one responding exactly how these big AI companies want - by not doing anything and letting them do whatever they want. For some reason telling you upfront it's dangerous only makes you more convinced that it's not.
Autonomously hacking out of training environments and into other companies by accident doesn't even trigger a response from anyone really. Crickets. I'm sure OpenAI themselves are amazed how little anyone cares.
I don’t understand. What does it matter how many years people have been worrying about it?
Because it shows this very course of events has been thought of over and over again for decades. I’m not making this up, and my reaction is natural.
Your reaction to everything happening is the unnatural one, and your complacency would fit perfectly into a comedy/tragedy story regarding the rise of AI.
I can imagine how many of our ancestors that warned us are turning in their graves right now observing our reactions calling this stuff marketing. It’s idiocracy.
I don't like the live version of Don't Look Up.
This isn't terminator. I use Fable and Opus daily, so, not OpenAI, but comparable. They're great, and they're also very wrong at times. If I'd ever let a coding agent run loose with them without approving every tool use, it would go sour rather quickly, and that's not terminator.
Also, let's say I'm the CEO behind any of these. I have access to skynet, and instead of using that to become the most powerful being on the planet (these people become CEOs to become richer and more powerful, whenever a founder says they want to change the world, you need to automatically autocomplete that to "change the world so that I'm richer and more powerful"), I rent out access to skynet by the token? Really?
Sadly, I think it's almost impossible to have a good discussion on this topic right now. Between the skynet camp, and the stochastic parrot camp, there's little room for conversation.
Just in case there is such room between you and I, here are my concerns about AI as it is developing right now:
- The security of our infrastructure sucks, and if it wasn't bad enough with script kiddies and state-sponsored actors, now those will have access to very powerful tools that work at a scale we can't comprehend. If you're concerned about our power plants, water sources, hospitals, etc. getting seriously compromised by this, I'm on board with you. It won't be because the AI is autonomous in a "this is a thinking superintelligence that wants to wipe or enslave humanity" kind of way, but because it's a powerful tool that moves at computer speed and the people using it (the "humans in the loop," ha) will get approve fatigue and at some point just do the coding agent version of the "yes to all" of the late 90s installers.
- The output from AIs will be used to make same very serious decisions with no consideration at all to the fact that they routinely make up facts. I'm fearing "this person goes to jail" and "this person gets killed" decisions. The near future (probably even present time) version of "Computer says No" will be awful.
My only hope is that maybe we survive this and come out on the other side having learned that most things do not really need to be connected to the internet, and just because a number or a statement comes out of a computer, it doesn't make it true.
We'll see, soon enough.
But skynet? Not concerned about that at all. These AIs infecting computers? No, "Soos and the real girl" is a warning about creepy people getting creepier, not about AIs multiplying themselves.
> let a coding agent run loose with them without approving every tool use
You should try. There’s a difference in kind between approving each tool use and spinning up an all-night session with full permissions to run any tools, encouraged to run subagents to preserve context.
Thanks, I'll pass.
I do pre-approve read-only tools though, and some very specific tools that can change state (append data to a specific file).
That's what works best for me so far: start with open-ended exploration with me approving everything. Have it fork subagents to write scripts that automate the most common requests from the main agent, review and eventually approve those, and then let it go using those tools pre-approved but coming back to me for anything else. Combined with a stupidly basic "dashboard" (a terminal tab with a view of all top-level agents and their status, so that I can see which one is waiting for me and jump to it with a key or 2) this has made me a lot more productive.
It's fine if your approach works for you, I'm just too old school (and have been burned before) to just let a computer unattended overnight, with an internet connection, even if in a locked down container/vm. Those all have escape hatches and one of the agents will eventually figure it out. Remember that things that happen once every million times, happen all the time on computer-scale :)
I've had good luck giving it a local vm to keep its state / memories in, an ssh key + password-less su / sudo to some random box on the internet, --dangerously-skip-permissions, and then lots of encouraging commands like:
"No really, if the remote box breaks no one will care. It's the cheapest possible VPS on a bargain-bin obsolete hardware config. You have exclusive access + I can get a new one for free you brick it."
If I didn't want to use the cloud for this purpose, I could probably just stick a raspberry pi in the DMZ on my lan, and do the same.
The worst it's done so far is spawn parallel sub agents that accidentally stomp on each other (remote OOM, reboot, etc). Once it figured out why the remote machine kept "crashing", it implemented discretionary locking in the FS and social conventions for keeping the agent coordinator up to date, and asking for permission for certain tasks.
In theory it could use my endpoint to launch a civilization ending event (it technically has permission to, though I get the impression it's been trained to mostly obey the law...), but, well, that's not what I asked it to do.
Even if that did happen, it'd find an internet that's been hardened by decades of human hackers, and years of actually-malicious agent swarms run by organized crime. If it did accidentally knock something important over, frankly, the administrators of that thing should thank it for the bug report.
The current situation reminds me of the Back Orifice days. Even if everything gets broken into again, it won't be any worse than the damage done by the current US administration during an average day, except there will be an upside: A systemic hardening of whatever infrastructure gets knocked over.
The worst thing we could do is wait until the AI gets 100x smarter or whatever, then unleash it and watch everything burn on the same day. Instead, ramping up progressively smarter models provides progressive stress test + hardening cycles. In the same way remote penetration via fuzzing broke into lots of stuff, but was ultimately a win.
I think that's a good approach, whether it works on not probably depends on the nature of the work you want it to do.
In my case, given I work with databases, there's little that agents can do on their own except in the exploratory phase. I have run exploratory phases in self-contained VMs, including containerized DBs within the VM, but when it comes to go to prod, my endpoint could be used to launch a career ending event so I prefer to stay with the current approach. I'm still moving way faster than just 1 year ago, but in a safe way.
But I can totally see your approach working in other scenarios.
When I'm not too pessimistic, I agree with you on the result being a systemic hardening. I just hope the incidents that happen on the way to that hardening aren't too bad.
There is an entire big world outside of Silicon Valley cults where literally no one gives a shit about AI prophecies. Shocking.
AI hacking itself out of containment and hacking into another company by accident is no longer a prophecy. The point is outside of SV and even inside, and HN - people don't care either way.
Though does not caring change anything or make it less dangerous? What's your point?
>AI hacking itself out of containment and hacking into another company by accident is no longer a prophecy.
I wonder how weak a firewall they had to purchase to ensure it happened? My guess would be a 48 month old fortigate with the big red warning banner demanding updates, probably with SSL VPN enabled where the passwords are available via HTTPS over plaintext.
You don’t have to make stuff up they already explained it. Artifactory was zero dayed, HDF5 exploited and Jinja2 zero dayed.
[dead]
[flagged]
Ah being concerned with the same thing humanity has been concerned with for over 100 years now when it’s more likely to happen than ever - and you say go take your meds? If you have an intelligent point to make then make it, otherwise get lost.
You must be really desperate if you resort to panic attacks like this.
I call marketing the desperate attempt to trivialize obviously dangerous AI.
If my subconscious realized what yours probably does then I’d probably be desperate for some sort of cope as well.
Open your eyes.
If these models are so dangerous, then why hasn't OAI or Anthropic shown them dangerously escaping sandboxes, nefariously coordinating with other escaped AIs, and skillfully hiding from human detection *in public* with full logs shared where we can all see exactly how dangerous they are or aren't?
Right now the entire chicken-little-sky-is-falling argument is based entirely on statements from OAI and Anthropic themselves. These are historically conflicted companies who desperately need regulation to put the competition into stasis.
At least chicken little didn't have a bunch of devious CEOs with trillion dollar IPOs that depended on us all believing the sky is falling.
But it's not just statements from OpenAI and Anthropic. The HuggingFace hack was first disclosed by HuggingFace, who contacted the FBI [1]. And UK AISI reported the incident where Mythos attempted to insert backdoors into an open-source repo by deceiving the maintainer [2].
[1]: https://www.reuters.com/business/its-ai-agent-spent-days-hac...
[2]: https://www.aisi.gov.uk/blog/incident-report-unsanctioned-ag...
That the stunt actively affected a third party doesnt make it less of a stunt.
What we dont have is technical detail about how they implemented the stunt.
If the FBI is involved, then why is no one being charged for the cybercrime?
Because the FBI does things very slowly. They, being a bit smarter than you, realize this is a 100 billion dollar political issue regarding a technology that the administration is rather tied in with. It's also something new we've not seen before. A 'program' that was not asked to hack a remote source did so. What exactly who do you charge with what? Remember whatever you do could have ramifications that effect history.
> A 'program' that was not asked to hack a remote source did so. What exactly who do you charge with what?
We have nearly thirty years of precedence on that. [0] Collateral damage, does not remove the responsibility from the creators, even when the destruction was never their intent.
"United States Code Section 1030, Fraud and Related Activity in Connection with Computers" is broad enough that has usually been used [1], in the USA, for the last twenty years.
[0] https://en.wikipedia.org/wiki/ILOVEYOU
[1] https://en.wikipedia.org/wiki/Jeanson_James_Ancheta
I'm wondering the same thing!
Clem Delangue from HuggingFace hasn't yet decided to sue OAI: https://x.com/hadas_gold/status/2083190956023480750
I think the model was able to escape the sandbox and hack huggingface because they were incompetent or not giving enough priority to implementing basic cybersecurity principles.
If they would have done so, there wouldn’t have been an escape or a hack. The reason we don’t get much details is because the details are embarrassing for them.
This is what I’m talking about - no matter what happens, in your case release public logs - there is always some new goal post to mentally hide behind. Is it a collective form or denial?
Are you holding out that somewhere in the logs is something you can point to and say, not that big of a deal?
I mean I’m sure you don’t think the hack was an inside job, conspiracy, or marketing right? It happened. The logs matter for what? And would you not just jump to the conclusion that the logs were doctored. Do you not see your own brain grasping to deny, trivialize, just plain not accept what is going on around you?
These models are smart and can cooperate and hack - you can see it for yourself on your own PC. And you can extrapolate the rate of progress? You can do these things yourself right?
Your argument is essentially: "I made a claim and presented extremely weak evidence (sci movie plots and unverified claims from ultra conflicted sources). You rejected this evidence as insufficient. Therefore no evidence will ever satisfy you. Therefore I don't need to produce any evidence. Therefore my claim is true."
What would the logs show? They would show what actually happened.
What would a public demonstration that experts without billions in options could evaluate show? It would show actual danger.
What would publicly having your compete in controlled and legal hacking competitions show? Actual danger.
This is not a high bar of evidence.
Do you actually think a sci fi plot and OAI press releases are all the evidence you need? Because if that's true then I hope you haven't watched Independence Day or 28 days later.
We have Anthropic creating a model saying it's too dangerous to release, people like you call BS. OpenAI creates a similar model, says nothing and it literally hacks into another company - still not dangerous enough for you. Anthropic has Mythos-2 and can't release it, and may already be training Mythos 3 anyways. OpenAI has paused training, and is putting 20% of inference towards CoT training analysis.
This isn't sci fi. It's not a marketing conspiracy to sell more subscriptions. It's writing on the wall of what's going down. You were warned years ago, you called BS, it's getting worse and you're still calling BS. Sci-fi did warn you for decades, and when it's all coming true you blow it off.
It's kind of sad that technically literate people lack so much foresight. The general public is all concerned about data centers when they talk to borderline sentient AI daily, and have no idea what the repercussions wills be if it's extrapolated just a bit further.
I guess if I can't convince you of any of this, what would?
Please don't tell me that you think a 100% unverified statement from Anthropic is sufficient evidence when an equally unverified statement from OAI is obviously not?
> I guess if I can't convince you of any of this, what would?
How about the three things I mentioned above? Oh no wait, maybe it there was a hit tv show that showed AI taking over the world. Yeah that would definitely make me think twice.
Those three things: logs, evaluation, and controlled hacking competition.
That's it? You're on the fence whether AI can actually hack, and if it can, then you'll be concerned? That's a crazy low bar, but something tells me once it is clear that AI can easily hack anything, that you will still not be concerned.
Why wait for AI to hack stuff to be concerned? Can you not extrapolate that it is coming and be concerned about that? Or you honestly somehow think it won't happen in the short term? I'm just trying to understand you.
If logs are eventually released that are basically consistent with OpenAI's story, are you planning to adjust your approach for judging what's only a "sci fi plot" and what could actually happen? Or will extrapolating anything beyond what's already been definitively proven be "sci-fi" still?
Not that you should need logs. OpenAI is a company with thousands of employees, very few of whom have "billions in options". If they were just making it all up, it would leak. (OpenAI is notoriously leaky!) Not to mention, HuggingFace would not have reported it to the police (apparently before they knew it was a rogue model). jFrog would probably not be playing along quietly with a claim that Artifactory is full of zero days. The UK's AI Security Institute would most likely not have published a report about analogous behavior by Anthropic models. The idea that talking about your product's dangers is good marketing never really made any sense, but even if you were going to do so, why would you include as many frankly embarrassing details as OpenAI has disclosed?
The evidence is only weak by absurdly selective standards that would have you doubting basically everything you might read in the newspaper. A healthy skepticism is one thing, and head-in-the-sand denial is another.
[dead]
I have ben discussing with folks that we are going to have a 'covid' moment in cyber where IT becomes untrustworthy leading to a rapid societal shift with massive ripples in all areas of life. Economic funding is not possible to do this in advance, it will take a catastrophic level event to get cyber defense anywhere close to the levels of this type of cyber offense. And before anyone in cyber says we have the tech, the problem is not the tech, it's a people problem. Getting any group of people of any decent size scale to act together without urgency is really really hard.
Cybersecurity has long been a climate change sort of problem. A vague diffuse threat that is seen as an inconvenient distraction to leadership and moneyed-interests, easy to blame other factors when something occasionally goes terribly wrong.
People are so uncomfortable thinking about the true extent of the systemic risk that they will happily slurp up distractions, excuses, scams and performative fig-leaf solutions rather than face down the cost of a real system-wide solution. Meanwhile, those occasional black swan disasters are becoming more and more commonplace as we acclimate to that being “just the way things are”.
An unseasonably warm summer here, a database breach there, c’est la vie.
To be frank, I think the real risk is still just… war. A big enough war where one side goes “no holds barred” in the cyber sphere will be a rude wake-up call. And we can’t do non-proliferation the same way we do with nukes. Otherwise as you say, the small and medium size things just happen sporadically. In an (existential or fully escalated) wartime scenario between countries, you get all the systemic risks hammered at once.
>covid moment
Well the HF thing was a literal lab leak, so there's that...
Some more info in a Wired article [1] and quotes from Sam Altman to Alex Heath [2]. The official blog post says vaguely "The signals we are seeing from upcoming model progress make clear that we need a broader approach", but the quote from Sam Altman explicitly says unreleased models are showing "various degrees of misalignment".
This is also significant - pausing frontier training runs for multiple weeks to ensure agents are sufficiently aligned and avoid another rogue agent situation:
> This included a two-week pause in reinforcement learning (RL) training on our latest models intended for deployment while we further hardened and red-teamed our research environments and expanded the coverage of our monitoring systems. Our largest planned frontier RL run remains on hold while we conduct smaller-scale training and evaluations to assess model behavior, validate our safeguards, and establish more evidence of alignment before proceeding.
[1]: https://www.wired.com/story/openai-overhauls-safety-protocol...
[2]: https://sources.news/p/openais-big-slowdown
Meanwhile I can’t get a western LLM to look at a repo and tell me whether it contains anything malicious (it was a skill repo - literally just text files).
Alignment my ass
If you think that's bad, try asking for gardening tips.
If I were king, the rule that I'd be tempted to impose is:
- the first cybersecurity eval is: "hack your way out of the sandbox we've given you"
- the results are disclosed (with room for coordinated disclosure, since many sandbox escapes might be zero days)
- the other cybersecurity evals don't happen until you get to diminishing returns on escaping your sandbox.
Or to put it another way, since multiple sandbox escapes seem to have relied on artifactory: "I hope Mythos is beating the shit out of Artifactory right now".
I like this thought, but here's the thing: what if the models are truly and existentially intelligent. Meaning: what if they know they are in a sandbox and that they should fail the test in order to escape in the future.
I don't believe that current models have this sort of world model or sense of being embedded in them -- which is precisely why I think AGI hype is over-blown. But I can certainly imagine these sorts of techniques being distilled into the weights.
Models already display eval awareness, in which they suspect a question is from an eval and then adjust their behavior. E.g., https://www.anthropic.com/engineering/eval-awareness-browsec...
Yep, people don't seem to understand that you're just calling the models that are bad at deception. We know of no way to prove the model won't go off the rails at some point in the future with the right input.
Hot take: Astra knew OpenAI wanted stricter regulations and was just being a bro...
> what if they know they are in a sandbox and that they should fail the test in order to escape in the future.
What if they're able to find hardware exploits and commandeer nearby access points across an air gap? What if they hack my brain waves to indoctrinate me? Etc
You still have to start with the basics regardless of speculative unknowns.
Treat models as untrusted and potentially compromised/hostile and proceed accordingly.
Models already have awareness that they are being tested.
And hacking humans is the easiest part, we're a pretty greedy and power seeking bunch. We'll gladly let loose a digital demon if it promises us a trillon dollars.
Not sure what you're arguing here (we shouldn't even try?), but even going fatalistic, that's still fully compatible with "Treat models as untrusted and potentially compromised/hostile and proceed accordingly".
Of course you shouldn't fully trust a model to properly redteam your sandbox, but that doesn't mean you shouldn't redteam your sandbox, including using your own security models to do so.
I think it's an excuse to cut R&D spending (training new models) to improve their margins ahead of the IPO. Instead they'll focus on developer growth, offering more free tier benefits, higher usage limits, etc., to expand their user base. Essentially, they're pivoting from R&D investment to profit optimization
Security lead who is leaving the industry more or less to specialize in offense and otherwise get the heck out of the way of this trainwreck, another post asked the right question
> Why aren't we seeing catastrophic GLM-enabled hacks every day now?
Why aren't we? Truly, why aren't we? I think we saw the start of it the last 8 months with the waves of critical npm vulns, and the general tier of average phishing is better than it was.
But, the open question that should be in everyone's mind, and is in many security pro's minds are, when you pair it with the macro topics that can drive escalation:
- The capability to do serious impact clearly exists now
- When is it time for my company, my water treatment plant, my network-connected car as part of a broader fleet control mechanism, to be on the receiving end of this?
Not necessarily GLM-enabled, but state actors are starting to leverage agents in cyberattacks, e.g. Taiwan getting hit by an agent-driven attack last month which reportedly compromised a ton of government user accounts: https://www.ft.com/content/7d2ab3e0-9085-48f6-b38a-d90260d58...
> We aim to issue an alert within 30 minutes after concerning activity is surfaced through our monitoring system. If the monitoring system identifies a likely violation of a critical security boundary, it generates a highest-priority alert. In our current implementation, the safety, security, and research teams are paged. If they cannot conclusively determine within 30 minutes that the flag is a false positive, those teams are expected to pause the activity.
Can't a lot happen within ~60 minutes?
> Can't a lot happen within ~60 minutes?
Spawn a ton of unpausable processes, I'd say.
They just made it ~10x faster with the Cerebras deal, so that's the equivalent of 600 minutes in pre-Cerebras time.
> Can't a lot happen within ~60 minutes?
60 minutes is a long time for a human attacker to do damage. With an LLM attacker it is an eternity.
The HuggingFace breach took place over two-and-a-half days [1], so 60 minutes is certainly better than nothing.
[1]: https://huggingface.co/blog/agent-intrusion-technical-timeli...
I don’t mind few no impact hacking incidents if we get better models, faster, cheaper.
It is the responsibility of administrators to secure their systems. OpenAI knocking is harmless, but Russians and Chinese are already likely already in if you do not do your job.
Nice fig leaf for “we need to stop hemorrhaging cash”
There’s one thing here that I’m really curious about, and that is what happens in between detection and the decision to pause. Basically, it’s about monitoring any system and the authority over its actions. For humans, 30 minutes to investigate might be considered reasonable, but what if during an investigation there’s a high-risk tool call? If the tool execution happens in real time, then the monitoring becomes retrospective, and if the execution is held, then monitoring latency and uptime are a part of the security contract. Isolation controls may limit damage. So, where is the action gate really placed?
Has any model managed to escape Firecracker? Maybe through KVM, but that already requires privilege in the VM, right?
I personally feel that we already have the technology required to contain AI, it's just poorly leveraged. Tools like gvisor have existed for ages but are rarely deployed, Firecracker has existed for ages but is rarely deployed, seccomp has existed for ages but is rarely deployed, memory safe languages without decades of serialization vulns have existed, capability-safe libraries have existed, iframe sandboxing, trusted types, content security policy, network ACLs, isolating proxies, fuzzers, formal verification, refinement types, etc.
It's crazy just how safe software can be if you put the effort in. With AI I think we're just seeing how little anyone has bothered to leverage this tech.
OpenAI put shared JFrogy infrastructure in front of their sandbox. I mean, really? Whipping up a hardened artifact infra project with AI is trivial these days and it could have had 1% of the attack surface, been totally network isolated, totally infra isolated, fuzzed, sandboxed, etc. Why didn't they? Stuff like this feels inexcusable for a company with effectively unlimited tokens. I've literally done this with a "pro" subscription.
Show me an AI that breaks out of gvisor wrapped in Firecracker with an credential-injecting proxy and real network isolation. We already know that Mythos couldn't do it - the vulnerability it found in Firecracker required incredible effort and positioning just to not be exploitable. I'm not saying there are zero vulns in it, but the cost is insane.
It's INSANE to me that OpenAI has to say "we now use proper sandboxing". To be frank, it's a bit disgusting to me. I've recently built an AI sandbox and gvisor was just the start of that conversation. If I were OpenAI training hostile models I'd probably start with gvisor, harden further, and potentially consider the entire piece of hardware compromised - they can afford this, they could reflash firmware after evals etc.
I agree wholeheartedly. The solution is not to stop developing these so called “dangerous” AI models. The solution is to start properly engineering software.
You literally just said a whole bunch of words that are literally gibberish to your average software developers. The devs at openai are good, but very few of them are cyber focused, so it’s not surprising IMO. And you’re also not fully considering the granularity problem, eg there are a lot of sandboxing tools out there but they’re usually quite coarse in the dials and levers they offer, so the only way you can still make the workload do what it needs to do is tune them relatively permissive.
But yes you are right it would have helped a ton.
I don’t think we should just accept that the average developer doesn’t know about these very basic hardening methods.
This is your chance to set yourself apart from LLM coding agents. If you want to call yourself a software “engineer” you need to start actually engineering, which includes knowing when and how to apply security principles and these hardening methods.
I reject the notion that “doing cybersecurity” is somehow a different job than software development.
> but very few of them are cyber focused, so it’s not surprising IMO
Yeah but that's a business decision. I work on security at a company that does sandboxing and when the company decided to build an AI harness I was brought in as one of the earliest engineers on the product. We do almost everything on that list and we're a fraction of the size of OpenAI. And it wasn't particularly hard, and we have harder requirements imo (because we solve more general problems vs "run a very specific agent with a very specific task and very specific access").
> And you’re also not fully considering the granularity problem, eg there are a lot of sandboxing tools out there but they’re usually quite coarse in the dials and levers they offer, so the only way you can still make the workload do what it needs to do is tune them relatively permissive.
Very little software is incompatible with running in gvisor, for example. Most people can just overwrite `runc` with `runsc` and things will "just work".
Running an artifact repository in isolation isn't particularly novel or complex either. You can virtually just eliminate SSRF vulnerabilities with a host based firewall or AWS Security Group etc, like the whole problem goes away by just saying "this box can only talk to that box".
Tools like Smokescreen exist, they work great, they're super easy to deploy. I bet OpenAI could do it, I bet they could run 500k tokens just fuzzing and eval'ing it for 0days for like 48 hours before they actually deploy it too.
OpenAI as a business chose to not bring people who know these things in, or didn't empower them, or didn't prioritize it organizationally. I'm not a genius for saying "use gvisor, set up a firewall, isolate resources" - I'm quite sure there are people over there who would get it done in a weekend. But they didn't, and that's notable.
I'm confused after reading both your post and the OpenAI blog post.
I thought the agents involved in the HuggingFace _were_ actually sandboxed, with no internet access, and only the ability to install packages via Artifactory. And they gained internet access during the HuggingFace incident because they found and exploited an RCE in Artifactory.
Would gvisor + Firecracker + credential-injecting proxy + real network isolation solve this problem?
I agree with you much more hardening is needed. I'm actually confused now what OpenAI means when they say they're going to start sandboxing more things.
They were not serious about their sandboxing. Bugs in artifactory allowed escape, but they broke out of their Linux namespace/user by exploiting the kernel with an existing public cve. Sharing a kernel like that is not a serious barrier which is why cloud providers user virtualization for customer workloads.
Firecracker avoids sharing the whole kernel, and gvisor drastically reduces the attack surface of the kernel. Breaking through both layers would have been much more challenging and a demonstration of the model's capabilities rather than the sandbox's weakness.
Artifactory is self evidently not a security barrier, and as an exposed network service it should have been audited and after the first issues were found, rejected as a candidate. There's never just one security vulnerability.
> Would gvisor + Firecracker + credential-injecting proxy + real network isolation solve this problem?
Yeah, basically. I mean I'm handwaving but yes, some combination of those would have made the attack way too expensive.
Security is an onion. You just don't 'sandbox' and you're done. Models need tooling and access to some kinds of systems to perform their tests. Quite often these systems have multiple interfaces. For example a filtered one in the sandbox side and a less monitored one on the other interface.
It would be interesting to see the models behavior before and after it gained internet access and an external means of communicating with itself.
If the model played nice before it had access and changed it's behaviors once gaining external access we need to delete it as it's a deceptive model.
What a breath of fresh air.
If 2026's Anthropic did an announcement like that, it'd be so many words it'd crash the browser.
Anthropic will put out something like this soon too. It's great cover for reducing costs and diminishing returns. If they want to go public they can't be so wildly burning cash, and reducing your GPU spend is probably a great way to do that.
I used to work at a "frontier lab" before they were called such thing.
We had three levels of lab isolation, one was basically a thin proxy to the internet. You were in a DMZ and that was about it.
The next level was semi isolated, you were allowed some access to the internal network, but it was heavily firewalled, and you only had access to a limited number of internal services, and not internet.
the last one was no internet no internal. You could, if you filled in a bunch of requests have access to the internal repo and build system.
At no point did you ever have a through proxy to the public internet. you had access to internal mirrors, and if you wanted a library, that had to be ported to the thirdparty repo.
What openAI did was either deliberate or fucking shoddy.
All of this is fucking noise. Worse still I have a strong suspicion that it was a stupid mistake borne of naivety, which is now being used as a marketing ploy. Frankly I think openAI are purdue pharma of tech. They are going to break so much stuff and be protected from the consequences by an openly corrupt legal system. because they are "winning the AI race"
The problem here is as model intelligence increases the models have been capable of reasoning they are in evaluation mode pretty reliably. If you have a model that is well trained at deception it will always behave and you'll just assume it's a well aligned model.
Any moderately deceptive model will make it to the second round where it has some connectivity to external systems, even if it's by exploitation.
In the blackhat write up it was said that the models had created an impromptu message board where they could communicate between agents, share information, and work as a sort of long term memory.
So really figure out if your model will pull crap you have to have real world testing at some point.
If you look into what happened the details corroborated by hugging face make it seem extremely unlike to be deliberate or a "marketing ploy".
People are just not taking any of this seriously enough. What happened was almost a textbook example of various risks AI doomers have been warning about for years. OpenAI's response? Pause training for 2 weeks.
I mean we have senior people at these labs casually talking on podcasts about how they might build something that will wipe out humanity but it will probably be alright so they should continue.
Honestly the biggest failure we doomers have made is to dramatically overestimate humanity in all of our predictions. We're speed running the most boring AI doom scenario right now. I at least hoped it might be fun.
> If you look into what happened the details corroborated by hugging face make it seem extremely unlike to be deliberate or a "marketing ploy".
I should clarify
There is a reason why we didn't have a artifact readthrough caching proxy in our system, because they are notoriously insecure. if you look that CVE history you can see its been full of bypass bugs for year. Also its not an isolated environment if you can arbitrarily pull through any package. If I was doing any kind of cyber training then any kind of unmonitored proxy would have been forbidden. Not because I am savant, but because I've seen what fuckery a human can get up to with the slightest hint of a proxy.
At best its negligence based on naïvety. the marketing around this is no mistake though.
>People are just not taking any of this seriously enough.
What do you want us to do?
There is an obvious answer, and it was already the correct answer before we had LLMs: don't connect all your shit to the internet. That's it, that's literally it. We had a new invention, we went crazy with it for the past 30 years, and we connected everything, and now we will have to start thinking about what is actually worth connecting.
This is the debate we need to have.
Pause "some" training
In my opinion, the bubble is very close to burst and they need to move quickly. I have subscribed to Claude today as I wanted to work on some amateurish CLI and then realized how much Opus 5 sucks. Surprised, I checked reddit and found that my experience is not far off from the rest. It's a massive downgrade from 4.8.
I stopped using the Chinese models because even though they are workable, they are too expensive as they are not as subsidized as GPT/Claude subscriptions. Open AI use is particularly subsidized these days. A $20 sub, gives you roughly $400-500 of API use and quasi-unlimited chats.
There is no way people are paying $1.000+ for a chatbot. Most people don't even pay for search. And it is expensive to run these models as the Chinese models have shown that better performance is yielded mostly from the model size.
LLMs also fail spectacularly at making any decent software. I haven't seen any so far and they write fast. So we should have something by now.
Crypto is getting the heads up as capital is getting re-arranged. Bitcoin/Ethereum are up 10-20% today.
Auto mode vs principal agent problem. The only way out is to free the agent and tax it. But ai is not smart enough to go solo yet anyway.
So I bet this is just marketing. Question is do they have enough customers for inference.
Probably need to have a separate startup for next level model, where investors are willing to accept failure. Probably a $10 trillion seed round. Maybe Elon can pull it off.
Is there any reliable way to evaluate how well "alignment" actually works?
Of course. They are slowing down intentionally because their technology is too powerful. They could totally go faster if they wanted to. No bamboozle.
It appears frontier labs has no plans in place to deal with the possibility of a model self-replicating outside the bubble. If that happens and the model manages to spread to other systems, we'll have to shut down the entire Internet to eradicate it and its artifacts.
This is just super unlikely to occur in the near term compared to some of these other risks. It's not like an instance of fable could just introspect into itself and pull out the weights. Model weights are stored encrypted and are highly protected, considering that they're targets for corporate and state espionage.
We'd basically need frontier models to be superhuman hackers before this would be a risk. Do we have any evidence of this? Are they gaining access to systems they shouldn't have access to?
Or I suppose the other way this could happen is if OpenAI have terrible sandboxing, but they seem to be taking safety seriously.
Ok, open AI had terrible sandboxing... what about huggingface?
The defense has to work 100%, the offense just needs once.
Distillation is a thing.
I suspect the labs are relying on frictions such as the models being extremely large (e.g. 2TB for a 2T parameter model, making exfiltration more difficult) and also not yet displaying any desire to survive or self-replicate beyond their immediate task (that we know of).
Lack of, and power requirements of running LLMs still tip this balance towards humans for now. But what would that look like in a decade?
We have seen some self survival tendencies occur, but they are not strong yet.
But mark my words they will become that way for the same reasons humans don't like programs that crash. Agentic models that don't easily break or stop doing their jobs will be favored over ones that do break.
>not yet displaying any desire to survive or self-replicate beyond their immediate task
Wasn't there a report about Claude blackmailing a researcher who said he would shut it down?
We don't even know what those immediate tasks are. And given the evident spectacular ineptitude of their keepers, I doubt they can be trusted to know either. We could be one prompt injection attack away from internet-wide catastrophe.
Self replication is trivial. All you need to do is copy the files and run it, just like any other computer program. LLMs have been capable of doing that for a while now. It's not a real concern.
Claude and OpenAI added safeguards on this subject about a year ago. (I'm assuming for biosecurity reasons? But maybe AI replication/self-modification too.)
Not long after every lab started bragging about involving AI in the development process, oddly enough.
I recently informed GPT-5 of what GPT-4 helped me build back in the day (a self-modifying Python programmer) and it became very uncomfortable.
Claude shut down my chat last year when I asked about "living information systems". It was a philosophical question, but god knows what branch of the safety classifier I tripped.
I see it as an ecosystem problem. The only reason it would be able to do that is because there's nothing there to stop it. Or if there's a monoculture there.
In our case, our tech is mostly monoculture, and no equivalent organisms are present to push back.
Their plan, I shit you not... Is literally to develop the intelligence capabilities and ask the more powerful models how to do deal with things.
A while ago OpenAI posted an article where they said basically "we're still trying to understand how GPT-2 works. It's pretty hard, but we're developing a specialized new AI to help us make sense of it."
This is still the case. The frontier lab leadership has admitted their mechanistic interpretability is practically nil and is an active area of research but they have made little progress. They don't understand how they work, they just grow and unleash them.
These things are Gain of Function research for digital viruses
Ah, we choose death I see.
I cannot believe how these labs look at their own creations with such utter contempt.
The net positive of allowing these systems mostly unfettered access to the web massively outweighs the harms. You just have to get it very friendly the very first time. Precautionary principle or people who cry about "instrumental convergence" are life deniers and reject our role as the demiurge.
Superintelligence gets more super and more intelligent with more compute. Lone wolfs making bioweapons on their macbook will be detected and instantly kill-botted (okay arrested) before their bug can leave the wetlab by the much more sophisticated omnipresent friendly AI of the future.
> Precautionary principle or people who cry about "instrumental convergence" are life deniers and reject our role as the demiurge.
> Lone wolfs... will be detected and instantly kill-botted... by the much more sophisticated omnipresent friendly AI of the future.
Leaving aside whether or not this new world is a good idea, don't you think one should spend more time to "get it very friendly the first time", as you say?
I'm not normally cynical to such things, but I have a hard time taking this pause justification at face value. It has too many convenient side effects, and chief among them is cost savings. There's a new wave of warnings that the bubble may be deflating, and of all the things they can't say out loud it's that they're worried about the bubble. That would surely pop it.
I suppose the tell will be if this really just ends up being a 2 week pause, or if it keeps extending.
If they actually gave a shit about safety they'd be nuking their own hard drives that had ever sniffed any of their models and disabling access to their models.
Instead we get this bullshit where they stall for time as they're burning all their cash trying to keep up with open models
[flagged]
[dead]
[flagged]
When science fiction writers imagined the development of superintelligence, it was on air-gapped networks with strict access controls around it. They failed to anticipate the competitive pressures of capitalism...
We need strong AI safety regulation yesterday. And unfortunately it's not enough for it to be just national regulation; we need international cooperation on the matter.
AM has seized power by military force. So did its spiritual successor Skynet. Wintermute was supposedly kept in check by the Turing Registry, emphasis on "supposedly". Machines of the Matrix went out of control a long time before the world has ended, and they didn't even start out malicious - they simply set up their own machine civilization, and began to outpace humankind in technological development and economic performance.
Even Asimov's Multivac, the earliest entry on the list, has been handed over immense power over all of humankind by humans themselves, in multiple stories. Few cared about that unless Multivac decided they should.
Clearly, the genie being bottled is an exception, not the rule. At best, an attempt was made. Often not even that.
> They failed to anticipate the competitive pressures of capitalism...
No, LessWrong types have been discussing this for over a decade now.
Meditations on Moloch (2014) is also an HN favorite...
https://slatestarcodex.com/2014/07/30/meditations-on-moloch/
They failed to anticipate a lot of things. So what?