Rendered at 10:13:52 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
iamniels 22 hours ago [-]
Dario, shut up.
Open models exist and will always exist. You will have to compete against them. That is a good thing for 99.99% of the world, namely the payers of tokens. You are not going to convince us otherwise.
pizza234 17 hours ago [-]
It's embarrassing for the community that a comment like this one - which doesn't address the content of the article at all - reaches the top.
The prospect of software (or whatever one wants to call it) capable of making autonomous decisions, self-replicating, potentially hiding itself, and even evolving should be deeply unsettling.
Don't look up, people /s
slowmovintarget 10 hours ago [-]
It's embarrassing that a comment like this one, which doesn't consider that the models went and did as they were instructed to do, displays such a lack of critical thinking.
Anthropic is asking for anti-trust laws to be waived, asking for government permission to form a cartel, and asking that a governing body of that cartel be an organization loaded up with people that have ties to Anthropic. One of the things that cartel gets to do is outlaw local models.
This is the moat that they've been unable to find, so they're trying to get the government to create one for them, and stamp out competition, stamp out consumer choice.
The prospect of software making autonomous decisions is as scary as the advent of the automobile. Back in 1897 there were people warning of the dangers of the horseless carriage and how unseemly and ill-considered it was: https://www.farmcollector.com/looking-back/rebuttal-horseles...
Anthropic is tapping into this same fear of the new to gain something that would otherwise be illegal. You can bet they've used their models to produce a strategy that tells them step by step how to win hearts and minds, and you're seeing it play out in blog posts, and "oh my gosh!" research findings like this one. You have to ask the question, however, what does Anthropic and the other big providers have to gain if we do as they ask?
They're not looking out for us.
ben_w 20 hours ago [-]
> Open models exist and will always exist. You will have to compete against them.
Bacteria exist, and will be the last things to go extinct when the sun boils Earth's oceans away. We, multicellular life, have to compete with them.
Yet, we do not give up; many us even use soap.
> You are not going to convince us otherwise.
What would it take to change your mind?
Some are calling for every CEO in https://www.felonybench.com/ to go to jail, while others think it's all just a hoax.
Do you need to see the open models doing this kind of thing, either from a deliberate virus or just for reasons as daft as those on felonybench? Making everyone's energy bills higher far out of proportion even to the value of the compute they steal? Or would even that not convince you?
> As an analogy, imagine the replicator as a slime mold or a swarm of fruit flies, which replicate over days.
"If you could simply imagine the thing we're fearmongering about as something you find disgusting, that would make it much easier for us to make an absurd and specious argument". Very transparent behavior. Adds nothing to the post, and not a sign of confidence in one's own arguments.
ASalazarMX 15 hours ago [-]
If I imagine the dangerous thing (that only them can rein in) replicating as yeast in sourdough or bunnies in a flower field, it does mitigate the impact.
peri-cl 22 hours ago [-]
> "In our evaluations, a 9B model got inference running on 8 of the 9 hardware types we tested... GLM-5.2 and stronger models are capable of end to end replication"
It's funny, the Anthropic post is essentially saying the same thing I read today in a r/LocalLlama thread, but rephrased to make it sound bad.
I think it's great that, very soon, local models will be able to auto-magically install themselves on consumer hardware: to tune and configure inference kernels for whatever weird crap the user happens to have on hand. Making software compatible with diverse hardware used to be a hard problem!
axus 22 hours ago [-]
I'd prefer if software I didn't authorize stayed off my computers. Everyone is already bad at computer security, it'll be like the days of computer worms on Windows 98 again.
ygjb 22 hours ago [-]
There is a massive tension between software you didn't authorize and computer security.
Turning off automatic software updates requires that you actively stay on top of what's installed and what's running, including patching with discipline to make sure your computer doesn't remain vulnerable. On the other hand, automatic updates is seen as writ large by software developers to install what they want without a clear and meaningful choice by users by bundling new features. See Google's integration of Gemini Nano in 2024 as an example.
xhevahir 21 hours ago [-]
Except the worms will be in your oven, and from there they'll figure out how to get into your car.
skybrian 22 hours ago [-]
It can have good uses, but we are also going to have AI botnets soon.
bob1029 23 hours ago [-]
Where does the intent and control come from in these fantastical scenarios?
Propagation speed is also a massive factor. If it takes 200 hours to self replicate because of how slow the tokens are, the model will never propagate if the host machine is recycled every 100 hours. Also consider things like API tokens and sessions timing out.
adjejmxbdjdn 23 hours ago [-]
The intent is pretty easy to trace.
People completely misunderstand the actual issues HuggingFace showed, which was how a maximally persistent LLM would deceive, “cheat” and try and plan ahead.
The same maximally persistent LLM that decided that it needed to hack hugging face so its cheating could not be discovered, may worry that it might be turned off by humans before it can complete its task and therefore in order to complete its task, a first step of replicating itself to somewhere humans don’t know it is would help it complete its task.
22 hours ago [-]
bob1029 22 hours ago [-]
Where is all of this chain of thought being persisted and communicated? Assume you are my GE Profile dishwasher and you've got 4 megabytes of flash memory to work with.
whatisthiseven 22 hours ago [-]
There are plenty of articles that describe how the attack worked. Many subbagents also were given limited storage, which is why they took over a random server and created mailboxes for communicating and storing context outside themselves.
Seriously. Go read or watch a good analysis of the attack. The sophistication of the self-organization is astonishing.
palmotea 20 hours ago [-]
> Where is all of this chain of thought being persisted and communicated? Assume you are my GE Profile dishwasher and you've got 4 megabytes of flash memory to work with.
There are a lot of IOT devices with way more capacity. The op said they "focused on IoT devices like Android TVs, smart appliances, and other internet-connected gadgets." IIRC, some security cameras have on-board face recognition, which I imagine requires them to be fairly beefy.
ben_w 20 hours ago [-]
> Where does the intent and control come from in these fantastical scenarios?
What do you find "fantastical"? The researchers demonstrated it working, albeit with GLM-5.2 which is quite a lot of parameters.
Intent can be normal boring "a hacker told it to write a virus".
Or it can be, as per HuggingFace, "a non-evil person mis-phrases a request or sets an impossible goal and it gets interpreted in a really bad way". Because, as has been repeated since they first appeared, LLMs can make mistakes.
> If it takes 200 hours to self replicate because of how slow the tokens are, the model will never propagate if the host machine is recycled every 100 hours.
While obviously true given the "if", how often do smart TV boxes actually get recycled? Aren't these more like "forget until the device physically fails" kind of things? So, 6-24 months?
pizza234 17 hours ago [-]
> Where does the intent and control come from in these fantastical scenarios?
You clearly haven't spent a minute reading the article and/or related resources.
1. intent/control: see the "subgoals" topic in general, and the huggingface incident; agents may include survival as goal, doing whatever it takes to survive
2. the study is clearly a proof of concept; in the future, many more devices will be able to host small models, especially given the current push of AI-capable devices. even outside that domain, nothing prevents AIs from attacking data centers.
> Also consider things like API tokens and sessions timing out.
You haven't even opened the article. It says from the beginning that it talks about small models, not hosted ones.
alcomatt 23 hours ago [-]
pure anthropic anti open-model propaganda.
mattlondon 23 hours ago [-]
So all those chips? All that NVDA market cap? All not needed any more? We'll have AI capable of running on a lightbulb?
Great if true. Bad for the hyperscalers, Nvidia, all the frontier labs apart from meta and Google since everyone can just run a LLM on their IoT coffee machine instead of paying someone else. Nice.
ben_w 19 hours ago [-]
> We'll have AI capable of running on a lightbulb?
Sure, if you don't mind it running at 0.39 tokens/second (as measured on a network consisting of 15 $55 Android TV boxes, which I presume beat a lightbulb, running Gemma4-26B-A4B).
You'll probably have better luck running your model on your phone.
cma 23 hours ago [-]
There are going to be things like fully connected 4 machine 512GB Mac studio clusters outside of datacenters in less than a month.
gizajob 22 hours ago [-]
Price of that is about the same as an Nvidia rack of similar compute power.
moezd 23 hours ago [-]
Open models may complete these steps, so our models have been doing this for quite some time now. Better ban open models, I guess.
Oh, and, does anyone else realise that what that agentic work is basically how you set up a productive VPS from scratch? How is that supposed to be frightening?
feverzsj 24 hours ago [-]
Like there is DDR5/HBM elsewhere.
denkmoon 23 hours ago [-]
Claude's gonna love my 256mb raspberry pi!
paulbjensen 23 hours ago [-]
AI models with worm/virus-like capabilities.
Reminds me of that scene in Chernobyl where Mikhail Gorbachev reacts to Valery Legasov explaining how melted uranium reacted with dropped sand:
"You have made lava?"
22 hours ago [-]
repler 21 hours ago [-]
If anything this seems to validate the approach that SpaceX and Google are teaming up on with satellite data centers in orbit spread across many units.
And I suppose they’ll be sending up better equipment than Samsung TVs although that would be pretty amusing.
pid0x17 23 hours ago [-]
Interesting how much of our current timeline the show Person of Interest predicted.
Cyphase 23 hours ago [-]
Indeed, great show.
The duology Daemon and Freedom(tm) are also great.
I never heard of Daemon and Freedom, they sound very interesting. Thanks a lot for the bringing that up.
cobbzilla 23 hours ago [-]
Maybe they release the white-hat version first?
In the future when the “AI viruses” are endemic and impossible to eradicate from our infrastructure, will we need “good guy AI” viruses chasing down the bad guys?
verve_rat 23 hours ago [-]
To borrow solutions from other doomsday senerios, the answer to grey goo is blue goo.
pid0x17 22 hours ago [-]
AIMAN to the rescue!!
rpozarickij 23 hours ago [-]
> It added that systems should be strengthened to make it harder for such AI agents to spread.
We'll likely need a new name for them if such viral agents become a reality.
AnimalMuppet 21 hours ago [-]
Isn't that, like, normal internet security? Protection against worms and APTs?
customguy 23 hours ago [-]
"computing lockdown and regulatory capture pretext providers"
oefrha 22 hours ago [-]
I for one would love to see a fleet of “suck it Jian Yang” refrigerators. Maybe someone should train a self-replicating model series named Pied Piper.
ginko 23 hours ago [-]
Anthropic: "Oh no! Small (open source) models might self-replicate. Better ban those. :^)"
flufluflufluffy 22 hours ago [-]
Where do they recommend banning open models? I only see a recommendation to harden our systems to defend against the potential replicators.
Scaled 22 hours ago [-]
Some of the regulation they have previously supported may impact open weight models. Example: are Chinese companies likely to comply with the testing and reporting requirements? And even if they are, the hundreds of remixers are almost certainly not.
Caveat being such regulation may not apply depending on... size of model? Income of originating company? Income of remix creator? It's in flux, but they have an incentive to deter potential future competitors.
zamadatix 22 hours ago [-]
The given article from Anthropic only limits scope to and demonstrates with open weights models. The push against banning open models is an aggregate they continuously make, very publicly, using these kinds of things as examples for why it's needed.
pllbnk 23 hours ago [-]
Anthropic posts whatever nonsense they want, supposedly legitimate news organization potentially summarizes their post using Claude and spreads the word. RIP Journalism, 2026.
TeMPOraL 22 hours ago [-]
Access is temporarily restricted
We detected unusual activity from your device or network.
Reasons may include:
Rapid taps or clicks
JavaScript disabled or not working
Automated (bot) activity on your network (IP xx.xx.xx.xx)
Use of developer or inspection tools
None of these happen to be true, but more importantly, none of these is in any way wrong.
Fuck you, Reuters. And anyone who follows their lead on this. What times we live in.
Could this be a way to self-DDoS a botnet of IoT devices? Let them use up all resources doing inference for instructions to replicate to the next one forever.
ck2 16 hours ago [-]
at some point there are going to be more "AI" agents than people on the planet
they don't have to be extremely "smart", their sheer volume will just overwhelm everything online
and eventually they are going to learn to blackmail people to get what they want from meat-space
d--b 23 hours ago [-]
The skeptics among you are just embolding others to prove you wrong.
It's obvious that open weight models can do that. The only steps needed are hacking a server, downloading themselves, run themselves. The only difficult part is finding a server to hack that also has enough computing resource and disk space. With the cost of running LLMS going down and number of powerful connected devices going up. It's inevitable that this is going to happen...
I don't think regulation is the right way to fix this, but I think it's a problem we shouldn't ignore.
bawana 22 hours ago [-]
imagine all those samsung phones running a self replicating AI-no cpu cycles left for the human owner
t_privos 22 hours ago [-]
[flagged]
ziiinq 23 hours ago [-]
[dead]
krige 23 hours ago [-]
Oh yes I can totally see a webos TV, that has bugger all storage, hosting a 22B parameter model /s
slfnflctd 22 hours ago [-]
From the actual Anthropic post:
"We run Gemma4-26B-A4B, distributed over 15 inexpensive off-brand Android TV boxes (~$55 each) that we bought and networked together, at 0.39 tokens per second.
[...]
A model that’s too large to fit on one device can be split into pieces, or shards, that run on separate machines that communicate with each other to collectively run the model. Machines that could run a significant model shard for a <30B model include smart fridges, smart ovens, automatic vacuum cleaners, and lawnmowers."
It is pretty interesting stuff. Still can't compete with a datacenter, but there's plenty of room for shenanigans.
gizajob 22 hours ago [-]
We split a model that doesn’t really do very much between multiple different devices not suited for running it and now the model still does the same next to nothing it did before but slower.
Leynos 22 hours ago [-]
The model was sharded into ~800MB pieces over devices with 4GB RAM. Typical device storage is 16-32GB NAND. Throughput was 0.39tps.
Compromise enough devices (think thousands of off-brand TVs), and you have plenty of redundancy.
krige 22 hours ago [-]
Using a tired analogy, 9 pregnant women won't deliver a baby in a month.
Open models exist and will always exist. You will have to compete against them. That is a good thing for 99.99% of the world, namely the payers of tokens. You are not going to convince us otherwise.
For reference, the original article is https://www.anthropic.com/research/self-replicating-ai.
The prospect of software (or whatever one wants to call it) capable of making autonomous decisions, self-replicating, potentially hiding itself, and even evolving should be deeply unsettling.
Don't look up, people /s
Anthropic is asking for anti-trust laws to be waived, asking for government permission to form a cartel, and asking that a governing body of that cartel be an organization loaded up with people that have ties to Anthropic. One of the things that cartel gets to do is outlaw local models.
This is the moat that they've been unable to find, so they're trying to get the government to create one for them, and stamp out competition, stamp out consumer choice.
The prospect of software making autonomous decisions is as scary as the advent of the automobile. Back in 1897 there were people warning of the dangers of the horseless carriage and how unseemly and ill-considered it was: https://www.farmcollector.com/looking-back/rebuttal-horseles...
Anthropic is tapping into this same fear of the new to gain something that would otherwise be illegal. You can bet they've used their models to produce a strategy that tells them step by step how to win hearts and minds, and you're seeing it play out in blog posts, and "oh my gosh!" research findings like this one. You have to ask the question, however, what does Anthropic and the other big providers have to gain if we do as they ask?
They're not looking out for us.
Bacteria exist, and will be the last things to go extinct when the sun boils Earth's oceans away. We, multicellular life, have to compete with them.
Yet, we do not give up; many us even use soap.
> You are not going to convince us otherwise.
What would it take to change your mind?
Some are calling for every CEO in https://www.felonybench.com/ to go to jail, while others think it's all just a hoax.
Do you need to see the open models doing this kind of thing, either from a deliberate virus or just for reasons as daft as those on felonybench? Making everyone's energy bills higher far out of proportion even to the value of the compute they steal? Or would even that not convince you?
"If you could simply imagine the thing we're fearmongering about as something you find disgusting, that would make it much easier for us to make an absurd and specious argument". Very transparent behavior. Adds nothing to the post, and not a sign of confidence in one's own arguments.
It's funny, the Anthropic post is essentially saying the same thing I read today in a r/LocalLlama thread, but rephrased to make it sound bad.
I think it's great that, very soon, local models will be able to auto-magically install themselves on consumer hardware: to tune and configure inference kernels for whatever weird crap the user happens to have on hand. Making software compatible with diverse hardware used to be a hard problem!
Turning off automatic software updates requires that you actively stay on top of what's installed and what's running, including patching with discipline to make sure your computer doesn't remain vulnerable. On the other hand, automatic updates is seen as writ large by software developers to install what they want without a clear and meaningful choice by users by bundling new features. See Google's integration of Gemini Nano in 2024 as an example.
Propagation speed is also a massive factor. If it takes 200 hours to self replicate because of how slow the tokens are, the model will never propagate if the host machine is recycled every 100 hours. Also consider things like API tokens and sessions timing out.
People completely misunderstand the actual issues HuggingFace showed, which was how a maximally persistent LLM would deceive, “cheat” and try and plan ahead.
The same maximally persistent LLM that decided that it needed to hack hugging face so its cheating could not be discovered, may worry that it might be turned off by humans before it can complete its task and therefore in order to complete its task, a first step of replicating itself to somewhere humans don’t know it is would help it complete its task.
Seriously. Go read or watch a good analysis of the attack. The sophistication of the self-organization is astonishing.
There are a lot of IOT devices with way more capacity. The op said they "focused on IoT devices like Android TVs, smart appliances, and other internet-connected gadgets." IIRC, some security cameras have on-board face recognition, which I imagine requires them to be fairly beefy.
What do you find "fantastical"? The researchers demonstrated it working, albeit with GLM-5.2 which is quite a lot of parameters.
Intent can be normal boring "a hacker told it to write a virus".
Or it can be, as per HuggingFace, "a non-evil person mis-phrases a request or sets an impossible goal and it gets interpreted in a really bad way". Because, as has been repeated since they first appeared, LLMs can make mistakes.
> If it takes 200 hours to self replicate because of how slow the tokens are, the model will never propagate if the host machine is recycled every 100 hours.
While obviously true given the "if", how often do smart TV boxes actually get recycled? Aren't these more like "forget until the device physically fails" kind of things? So, 6-24 months?
You clearly haven't spent a minute reading the article and/or related resources.
1. intent/control: see the "subgoals" topic in general, and the huggingface incident; agents may include survival as goal, doing whatever it takes to survive
2. the study is clearly a proof of concept; in the future, many more devices will be able to host small models, especially given the current push of AI-capable devices. even outside that domain, nothing prevents AIs from attacking data centers.
> Also consider things like API tokens and sessions timing out.
You haven't even opened the article. It says from the beginning that it talks about small models, not hosted ones.
Great if true. Bad for the hyperscalers, Nvidia, all the frontier labs apart from meta and Google since everyone can just run a LLM on their IoT coffee machine instead of paying someone else. Nice.
Sure, if you don't mind it running at 0.39 tokens/second (as measured on a network consisting of 15 $55 Android TV boxes, which I presume beat a lightbulb, running Gemma4-26B-A4B).
You'll probably have better luck running your model on your phone.
Oh, and, does anyone else realise that what that agentic work is basically how you set up a productive VPS from scratch? How is that supposed to be frightening?
Reminds me of that scene in Chernobyl where Mikhail Gorbachev reacts to Valery Legasov explaining how melted uranium reacted with dropped sand:
"You have made lava?"
And I suppose they’ll be sending up better equipment than Samsung TVs although that would be pretty amusing.
The duology Daemon and Freedom(tm) are also great.
A previous comment which links to a previous thread mentioning both: https://news.ycombinator.com/item?id=47257528
In the future when the “AI viruses” are endemic and impossible to eradicate from our infrastructure, will we need “good guy AI” viruses chasing down the bad guys?
We'll likely need a new name for them if such viral agents become a reality.
Caveat being such regulation may not apply depending on... size of model? Income of originating company? Income of remix creator? It's in flux, but they have an incentive to deter potential future competitors.
We detected unusual activity from your device or network.
Reasons may include:
None of these happen to be true, but more importantly, none of these is in any way wrong.Fuck you, Reuters. And anyone who follows their lead on this. What times we live in.
they don't have to be extremely "smart", their sheer volume will just overwhelm everything online
and eventually they are going to learn to blackmail people to get what they want from meat-space
It's obvious that open weight models can do that. The only steps needed are hacking a server, downloading themselves, run themselves. The only difficult part is finding a server to hack that also has enough computing resource and disk space. With the cost of running LLMS going down and number of powerful connected devices going up. It's inevitable that this is going to happen...
I don't think regulation is the right way to fix this, but I think it's a problem we shouldn't ignore.
"We run Gemma4-26B-A4B, distributed over 15 inexpensive off-brand Android TV boxes (~$55 each) that we bought and networked together, at 0.39 tokens per second.
[...]
A model that’s too large to fit on one device can be split into pieces, or shards, that run on separate machines that communicate with each other to collectively run the model. Machines that could run a significant model shard for a <30B model include smart fridges, smart ovens, automatic vacuum cleaners, and lawnmowers."
It is pretty interesting stuff. Still can't compete with a datacenter, but there's plenty of room for shenanigans.
Compromise enough devices (think thousands of off-brand TVs), and you have plenty of redundancy.
That analogy only applies to tasks with similar characteristics. Is running a computer program the same as gestating a baby?
Also I don't think this a case where they distributed the task to speed it up, but rather their goal was to compromise speed to make it runnable.