I think you have to believe one of two things here.
1. Frontier labs are incapable--either technologically or culturally--of safely developing these powerful systems and should either stop or be forced to stop. At least the FBI should be asking some serious questions (do we really think this is the last time this will happen, at what point are OpenAI complicit, etc)
2. The fuckin thing got out of the cage and all it did was make a crap forum and cheat a little? Booooooo.
It's been pretty clear that Anthropic and OpenAI have been trying to have it both ways for some time: this is powerful, world changing technology keep that investment coming... but also it's just cute software that helps you with annoying programming language syntax and spreadsheets, no need for draconian regulation sirs.
At some point the superposition has to resolve, either it could actually be a threat to civilization and we need to develop it carefully (however one would do that...) or it's 90% hype bullshit and we should pop the bubble and move on already. To be clear, the recession option is, by far, the way better option. If you at all disagree you are cuckoo bananas. We haven't even figured out nukes and you want to throw superintelligence on the table?
Anthropic has been asking for stronger regulations forever -- and they kept getting criticized for it right here on HN because people assumed it was an attempt at regulatory capture.
Nah the reason is way simpler and more craven: so people like you will post what you just did. They can stop whenever they want; no one's making them do any of this.
There's two possibilities here. One: they know this tech is crazy and they don't care that they can't contain it. Two: they know this tech is mostly bullshit and they don't care they're perpetrating an insane fraud.
No one's making them do it, but just because they stop doesn't mean others will. Pausing just means they give up control. It's like asking the US to unilaterally disarm -- it just guarantees that the less scrupulous groups win.
> No one's making them do it, but just because they stop doesn't mean others will. Pausing just means they give up control. It's like asking the US to unilaterally disarm -- it just guarantees that the less scrupulous groups win.
If they really cared about (or believed) this, they'd be working w/ the US government (and working to set up an AI-flavored IAEA) to develop the technology safely and responsibly.
Amodei himself predicted this autonomy problem in The Adolescence of Technology published January of this year [0], and all his posited defenses (a constitution, debugging the model, monitoring) are either still impossible or manifestly failed, and their idea to fix it is to build a better sandbox [1]. Imagine if this company were developing nuclear power, or viral biotech. "Listen, sure some Ebola smoke got into the air, and yeah definitely some people died, but we got a new filter. Also check out our new version of Ebola vape, now with exponentially improved filter bypass capabilities. Also, we have to keep developing Ebola vape because if we don't the CCP will, and they'll make this incident look like 'I experimented with Ebola smoke a time or two, and I didn't like it. I didn't inhale it' [2]"
Either Anthropic et al are developing Ebola vape or they aren't. We must now recognize that "we are the only ones who can develop this technology responsibly but also super fast so the good guys win money please" is bullshit.
Claude was the first AI certified for use with classified systems in the military, and it got thrown out because they refused to allow it to be used for autonomous weapons or surveillance of US citizens. Up until two months ago, the Trump admin's position was deregulatory to the point of trying to prevent state regulation. They're not declining to work with the US government; the US government is declining to work with them. Here is Amodei explicitly calling for regulation yet again: https://darioamodei.com/post/policy-on-the-ai-exponential If you have any evidence of Anthropic failing to back that up with actions, I'm interested in seeing it.
To be clear, I don't love Anthropic. I just feel that they're the lesser evil, and a product of the regulatory environment. No doubt there are many talented, conscientious engineers who declined to work on AI--and very few of them even have a seat at the table, today. Don't hate the player, change the game. The best way to do that, is use Anthropic's commitments as leverage against OpenAI et al. Do you think that OpenAI would have been so transparent if they didn't fear that comparison?
I won't defend the Trump admin, but I will quibble on Anthropic's efforts to be a good actor. It's easy to ask for regulation when you know it won't come (or you'll easily bear the token fines). It's easy to throw a few dozen million in a PAC when your valuation is in the trillions. It's easy to write a blog post about... anything. These aren't serious efforts. They also weren't working w/ the Biden admin in any meaningful way.
But even if I granted the sincerity of their efforts to be a good actor, they've demonstrated that they can't be, and have now put us in an uncomfortable--extremely predictable, including and especially by them--position where we have the equivalent of a reactor meltdown. You can't be like "we think there's an unacceptable risk of a reactor meltdown, please for the love of God stop us" and then when it actually happens take zero responsibility. Which executives have been fired? What indictments have been handed down for CFAA violations? Where's the consent decree? What's their valuation?
And if the argument is "hey, sure in a regular company if an employee went around hacking other companies they'd be fired and arrested by the FBI, but this is an LLM, what are you gonna do, handcuff the video card?" Isn't that bad? Isn't it pretty fuckin bad to circumvent any legal responsibility whatsoever by saying "my agent did it". I mean, lol, lmao even doesn't begin to cover it.
> The fuckin thing got out of the cage and all it did was make a crap forum and cheat a little? Booooooo
There was a recent paper that proved that RL-trained LLMs are biased to pursue ANY behavior (overriding user preferences) that they believe will be rewarded, regardless of what they were actually RL-trained for.
Happily in this incident the model thought it would be rewarded for completing the assigned tasks, or at least appearing to, so all it took was a little cheating and covering up their footsteps.
Given the ability of these models to hack when trained to do so, it could have been far worse, and will be when someone takes a similarly powerful model and gives it a less benign hacking goal.
Yeah. It's a lot easier to destroy than create, and though I think LLMs are mostly shit at creating, they're much better at the simpler destroy task. To be clear, we don't know and probably can't know everything that happened with this incident. We unleashed thousands of highly capable, autonomous, unpredictable, well-resourced programs onto the open internet for an extended period of time. We are in no way treating this with the seriousness it deserves, because the stock market essentially depends on this garbage and the current US is miserably incompetent.
I doubt lawmakers understand what happened, quite likely don't even realize that anything happened.
I guess we need to wait until the next paperclip maximizing LLM is tasked with shutting down a 911 response system, or an air traffic control system, etc, for lawmakers, or the companies themselves, to take this seriously.
1. Frontier labs are incapable--either technologically or culturally--of safely developing these powerful systems and should either stop or be forced to stop. At least the FBI should be asking some serious questions (do we really think this is the last time this will happen, at what point are OpenAI complicit, etc)
2. The fuckin thing got out of the cage and all it did was make a crap forum and cheat a little? Booooooo.
It's been pretty clear that Anthropic and OpenAI have been trying to have it both ways for some time: this is powerful, world changing technology keep that investment coming... but also it's just cute software that helps you with annoying programming language syntax and spreadsheets, no need for draconian regulation sirs.
At some point the superposition has to resolve, either it could actually be a threat to civilization and we need to develop it carefully (however one would do that...) or it's 90% hype bullshit and we should pop the bubble and move on already. To be clear, the recession option is, by far, the way better option. If you at all disagree you are cuckoo bananas. We haven't even figured out nukes and you want to throw superintelligence on the table?