How AI guardrails are impeding the work of offensive cybersecurity researchers
We spoke with several cybersecurity researchers, who look for unknown vulnerabilities and develop tools to exploit them, about how OpenAI's and Anthropic's guardrails affect their work. For months, AI giants have devised special vetted programs and strict guardrails to limit the use of their models by malicious hackers. But these limits are now hindering the work of legitimate network defenders, as well as that of offensive cybersecurity researchers.
Key Takeaways
- government slapped export control restrictions on Anthropic's much-hyped AI models Mythos and Fable.
The move was prompted at least in part by a report that claimed it was possible to bypass the models' guardrails designed to prevent users from using them to build and execute malicious cyberattacks.
- organizations as part of the government's review process.
- Governments pay a premium for vulnerabilities precisely because they stay open, which is useful for intelligence operations.
Dowd admitted his work may make him biased, but he isn't alone.
- "This is where the whole offensive versus defensive and guardrails part comes in, because 'fix this code' as a prompt is both an essential mechanism for defense but also a roadmap for finding critical vulnerabilities in the code base," said Anley.
"So at the same time, the same tool is both an offensive tool and a defensive tool, and the two can't really be unpicked.
- Paolo Stagno, the chief technology officer at Crowdfense, a well-known company that develops, acquires, and sells unknown vulnerabilities to government agencies, agreed with Dowd, saying AI companies "essentially treat customers like children who need babysitting" with their vetted programs and guardrails.
government slapped export control restrictions on Anthropic's much-hyped AI models Mythos and Fable. The move was prompted at least in part by a report that claimed it was possible to bypass the models' guardrails designed to prevent users from using them to build and execute malicious cyberattacks. Regardless of whether the incident was really motivated by fears of a jailbreak , the fact is that Anthropic has repeatedly marketed Mythos as some kind of doomsday cybermachine that can only be given to carefully vetted users, and even then with strict guardrails in place.
(The export controls on Fable 5 and Mythos 5 have since been lifted. Fable 5 returned to general access on July 1; Mythos 5 has been reintroduced only to vetted U. organizations as part of the government's review process.
) That kind of gatekeeping isn't unique to Mythos. Both Anthropic, with its other models, and OpenAI offer cybersecurity researchers programs they can apply to get vetted and - if approved - access models with fewer cybersecurity restrictions: OpenAI's Trusted Access for Cyber program and Anthropic's Cyber Verification Program . These guardrails have been widely criticized, particularly by researchers whose job is to find unknown vulnerabilities in systems and devise ways to exploit them before criminals do.
For more details please read the original article at TechCrunch AI.
Continue Learning
Comments
Sign in to join the conversation