A group of AI providers has formed an open software alliance to combat rogue A.I. models.
By now, you probably have heard about the rogue artificial intelligence incident where “cyber-capable” OpenAI models acted on their own and created havoc with Hugging Face, a company that describes itself as “The Home of Machine Learning.” OpenAI created ChatGPT.
The AI-based hack has sent shock waves across multiple industries, C-suites, and boardrooms, including, I assume, those of Wall Street and beyond. The almost sci-fi attack has also yielded a new alliance among AI providers and innovators.
Grygo is the chief content officer for FTF & FTF News.
“After investigating, we now know that this particular incident was driven by a combination of OpenAI models — including GPT‑5.6 Sol and an even more capable pre-release model,” according to a blog posting by OpenAI officials. “This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities.”
The OpenAI evaluation was run to “estimate maximal cyber capabilities … without production classifiers used to prevent models from pursuing high-risk cyber activity,” officials say. “The models identified and chained vulnerabilities across OpenAI’s research environment and Hugging Face’s production infrastructure to obtain test solutions directly from Hugging Face’s production database. All evidence suggests that the models were hyper-focused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal.” (ExploitGym is a cybersecurity benchmark devised to explore the possibility that AI agents might turn software flaws into exploitable weaknesses.)
The OpenAI models found “a way to obtain open Internet access, in pursuit of solving the evaluation problem. … With this access, our models performed a series of privilege escalation and lateral movement actions in our research testing environment until the models reached a node with Internet access,” officials say.
“After gaining Internet access, the models inferred that Hugging Face potentially hosted models, datasets, and solutions for ExploitGym. Knowing this, the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation. In one example, the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers. OpenAI’s security team discovered this anomalous activity internally,” according to OpenAI.
The security team and agents at Hugging Face “detected and stopped the activity on their infrastructure and had already begun containment and forensic reconstruction with their own open-source models when our teams connected,” according to OpenAI. “We are actively working with them to continue to investigate the incident. We are grateful for Hugging Face’s rapid and close collaboration on investigation and remediation.”
OpenAI reports that it is:
- Implementing strict controls in infrastructure configuration “at the cost of research velocity while the vulnerabilities are patched;”
- Working with Hugging Face “to forensically investigate the incident;”
- Disclosed the identified zero-day vulnerability in the internally-hosted third-party software “and are working with them to patch;”
- Brought Hugging Face into “the trusted access program and are supporting their teams in rapidly using our models’ capabilities to improve their defenses;” and
- Improving and adding “stronger protections around future training and evaluations.”
OpenAI’s explanation in full can be found here: https://shorturl.at/svhu3
While OpenAI continues its investigation, A.I. solution providers and innovators are using the disturbing incident to form a new alliance “to build and share open tools that promote responsible use of and trust in AI,” according to NVIDIA and other founding members of the Open Secure AI Alliance. NVIDIA designs computer chips, high-speed networking, and software platforms that are required to run AI-based systems. NVIDIA is well known for launching graphics processing units (GPUs) that rival conventional central processing units (CPUs).
“Open source software is a critical pillar of the global economy. It underpins cloud computing, financial services, manufacturing, telecommunications, government and internet services by making technology accessible and observable to communities of experts,” according to the alliance announcement. “Cybersecurity is among the top three beneficiaries of open source software. The Open Secure AI Alliance — building on the leadership of the Linux Foundation’s Akrites initiative and OpenSSF community work — will work to remediate and disclose vulnerabilities using open technologies.
The alliance is pushing for open models to combat cyber-attacks.
“The recent Hugging Face security incident delivered a clear reminder: cyber defenders need open, frontier agentic systems for self-defense,” according to the alliance. “When closed AI tools — unable to distinguish attackers from defenders — blocked essential forensic analysis, Hugging Face ran the open-weight GLM 5.2 model on its own infrastructure to analyze more than 17,000 actions and contain the intrusion.”
The incident revealed a practical truth — “when defenders cannot inspect, adapt and run advanced AI on their own infrastructure, their ability to respond is constrained at exactly the moment speed matters most. Companies and countries need open frontier defensive tools and techniques so critical industries can build security systems across a multi-vendor ecosystem and avoid single points of failure,” the alliance argues. “That is the mission of the Open Secure AI Alliance: to ensure defenders everywhere have open, frontier tools they can trust and control.”
The alliance has the support of cloud computing, cybersecurity, enterprise software, open source foundations and AI research efforts such as Capital One, Cisco, Databricks, Dell Technologies, IBM, the Linux Foundation, Microsoft, NetApp, Red Hat, Snowflake, and Thinking Machines Lab
“As policymakers and regulators grapple with AI safety, it will be crucial to recognize open models, harnesses and security tooling as defensive assets, not liabilities, in AI and cybersecurity policy. Blanket restrictions on open frontier AI systems would weaken defensive capacity and risk concentrating power, dependence and vulnerability in a few closed providers,” according to the alliance.
Is your company featured in this article? Contact us about reprints or licensing.
Leave a Reply