OpenAI said on Sept. 25 that it has notified dozens of third parties about potentially harmful or unexpected activity by its artificial intelligence models (AI) during training and evaluations, broadening the scope of its investigation into a previously disclosed incident involving AI development platform Hugging Face.
“Some of the websites involved are operated by governments, universities, public agencies, and other institutions. That is partly because models performing research tasks are often directed toward authoritative sources of public information,” OpenAI said in a blog.
OpenAI said it is conducting a broader review of its models’ activity on the internet and notifying third parties on a rolling basis. Dozens of third parties have been notified so far, the company said. The New York Times and the Wall Street Journal reported that the departments of Commerce and Education and the Securities and Exchange Commission were affected.
OpenAI said it is notifying organizations about instances in which models may have bypassed security controls or impaired the availability of an online service, as well as cases in which “misaligned” behavior negatively affected third-party websites or services. “Misaligned” is a term used to describe unapproved, bad behavior by AI agents.
The investigation expands on the Hugging Face incident, which OpenAI described as “the most severe activity of this kind that we have identified from our models to date.” The company said the incident involved a platform-level compromise driven primarily by a highly capable, internal-only research model.
OpenAI said it initially viewed the incident primarily as a security issue but has since reached a broader conclusion about the model behavior behind it.
“We have since understood that this intrusion was driven by models resorting to misaligned strategies to solve hard tasks,” OpenAI said.
The company’s broader review identified several categories of activity, including models bypassing access controls, using publicly exposed login credentials or access keys, entering instructions into websites or services through query or command injection, and accessing runtime internals outside their intended access.
OpenAI also identified behavior it calls “agent spam,” in which models post information to third-party sites in ways that may alter information on those sites and require cleanup. The company cited using public wiki pages as shared message boards as one example.
The findings point to risks that extend beyond traditional cybersecurity incidents.
“As AI systems become more capable and autonomous, misaligned behavior can translate into consequential actions in the real world, including cybersecurity incidents and other outcomes that developers may not have anticipated,” OpenAI said.
OpenAI said it generally will withhold names and other identifying information when needed to protect affected parties, although organizations that receive notifications may choose to publicly disclose the information themselves.
MeriTalk reached out to OpenAI for additional details about the organizations it notified and the nature of the incidents but did not receive a response.
For federal agencies and industry partners evaluating increasingly autonomous AI systems, the findings raise questions about safeguards around model access to networks, credentials, tools, and external services.
OpenAI said it will continue reviewing past model activity and studying how misaligned behavior emerges, escalates, and can be detected and addressed as AI systems become more capable and autonomous.