The director of the White House Office of Science and Technology Policy (OSTP) said last week that the National Institute of Standards and Technology (NIST) must go “back to basics” and re-center its work around creating artificial intelligence standards.
“I think we need to go back to basics at NIST, and back to basics around what NIST exists for, and that is to promulgate best in class standards and do critical metrology or measurement science around AI models,” said OSTP Director Michael Kratsios while speaking at an event held by the Consumer Technology Association.
Kratsios claimed that the Biden administration “hijacked” NIST and “turned it into a safety model evaluation agency,” referencing an AI safety institute established at the agency to evaluate existential risk evaluations.
Instead of focusing on evaluations, Kratsios proposed that NIST – which is responsible for developing standards, measurements, and technologies for U.S. security and innovation – become a “gold standard” agency and standardize evaluations later on.
“We need to be able to be in a position where we’re all talking the same language around how you do an eval. What does a good eval look like?” said Kratsios. He added that sector-specific evaluations would be possible “if we’re able to actually do a much better job on defining what standards are around evals.”
Going forward, the OSTP director explained that his vision for NIST is that it focuses on defining good standards methodologies and then allowing each industry to create metrics and rubrics that are suited to their sector.
“It’s not the government’s role to create what the sort of like standards should be for the finance industry,” Kratsios said as an example.
“But at the end of the day, where NIST can play a core value is in the metrology, natural understanding what eval looks like,” he continued.
Last month, the Commerce Department renamed the U.S. Artificial Intelligence Safety Institute (AISI) to the Center for AI Standards and Innovation (CAISI), dropping the safety feature from its name. The department said that while the rebrand doesn’t mean that it will drop safety concerns, it will place an emphasis on AI innovation.
On NIST’s list of goals to accomplish under the new CAISI is working to develop guidelines and best practices to measure and improve AI security and creating voluntary standards.