“To properly secure machine learning, the enterprise needs to be able to do three things: find where machine learning is being used, threat model the risk based on what was found, and put in controls to manage those risks.
‘We need to find machine learning [and] do a threat model based on what you found,’ McGraw says. ‘You found some stuff, and now your threat model needs to be adjusted. Once you do your threat model and you’ve identified some risks and threats, you need to put in some controls right across all those problems.’
There is no one tool or platform that can handle all three things, but McGraw happens to be on the advisory boards for three companies corresponding to each of the areas. Legit Security finds everything, IriusRisk helps with threat modeling, and Calypso AI puts controls in place.
‘I can see all the parts moving,’ McGraw says. ‘All the pieces are coming together.'”
Apparently there are many CISOs out there who believe that their enterprise policies prohibit the use of ML, LLMS, and AI in their organization. Little do they know what’s actually happening.
BIML provided a preview of our upcoming LLM Risk Analysis work (including the top ten LLM risks) at a Philosophy of Mind workshop in Rio de Janeiro January 5th. The workshop was organized by David Chalmers (NYU) and Laurie Paul (Yale).
Once you learn that many of your new applications have ML built into them (often regardless of policy), what’s the next step? Threat modeling, of course. Irius Risk, the worldwide leader for threat modeling automation, announced a threat modeling library covering ML risks identified by BIML on October 26, 2023.
This is the first tool in the world to include ML risk as part of threat modeling automation. Now we’re getting somewhere.
Darkreading was the first publication to cover the news, and remains the best source for cutting edge stories about machine learning security. Read the original story here.
Much of what the executive order is trying to accomplish are things that the software and security communities have been working on for decades, with limited success.
“We already tried this in security and it didn’t work. It feels like
we already learned this lesson. It’s too late. The only way to
understand these systems is to understand the data from which they’re
built. We’re behind the eight ball on this,” said Gary McGraw, CEO of
the Berryville Institute of Machine Learning, who has been studying software security for more than 25 years and is now focused on AI and machine learning security.
“The big data sets are already being walled off and new systems can’t
be trained on them. Google, Meta, Apple, those companies have them and
they’re not sharing. The worst future is that we have data feudalism.”
Another challenge in the effort to build safer and less biased models
is the quality of the data on which those systems are being trained.
Inaccurate, biased, or incomplete data going in will lead to poor
results coming out.
“We’re building this recursive data pollution problem and we don’t
know how to address it. Anything trained on a huge pile of data is going
to reflect the data that it ate,” McGraw said. “These models are going
out and grabbing all of these bad inputs that in a lot of cases were
outputs from the models themselves.”
“It’s good that people are thinking about this problem. I just wish
the answer from the government wasn’t red teaming. You can’t test your
way out of this problem.”
BIML was invited to Oslo to present its views on Machine Learning Security in two presentations at NBIM in October.
The first was delivered to 250+ technologists on staff (plus 25 or so invited guests from all around Norway). During the talk, BIML revealed its “Top Ten LLM Risks” data for the first time (pre-publication).
BIML presented two talks at NBIM
The second session was a fireside chat for 19 senior executives.
The idea that machine learning security is exclusively about “hackers,” “attacks,” or some other kinds of “adversary,” is misguided. This is the same sort of philosophy that misled software security into a myopic overfocus on penetration testing way back in the mid ’90s. Not that pen testing and red teaming are useless, mind you, but there is way more to security engineering that penetrate and patch. It took us forever (well, a decade or more) to get past the pen test puppy love and start building real tools to find actual security bugs in code.
That’s why the focus on Red Teaming AI coming out of the White House this summer was so distressing. On the one hand…OK, the White House said AI and Security in the same sentence; but on the other hand, hackers gonna hack us outta this problem…not so much.
This red teaming nonsense is worse than just a philosophy problem, it’s a technical issue too. Just take a look at this ridiculous piece of work from Anthropic.
Red teaming sounds high tech, mysterious and steeped in hacker mystique, but today’s ML systems won’t benefit much from post facto pen testing. We must build security into AI systems from the very beginning (by paying way more attention to the enormous swaths of data used to train them and the risks these data carry). We can’t security test our way out of this corner, especially when it comes to the current generation of LLMs.
It’s tempting to pretend we can sprinkle some magic security dust on these systems after they are built, patch them into submission, or bolt special security apparatus on the side. Unfortunately the world well knows what happens when we pretend to be hard at work on security yet what we’re actually doing is more akin to squeezing our eyes shut and claiming to be invisible. Just ask yourself one simple question, who benefits from a security circus in this case?
AP reporter Frank Bajak covered BIML’s angle in this worldwide story August 13, 2023.
