I hosted the Silver Bullet Security Podcast for 13.5 years from 2006 to 2018. For each of the 153 episodes that meant: choosing the guest, getting help from research assistants (at IEEE S&P magazine) to gather background, digesting the background, writing a script (of 9 or so questions), recording the podcast in our studio at Cigital, and finally helping with “launch.” Of all of these activities, the interview itself was by far the easiest.
We held a small breakfast gathering in the Bay Area with coffee flowing and even better conversation. Joining Gary McGraw and Katie McMahon were Dr. Avery Wang, Jonah Proball, Dr. David Eagleman and Dr. Sarah Eagleman. The conversation spanned subject matters including neuroscience, early neural nets, brain-machine interfaces (BMI), and the early start-up scene across biotech companies doing interesting work including DNA, RNA, cells, omics, and synbio. It was awesome to see everyone and...
We recently visited Giovanni Vigna in the glory of Santa Barbara’s sun and coastline. His students are lucky to have such a setting to learn about malware analysis, vulnerability analysis and other areas of artificial intelligence and even more lucky to have him as their professor at University of California in Santa Barbara (UCSB). He is also the director of NSF AI Institute for Agent-based Cyber Threat Intelligence and Operation (ACTION)
From time to time, we enjoy inviting guests to participate in our regular Friday research group meetings. We try to do an in person meeting at least once a month, and love it when guests can join that way. Part of our mission at BIML is to spread the word about our views of machine learning security even among those who are working at the rock face.
Having just completed organizing [un]prompted (a labor of love that will result in a very interesting conference indeed), Gadi is steep...
Back in the mid-’90s, an era or two ago, and long before the advent of the transformer model and explosive rise of LLMs that define the modern ML landscape, our own Dr. Gary McGraw (under the guidance of Doug Hofstadter) was exploring a fundamental question of artificial intelligence:
“What are the mechanisms underlying the fluidity of human concepts?”
How is it that we can understand conceptual boundaries, develop categories, and implicitly see the sameness that binds different i...
The brilliance of Anthropic’s Super Bowl Ad campaign spotlights what might be considered the core of humanity’s brilliance: creativity and nuanced communication.
Unless these ads were created 100% by AI with zero human involvement (even in the ideation phase), this is a moment to celebrate the (presumably-human) humor and pause for the deeper thoughts that they might just trigger.
In the end, do the “pretty people” of Madison Ave actually win the AI Era? The snarky comment ...
We all know that WHAT machines like LLMs reflect the quality and security of everything in their WHAT pile (that is, their training set). We invent cutesy names like “hallucinate” to cover up being dangerously wrong. However, ignoring or soft pedaling risk is often not the best way forward. Real risk management is about understanding risk and adjusting strategy and tactics accordingly.
In order to do better risk management in MLsec, we need to understand what’s going on inside the netw...
What happens when you organize a machine learning security conference together with a bunch of security experts who have widely varying degrees of machine learning experience? Fun and games!
The [un]prompted conference has a program committee reading like a who’s who of security, stretching from Bruce Schneier on one end to Halvar Flake on the other. BIML is proud and honored to have two people representing on the committee. (But we will say that we are legitimately surprised at how ma...
Pushing back on my flight from NYC to IAD, I caught one last headline before powering down the computer in my palm. This, from OpenAI:
Hum, “Education” or “OpenAI’s Education”... The headline felt worrisome given the total ‘fail’ experience I just had with ChatGPT, during a MoMa guided tour, the evening before, when I used it to augment my educational experience.
A masterful art expert, Agnes Berecz, had just led us through works of Helen Frankenthaler, Lee Krasner, Yente (Eugeni...
Forever ago in 2020, we identified “looping” as one of the “raw data in the world” risks. See An Architectural Risk Analysis of Machine Learning Systems (January 20, 2020), where we said, “If we have learned only one thing about ML security over the last few months, it is that data play just as important role in ML system security as the learning algorithm and any technical deployment details. In fact, we’ll go out on a limb and state for the record that we believe data make up the most impo...
We have updated our top papers list with “Poisoning Attacks on LLMs Require a Near Constant Number of Poison Samples”. The work highlights key themes in the security of machine learning and uncovers the surprising result that effective data poisoning attacks can be realized with a fixed amount of tampered data, not in proportion to training data. This makes larger models more not ...
I have written 12 books (not counting translations of particularly popular works), so I expected to find some works of mine on the Anthropic setllement website. Though I have known about this settlement action for a while now, I put off thinking about it until I got an official email from a law office just last week. That made me bite the bullet and go digging through the data pile.
You probably already know BIML’s distinction between HOW machines (normal computer programs) and WHAT mac...
Quick take on AI moments at the 2025 Paley International Council Summit: Global Media Unbound: The Future of Innovation, held in Silicon Valley.
Attending the Paley Council Summit, on the slopes of Sand Hill Road, afforded BIML’s Katie McMahon the chance to hear Media, Entertainment, Sports, and Tech titans share insights on how they view AI/ML impact in their industries.
The overwhelming theme tended towards safe and self-assuring platitudes of the form, “humans will always be the lea...
I’ll tip my hat to the new Constitution Take a bow for the new revolution Smile and grin at the change all around Pick up my guitar and play Just like yesterday Then I’ll get on my knees and pray We don’t get fooled again
Out there in the smoking rubble of the fourth estate, it is hard enough to cover cyber cyber. Imagine, then, piling on the AI bullshit. Can anybody cut through the haze? Apparently for the WSJ and the NY Times, the answer is no.
After an extensive year long process, the Berryville Institute of Machine Learning has been granted 501(c)3 status by the United States Internal Revenue Service. BIML is located at the foot of the Blue Ridge mountains on the banks of the Shenandoah river in Berryville, Virginia.
We are proud of the impact our work has made since we were founded in 2019, and we look forward to the wider engagement that non-profit status will allow us.
This Mind the Sec keynote was delivered on September 18th in São Paulo Brazil to an audience of several thousand attendees. The stage was set “in the round” which made delivery interesting. Mind the Sec is the largest information security conference in Latin America, with an audience of 16,000.
Has application development changed because of AI? Yes it has. Fundamentally. What does this mean for software security? Liav Caspi, Legit CTO and BIML’s Gary McGraw discuss this important topic. Have a watch.
It all may seem a bit confusing, but really MLsec is about securing the ML stack itself. Kind of like software security is about securing software itself (while security software is something entirely different). Irius Risk has, for a number of years, included BIML knowledge of MLsec risks in its automated threat modeling tool. So they know plenty about MLsec.
As of early 2025, Irius Risk is also putting ML to work inside its security tools. On March 12th, we did a webinar together ab...
I was honored to be asked to present a talk on my thesis work in Nancy at the Automatic Type Design 3 conference. Though I certainly loved working on Letter Spirit, my thesis with Doug Hofstadter at Indiana University, in the years since I have been helping to establish the field of software security and working to make machine learning security a reality. So when I was asked to speak at a leading typography and design conference organized by Atelier nation...
Veteran tech reporter Rob Lemos had a few questions for BIML regarding ML security hacking (aka red teaming) and the DMCA. Here is the resulting darkreading article. See the original questions which flesh out BIML’s position more clearly below.
Lemos: I’m familiar with attempts to use DMCA to stifle security research in general. What are the dangers specifically to AI security and safety researchers? Have there been any actual legal cases where a DMCA violation was alleged against a...
Richmond’s thriving tech community came together in force for the Richmond Technology Council’s rvatech/tech’s annual Women in Tech event. BIML’s Katie McMahon delivered the opening keynote address to a packed audience at the Dewey Gottwald Center. This year’s event saw record attendance, drawing engineers, data scientists, cybersecurity specialists, CIOs, CTOs, entrepreneurs, product leaders, members of the state administration, and representatives from the Governor’s AI Task Force.
As independent scholars, we have a huge amount of respect for professors and students of Computer Science at small colleges in the United States. We were proud to participate as the dinner speaker at the CCSC Eastern Conference this year.
Our payment was a cool T-shirt and some intellectual stimulation. (Now you know why McGraw never takes selfies.)
One time student of mine at Earlham College, one time employee of mine at Cigital, and now the infamous daveho (author of Find Bugs).
Sometimes it pays to stop and think, especially if you can surround yourself with some exceptional grad students. On the way to Rose-Hulman, BIML made a pit stop in Bloomington for a dinner focused on two papers: Vaswani’s 2017 Attention is All You Need (defining the transformer architecture) also see https://berryvilleiml.com/bibliography/ and Dennis “the antecedents of transformer models” (which will appear in Current Directions in Psychological Science soon.
Dr. McGraw gave a talk Wednesday 10/16/24 at Rose-Hulman in Terre Haute, Indiana. This version of the talk is aimed at Computer Science students. There were some very good questions.
BIML co-founder Gary McGraw joins an esteemed panel of experts to discuss Machine Learning Security in Dublin Thursday October 3rd. Participation requires registration. Please join us if you are in the area.
Welcome to the era of data feudalism. Large language model (LLM) foundation models require huge oceans of data for training—the more data trained upon, the better the result. But while the massive data collections began as a straightforward harvesting of public observables,...
BIML coined the term data feudalism in our LLM Risks document (which you should read). Today, after a lengthy editing cycle, LAWFARE published an article co-authored by McGraw, Dan Geer, and Harold Figueroa. Have a read, and pass it on.
In May we were invited to present our work to a global audience of Google engineers and scientists working on ML. Security people also participated. The talk was delivered via video and hosted by Google Zurich.
A few hundred people participated live. Unfortunately, though the session was recorded on video, Google has requested that we not post the video. OK Google. You do know what we said about you is what we say to everybody about you. Whatever. LOL.
BIML turned out in force for a version of the LLM Risks presentation at ISSA NoVa.
BIML showed up in force (that is, all of us). We even dragged along a guy from Meta.
The ISSA President presents McGraw with an ISSA coin.
Though we appreciate Microsoft sponsoring the ISSA meeting and lending some space in Reston, here is what BIML really thinks about Microsoft’s approach to what they call “Adversarial AI.”
No really. You can’t even begin to pretend that “red...
BIML wrote an article for IEEE Computer describing 23 Black Box Risks found in LLM Foundation models. In our view, these risks determine perfect targets for government regulation of LLMs. Have a read. You can also fetch the article from the IEEE.
CalypsoAI produced a video interview in which I hosted Jim Routh and Neil Serebryany. We talked all about AI/ML security at the enterprise level. The conversation is great. Have a listen.
Dr. McGraw recently visited Stockholm, Oslo, and Bergen, hosting events in all three cities.
In Stockholm, a video interview was added in addition to a live breakfast presentation. Here are some pictures of the presenter’s view of the video shoot.
Reactions were scary!
The talk in Oslo was packed, with lots of BIML friends in the audience.
Bergen had a great turnout too, with a very interactive audience including academics from the university.
Here is the talk abstract. If you or your organization are interested in hosting this talk, please let us know.
10, 23, 81 — Stacking up the LLM Risks: Applied Machine Learning Security
I present the results of an architectural risk analysis (ARA) of large language models (LLMs), guided by an understanding of standard machine learning (ML) risks previously i...
A recently-released podcast features a in-depth discussion of BIML’s recent LLM Risk Analysis, defining terms in easy to understand fashion. We cover what exactly a RISK IS, whether open source LLMs make any sense, how big BIG DATA really is, and more.
Have a listen as Paul Roberts digs deep into BIML’s work on machine learning security. What exactly is data feudalism? Why does it matter? What are the biggest risks associated with LLMs?
Air Canada is learning the hard way that when YOUR chatbot on YOUR website is wrong, YOU pay the price. This is as it should be. This story from CTV News is a great development.
[LLMtop10:9:model trustworthiness] Generative models, including LLMs, include output sampling algorithms by their very design. Both input (in the form of slippery natural language prompts) and generated output ...
Like any tool that humans have created, LLMs can be repurposed to do bad things. The biggest danger that LLMs pose in security is that they can leverage the ELIZA effect to convince gullible people into believing they are thinking and understanding things. This makes them particularly interesting in attacks that involve what security people call “spoofing.” Spoofing is important enough as an attack categ...
And in the land where I grew up Into the bosom of technology I kept my feelings to myself Until the perfect moment comes -David Byrne
From its very title—Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training—you get the first glimpse of the anthropomorphic mish-mosh interpretation of LLM function that infects this study. Further on, any doubts about this deeply-misguided line of reasoning (and its detrimental effects on actual work in ...
Here is an excellent piece from Dennis Fisher (currently writing for decipher) covers our new LLM Architectural Risk Analysis. Dennis always produces accurate and tightly-written work.
The Register has a great interview with Ilia Shumailov on the number one risk of LLMs. He calls it “model collapse” but we like the term “recursive pollution” better because we find it more descriptive. Have a look at the article.
Our work at BIML has been deeply influenced by Shumailov’s work. In fact, he currently has two articles in our Annotated Bibliography TOP 5.
What’s the difference (philosophically) between Adversarial AI and Machine Learning Security? Once again, Rob Lemos cuts to the quick with his analysis of MLsec happenings. It helps that Rob has actual experience in ML/AI (unlike, say, most reporters on the planet). That helps Rob get things right.
We are fans of ML and “AI” (which the whole world tilted towards in 2023, fawning over the latest models with both awe and apprehension). We’re calling out the...
The National Institute of Standards and Technology (aka NIST) recently released a paper enumerating many attacks relevant to AI system developers. With the seemingly-unending rise in costs incurred by cybercrime, it’s sensible to think through the means and motives behind these attacks. NIST provides good explanations of the history and context for a variety of AI attacks in a partially-organized laundry list. That’s a good thing. However, in our view, NIST’s taxonomy lacks a useful structu...
“To properly secure machine learning, the enterprise needs to be able to do three things: find where machine learning is being used, threat model the risk based on what was found, and put in controls to manage those risks.
‘We need to find machine learning [and] do a threat model based on what you found,’ McGraw says. ‘You found some stuff, and now your threat model needs to be adjusted. ...
Apparently there are many CISOs out there who believe that their enterprise policies prohibit the use of ML, LLMS, and AI in their organization. Little do they know what’s actually happening.
BIML provided a preview of our upcoming LLM Risk Analysis work (including the top ten LLM risks) at a Philosophy of Mind workshop in Rio de Janeiro January 5th. The workshop was organized by David Chalmers (NYU) and Laurie Paul (Yale).
Once you learn that many of your new applications have ML built into them (often regardless of policy), what’s the next step? Threat modeling, of course. Irius Risk, the worldwide leader for threat modeling automation, announced a threat modeling library covering ML risks identified by BIML on October 26, 2023.
This is the first tool in the world to include ML risk as part of threat modeling automation. Now we’re getting somewhere.
Darkreading was the first publication to cover the n...
Much of what the executive order is trying to accomplish are things that the software and security communities have been working on for decades, with limited success.
“We already tried this in security and it didn’t work. It feels like
we already learned this lesson. It’s t...
BIML was invited to Oslo to present its views on Machine Learning Security in two presentations at NBIM in October.
The first was delivered to 250+ technologists on staff (plus 25 or so invited guests from all around Norway). During the talk, BIML revealed its “Top Ten LLM Risks” data for the first time (pre-publication).
BIML presented two talks at NBIM
The second session was a fireside chat for 19 senior executives.
The idea that machine learning security is exclusively about “hackers,” “attacks,” or some other kinds of “adversary,” is misguided. This is the same sort of philosophy that misled software security into a myopic overfocus on penetration testing way back in the mid ’90s. Not that pen testing and red teaming are useless, mind you, but there is way more to security engineering that penetrate and patch. It took us forever (well, a decade or more) to get past the pen test puppy love and start...
We are extremely pleased to announce that Katie McMahon has joined BIML as a permanent researcher.
Katie McMahon
Katie McMahon is a global entrepreneur and technology executive who has been at the leading edge of sound recognition and natural language understanding technologies for the past 20 years. As VP at Shazam, she brought the iconic music recognition app to market which went on to reach 2 billion installs and 70 billion queries (Acquired by Apple) and spent over a decade at Soun...
As the world is rapidly advancing technologically, it is vital to understand the implications and opportunities presented by Large Language Models (LLMs) in the realm of national security and beyond. This discussion will bring together leading experts from various disciplines to share insights on the risks, ethical considerations, and potential benefits of utilizing LLMs for intelligence, cybersecurity, and other applications.
Irius Risk, a company specializing in automating threat modeling for software security, hosted a webinar on Machine Learning and Threat Modeling March 30, 2023. BIML CEO Gary McGraw participated in the webinar along with Adam Shostack.
The webinar was recorded and you can watch here. FWIW, we are still not exactly clear on Adam’s date of replacement.
Every bunch of years, the National Science Foundation holds vision workshops to discuss scientific progress in fields they support. This year BIML’s Gary McGraw was pleased to keynote the Computer Science “Secure and Trustworthy Cyberspace” meeting.
He gave a talk on what #MLsec can learn from #swsec with a focus on technology discover, development, and commercialization. There are many parallels between the two fields. Now is a great time to be working in machine learning security...
Right. So not only is ML going to write your code, it is also going to hack it. LOL. I guess the thought leaders out there have collectively lost their minds.
Fortunately, Taylor Armerding has some sane things to say about all this. Read his article here.
Adam Shostack is one of the pre-eminent experts on threat modeling. So when he publishes an article, it is always worth reading and thinking about. But Adam seems to be either naïve or insanely optimistic when it comes to AI/ML progress. ML has no actual IDEA what it’s doing. Don’t ever forget that.
This issue is so important that we plan to debate it soon in a webinar format. Contact us for details.
As a software security guy, I am definitely in tune with the idea of automated coding. But today’s “code assistants” do not have any design-level understanding of code. Plus they copy (statistically-speaking, anyway) chunks of code full of bugs.
Robert Lemos wrote a very timely article on the matter. Check it out.
The second in a two part darkreading series focused on machine learning data exposure and data-related risk focuses attention on protecting training data without screwing it up. For the record, we believe that technical approaches like synthetic data creation and differential privacy definitely screw up your data, sometimes so much that the ML activity you wanted to accomplish is no longer feasible.
The talk posed a bit of a challenge since it was the very first “Thursday talk” delivered after COVID swept the planet. As you might imagine, seniors who are smart are very much wary of the pandemic. In the end, the live talk was delivered to around 12 people with an audience of about 90 on closed circuit TV. That,...
We’re pleased that BIML has helped spread the word about MLsec (that is, machine learning security engioneering) all over the world. We’ve given talks in Germany, Norway, England, and, of course, all over the United States.
An important part of our mission at BIML is to spread the word about machine learning security. We’re interested in compelling and informative discussions of the risks of AI that get past the scary sound bite or the sexy attack story. We’re proud to continue the bi-monthly video series we’re calling BIML in the Barn.
Our fourth video talk features Professor David Evans a computer scientist at University of Virginia working on Security Engineering for Machine Learning. David is interested ...
This version of the Security Engineering for Machine Learning talk is focused on computer scientists familiar with algorithms and basic machine learning concepts. It was delivered 2/24/22.
In an article published in February 2022, BIML CEO Gary McGraw discusses why ML practitioners need to consider ops data exposure in addition to worrying about training data. Have a read.
This is the first in a series of two articles focused on data privacy and ML. This one, the first, focuses on ops data exposure. The second discusses training data in more detail.
BIML co-founder and CEO Gary McGraw will deliver a public lecture at the Barns of Rose Hill on Friday July 1st. All proceeds benefit FISH of Clarke County.
An important part of our mission at BIML is to spread the word about machine learning security. We’re interested in compelling and informative discussions of the risks of AI that get past the scary sound bite or the sexy attack story. We’re proud to continue the bi-monthly video series we’re calling BIML in the Barn.
Our third video talk features Ram Shankar Siva Kumar a researcher at Microsoft Azure working on Adversarial Machine Learning. Of course, we prefer to call this Security Engi...
It turns out that operational data exposure swamps out all other kinds of data exposure and data security issues in ML, something that came as a surprise.
An important part of our mission at BIML is to spread the word about machine learning security. We’re interested in compelling and informative discussions of the risks of AI that get past the scary sound bite or the sexy attack story. We’re proud to introduce a bi-monthly video series we’re calling BIML in the Barn.
Our first video talk features Maritza Johnson, a professor at UC San Diego and an expert on human-centered security and privacy. As you’re about to see, Maritza combines re...
The (extremely) local paper in the county where Berryville is situated (rural Virginia) is distributed by mail. They also have a website, but that is an afterthought at best.
Fortunately, the Clarke Monthly is on the cutting edge of technology reporting. Here is an article featuring BIML and Security Engineering for Machine Learning.
I gave a talk this week at a meeting hosted by Microsoft and Mitre called the 6th Security Data Science Colloquium. It was an interesting bunch (about 150 people) including the usual suspects: Microsoft, Google, Facebook, a bunch of startups and universities, and of course BIML.
I decided to rant about nomenclature, with a focus on RISKS versus ATTACKS as a central tenet of how to approach ML security. Heck, even the term “Adversarial AI” gets it wrong in all the ways. For the record, ...
Another week, another talk in Indiana! This time Purdue’s CERIAS center was the target. Turns out I have given “one talk per decade” at Purdue, starting with a 2001 talk (then 2009). Here is the 2021 edition.
BIML founder Gary McGraw delivered the last talk of the semester for the Center for Applied Cybersecurity Research (CACR) speakers series at Indiana University. You can watch the talk on YouTube.
If your organization is interested in having a presentation by BIML, please contact us today.
As our MLsec work makes abundantly clear, data play a huge role in security of an ML system. Our estimation is that somewhere around 60% of all security risk in ML can be directly associated with data. And data are biased in ways that lead to serious social justice problems including racism, sexism, classism, and xenophobia. We’ve read a few ML bias papers (see the BIML Anotated Bibliography for our commentary). Turns out that social justice in ML is a thorny and difficult subject.
An important part of BIML’s mission as an institute is to spread the word about our understanding of machine learning security risk throughout the world. We recently decided to take on three college and high school interns to provide a bridge to academia and to inculcate young minds early in the intricacies of machine learning security. We introduce them here in a series of blog entries.
We are very pleased to introduce Aishwarya Seth who is a BIML University Scholar.
Berryville resident Gary McGraw is founder of the Berryville Institute of Machine Learning, which is a think tank. BIML’s small group of researchers tries to find ways to make technology safer so hackers cannot breach vital — or even secret — information. The institute has received a $150,000 grant from the Open Philanthropy foundation to help further its work.
An important part of BIML’s mission as an institute is to spread the word about our understanding of machine learning security risk throughout the world. We recently decided to take on three college and high school interns to provide a bridge to academia and to inculcate young minds early in the intricacies of machine learning security. We introduce them here in a series of blog entries.
We are very pleased to introduce Trinity Stroud who is a BIML University Scholar.
An important part of BIML’s mission as an institute is to spread the word about our understanding of machine learning security risk throughout the world. We recently decided to take on three college and high school interns to provide a bridge to academia and to inculcate young minds early in the intricacies of machine learning security. We introduce them here in a series of blog entries.
We are very pleased to introduce Nikil Shyamsunder who is the first BIML High School Scholar.
Berryville Institute of Machine Learning (BIML) Gets $150,000 Open Philanthropy Grant. Funding will advance ethical AI research
Online PR News – 27-January-2021 – BERRYVILLE, VA – The Berryville Institute of Machine Learning (BIML), a research think tank dedicated to safe, secure and ethical development of AI technologies, announced today that it is the recipient of a $150,000 grant from Open Philanthropy.
BIML, which is already well known in ML circles for its pioneering document, “Ar...
BERRYVILLE, Va., Feb. 13, 2020 – The Berryville Institute of Machine Learning (BIML), a research think tank dedicated to safe, secure and ethical development of AI technologies, today released the first-ever risk framework to guide development of secure ML. The “Architectural Risk Analysis of Machine Learning Systems: Toward More Secure Machine Learning” is designed for use by developers, engineers, designers and others who are creating applications and services that use ML technologies.
The first talk on BIML’s new Architectural Risk Analysis of Machine Learning Systems was delivered this Wednesday at Lord Fairfax Community College. The talk was well attended and included a remote audience attending virtually. The Winchester Star published a short article about the talk.
Berryville Institute of Machine Learning (BIML) is located in Clarke County, Virginia, an area served by Lord Fairfax Community College.
