Extraordinary demand for this gathering: [un]prompted, AI security practitioner conference. Having been part of the submission review board and sitting in several of the speakers’ dry-runs this week, I’m looking forward to seeing everyone in person, hearing the talks and, perhaps most critically, what unfolds in the hallway conversations.
Huge thank you to Gadi Evron and the team putting this event together, including Ida Vass. The overflow has us moving from Salesforce Tower to a downtown historic venue and online being spun up.
Harold and I will be there representing BIML (Berryville Institute of Machine Learning) and we’re excited to talk-up the focus of machine learning security within this crowd. See you in San Francisco!
I hosted the Silver Bullet Security Podcast for 13.5 years from 2006 to 2018. For each of the 153 episodes that meant: choosing the guest, getting help from research assistants (at IEEE S&P magazine) to gather background, digesting the background, writing a script (of 9 or so questions), recording the podcast in our studio at Cigital, and finally helping with “launch.” Of all of these activities, the interview itself was by far the easiest.
Know why Silver Bullet was so good with such in-depth questions? Because the script writing took 4-5 hours per episode (not counting the background research…which was often much more involved than just googling the person). All this for a 20 minute show.
We are rebooting Silver Bullet after a few years off with a new focus on Machine Learning security. Our first guest will be Gadi Evron. We’ve redesigned the logo, built an initial distribution list, created a landing zone with proper feeds to the usual channels, and yes..written a script. But this time I decided to use Gemini as my research assistant. TL/DR it was great.
I started with a bunch of ideas in an amorphous blob. This got me thinking about show story arc, coverage of various aspects of MLsec, etc. Here is what my notes looked like.
Then it was time to invoke Gemini. Fortunately, Gemini knows lots about me and about Silver Bullet. Eerily so. It knew where the archive was, and was able to garner a meta-pattern for the show with some insight into its philosophy. Was it absolutely spot on? Nope. Was it sycophantic and overly agreeable? Yes. But hey, the show’s creator is here driving the laser pointer (which, like a good cat, Gemini was happy to pounce after).
I worked through the script in order with Gemini for about an hour, during which I was impressed with its up-to-date (like yesterday) access to things happening in the world…like on this very website. For example, Gemini knew that Gadi had just visited BIML and that [un]prompted was something we had worked on together. It was very helpful, sometimes wrong, often using the wrong words…but, question by question, the show arc emerged. It kept track of where we were, sometimes suggesting new directions (which I rejected every time), but always knowing where we were in the work. After the session, I asked it to dump the script to one place for copy/paste and then did a fine tuning edit pass (including real fact checking on a couple of things).
All told, my bet is Gemini saved me about a factor three or four times the usual amount of work I used to do. Will the show be just as good? Obviously, the proof is in the pudding. We will be launching the first episode on March 2nd.
Here’s how it will all start…
Silver Bullet Intro (BIML Focused)
[MUSIC: Classic Silver Bullet Theme – Up and Under]
gem: Welcome to the Silver Bullet Security Podcast episode 154. I’m your host, Gary McGraw, coming to you from the Berryville Institute of Machine Learning where we are defining the future of machine learning security.
From 2006-2018, Silver Bullet explored the nascent field of software security through the lens of building security in. But today, the frontier has moved. As we integrate machine learning into the fabric of our essential systems, we find ourselves facing a new set of architectural flaws and security challenges that traditional software security can’t touch.
On Silver Bullet, we’re shifting our focus to the security of machine learning—bringing the same deep-dive, “no-silver-bullet” philosophy to the world of AI.
To help me kick off this new era, I’m joined by my new friend Gadi Evron. Gadi is a veteran of the botnet wars, a community builder, and the chair of the new [un]prompted conference. Gadi, welcome to the show.
[MUSIC: Swells briefly then fades out]
1. The [un]prompted Vision
Gadi, you’re chairing the [un]prompted conference, and I’m really pleased to be working on the committee with you. We’ve both seen the security conference circuit evolve over the decades, but [un]prompted feels like it’s trying to capture lightning in a bottle for the ML security space. What was it about the current state of AI security that made you feel we needed a dedicated, practitioner-first venue—something beyond just another “AI track” at a traditional security show?
We held a small breakfast gathering in the Bay Area with coffee flowing and even better conversation. Joining Gary McGraw and Katie McMahon were Dr. Avery Wang, Jonah Proball, Dr. David Eagleman and Dr. Sarah Eagleman. The conversation spanned subject matters including neuroscience, early neural nets, brain-machine interfaces (BMI), and the early start-up scene across biotech companies doing interesting work including DNA, RNA, cells, omics, and synbio. It was awesome to see everyone and hear what they are up to and to share a little bit about what we’re doing at BIML.
We recently visited Giovanni Vigna in the glory of Santa Barbara’s sun and coastline. His students are lucky to have such a setting to learn about malware analysis, vulnerability analysis and other areas of artificial intelligence and even more lucky to have him as their professor at University of California in Santa Barbara (UCSB). He is also the director of NSF AI Institute for Agent-based Cyber Threat Intelligence and Operation (ACTION)
Gary and Giovanni traded stories which sounded like the good old days, but then we got serious in talking through the new, uncharted territory of machine learning security. We discussed what’s to be explored with emergent behavior in this “Agentic AI” phase and more.
Giovanni is also the founder of Shellphish which, apparently, has participated in more DEF CON CTF competitions than any other team in the world and an advisor to Artiphishell. Very cool.
From time to time, we enjoy inviting guests to participate in our regular Friday research group meetings. We try to do an in person meeting at least once a month, and love it when guests can join that way. Part of our mission at BIML is to spread the word about our views of machine learning security even among those who are working at the rock face.
Having just completed organizing [un]prompted (a labor of love that will result in a very interesting conference indeed), Gadi is steeped in the cybersecurity perspective of machine learning (as an offensive tool, a defensive tool, an attack surface, and an enterprise challenge). Of course we have our own BIML perspective on this, more focused on building security in than anything else.
Our meeting this week focused on tokenization first (an under-studied aspect of MLsec), and then tried to make sense of the absolute flood of stuff coming out of Anthropic these days. Bottom line?
There is lots more work to be done in tokenization
The C-compiler that Carlini tried to build with Claude is interesting, incomplete, and angled toward a reality check on the usual hyperbole. Good for Carlini for addressing the reality head on!
The Zero-day work (on the other hand) is hyperbolic, involving a breathless treatment of three well known and pretty boring attack pattern instances as applied in the face of blackbox fuzzing? We do acknowledge that automating exploit finding is a great thing to cover. Lets just do it without the razzle-dazzle.
Dario’s The Adolescence of Technology would better be described as the philosophy of an adolescent. Our main concern here is not counterfactualizing about AI apocalypse so much as how much of the real security conversation we need to have in MLsec gets ignored by this “look over there” kind of stuff.
We have lots more work to do to understand transformer circuits. You should look into it too. We must get into these networks and see what exactly they are doing INSIDE.
Anyway, it was great to have Gadi join us for the meeting and for a delightful lunch afterwards. This MLsec stuff is so fun.
Gadi Evron is Founder and CEO at Knostic, an AI security company, and chairs the ACoD cyber security conference. Previously, he founded (as CEO) Cymmetria (acquired), was CISO of the Israeli National Digital Authority, founded the Israeli CERT, and headed PwC’s Cyber Security Center of Excellence. He wrote the post-mortem analysis of the “First Internet War” (Estonia 2007), founded some of the first information-sharing groups (TH-Research, 1997, DA/MWP, 2004), wrote APT reports (Rocket Kitten – 2014, Patchwork – 2016, etc.), and the first paper on DNS DDoS Amplification Attacks (2006). Gadi has written two books on cybersecurity, is a frequent contributor to industry publications, and speaker at industry events, from Black Hat (2008, 2015) to Davos (2019) and CISO360 (2022).
Back in the mid-’90s, an era or two ago, and long before the advent of the transformer model and explosive rise of LLMs that define the modern ML landscape, our own Dr. Gary McGraw (under the guidance of Doug Hofstadter) was exploring a fundamental question of artificial intelligence:
“What are the mechanisms underlying the fluidity of human concepts?”
How is it that we can understand conceptual boundaries, develop categories, and implicitly see the sameness that binds different instances of a concept together? And what might we learn by building a machine that simulates this behavior? Or, rather, what is an A?
The perceptual hypothesis behind the Letter Spirit project is that letter-concepts are composed of constituent roles. That is, letter concepts, in turn, have letter-part concepts.
The Letter Spirit project approached these questions from the angle of letter perception. While easy to take for granted, we literate apes possess the ability to differentiate letters and letter categories displayed a huge variety of fonts, handwriting styles, and artistic styles. Our gut instinct may tell us that the letter “a” is a mere shape made up of a bunch of tiny dots; but just a few examples can reveal a much greater depth to what constitutes our concept of the letter ‘a’.
This role model hypothesis is implemented here as the Letter Spirit Examiner program (a program written in scheme in 1995). It works through emergent computation—by segmenting letters into natural, constituent parts that correspond to the conceptual roles of the very concept of a letter—that is, different conceptual rules that when satisfied lead us to identify a letter. The examiner does this by running hundreds of micro-agents (called codelets) that are instantiations of sixteen codelet types. The asynchronous, parallel, local processing done by the codelets implements a parallel terraced scan of possible structures (as in the role model’s predecessor, Copycat). From these codelets emerges a high-level perception—the categorization of a letter shape into an idea.
Just a couple of ‘a’s – Letter Spirit Ch. 1
To our great pleasure and delight, we recently learned that Paul Geiger has developed a JavaScript implementation of the Letter Spirit Examiner based on the original Scheme code developed originally by McGraw and then adapted by Dr. John Rehling (standard-bearer of Letter Spirit – Part 2). This version is now accessible on the web, for curious people to play with.
The brilliance of Anthropic’s Super Bowl Ad campaign spotlights what might be considered the core of humanity’s brilliance: creativity and nuanced communication.
Unless these ads were created 100% by AI with zero human involvement (even in the ideation phase), this is a moment to celebrate the (presumably-human) humor and pause for the deeper thoughts that they might just trigger.
In the end, do the “pretty people” of Madison Ave actually win the AI Era? The snarky comment one would hear way back when in Silicon Valley was always “when the pretty people (aka Creatives, agency types, ad people, etc.) got involved, the tech was toast.” Remember when Google was a search company and not an ad serving company? We do.
All that said, these ads are targeted at B2C and mass market—the consumer base that loves to lap up beige. Now why would that be? Well, the WHAT pile has a great deal of their thinking (and hopes and dreams and aspirations) encoded in it through text. That’s why AI will do really well with the great American consumer: because it reflects their (beige) worldview.
Another quick question, will we use ad tech to run our banks and nuclear power plants? Probably not. So MLsec is just as critical as it ever was.
We all know that WHAT machines like LLMs reflect the quality and security of everything in their WHAT pile (that is, their training set). We invent cutesy names like “hallucinate” to cover up being dangerously wrong. However, ignoring or soft pedaling risk is often not the best way forward. Real risk management is about understanding risk and adjusting strategy and tactics accordingly.
In order to do better risk management in MLsec, we need to understand what’s going on inside the network. Which nodes (and node groups) do what, what is the nature of representation inside the network, can we spot wrongness before it comes out? Better yet, can we compare networks and adjust networks from the inside before we adopt them?
These are the sorts of things that Starseer is looking into. At BIML we are bullish on this technical approach.
What happens when you organize a machine learning security conference together with a bunch of security experts who have widely varying degrees of machine learning experience? Fun and games!
The [un]prompted conference has a program committee reading like a who’s who of security, stretching from Bruce Schneier on one end to Halvar Flake on the other. BIML is proud and honored to have two people representing on the committee. (But we will say that we are legitimately surprised at how many people claim to have deep knowledge of machine learning security all lickety split like. Damn they must be fast readers.)
Ultimately all the experts had to slog through the 461 submissions, boiling the pile down to 25 or 30 actual talks. Did the law of averages descend in all its glory? Why yes, yes it did.
I have served on some impressive and diligent academic program committees over the decades (especially Usenix Security, NDSS, and Oakland). The [un]prompted approach is apparently more like Blackhat or DEFCON than that, with lots of inside baseball, big personalities, seemingly-arbitrary process, really smart people who actually do stuff, and much much more fun. And honestly the conference is going to be great—wide and deep and very real with a huge bias towards demos. ALL of the talks will be excellent.
I took it on myself to review everything submitted to my track (TRACK 1: Building Secure AI Systems) and also track 5 (TRACK 5: Strategy, Governance & Organizational Reality). Though I did get track 1 done (three times no less), I did not get through everything that came in during the deadline tidal wave. Lets just say A&A for Agents is over-subscribed and under-depth, prompt injection is the dead horse that still gets beaten, MCP and other operations fun at scale is the state of the practice, and wonky government types still like to talk about policy (wake me up when it’s over). If you want to see what’s next in building security in for ML, well it is only very slimly represented by two “lets get in there and see what the network is actually doing” proposals (one from Starseer and one from Realm labs). Yeah, submissions were “anonymous,” but everybody knows who is doing what at this end of the security field, so that’s just pretend.
Not only do we desperately need more whitebox work (leveraging the ideas behind transformer circuits you can find here), we also need to stop and think in MLsec. Where does recursive pollution (our #1 risk is BIML) fit in [un]prompted? Nowhere. How about model collapse? Nope. Data poisoning a la Carlini? Not even. Anything at all about data curation and cleaning (and its relationship to security)? Nah. Representation issues and security engineering? Well, there was one proposal about tokens…
Hats off to the outside–>in ops guys, they’re grabbing hold of the megaphone again! Just raw hacker sex appeal I guess.
Anyway, if you’re looking for a reason that BIML exists in all of our philosophical glory, it’s to peer as far into the MLsec future as possible. Somewhat ironically, we can do that by remembering the past. This [un]prompted experience feels so much like early software security (everyone was talking about buffer overflows in 1998 and penetration testing was an absolute wild west blast) that we can confidently predict MLsec is going to evolve from blackbox outside->in malicious input stuff, through intrusion detection, monitoring and sandboxing, eventually discovering that networks have lots of actual stuff you can try to make sense of inside the black box. Meanwhile the ops guys will paint a little number on each agentic ant, not thinking once about what the ant colony might be up to.
Do you remember when we decided to start looking at code to find bugs before it was even compiled? Because I do…it was my DARPA project. It will happen again. Not through static analysis…but through understanding just what the heck is going on INSIDE the networks we are building as fast as we can.
Pushing back on my flight from NYC to IAD, I caught one last headline before powering down the computer in my palm. This, from OpenAI:
Hum, “Education” or “OpenAI’s Education”... The headline felt worrisome given the total ‘fail’ experience I just had with ChatGPT, during a MoMa guided tour, the evening before, when I used it to augment my educational experience.
A masterful art expert, Agnes Berecz, had just led us through works of Helen Frankenthaler, Lee Krasner, Yente (Eugenia Crenovich), Louise Bourgeois, and Joan Mitchell.
Then we stopped to view this piece by Niki de Saint Phalle:
I was using the ChatGPT App as a companion during the tour because the human tour guide had mesmerizing knowledge of details of each artist, their style, inspiration, and wider impact on the art scene of the times. I wanted to hoover it all up.
I jotted notes on my phone and also shot queries into ChatGPT, to further dig for nuggets that could add to my knowledge.
Now, we all know ChatGPT can ‘get it wrong’, but it’s all too delightful to not lean on it and expect rightness.
I fed the photo into ChatGPT (in part, to log the artist’s name and spelling correctly).
What unfolded was a shocking reminder that no matter how spectacularly confident outputs read, you must keep your critical brain switched ‘on’!
ChatGPT responded with crediting the artwork to Robert Rauschenberg, not Niki de Saint Phalle.
Had it really just attributed a work hanging in the MoMa to the wrong Artist?
It had.
So I chose a prompt to suggest a sense-check.
Next, this output reponse:
For now and on so many levels, Agnes as teacher far exceeds the machine.
At BIML, we are pushing on the topic of Recursive Pollution as a very real thing.
If it plays out, the museum of the future may be full of Mona Lisas.
