gem
This is the Silver Bullet Security Podcast with BIML. I’m your host, Gary McGraw, CEO of the Berryville Institute of Machine Learning and author of Software Security. This podcast series is sponsored by BIML, a nonprofit science and technology organization whose research focuses on machine learning security. For more, see berryvilleiml.com/podcast.

This is the 160th in a series of interviews with security gurus and machine learning people, and I am pleased to have with me today somebody who’s both, Aaron Bedra. Hi, Aaron.

AARON
Hi, thanks for having me. I looked back — it was 2014 when we did this last time.

gem
Wow. Aaron Bedra is the CTO of Cumberland, a digital asset business of DRW, operating at the intersection of high-frequency trading, software engineering, and system security.

Over a distinguished technical career, he served as chief security officer, chief technology officer, chief scientist, and principal architect across healthcare, financial infrastructure, and retail e-commerce. A deeply influential figure in functional programming and language design, Bedra was a core contributor to Clojure and ClojureScript, and co-authored the definitive reference book, Programming Closure. Grounded in functional programming principles, language theory, and quantitative risk modeling, his work focuses on building high-assurance, resilient architectures that satisfy complex business and regulatory demands without compromising speed or system integrity.

And the thing that’s left out is that Aaron used to work at Cigital with me 25 million years ago, so we know each other well.

AARON
That’s right.

gem
And it’s great to have you on. Thanks for joining us.

AARON
Thanks for having me back again.

gem
Aaron, you’ve spent your career at the intersection of hardcore software engineering and executive security leadership, which is a hilarious gap to try to straddle. Over the last couple of years, the rise of generative AI and LLMs has completely reshaped the software landscape.

Looking at how rapidly these capabilities have evolved — what surprised you the most about where AI actually delivered versus the hype? And are enterprise security organizations adapting their core architectures properly, or are they still just scrambling to react?

AARON
A big loaded question to start. I love it. The thing that surprised me most was what you can do when you give people agency. There’s always been a struggle in building software about getting it right — listening to people, figuring out what they really mean, not giving people what they say they want, by giving people what they need. And there’s a whole career in that sentence, to do it right.

gem
Yep.

AARON
And what LLMs provided was a path for people to express that in executable form. That part was awesome. And it’s so powerful — I’ve seen so many amazing things happen, kind of just cutting out the back and forth and the time to, like, a good idea being a great idea in production.

gem
Right.

AARON
Now, on the other side of that, you know, the adaptation of this came so fast that it’s impossible for people to have changed their architectures, their systems of security, to respond, right? This was something you would have had to have started 10 years ago.

gem
Especially if you consider what happened maybe three months ago, with the step change in frontier models. I mean, it’s been remarkable how fast things have developed.

AARON
Yes. Yeah, it is, you know — I think February, March this year was a big one, and then even just a few months ago, another big one as well. This was something people we talked about years ago, of changing how we do architecture, how we design things. Now the LLMs are really great at pointing out where that’s not great and helping us find them faster.

gem
Right. Yeah. And, you know, with regard to organizations adapting — do you think they’re… most of them are scrambling? What’s your feel?

AARON
There’s a lot of scramble — people I talk to are certainly scrambling. But what I also think is happening, and I think it’s kind of interesting, is rather than scramble to bolt it on, I think this has been big enough that people realize it fundamentally has to change to be solved. So I think it’s becoming kind of a slingshot effect — you actually can’t just go fix it, you kind of have to redo it.

gem
Right.

AARON
So people are thinking past the incremental fix and thinking about the fundamental change that actually solves the problem.

gem
That’s cool. I suppose it’s complicated because every enterprise exec team is demanding faster AI integration, but security teams are like, whoa, whoa, whoa — new risks, some directly stemming from ML, and some, you know…

AARON
Yes.

gem
…because of use of ML. When you look at the actual front lines now — how are defenders successfully deploying AI inside their security ops to do security? How are attackers leveraging it on the other side? And are AI-driven attacks fundamentally novel, or are they just hyper-automated versions of the same stuff we’re already familiar with?

AARON
They are hyper-automated versions of the same stuff. But, you know, the interesting part is they are relentless. They don’t sleep, they don’t check, they don’t stop, they don’t take a break — they just keep going, as long as the token furnace can burn tokens, you can keep going. The other thing is, now that you have the ability to chain ideas together, there’s nothing really stopping them from stopping.

gem
Right.

gem
You know, what’s interesting is it reminds me of grinding in a video game.

AARON
Yeah. But they can grind harder, better, faster, longer, more successfully, and more multiplicatively.

gem
They like it — they like to grind.

AARON
Yeah.

gem
And, you know, the part that you’re just like, oh, God, I don’t want to do that — it gets done for you.

AARON
Yeah. They will leave no stone unturned. A human researcher has bias and a time commitment they have to spend or not spend, and an LLM has tokens, and as long as the tokens…

gem
Right.

AARON
Tokens are infinite — the stones, all the stones get turned over.

gem
Right. Really interesting way of looking at it, I think that’s right.

AARON
Yeah, all of the above apply. The things that stand out most to me — the most appropriate name for this group, Jack Jones talked about it many years ago: the non-malicious insider threat group. The ones with all the access who don’t mean any harm, but do cause oopsies — where oopsies could be lots of things.

You know, I do think there’s a lot to consider around the relationship between what goes into the LLMs and what the providers have access to. We’ve talked about that for a long time — there was a pretty famous interview recently with the CEO of Palantir about that. But I think there are lots of alternatives here, where you can kind of rent inference — where you’ve got the hyperscalers who offer the frontier models, with a sandbox or a way to kind of cordon data off.

There are things you can do to still use these really interesting models in safer ways, and so it’s not that no option exists, but it takes some effort to get right.

gem
Right. Yeah, and I think the effort part has to be done over and over as the models really evolve. When they go through a step function, you’ve got to change your position.

AARON
Yeah, that’s right — that’s very right. But to the part of people trusting the models, I think they trust models for different reasons. Some of it is, you know, they know the work product really well. They can validate the output — they’ve seen it, they know what it’s supposed to look like. That’s great, but they may not be as familiar with the software side of it. And so they might not understand that that thing that they… the button they clicked, or the script they ran, is doing something that’s suboptimal, or they didn’t intend it to. And so, where the understanding gap is, is where the risk tends to lie — whether that’s software folks doing more business-focused things without understanding, or business-focused folks doing more software stuff without the software understanding. Where you have the gap, you have the fundamental risk.

gem
Yeah, and in some sense that leads to what I’d like to call MacGyverism. You know, the thing works great — it’s built out of bailing twine and bubble gum, and it did what you needed it to do once, and you’re like, “Dude, I can use this all the time,” and then you’re like, “Wait a minute, that’s bubble gum.”

AARON
Yep. Yeah, that’s right — that’s where the high agency is so powerful, but you have to have a way to hand that off to a place where it can be turned into something a bit more real. We don’t have to go through the hardcore productionalization ideas that I think have existed for many years — I think there’s a looser form of that that could cover the 80% case and make things much simpler.

gem
Let’s dive deeper into machine learning security itself — the security of the system, as opposed to using it for security. Too much of the popular security discourse around AI gets stuck on surface-level issues like prompt injection or system information leakage, but real machine learning security is an entirely different discipline. How do we shift the conversation towards architectural risks — protecting model integrity, securing training data, defending against subtle data poisoning, watching out for model drift? Stuff like that, which is not in the press.

AARON
Yes. I mean, to an extent, a really smart person told me the answer to this question was, “I have no idea.” And I also say I have no idea to the answer to this question.

gem
But…

AARON
However, I think there are still some things we can think about and they start with the most fundamental concepts. This is things we’ve been talking about for years, right? Data cataloging, data labeling, classification, ownership — the things that we’ve talked about in data and enterprise data security for many, many years have now become one of the most important topics in machine learning, because garbage in, garbage out. Or poison in, poison out, in this case, in some cases as well.

gem
Well, and in some sense, it can get worse with machine learning, because it’s poison in, poison out, and then more poison — because the thing poured poison back into its own data ocean.

AARON
Yes — eating your own tail can be a really, really dangerous thing, as the…

gem
We think that’s kind of number one at BIML, but we’re having a hard time convincing everybody else that it is, even though it’s beginning to show up in the real world.

AARON
It is, it is. It’s a very, very difficult problem. And I do believe we are headed towards an “I told you so” moment in the future. Like you said, signs of it are already appearing. But it’s going to have to get a bit more catastrophic before, I think, people catch back up.

gem
Yeah — you heard it here, third or fifth or 3,300 millionth time: recursive pollution does in fact matter.

So, building on that kind of architectural perspective — the threat landscape shifts dramatically when we move from single AI models to agentic AI, and especially agentic AI swarms. If you analyze a single autonomous agent, its capabilities or failure modes might look manageable — kind of like looking at an individual ant with a tiny number painted on its back. But as we saw in the OpenAI/Hugging Face attacks, when dozens of specialized agents interact autonomously, emergent behaviors appear that no individual agent was explicitly programmed to perform. Now multiply that by 10,000, and we have a serious problem. How do we build an ecosystem approach to security that governs these swarms, where the risk isn’t just one compromised agent or five agents colluding, but the collective behavior of a colony of agents?

AARON
This is where we start thinking about the evolution of all of this. We started with prompts, then we got into planning, and then loops, and now I think graphs is like this week’s hot one. Swarms came around and then kind of faded, and now they’re back again. But you said the magic word: emergent properties.

gem
Yeah.

AARON
These are systems, and there’s a systems thinking — like the next thing is going to be this kind of emergent systems thing that evolves out of graphs. Graphs are interesting, but they don’t capture where I think we’re really headed here, which is systems.

gem
I totally agree. And I also think that we’re not so great at understanding our own emergent systems — like, say, oh, I don’t know, the stock market, or the world economy, or how the ecological zone of the planet actually works, you know, as a whole planet.

AARON
Yeah, when you have large pressure on a system, new things happen — things that you didn’t think were possible, or didn’t clock as probable, become more likely. I mean, even things like NetLogo have shown us what happens, what kind of emergent properties are possible. We’ve been able to model these ideas for many, many years.

gem
Yeah.

AARON
We should be thinking about all the research done there, and where this kind of thing is headed, and see what overlaps — I’ll bet you there’s a decent amount of overlap in those two ideas.

gem
I think so, too. We’re actually working on that at BIML, as you probably know, but it’s summer and there’s so much going on. And, you know, AI is continuing without us.

AARON
It is. It’s not gonna stop.

gem
So, as these models and agentic swarms — or whatever you want to call them — emergent properties of the systems become capable of doing stuff like generating code and automating routine tasks by externalizing state, that’s the thing. Over time, there’s a lot of loose talk about replacing human engineers and analysis with AI, but a statistical model is fundamentally a reflection of past data. When navigating unprecedented system shocks, novel threat vectors, or complicated architectural trade-offs, how do we preserve the human core? I like to call those the three I’s — insight, intuition, and ingenuity — that no probabilistic engine seems to replicate at the moment. Or do we just have to wait around a month or two?

AARON
It’s probably longer than a month or two, I think. Probably a year or two is a safer bet, maybe longer, but who knows. I certainly can’t predict the speed at which it evolves. It’s shocked me continually. But it’s probably more than a month or two. I mean, in my job day to day, we think a lot about predicting the future.

gem
I think so, too. I mean, I’m… I’m just trying to be silly. But we do — we’re like hanging on by the fingernails of insight, intuition, and ingenuity. How big are those fingernails?

AARON
They’re big. They’re really big.

gem
Good.

AARON
You know, for many years — I mean, my job day to day, without going into all the details — is part and parcel about trying to forecast what we think the probability of future events might look like, and outcomes. You know, the best Bayesian models, the best filters, the best intuition training still have lots of flaws. You know, like you said, you could only look at what you’ve had historically as input. And we do, you know, as a species, we’ve done okay with this, but we certainly haven’t cracked it. So, until I think these machines evolve more — if you think about the architecture and existence of what we know about LLMs today, I don’t think we get AGI out of that. I think we get something that can mimic intelligence, that can emulate it, but can actually reason about it, think about it, evolve in novel ways — I haven’t seen that emerge yet. And who knows, maybe somebody’s already got this and we haven’t seen it, but what I’ve had access to, you know, personally, I haven’t seen that capability yet.

gem
I totally agree. I mean, I was talking to Melanie Mitchell on the last episode of Silver Bullet — the previous one to this. And, you know, she and I both think the same thing you do: we’re not anywhere near AGI. And in some sense, it’s very hard to discriminate whether this is a simulation of the thing, or the thing, in a lot of cases. But maybe that’s the case with employees too — we’re not going to go there.

AARON
Yeah. I mean, it’s a great parrot. I mean, call me Blackbeard, I’m happy to sign up for the parrot. It’s quite good at repeating what you teach it.

gem
Yeah.

AARON
And quite useful at it, too. I just don’t think we’ve hit that moment where the intuition, and the ability to absorb all the patterns at once and make novel reflections of what exists in front of us, is real.

gem
Right. So organizations are trying to figure out how to adapt all this stuff in real time. They’re building policies, setting guardrails, evaluating new security capabilities, thinking about what they’ve got to do versus what the vendors have got to do. But heavy-handed security bans can drive usage underground.

AARON
Yes, that’s right.

gem
You know, and for a long time, people talked about shadow AI. I know it’s a little boring to talk about that now, but guess what’s happening now? So, how do you construct a security posture that doesn’t act as sort of a brick wall, but actually encourages safe experimentation and empowers teams to build stuff responsibly?

AARON
It’s all about the harness. I guess it’s probably a cliché word at this point, but it really is about the harness. It’s the sandbox. What environment are you letting these tools run in? Whether it’s at the fringe, where you’re doing that data inference renting — whether that’s through hyperscalers, or the hyperscalers is your own local models — that’s probably your best outward line. But even inside of your world, are you running in the “don’t ask me anything about it, just do it” YOLO mode? Or, if you’re doing that, are you doing it in well-contained sandboxes?

gem
Right.

AARON
Do you have your data well classified and labeled? But really, visibility, I think, probably more than anything, is the ability to alert, to flag things that are happening, and then react to them, and do more point-in-time remediation of those things as they come up. But you have to, at some point, limit the blast radius of what something can do when something goes wrong.

gem
Right, so, you know, it’s kind of a balance. So when you look at people that are getting it right today, what separates the organizations that are really doing it from the ones that are struggling? Is it better to do internal tooling and clear data classification — like you’re saying, embedding security directly into developer workflows — or to change how leadership views software risk? I mean, this reminds me so much of software security — it’s just like a continuation of the battle we’ve been fighting, you and I, personally, for 30 years.

AARON
You know, it is, but I also have a little bit of optimism on this.

gem
Good.

AARON
Because we’ve also seen LLMs be capable of producing pretty good, like, built-in security measures too, right? A lot of times security is harder, it takes a little longer, you have to spend some extra time, there’s other things that are more important. When you don’t need to spend the time, because it can be produced for you when you have the tools that can emit the things that are supposed to happen by default. It’s all like water, it’ll take whatever path is the least resistant. So if you can put LLMs early in, you get a better result.

gem
But, see — yeah, software security was like that too, though. Remember when we could say, “here’s the architectural blobs that I need you to use, and by the way, here’s some cryptographic stuff for you to use, and don’t roll your own”? I mean, we did that before. And so that’s what I mean by continuation, I suppose.

AARON
We did.

gem
I wasn’t being negative about it. Believe it or not, I think we made real progress in software security.

AARON
Yes, yes, very much so. It was night and day different. And we can put that into the LLMs and let them do better things for us, right? It actually can become (unclear).

gem
Right, but what we need them to do — yeah, we still need to provide the intuition, and we still need to provide the architectural kung fu that nobody can seem to write down, even though we’ve tried it.

AARON
Yeah, the boxes and arrows are hard.

gem
Yeah, and the boxes and arrows — you know, back in the old days, degenerated to UML, for God’s sake. So we can’t do that, I don’t think. Interesting times. We’ll see how long it takes for software architecture, to fall — not to fall, to be picked up. It already fell, but by machine learning.

Let’s close this by looking way down the road. So, imagine five years — like, you know, five years ago we barely had, what, AlphaGo, and reinforcement learning coming down the pike? That was maybe six years ago, but pretty close — it seems like infinity when you think about AI. So let’s try to project that far. What will the enterprise security landscape look like? We know new risks are going to come out of fundamental problems, and they’re going to be hard. But where do you see the biggest reasons for, say, this optimism that we’ve been talking about in the last question? Like, what longstanding security challenges might we maybe solve if we get this right?

AARON
The one that comes to mind that stands out more than anything else is visibility. Visibility has been so hard — and part of it is proper inventory, knowing what’s there, what’s not there, where things are — just knowing what you even have. The best tools in the world still fail at this. If you have a relentless machine that will scour everything forever and never give up, you’ve got a better chance of looking through all your code bases, your network, your systems, your tools, and putting it all together and synthesizing the right information. So once you have visibility, the other thing is prioritization — what are the things you’re supposed to tackle first? And not just from a risk register, or from somebody’s kind of thumb in the wind — “I think this is what we should focus on” — but, “oh, no, I checked this thing, it actually is vulnerable, all right, here’s the proof, I already did it,” or “here’s how I can verify this risk,” not “I think it’s bad.”

gem
I want to push on that a little bit. So, imagine that we’re good at going down — we can get to the molecular level of security, we can watch all the stuff, do the grinding. But we’re not equally getting good at going up. And so it’s that kind of fusion, integration, architecture that I think we’re not prepared for — because I do think that we can go down to the molecular level and make a lot of progress, but I also think that we’re not very good at looking at a rainforest on a planet and figuring out why it’s screwed up, from an ecology perspective.

AARON
Yes, yes, that’s right.

gem
If we can’t even do that, like, how the heck are we going to do this thing where it’s all virtual and a bunch of agents in a field we don’t really understand?

AARON
We also have a high cost of change problem. Change is very expensive — right, you found the problem, now you need to do something about it, but you have to have a change control committee, a sign-off process, a time you can do it, a rollback strategy — all these processes that were in place to prevent rapid change, or protect from rapid change, are now tossed into a world where rapid change is what you’re supposed to do every day.

gem
Yeah. I mean, it’s kind of like middle management is an inertia machine.

AARON
Yeah.

gem
If the inertia is slowing, then they’re gonna slow you down some, but if it’s going too fast, they’re going to make you go too fast also, some.

AARON
Yes.

gem
It’s…

AARON
Yeah, the bureaucracy of ITSEC has to evolve with what we are doing.

You have to have controls, you have to have things that can provide a safety net and provide the right guidance. But I don’t think it’s checklists and humans and orthogonal systems — I think it’s something much more deeply embedded into the systems themselves.

gem
Yeah, okay. So, I don’t think we’ve got five years — I think we’ve got like maybe one year. So, let’s look…

AARON
Yes.

gem
Let’s look five years. Imagine — why do we need programming languages? Why does a machine learning system need that representation? Why do we need software at all? Why can’t we just fab hardware to do the thing, and only the thing that we wanted to do — and then, of course, we’ll get sloppy about the hardware, so that’s… it’s a weird way to approach it, but what do you think about that, like, re-representing the way we do computation, so that we take advantage of these models that work differently than Turing machines?

AARON
Yeah. It’s a great question. I mean, fab — fab is expensive, and long. So I don’t know if fab is the… unless we can fix the physical, you know, moving atoms, not bits.

gem
Well, the more we fix it, though, the sloppier we’re going to get.

AARON
That’s true, that’s true, yes. But it’s a great question — like, do we need programming languages?

gem
Right.

AARON
Or can we write machine code? Or can we write something closer to machine code? Are we actually on an arc back to assembly, or something like that, instead of, you know, Python and other languages? Like — this is for humans, this is for machines. We do all this stuff for humans to get to the machines.

gem
Yeah, yeah.

AARON
I think there’s a very valid point that you need less of it going forward. I think there will be lots of signs on whether or not this is a good idea or a bad idea, but I’ve already seen really significant strides towards moving back to C and C++ — back to more machine-friendly, or sympathetic, languages. It’s a matter of time before what we’re really doing is coding rules, proofs, verification systems — things that can govern what the production of code is, versus the code itself.

gem
Well, and then there’s the code looking at the code — pile, like, you and I just think of that naturally. There’s the code, I understand this system. But some people — the API to the code is now, you know, natural language.

AARON
Yes.

gem
And, as we know, you can’t specify stuff in natural language. Like, English is notoriously awful at that.

AARON
Yes. But can we specify things as finite state machines?

gem
So we… yeah, maybe.

AARON
Can we specify them as, like, closed systems? I have formal verification.

gem
Yeah.

AARON
A formal language is awful at this — you don’t want Turing completeness for this, what you want is a closed loop that you can verify.

gem
Yeah. Yeah, yeah. So, it’s kind of a weird time, because we’re bifurcating — we’re going both directions at once. And that’s what makes thinking about five years from now so hard, but also so fun.

AARON
Agreed. Yeah, but I think that’s what humans are still in control of — what are the rules of the system, and how can we express that in an ML-sympathetic way, such that you can provide a verification mechanism to the ML that will net the right result? Even if the machine keeps spinning on something, the thing it produces has some verifiable outcome.

gem
Right, right. Cool. Well, I sort of share your optimism. Thanks for joining us.

AARON
Thanks for having me.

gem
This has been the Silver Bullet Security Podcast with BIML. Silver Bullet is sponsored by the Berryville Institute of Machine Learning, a nonprofit science and technology organization whose research focuses on machine learning security. You can find a permanent archive of all of our episodes, dating back to 2006, at garymcgraw.com/technology/silver-bullet-podcast. Show links, notes, and an online discussion can be found on the Silver Bullet webpage at berryvilleiml.com/podcast. This is Gary McGraw.