gem
This is the Silver Bullet Security Podcast with BIML. I’m your host, Gary McGraw, CEO of the Berryville Institute of Machine Learning and author of Software Security. This podcast series is sponsored by BIML, a nonprofit science and technology organization whose research focuses on machine learning security. For more, see berryvilleiml.com/podcast.

This is the 161st in a series of interviews with security gurus, and I am really pleased to have today with me, Christoph Kern. Hi!

CHRISTOPH
Hello! How’s it going? Glad to be on the podcast.

gem
It’s great. Christoph Kern, widely known as X2F, is a principal software engineer in Google’s information security engineering organization, where he leads research and engineering initiatives focused on scalable principled approaches to software security.

A founding member and steering committee member of the IEEE Center for Secure Design, where I met him, Kern has played a central role in shifting the industry paradigm from reactive patch-based bug hunting towards secure-by-construction design principles that eliminate entire classes of defects at scale.

His influential research published across the ACM and IEEE focus on type safe abstraction, static analysis, and secure developer ecosystems that systematically prevent vulnerabilities like cross-site scripting and injection attacks across massive industrial code bases. Recently, his work has expanded into establishing structural security frameworks for AI agent architectures and modular software safety.

Which is… totally great. You did the exact same segueway that I did. So, Christoph, welcome to the show.

About a decade ago now, you helped fundamentally reshape web security with your seminal work on killing cross-site scripting at scale, moving the industry away from the endless reactive cycle of finding and patching individual vulnerabilities towards… a secure-by-construction thing.

By leveraging type safe APIs and hardened web frameworks and compile time guarantees, Google essentially proved that you can systematically eliminate an entire class of bugs before code even hits production. So looking back at the progress made since those early framework hardening days, a million trillion years ago, how has the security by design methodology evolved? And what are the major engineering hurdles teams still face when trying to enforce these hard structural guarantees across modern heterogeneous software attacks?

CHRISTOPH
Yeah, that’s a good question. I think it’s become perhaps more mainstream accepted. And one thing that was actually really nice to see, I think, is that a lot of the underlying philosophy of how to do this evolved separately and independently in the Rust community. I really like seeing the work there and seeing, oh, this is exactly how I think about the problem, which is really nice to have sort of independent cooperation that this is a practical path between the spectrum of full rigor of formal methods and just sort of software crafting without any rigor at all, right? And it seems to be kind of a sweet spot that gives you high confidence outcomes at very, very reasonable costs in practice for large scale practical software projects.

And since you were asking about challenges, I think where we’ve seen this work really, really well is for new projects, for greenfield projects that basically start on the secure by design stack, because then the extra work you need to do to stay within the guardrails of the secure stack is really very, very minimal, right? As a developer you hardly ever notice, ideally, if it’s designed well. And this is sort of part of the usability question of these stacks, like the developer friendliness and ergonomics aspects of these. And this is manifested in things like, you know, Rust has really nice error messages that tell you what to do, right? And we’ve done similar things in our tooling where, if you hit one of those compile time checks, you get an error message that tells you what to do next and how to get rid of it. And then once you’ve done it, you know how to do it. And then from then on, the friction is very, very minimal, right?

Where it gets tricky is if you have a very large code base that is very far away from this style, right? We’ve actually had some successes with existing code bases. So for instance, in the early days of the web security work, one of our first large early adopter programs was Gmail, which was a very large code base, a very large complex web application, probably one of the most complex ones you would find out there at that time, right? And we were actually able to refactor it in place to adopt these secure by design APIs with, considering the size of the project, a very reasonable amount of effort, right? It was on the order of a couple of suite years, maybe, which is a really small cost compared to the size of that project. So in that case, it was possible.

gem
That’s amazing. I know that DARPA was working on some Rust and so on projects too, because Dan Wallach is a friend of mine, so he’s always telling me about that.

CHRISTOPH
Yeah, exactly. Right. And so this was going to be my example of where this is much harder, right? Like we have, I don’t know how many, like millions of lines of C++ code, and getting that transferred to a secure by design platform with respect to memory safety in this case is much, much, much harder, right? And so the interesting thing, though, was that the Android team has made that transition for their code base over the years. And I think last year they announced that they have basically crossed the threshold where most new code is written in a safe language, even though there is still a lot of existing legacy code that’s written in C++. It doesn’t change much. And that has actually resulted in a disproportionate drop in the number of vulnerabilities that are being found, right? So there does seem to be sort of an effect that vulnerabilities have a half-life.

And if you don’t touch the code, you’re not introducing new ones. And so eventually, you’ll kind of shake them out, right? So you’re actually getting a lot of progress by just focusing on new code. But you still have residual risk in all that old code. And this is sort of, since we’re talking in the context of AI, this is where it gets interesting now, because we now actually have models that are capable of doing high quality rewrites of existing code in different languages, right? And so I think there is actually a unique chance that the cost of migrating brownfield code that is not very close to a secure by design style is becoming maybe much more palatable, or much more practical. So that’s a really exciting opportunity, I think.

gem
So I agree with you, but I think there’s some flaws that are still really hard, like, I don’t know, deep logic errors, broken access control, subtle time and state mistakes. They’re harder to constrain through language or framework design alone.

From your experience, what structural properties determine whether a bug class is amenable to being wiped out by architectural correction, and where do traditional type systems and frameworks sort of hit the wall?

CHRISTOPH
Yeah. So that’s a very good question, right? I think it kind of depends on how far you lean into it. An expressive type system like Rust can actually encode a remarkable degree of correctness invariance that you want to express. There was a really interesting talk by a colleague who was working on the Fuchsia network stack rewrite in Rust. And they really sort of leaned into this, using the type system to encode invariants. And I think I recall, it’s been a while since I’ve looked at this, but I think I recall that they were able to express a lock ordering discipline at the type system level, where basically taking out locks in the wrong order becomes a type error. And this relies on Rust linear types to actually express. And so they basically were able to statically eliminate at compilation time the possibility of deadlocks, right?

gem
I mean, time and state is always to me been the next big thing that we’re all going to have to deal with.

CHRISTOPH
Yeah. And so they found that when they finished the code, they were going to roll it out internally for team fooding. Right. They expected to get, like, tens or hundreds of bugs, because you’re writing a new TCP stack. There’s going to be a lot of problems. And they reported they had like three bugs.

gem
Oh my god.

CHRISTOPH
So I think there is a lot of scope for doing this, but you really have to think about how you structure.

gem
All you’re saying is that they rolled three D20s and got, you know, 60.

CHRISTOPH
Yeah. So there is, I think, really a pretty significant possibility. The other question is, you know, not only what can you do, but what can you do cost effectively, right? And so I think what’s been, in the past at least, without the benefit of highly capable automation through AI, the determining factor is whether or not the bug class is common to a large number of applications of a similar archetype, right, like a similar structure. And typically, this relates to bugs that are actually at the sort of underlying platform level. Like web application bugs apply to every web application irrespective of what it does, right? Like it doesn’t matter if it’s a banking app or a note-taking app or a photo app or whatever.

gem
Sure, like a system call in C, for example.

CHRISTOPH
Exactly, right? And the same applies to memory safety, right? That pretty much applies to any program. And so then if you make this investment in the platform, it essentially amortizes across all the applications of that archetype, right? So our work on secure web stacks got amortized across hundreds of web front ends. And so then it became very, very cheap. And of course, it only works if the sort of variable cost of it is also small.

gem
Well, I mean, you got to be able to shove stuff into types, right? So now we’re moving towards stuff like RAG, or it’s everywhere now. You know, so you got like search grounded AI systems. We’re seeing kind of untrusted external content fed directly into model contents. So what do you do in that situation? Like, how do we take lessons learned from securing traditional search architecture, say, and apply that to LLM context pipelines, and making sure that retrieved data actually is the kind of data that you want, with the right types, and never gets interpreted as a control instruction, for example?

CHRISTOPH
Yeah. So that’s, I think, a very, very difficult problem. And as far as I can tell, with the current architecture of LLMs, essentially unsolvable, right? Because there is no rigorous delineation between data and control in an LLM. In the end it just becomes one big vector of tokens, and you’re relying on the weight matrix of the model to kind of separate them, but it’s always going to be an imperfect level of assurance, right?

gem
Well, but then we get stupid things like filters and system prompt instructions and guardrail models. But from a secure by design perspective, those feel like the old days of trying to sanitize inputs instead of finding, you know, the underlying paradigm. So, but I think you’re getting somewhere with your architecture like CaMeL, for example.

CHRISTOPH
Exactly. Right. So I think, basically, for properties that we want to be assured with very high degrees of confidence, as far as I can see, the only way to do that is to essentially treat the LLM itself as a fallible component of a larger system, and then enforce the property through something that is implemented as classical, understandable, trustworthy code outside of the model, right? So this is where policy engines come in, and things like CaMeL, where if you want to track and reason about the provenance of certain data, you have to keep it in a separate context, and use a classical system, like, in the CaMeL example, a data flow aware execution engine for the code that is emitted by the main planning model, to actually keep track of, right?

gem
Right. But I mean, honestly, it’s just the separation of control and data, and back in the Center for Secure Design days, we had a lot to say about that. Here we go again. It’s like back to the future.

So let’s kind of push on that. You know, when things get agentic, it gets even more complicated, because right now developers can wire LLMs directly into APIs and code interpreters and file systems. We have a way of handling that. But when you look at designing tool abstractions for agentic systems to use, how do we construct capability-based kind of security boundaries around an agent so that, even if its internal reasoning is hijacked by, say, some adversarial input, it still has this underlying runtime environment limitation, and so it can’t execute really terrible stuff?

CHRISTOPH
Yeah. So, I mean, this is basically the crux of AI security and agent security, right? And I think the really underlying challenge is in finding the right trade-off between adequate bounds on risk and utility and usability, right?

gem
Well, the good news is humans are so good at risk management, I tell you. Okay, never mind. They’re bad at it.

CHRISTOPH
So this is, I think, really the trick. And I think actually, in some sense, a lot of folks who come from a pure security engineering background have sort of a tendency to not see that trade-off. And you hear a lot of people say, oh, agents should have…

gem
Some, I think academics for sure, but the people in commercial land, we got to build stuff that actually works and we have to ship. You know, the Visa card has to ship. People have to carry that around.

CHRISTOPH
Exactly. Right. And so I think, you know, you do hear a lot of security people say, oh, agents should have least privilege. Right. And then on the other hand, you see…

gem
You’re like, what does that mean?

CHRISTOPH
Agent people and, like, AI developers and users also kind of come from the exact opposite spectrum. They say, well, my agent…

gem
What they do is they’re like, okay. Least privilege sounds great. Let me just get OpenClaw to build that for me.

CHRISTOPH
Exactly. Right, like they basically come from the position, you know, I want my agent to do everything that I can possibly do. And of course, neither one of those is practical, right? Like you don’t really want your agent to, you know, buy a house or sell your house on your behalf without asking you, right? I mean, that’s like, or liquidate your 401k or things like that, right? Or even make a major purchase. I probably don’t want my agent to buy a car without talking to me. But at the same time, I don’t want to be asked every time my agent buys groceries for the week to approve that, right?

And so I think the key is to find the place in that spectrum where you have a reasonable bound on tail risk, and you also at the same point have a good amount of utility, right? So if you look at an agent that can make purchases, what I kind of want is an agent that can make small purchases that are, you know, rate limited and limited per transaction. So it can go buy my groceries automatically. And if I tell it, my toaster is broken, buy me a new one, it can go out and buy me a toaster and not ask me a lot of questions, but it can’t just go and buy a car. And that’s me, right? Like if you’re Elon Musk, maybe you’re okay with your agent buying an airplane, right?

gem
Then it’s a small country or maybe democracy. That’s for sale.

CHRISTOPH
Right. But I think there’s an interesting lesson there, is that what is an appropriate bound on risk is a very user specific consideration, right? So I think the system needs to be built where, as a user, I can tell it once, okay, this is what I’m comfortable with. And if you buy me the wrong thing, you know, once a month and I have to go back to the UPS store and return it, I’m okay with that, right? If most of the time it does the right thing and it saves me time, right? So I’m actually okay with a small risk of an incorrect action or an undesired action by the agent, as long as I’m very, very sure that there’s a hard bound on how bad it can be. Right.

gem
Okay, so we’ve barely scratched the surface, because agents that are by themselves, we still have a hard time reasoning about. Now scale it up from one of those to multi-agent swarms where specialized agents pass contexts and delegate tasks and trigger actions and have little societies and trick each other and use each other’s privilege and sleep with each other’s agentic spouses and stuff like that.

So if you evaluate a single autonomous agent, its capabilities or failure modes might sort of seem bounded to those of us who are not supreme formal methodists, but much like inspecting an individual ant, that’s why it’s possible. But when you deploy a colony of tens of thousands of these things, you get emergent behavior that no engineer programmed into a single prompt or some sort of system instruction. So how do we shift our security architectures to govern those things, agentic swarms, where the primary risk is no longer just one compromised screw up or node, but the emergent dynamics of the whole dang colony?

CHRISTOPH
I don’t know if I have an answer to this question, right?

gem
Come on, this is the fun question.

CHRISTOPH
Yeah, this is the fun question. I mean, one thing that does come to mind is that at this point, or at that level, I think the fundamental considerations aren’t even technical to start with, right? I think they’re more sort of societal. I mean, basically, you know, this is the same thing if you have a country full of people that has emergent behavior, and societies have evolved mechanisms of governance and things like morals and ethics to kind of govern that in a reasonable way. And it kind of worked most of the time, sort of, right?

gem
Well, and we have some sciences for studying those same things when we don’t understand them, like ecology. You know, we can go to a new zone on the planet and figure some stuff out scientifically. Maybe some of those methods are things that we can use.

CHRISTOPH
Yeah. And then once you figure out at a high level what that governance looks like, then you have to build the mechanisms to actually implement it, right? And so, you know, maybe that takes the form of delegation mechanisms for capabilities or privileges, or perhaps, you know, if the other agent is running in a different runtime, but it is actually running on top of a trustworthy platform that can certify or vouch for bounds that are placed on the behavior of that particular agent, maybe then, as a user of that, that’s enough for me, and I don’t actually need to know the sort of intellectual property that went into the agent and the prompt and whatnot. Right?

gem
Maybe. I mean, I sort of worry about atomicity and a thing like that. I mean, we worry about time and state where we understand the atoms and they have types. But imagine the atoms are so small we can’t really say what happens when you assemble them in new ways. Yeah, that’s the challenge. All right, so… All right. That was a little too much.

CHRISTOPH
Yeah, that’s above my pay grade.

gem
But let’s back it back a little… sort of. So as these agentic swarms and self-organizing computational ecologies take on more of the routine software lifecycle, it frees up human engineers to focus on highest levels of design. So probabilistic engines are really good at sweeping through a vast pattern space. But defining true architectural elegance and anticipating human context and inventing entirely new security paradigms still requires something fundamentally distinct.

So when you look at how security engineering evolves alongside of these autonomous systems, how do these tools amplify the uniquely human thing that we bring, our capacity for insight or architectural intuition or creative ingenuity, to build ecologies that are more resilient than anything we would have constructed by hand?

CHRISTOPH
Yeah, so that is a very good question, right? I think, before we can actually answer that question, I think we need to be able to answer the question of how the human observer and accountable person for the system that’s being built can actually exercise this accountability in a meaningful way when the code, or almost all the code, or all the code, is written by an agent and really no longer consumable by humans because it’s just the sheer volume of it, right?

And so one interesting way of looking at it, which kind of comes out of the way I think about systems through platform and developer ecosystem enforced invariance, is that essentially what you want, the outcome that you want, is high confidence, understandable bounds on the behavior of the system that came out of it. And an important, or maybe valuable, thought there is that not all bounds are equally important and need to be assured at the same level.

So if I have a system, I kind of want to be very, very sure that it doesn’t have misbehaviors that would be very, very bad in impact and outcome, right? Like security bugs or catastrophic data loss defects and things like that. On the other hand, if that system has a behavior that’s prescribed in the typical sort of type of spec where you have a user journey that says, you know, I fill out this form and I click a button and then something appears in the database, if the system doesn’t always do this, and maybe it doesn’t do it in certain obscure circumstances that are never reached in practice, and it fails in a safe fashion, it just, nothing happens, basically, that’s actually maybe an acceptable defect, right? Because we’ll never hit it in practice, and if it does happen frequently we’ll know, and if it doesn’t happen maybe we don’t care, right? And so if…

gem
Unless it’s an application of your brakes, but yeah.

CHRISTOPH
Yeah, and so exactly, right? So we basically need to identify… And the software equivalent is like, it doesn’t let one user access another user’s data, or it doesn’t crash and destroy your disk or your computer or whatever, right? And so I think if we can get to a way of building systems where these important invariants, that if they fail or if they are violated, it results in a defect that has a very high negative impact, if we can ensure those very, very confidently, but at the same time we have, you know, high confidence but not highest confidence assurance of the less important properties or less high stakes properties of the system, we’re actually maybe in a good place, right? And so this leads to sort of a…

gem
Yeah. And if… it frees up your creative juices as an architect to do some cool stuff. One would hope.

CHRISTOPH
Exactly, right. So then, you know, you can basically describe the system, describe what it wants, and you impose platform-insured invariants that really bad stuff doesn’t happen, you can maybe at that point actually trust both an implementing and a reviewing agent to do a very adequate job on building the thing, right? And then where you impose your judgment is on the architectural structure of the system, and you’ll want some way of actually expressing this rigorously, like with a rigorous expression of what can depend on what subcomponent and things like that, right, or what certain subcomponents can do.

gem
Sure. But you can also have a sense of aesthetics that, you know, you might not be able to formally describe, but you know when you see it that that’s a beautiful piece of architecture, and that’s the part that’s that ephemeral human stuff. So the good news is we’re still going to need really extremely good architects forever. I think.

CHRISTOPH
Yeah, so I think one thing I’ve actually been thinking about a lot is how can we encode, sort of, how can we ensure that the architecture of the system as built is legible and understandable to the architect, to the human? And we can be very sure that the system as built actually matches, or its architecture actually matches, the understanding that the human has based on whatever artifacts they see, right?

gem
Think about this. It’s kind of like growing a building instead of building it out of pieces. You know, more organic structure. We can reason about those and think about those, but maybe we might have to throw out our crystals.

CHRISTOPH
Yeah, that building analogy is interesting, right? So you may have a very high confidence understanding of the basic structure, right? Like where’s the load bearing pillars or whatever. And then you have like some organic goo that grows the walls, but they stick to the scaffold, and so you don’t really care exactly what they look like, right, because they’re going to be good enough.

gem
Right. Exactly. Stuff like that. And then you end up with, like, 1970s Planet of the Apes architecture, and everybody’s happy.

Okay, let’s wrap up by looking out over the next five years, which is an absolutely ridiculous exercise, but because this is a podcast, we have to do it. So we’re standing at this unique inflection point where the speed of software generation is skyrocketing, and yet our traditional defensive frameworks were built for a way slower manual era.

On one hand, we risk flooding the ecosystem with probabilistic code that resurrects long vanquished bug classes. On the other hand, we have a rare opportunity to embed secure by construction constraints directly into the generative substrate itself. So ensuring that systems are safe by default as they’re created is our goal. As you look forward, like, over the next five-year horizon, where do you see the most profound shift happening?

Are we going to be integrating rigorous structural invariance with AI-driven reasoning, finally allowing us to outpace the race of new vulnerability creation? Or is it going to be fundamentally rewriting everything by system of correctness? Or… dot dot dot. What do you think?

CHRISTOPH
Yeah, I mean, one thing I’m quite excited about is that it seems to me that when we use models, it’s very, very difficult to be sure that it won’t do something undesired or that it always will do the right thing. However, if we let the model produce something that can be mechanically verified with a high degree of confidence, then we’re in a good place, right? And it sort of does seem to be emerging.

gem
I mean, we’re in a good place, but we also decided Turing completeness is what we want all the time. And I don’t think we need to make that decision.

CHRISTOPH
Well, I mean, maybe we don’t, right? Like, I mean, so the interesting thing is that finding the solution might be very, very hard, and it might take a very, very capable model to do, and burn a lot of tokens on. But once it gives you a solution, checking that the solution is right is a mechanical thing, right? And we’re starting to see this.

gem
That’s old Necula and Lee come back at us, you know.

CHRISTOPH
No. Yeah, yeah, and I think this is a really promising direction, right? Like, I mean, basically, I think doing rigorously verified development used to be impossibly expensive, right? The verification was like orders of magnitude more expensive than just writing the code. But we are getting to the point. I mean, like, OpenAI just published that their model solved, like, a Millennium Prize problem, right? That’s extreme.

gem
Uh… Don’t go there.

CHRISTOPH
I mean, we’ll see where that lands. But I think this is a very good example of where the formalization of what is proved is relatively compact and should be understandable to experts. And then the proof is like insanely complicated and tedious. But if you trust your proof checker, if you trust Lean, you can trust this whole thing, which I think is a very plausible place to be, right, because…

gem
If you were watching video at this point, you’d see me going with my mouth, like, of course this is audio so you can’t see that.

CHRISTOPH
Yeah, I mean, I’m somewhat optimistic about this, right? Like, because I mean, you know, maybe there’s a flaw in the core theory behind the prover, but they’re so small and so well thought through by experts in the field that a discovery of a flaw in that would be like an earth-shattering discovery, right?

gem
Of course. I mean, it’s like choosing your right tools all over again and putting your invariance into the tools.

CHRISTOPH
And so I think it is perhaps not inconceivable that the use of rigorous machine check verification will become much more expansive, right? Like, it used to be basically what we had so far, because it could only be done by a very small number of experts and was insanely labor intensive, we would get this for, like, a microkernel like seL4, and they spent like 20 person years verifying it. And this is maybe something we can actually, at some point in the not so far future, afford to do for more of the core trusted components of our stacks, right? Like we might…

gem
I like it. So building better crystals to use.

CHRISTOPH
Yeah, exactly. Right. And then those things are what place very firm security boundaries of what is built on top of them, and is a firm foundation for what is built on top of it. And then maybe those things on top of it don’t need to actually be verified, right? Because that’s actually very hard, like verifying the property or, like, formally specifying the properties of a web-based application is really difficult, right, and maybe not worth it. But if you know…

gem
That’s because nobody knows what it’s actually supposed to do.

CHRISTOPH
Exactly. Right. But I mean, if we can have foundations, again, this comes back to, if we have the foundations that can give us very strong guarantees that large classes of well-understood, very bad behavior really are ruled out with a very high degree of confidence, and what’s on top of it is perhaps okay if it’s not absolutely perfect. Right.

gem
I love this. You’re not going to believe this, but we’ve been talking for about 35 minutes, so we have to stop. But I could talk to you forever about this stuff. It’s really, really very fascinating. Thanks so much for being on the show.

CHRISTOPH
It is. Yeah, thanks for having me on. It was a great discussion. Good questions. You’re, you’re… there were a couple that stumped me.

gem
Thank you. Well, I want you to sit at the highest point where you are and take a look over the mountain. The rest of us are too short to see over there. So thanks for helping.

This has been the Silver Bullet Security Podcast with BIML. Silver Bullet is sponsored by the Berryville Institute of Machine Learning, a nonprofit science and technology organization whose research focuses on machine learning security. You can find a permanent archive of all our episodes dating back to 2006. GaryMcGraw.com/technology/Silver-Bullet-Podcast. Show links, notes, and an online discussion can be found on the Silver Bullet webpage at berryvilleiml.com/podcast.

This is Gary McGraw.