Author: Silver Bullet

gem
This is a Silver Bullet Security Podcast with BIML. I’m your host, Gary McGraw, CEO of the Berryville Institute of Machine Learning and author of Software Security. This podcast series is sponsored by BIML, a nonprofit science and technology organization whose research focuses on machine learning security. For more, see berryvilleiml.com/podcast.

This is the 156th in a series of interviews with security gurus, and I’m pleased to have with me today my old friend, Phil Venables. Hi, Phil.

PHIL.
Hey, good to be here.

gem
Phil Venables is a venture partner at Ballistic Ventures and a senior advisor at Google Cloud, where he previously served as the inaugural CISO. Over a distinguished 30-year career, Phil has held foundational leadership roles, including 17 years as the first CISO of Goldman Sachs and subsequent positions as their CRO, Chief Operational Risk Officer, and the Director on the board. A globally recognized authority on information security and risk, Phil co-founded the Center for Internet Security, CIS, and has advised world leaders as a member of the President’s Council of Advisors on Science & Technology. His current work focuses on early stage cybersecurity investments and the architectural intersection of AI, resilience, and business modernization. He holds an MSc in computation from Oxford University and a BSc in computer science from the University of York.

Thanks for joining us today.

PHIL
Yeah, it’s a pleasure.

gem
Phil, you’ve had one of the longest running tenures at the top of the security field. Starting out as CISO at Goldman Sachs, I think you were in your mid-20s. Through your path-defining role as architect of systemic resilience at Google Cloud at age, well, never mind. How has being a CISO itself evolved? And what have we learned about trust and technology? And did that stuff we wrote down in the CISO report, that’s still true?

PHIL
You know, it’s kind of interesting because I think it’s to some degree the CISO role has kind of come full circle. So like when I when I–if I’m being honest–got dragged into being a CISO many, many years ago, my background was software engineering and I just happened to stumble into building secure software, building various things, and then after that stayed in the security team and inevitably at that stage, and you know, this was in the 1990s. There were some products you could buy, but a lot of the stuff you had to build yourself. And even if you didn’t build it, the stuff you bought needed so much like wiring together that pretty much the whole security team was essentially an engineering team.

And then as things matured, you got into things being more regulated. There was things like Sarbanes-Oxley, and all of these other regulations in the 2000s. And the security teams kind of morphed into being more of a risk and compliance team. And then with the presence of nation-state threats, in the mid-to-late 2000s and then beyond that and criminal activity, teams evolved into being kind of government national security-oriented teams and you got these kind of new waves of leaders coming out of security services and law enforcement.

And then in the past few years, though, we’ve all learned that one of the the right ways to do security is to kind of build it into the platforms and the engineering frameworks and to provide solutions and not just throw policies at people.

gem
Gosh, that sounds like software security.

PHIL
Well, teams today are this mix of compliance, risk, security engineering, operations threat, the whole thing, the character of, I think, some of the best security teams out there, even in like Fortune 50 major businesses, look and feel more like engineering teams did in the past than they did the primarily kind of risk and compliance, or even though they do risk and compliance. So I think it’s come full circle, and I don’t think… not necessarily the best security leaders have always got engineering backgrounds, but they many do, and even the ones that don’t now take a very engineering solutions oriented approach to to make the secure path the easier path by delivering things. And I think in that respect, it’s come full circle. It’s kind of fascinating.

gem
Oh, that makes a lot of sense, but I think we’re in for a world of change. We’ll talk about that. In 2008, which was a long time ago, you and I wrote about developing an internal market inside of an organization in order to manage the hidden costs of software maintenance, or let’s just call it technical debt.

gem
Now in 2026, we’re seeing AI development move at a velocity that a traditional kind of TCO model might have trouble tracking. When an agent can spin up, say, thousands of microservices in an afternoon, how do we apply that sort of economic rigor? And, you know, are we automating technical debt creation? How do we handle that?

PHIL
Well, I think for many organizations I’ve seen, you know, the naively adopting models to generate software, you certainly see that being a big factory for technical debt and architectural debt and all sorts of other debt.

However, as as they say, the future is already here, it’s just unevenly distributed. You see some organizations, and it’s not just the big tech companies or the foundation labs, it’s a number of other companies that have quickly adapted to make use of agent driven software development and have adapted it to make sure it’s producing things according to their technical standards, designs, coding practices and are making substantial progress and actually using that to refactor existing code bases and pay down technical debt.

So I think there’s there’s there’s currently kind of a world of haves and have nots and I think ultimately everybody gets to the haves status. It’s just a question of time.

But one of the things that is interesting is, again, we talk about a world where there’s going to be 100x, 1000x, maybe even more software than we’ve had before. And that’s not necessarily a bad thing, but it does point to the fact that organizations have got to have that basis software control, control of their software production pipelines, testing pipelines. There was an interesting Google Developer Operations research report a few months ago about this kind of impact. And you know the conclusion is totally unsurprising, but it’s still a useful conclusion anyway, that if you if you sprinkle AI onto a chaotic software development environment, you’re going to get chaos amplified. If you sprinkle it into a well-maintained software development pipeline, you’re going to get productivity amplified. And I think a lot of organizations are just realizing now that their underinvestment in that base level of software control is hampering their ability to get advantage out of AI without introducing chaos. And so I think there’s a lot of tone in the industry as you see that about kind of using AI to manage AI. Well, you know, kind of a lot of what we need to do with AI is just implement the foundational controls and the baseline controls and reliability and software management practices organizations should have done years ago. You’ve really got to do that now and it’s to get the best out of AI or at least to stop AI introducing so much chaos.

gem
Yeah, I agree with that, but I still wonder where, say, software architects come from, which we’ll get to in a minute.

We’re moving away from… the problem of, say, developers with compilers being inside of our network towards this weird Hofstadarian strange looper agents, which are tools, autonomously build and test and deploy their own specialized tools. When the tool user and the tool become the same part of some sort of recursive process, where does the security checkpoint live? Can humans, like, stay in the loop when the the loop is crashing over itself so quickly?

PHIL
Yeah, I mean, definitely…. sso you know, we’re in this kind of, this whole space at the moment requires you to keep two conflicting ideas in your head at the same time. And so we’re definitely in that world where the production of code from AI requires code review. But at the same time, there’s so much code, the existing code review practices just cannot scale and never hope to scale.

And so you take a step from back from those two conflicting things is there probably still is always going to be some absolutely centrally critical code that will always need human eyes on it.

What percentage of an organization that is, but you can imagine like the core code on cryptographic signing of trillion dollar payments in a bank probably is going to need some human review constantly as would be the ability to apply root signing keys to all of your software. So there’s elements that are always going to need that. Then the rest of it, I think the agents, verifying the output of agents as long as there’s sufficient independence of that, it could actually have a lot of utilities.

So I think there’s there’s still a lot to be done.

The other thing I think as well, and this kind of always reminds me, and you know we’re we’re going to start dating ourselves pretty quickly. This reminds me of the early days of the internet when we had web servers, web browsers, firewalls, databases, but nobody had really got consistency of how all this stuff hangs together. And the real gotchas were not in the components, they were in the arrangements and design patterns of how you built it all. And I think we’re at the same stage now with agents and AI in general that there’s so many different potential design patterns of how you do this that nobody’s really figured this out.

So again, we we talk about agents, but we should also be talking about skills. So there’s all of these markdown files full of skills that agents use that need management and checking. And then there’s all of the the rag input to the models for the agents to do things. And then there’s all of the context memory that’s built up and needs managing and pruning occasionally. So we’ve got this loose flying collection of agents, skills, memories, models, and those act as this complex system. And we’ve not quite nailed down what’s the design pattern for managing all of that. And at the moment, we just say things like, “We should do agent security” and everybody goes, “Yes, we should do agent security!”

But then nobody really knows what that means. And you have to drop down a level of abstraction and say, okay, this means we’re going to have to manage the integrity of skills documents, going to have to manage the memory context. We’re going to have to figure out local and global memory. We’re going to figure out after how the agents interact with ?. So there’s just so much still to do. And that’s where all the devil is going to be in those details.

gem
Yep. As we watch the rise of machine learning security, the industry seems over focused on AI _for_ security, using AI to _do_ security, to the detriment of security _of_ AI. And I wonder how you yourself define machine learning security as an engineering discipline. Do we treat the model as a black box? Do we get inside it? Can we change its architecture? You know, can we control training sets or are we done with that?

PHIL
Well, again, this is another one of those spaces where the conversation about what is security of AI quickly over-fixates on micro exploits and threats. So we quickly get into talking about, you know, these are good things to talk about and manage like model poisoning, tool injection, all of those things are important risks to manage. But we’re going to have thousands of those attack techniques and threat vectors over many years. And as we’ve learned from our past in software security and other security, at some point, and this is the opposite of what I just said before, in this space, you’ve got to abstract up and figure out what’s classes of control that can mitigate all of those things.

Certainly my prior experience at Google and now, when you step back and you look at what does it take to control and govern the security and safety of AI, it really comes down to good old-fashioned software lifecycle risk. At the end of the day, this is a bunch of software, and so you’ve got to manage the integrity of the software supply chain, the integrity of the software, how it hangs together, all the traditional good stuff we know we need to do. And quite frankly, many organizations don’t currently do a great job of.

Secondly, it becomes a big data governance problem. The the training data, the fine tuning data, the model weights, the parameters, the test data that constantly keeps a model in shape. Again, that’s another space where very few organizations have well-developed data governance and data lineage management practices.

There’s a third pillar, which is managing the operational risk. So what’s the guardrails, the circuit breakers, the checks that that put some deterministic controls on what are non-deterministic things.

And then you’ve got this base layer of identity and access. So what are the applications and the agents, what are they permission to do? What privileges do they have? How do we think about their identity? And then to your point, this broader problem of testing and validation. So in as you know in our traditional world of software engineering, you test at you’ve done a change, and then you don’t really test constantly in production. You test on change, whereas with AI systems, because of drift and other factors, you just have to test continuously. And so if you do those five things, software lifecycle, data of governance, operational risk, identity and access, and testing and validation, all the things that emanate from that are the things that mitigate all of the micro issues. And I think focusing on that is is the right thing.

gem
So when we built the BSIMM in 2009, which you really helped with, we were careful to define it as a descriptive model, sort of reflecting on what the world’s best security teams were actually doing rather than telling what they should do. I have a couple of questions around that. So first of all, who is actually doing machine learning security? Are there groups for that? Like, is there an AI equivalent of the SSG? And second, is there enough stable data from people that are actually getting this done today effectively to build an AI BSIMM or is it too early?

PHIL
First things first, so I think you know there are a number of pretty good organizations. So at at Google, we created this thing called the Secure AI Framework, which went from principles to actually some decent level of detail. And then we handed that over to ah an open foundation to create the Coalition for Secure AI. There’s other things happening in the ML Commons that are at more detailed level. There’s the Frontier Model Forum, where the the foundation labs collaborate on controls around protecting the high-end models.

And then there’s some other groups. So there’s a company called the AI Underwriting Company that have been doing work with a bunch of us to create a standard called AIUC1, which is a specification of controls that can be used by audit firms to audit in a SOC2 framework. And so and there’s many other things… the Cloud Security Alliance have done some pretty amazing work. And there’s another thing called AARM.dev that is like an agent action open standard. So there’s loads of things that have gone from these kind of abstract into specific controls.

And there are, to your last question, there are some organizations that are doing well to implement that. And there are also quite a lot of tech companies and cloud providers are baking these things into their platforms that they’re delivering. But there’s still a ways to go. And I think we’re probably on the the cusp of being able to do an AI SIMM and get some early indication of which companies are doing well because of what reasons. So I think it’s it’s probably something worth BIML could be a useful useful venue for doing that.

gem
Maybe. I was really thinking less about these organizations that are umbrella organizations and more about, you know, is the AI risk control group inside the CISO organization in an enterprise? Or I know you’re thinking much bigger, like let’s just take a bank, for example. Is there one place where ai security lives or is it just all over the place?

PHIL
Well, so that the the banks are a good example, in my experience, of those of those companies that are doing it well because they’ve they’ve got this established structure for doing it because of what they’ve done on algorithms and the use of machine learning in the past, yeah they they have a security team that’s well aware of this. They have a compliance team that’s quite sophisticated. Their lawyers are quite sophisticated in these things. And they have these model risk validation teams that are very well equipped to validate and test models, and even the later, more generative based models.

So I think they’re in good shape, but it doesn’t live in one place. It’s a collection of teams that are part of the process of validating and vetting.

Now, when you get outside of banks, it’s rare to have that degree of sophistication and structure. And to the point of your question, the CISOs are having to pick up this. And so CISOs are becoming more like chief digital risk officers in a world of AI because the board and the rest of the C-suite is turning to the CISO. So I’ve seen this situation a few times is the CISO will be sat in a board meeting and the CISO is presenting, here’s my AI for security work and security for AI work. And the board goes, that’s great. What about safety, bias, compliance, privacy, industry? And the CISO goes like, what me and not the legal people or the compliance people. And they go, no, no…

gem
As long as you’re holding the bag, hold all the bags.

PHIL
Yeah, and so you know it’s you know it’s a great opportunity for CISOs to actually kind of take that broader business risk manual because they’re being seen as the places where that central function should evolve. Some CISOs are grasping that and doing it amazingly. Others are kind of pushing back on it. And so you know it might be one of those moments where CISOs need to embrace the desire of their organization to have more ownership over the entire risk set.

gem
Right, right, right. Yeah, I mean, you’ve always been way ahead of the curve. So I think the curve is just catching up to you on that one. In light of agentic AI, which we’ve been talking about mostly, do we reassert the essential nature of the architect, like the software architect, information architect, security architect, and so on in building systems? Because in my view, architects can envision the right sorts of harnesses for AI and play the role of kind of whiteboard overseer. And in some sense, the there’s a lack of, if you don’t have that architectural rigor already in place, it’s difficult to handle the architectural requirements of agentic AI properly.

And so I’m concerned about enterprise architects, you know, that are coming from sales, say, a sales guy who says, I want to build this thing this way. And the AI does it, but anybody from technology who looked at the architecture would go, oh come on, we can’t do that.

PHIL
And you certainly see some of that. So organizations that have appropriately encouraged vibe coding in their user base, have a process by which some of the systems that emerge from that get re-architected to be production ready.

gem
Right.

PHIL
Many of these vibe-coded things just stay as like end-user applications, just like advanced macros in Excel or scripts in a Databricks data warehouse.

gem
Right. Don’t tell that to the trading floor guys.

PHIL
That’s right. But like even in those environments historically where you did get user-developed apps, those that emerged to be critical got refactored into being a real production application.

The interesting part of that question, though, is that, you know because I actually don’t know how this should turn out. So when you look at some of these super-developers, super-software engineers that have like basically just not writing code anymore, they’re orchestrating agents to write code. They’ve gone from being software engineer to being software architect almost by definition.

gem
Yes. Yes.

PHIL
So the real question is not so much whether we have enough software architects, but rather whether we’re equipping our software engineers to actually understand and drive architecture and whether the models themselves have enough understanding of architecture so that they’re not just generating code, they’re generating architecturally good sets of code. And I think that’s and that’s a very good open question.

gem
Yeah, I think so too. I think that it actually is very hopeful for people that are software professionals. We never really did understand where architects come from, but they’re going to be even more important and we need lots more of them now.

So you’ve recently written about the second order risks of trillions of interacting agents. This feels like the moment to revisit Stephanie Forrest’s DARPA work on computer immunology. In the world of agentic AI swarms where behavior can be emergent rather than designed, are we moving towards a security model that looks less like a set of well-defined rules and policies, which we’ve been talking about, and more like a biological immune system? And, you know, how do we build self-tolerance into these agents so that we don’t have an autoimmune reaction inside of our own systems?

PHIL
Yeah, I mean, I do think, and I’ve long thought that an immune system analogy to cyber is probably one of the better analogies.

And certainly one of the things when I was at Google Cloud, we used to talk about the cloud being a digital immune system in the sense that if you can early enough detect an attack in one place and then analyze and propagate controls to defend against that faster than the attackers can get to other parts of the environment, then you have in effect got this loop that creates this immune system effect.

gem
Yeah.

PHIL
I think in a world of trillions of agents, either broadly or in your own environment, having the ability to constantly update and refine that in response to detected problems in part of the environment, it makes that a necessity, not just a nice-to-have.

gem
It’s sort of loop kind of, if you think about it.

PHIL
Clearly, No, no, exactly. But the yeah the real thing I worry about, I mean, it’s the reason I kind of wrote about these second order effects things is in every wave of technology change, we get fixated on the risks in front of us, which is, you know, all you really can do.

But the really interesting risks come as a second order effect. So like when smartphones happened in 2007 and 2008, we were worried about all sorts of risks. We weren’t worried about the risks of social media platforms because they didn’t really come until they were enabled by phone. We weren’t really worried about all sorts of things that smartphones gave us. And the same thing is going to be here in AI. So again, in that world of billions of agents, all of which are operating independently with different models, with different reward functions, competing with each other and interacting with each other. I mean, that’s like an evolutionary soup that is going to cause all sorts of weird emergent properties that I don’t think we’re really prepared to monitor or even know how to control. And by definition, you can’t control a complex system. You can just nudge it in various ways.

gem
I think there are two metaphors that are helpful. The immune system is the one that I know that you have talked about an Royal too, of course. And I think another one is the colony metaphor, you know, ants versus, versus the whole colony. What’s the colony doing? I don’t need to talk about individual ants or paint little numbers on each ant.

And, and that’s the future. So we have to think about that starting pretty soon.

PHIL
No, that’s right. I intuitively think that AGI will come from as an emergent property of agents. You know, yeah as you said, it’s the ants versus anthills is going to be a useful way to think about this.

And it’s just at what point… Again, going back to kind of our financial analogy, you know when are we going to see the first agentic flash crash of a trillion agents descending on some business’s agent because it mistakenly offered up a cheap deal? I think that I think we’re like 12 months away from something like that happening, if not sooner.

gem
Don’t say it. You’re too good of a prognosticator to say that, Phil.

PHIL
LAUGHTER

gem
This has been really fun. So, we’ve talked about what’s right around the bend, agents and swarms operating at machine speed. I want to close on what I think is an optimistic note about the three I’s: intuition, inspiration, and insight. A model can predict the next token, but it doesn’t really have a hunch. Sometimes the design feels wrong and it doesn’t experience some sort of eureka moment where it connects two unconnected domains. So as we get the busy work of security taken care of by automation through AI, do human traits like intuition, inspiration and insight, is that like what the ultimate premium is going to be for a high end professional in our field?

PHIL
Yeah, I think that, and it’s kind of a corollary to that, is taste. So like design, taste, which is sort of like how you direct things to happen. And I and I think the the question, I don’t really know. So I think there’s going to be a premium on security people that can know how to direct a set of technologies to a goal by still having their imagination of what are the attacks, what are the potential offensive and defensive tactics to harness AI to do that.

But then I also think, by definition, the ability of AI to create boundless scenarios and possibilities and then analyze them and score them, can that, in effect, stimulate an even better intuition and I just don’t know the answer to that. The only thing the only thing I know in the use of AI is the ability to have a bunch of software is not going to be a competitive edge anymore. The ability to have managed perception, taste, understand what problem you’re solving, and the same thing applies in security, is going to be the premium.

I think if anything, it’s going to, back to that CISO is becoming the chief digital risk officer point, think it compels security people to do what they always should have been doing, which is to think broadly about their business risk, their business resilience or their mission risk if you’re in a different environment and not just focus on the security because I think they’re going to be called upon to be much more multidisciplinary, to remain useful and actually just to amplify what they can do on security.

gem
Fantastic. Thanks so much for your insights today. It’s awesome.

PHIL
Yeah, always a pleasure to talk.

gem
This has been the Silver Bullet Security Podcast with BIML. Silver Bullet is sponsored by the Berryville Institute of Machine Learning, a nonprofit science and technology organization whose research focuses on machine learning security.

You can find a permanent archive of all of our episodes dating back to 2006 at garymcgraw.com/technology/silverbulletpodcast. Show links, notes, and an online discussion can be found on the Silver Bullet webpage at berryvilleiml.com/podcast.

This is Gary McGraw.