The idea behind defense in depth is to manage risk with diverse defensive strategies, so that if one layer of defense turns out to be inadequate, another layer of defense hopefully prevents a full breach.

Let’s go back to our example of bank security. Why is the typical bank more secure than the typical convenience store? Because there are many redundant security measures protecting the bank, and the more measures there are, the more secure the place is.

Security cameras alone are a deterrent for some. But if people don’t care about the cameras, then a security guard is there to defend the bank physically with a gun. Two security guards provide even more protection. But if both security guards get shot by masked bandits, then at least there’s still a wall of bulletproof glass and electronically locked doors to protect the tellers from the robbers. Of course if the robbers happen to kick in the doors, or guess the code for the door, at least they can only get at the teller registers, because the bank has a vault protecting the really valuable stuff. Hopefully, the vault is protected by several locks and cannot be opened without two individuals who are rarely at the bank at the same time. And as for the teller registers, they can be protected by having dye-emitting bills stored at the bottom, for distribution during a robbery.

Of course, having all these security measures does not ensure that the bank is never successfully robbed. Bank robberies do happen, even at banks with this much security. Nonetheless, it’s pretty obvious that the sum total of all these defenses results in a far more effective security system than any one defense alone.

The defense-in-depth principle may seem somewhat contradictory to the “secure-the-weakest-link” principle because we are essentially saying that defenses taken as a whole can be stronger than the weakest link. How­ever, there is no contradiction. The principle “secure the weakest link” applies when components have security functionality that does not overlap. But when it comes to redundant security measures, it is indeed possible that the sum protection offered is far greater than the protection offered by any single component.

ML systems are constructedout ofnumerous components.  And, as we pointed out multiple times above, the data are often the most important thing from a security perspective. This means that bad actors haveasmany opportunities to exploitan ML system as there are components, and then some. Each and every component comes with a set of risks, and each and every one needs to address those risks head on.  But wait, there’s more. Defense in depth teaches that vulnerabilities not addressed (or attacks not covered) by one component should, in principle, be caught by another. In some cases a risk may be controlled “upstream” and in others “downstream.”  

Lets consider anexample: a given ML system designmay attempt to secure sensitive training data behind some kind of authentication and authorization system, only allowing the model access to the data while it is actually training. While this may well bea reasonable and well-justified practice, it is by no means sufficient to ensure that no sensitive information in the dataset can be leaked through malicious misuse/abuse of the system as a whole. Some ML models are vulnerable to leaking sensitive information via carefully selected queries made to the operating model.^[i] In other cases, lots of know-how in “learned” form may be leaked through a transfer attack.^[ii] Maintaining a history of queries made by users, and preventing subsequent queries that together could be used to divine sensitive information can serve as an additional defensive layer that protects against these kinds of attack.  

Practicing defense in depth naturally involves applying the principle of least privilegeto users and operations engineers of an ML system. Identifying and preventing security exploits is much easier when every component limits its accessto only theresources it actually requires.  In this case, identifying and separating components in a design can help, because components become natural trust boundaries where controls can be put in place and policies enforced.

Defense in depth is especially powerful when each component works in concert with the others.

Read the rest of the principles here.

^[i]M. Fredrikson, S. Jha, and T. Ristenpart, “Model Inversion Attacks That Exploit Confidence Information and Basic Countermeasures,” Proceedings of the 22Nd ACM SIGSAC Conference on Computer and Communications Security, 2015, pp. 1322–1333.

^[ii]B. Wang, Y. Yao, B. Viswanath, H. Zheng, and B. Y. Zhao, “With Great Training Comes Great Vulnerability: Practical Attacks against Transfer Learning,” 27th USENIX Security Symposium, 2018, pp. 1281–1297.