Early work in security and privacy of ML has taken an “operations security” tack focused on securing an existing ML system and maintaining its data integrity. For example, Nicolas Papernot uses Salzter and Schroeder’s famous security principles to provide an operational perspective on ML security¹. In our view, this work does not go far enough into ML design to satisfy our goals. Following Papernot, we directly address Salzter and Schroeder’s security principles as adapted in the book Building Secure Software by Viega and McGraw. Our treatment is more directly tied to security engineering than to security operations.

Security Principles and Machine Learning

In security engineering it is not practical to protect against every type of possible attack. Security engineering is an exercise in risk management. One approach that works very well is to make use of a set of guiding principles when designing and building systems. Good guiding principles tend to improve the security outlook even in the face of unknown future attacks. This strategy helps to alleviate the “attack-of-the-day” problem so common in early days of software security (and also sadly common in early approaches to ML security).

In this series of blog entries we present ten principles for ML security lifted directly from Building Secure Software and adapted for ML. The goal of these principles is to identify and to highlight the most impor­tant objectives you should keep in mind when designing and building a secure ML system. Following these principles should help you avoid lots of ­common security problems. Of course, this set of principles will not be able to cover every possible new flaw lurking in the future.

Some caveats are in order. No list of principles like the one pre­sented here is ever perfect. There is no guarantee that if you follow these principles your ML system will be secure. Not only do our principles present an incomplete picture, but they also sometimes conflict with each other. As with any complex set of principles, there are often subtle tradeoffs involved.

Clearly, application of these ten principles must be sensitive to context. A mature risk management approach to ML provides the sort of data required to apply these principles intelligently.

Principle 1: Secure the Weakest Link

Principle 2: Practice Defense in Depth

Principle 3: Fail Securely

Principle 4: Follow the Principle of Least Privilege

Principle 5: Compartmentalize

Principle 6: Keep It Simple

Principle 7: Promote Privacy

Principle 8: Remember That Hiding Secrets Is Hard

Principle 9: Be Reluctant to Trust

Principle 10: Use Your Community Resources

What will follow in the next few blog entries is a treatment of each of the ten principles from an ML systems engineering perspective.

We’ll start with the first two tonight

1. N. Papernot, “A Marauder’s Map of Security and Privacy in Machine Learning,” arXiv:1811.01134, Nov. 2018. (see https://berryvilleiml.com/references/ for more)