Uncategorized

Berryville Institute of Machine Learning (BIML) Gets $150,000 Open Philanthropy Grant. Funding will advance ethical AI research

Online PR News – 27-January-2021 – BERRYVILLE, VA – The Berryville Institute of Machine Learning (BIML), a research think tank dedicated to safe, secure and ethical development of AI technologies, announced today that it is the recipient of a $150,000 grant from Open Philanthropy.

BIML, which is already well known in ML circles for its pioneering document, “Architectural Risk Analysis of Machine Learning Systems: Toward More Secure Machine Learning,” will use the Open Philanthropy grant to further its scientific research on Machine Learning risk and get the word out more widely through talks, tutorials, and publications.“In what is by now an all too familiar pattern our embrace of advanced ML technology is outpacing an understanding of the security risks its use drags along with it. AI and ML automation continues to accelerate at an alarming pace. At BIML we’re dedicated to exposing and elucidating security risk in ML systems. We are pleased as punch that Open Philanthropy is pouring accelerant on our spark.”

“In a future where machine learning shapes the trajectory of humanity, we’ll need to see substantially more attention on thoroughly analyzing ML systems from a security and safety standpoint,” said Catherine Olsson, Senior Program Associate for Potential Risks from Advanced Artificial Intelligence at Open Philanthropy. “We are excited to see that BIML is taking a holistic, security-engineering inspired view, that considers both accidental risk and intentional misuse risk. We hope this funding will support the growth of a strong community of ML security practitioners at the intersection of real-world systems and basic research.”

Early work on ML security focuses on specific failures, including systems that learn to be sexist, racist and xenophobic, and systems that can be manipulated by attackers. The BIML ML Security Risk Framework details the top 10 security risks in ML systems today. It is designed for use by developers, engineers, designers and others who are creating applications and services that use ML technologies, and can be practically applied in the early design and development phases of any ML project.

“In what is by now an all too familiar pattern, our embrace of advanced ML technology is outpacing an understanding of the security risks its use drags along with it. AI and ML automation continues to accelerate at an alarming pace,” said Dr. Gary McGraw, co-founder of BIML and world renowned software security pioneer. “At BIML, we’re dedicated to exposing and elucidating security risk in ML systems. We are pleased as punch that Open Philanthropy is pouring accelerant on our spark.”

About BIML

The Berryville Institute of Machine Learning was created in 2019 to address security issues with ML and AI. The organization was founded by Gary McGraw, author, long-time security expert and CTO of Cigital (acquired by Synopsys); Harold Figueroa, director of Machine Intelligence Research and Applications (MIRA) Lab at Ntrepid; Victor Shepardson, an artist and research engineer at Ntrepid; and Richie Bonett, a systems engineer at Verisign. BIML is headquartered in Berryville, Virginia. For more information, visit http://berryvilleiml.com/.

About Open Philanthropy

Open Philanthropy identifies outstanding giving opportunities, makes grants, follows the results, and publishes its findings. Its mission is to give as effectively as it can and share the findings openly so that anyone can build on them.

BERRYVILLE, Va., Feb. 13, 2020 – The Berryville Institute of Machine Learning (BIML), a research think tank dedicated to safe, secure and ethical development of AI technologies, today released the first-ever risk framework to guide development of secure ML. The “Architectural Risk Analysis of Machine Learning Systems: Toward More Secure Machine Learning” is designed for use by developers, engineers, designers and others who are creating applications and services that use ML technologies.

Early work on ML security focuses on specific failures, including systems that learn to be sexist, racist and xenophobic like Microsoft’s Tay, or systems that can be manipulated into seeing a STOP sign as a speed limit sign using a few pieces of tape. The BIML ML Security Risk Framework details the top 10 security risks in ML systems today. A total of 78 risks have been identified by BIML using a generic ML system as an organizing concept. The BIML ML Security Risk Framework can be practically applied in the early design and development phases of any ML project.

“The tech industry is racing ahead with AI and ML with little to no consideration for the security risks that automated machine learning poses,” says Dr. Gary McGraw, co-founder of BIML. “We saw with the development of the internet the consequences of security as an afterthought. But with AI we have the chance now to do it right.” 

For more information about An Architectural Risk Analysis of Machine Learning Systems: Toward More Secure Machine Learning, visit https://berryvilleiml.com/results/.  

A link to the PR on the wire: https://onlineprnews.com//news/1143530-1581535720-biml-releases-first-risk-framework-for-securing-machine-learning-systems.html

Recently there have been several documents published as guides to security in machine learning. In October 2019, NIST published a draft called “A Taxonomy and Terminology of Adversarial Machine Learning”. Then in November, Microsoft published several interrelated webpages laying out a threat model for AI/ML systems and tying it to MS’s existing Software Development Lifecycle. We took a look at these documents to find out what they are trying to do, what they do well, and what they lack.

The NIST document is a tool for navigating MLsec literature, somewhat in the vein of an academic survey paper but accessible to those outside the field. The focus is explicitly “adversarial ML”, i.e. the failures a motivated attacker can induce in an ML system through input. They present a taxonomy of concepts in the literature rather than covering specific attacks or risks. Also included is a technical terminology with definitions, synonyms and references to the originating papers. The taxonomy at first appeared conceptually bizarre to us, but we came to see it as a powerful tool for a particular task: working backward from an unfamiliar technical term to its root concept and related ideas. In this way the NIST document may be very helpful to non-ML experts concerned with security attempting to wrangle the ML security literature.

The Microsoft effort is a three-headed beast:

• “Failure Modes in Machine Learning”, a brief taxonomy of 16 intentional and unintentional failures. It supposedly meets “the need to equip software developers, security incident responders, lawyers, and policy makers with a common vernacular to talk about this problem”. To this end the authors avoid technical language where possible. Each threat is classified using the somewhat dated and quaint Confidentiality/Integrity/Availability security model. This is easy enough to understand, though we find the distinction between Integrity and Availability attacks unclear for most ML scenarios. The unintentional failures are oddly fixated on Reinforcement Learning, and several seem to boil down to the same thing. For example #16 “Common Corruption” appears to be a subcategory of #14 “Distributional Shifts.”
• “AI/ML Pivots to the Security Development Lifecycle Bug Bar”, similar to the above but aimed at a different audience, “as a reference for the triage of AI/ML-related security issues”. This section presents materials for use while applying some of the standard Microsoft SDL processes.  Of interest is the fact that threat modeling is emphasized in its own section.  We approve of that move.
• “Threat Modeling AI/ML Systems and Dependencies”  is the most detailed component, containing the meat of the Microsoft MLsec effort. Here you can find security review checklists and a survey paper-style elaboration of each major risk with an emphasis on mitigations. The same eleven categories of “intentional failures” are used as in the other documents. However, (at the time of writing) the unintentional failures are left out. We found the highlighting of risk #6 “Neural Net Reprogramming” particularly interesting, as it had been unknown to us before. This work shows how adversarial examples can be used to do a kind of arbitrage where a service provided at cost (say, automatically tagging photos in a cloud storage account) can be repurposed to a similar task like breaking CAPTCHAs.

The Microsoft documents function as practical tools for securing software, including checklists for a security review and lists of potential mitigations. However, we find their categorizations confusing or redundant in places. Laudably, they move beyond adversarial ML to the concept of “unintentional failures”. But unfortunately, these failure modes remain mostly unelaborated in the more detailed documents.

Adversarial/intentional failures are important, but we shouldn’t neglect the unintentional ones. Faulty evaluation, unexpected distributional shifts, mis-specified models, and unintentional reproduction of data biases can all threaten the efficacy, safety and fairness of every ML system. Both the Microsoft and NIST documents are tools for an organization seeking to secure itself against external threats. But equally important to secure against is the misuse of AI/ML.

Community resources can be a double-edged sword; on the one hand, systems that have faced public scrutiny can benefit from the collective effort to break them. But nefarious individuals aren’t interested in publicizing the flaws they identify in open systems, and even large communities of developers have trouble resolving all of the flaws in such systems. Relying on publicly available information can expose your own system to risks, particularly if an attacker is able to identify similarities between your system and public ones.  

Transfer learning is a particularly relevant issue to ML systems. While transfer learning has demonstrated success in applying the learned knowledge of an ML system to other problems, knowledge of the base model can sometimes be used to attack the student [wang18]. In a more general sense, the use of publicly available models and hyperparameters could expose ML systems to particular attacks. How do engineers know that a model they use wasn’t deliberately made public for this very purpose? 

Public datasets used to train ML algorithms are another important concern. Engineers need to take care to validate the authenticity and quality of any public datasets they use, especially when that data could have been manipulated by unknown parties.  At the core of these concerns is the matter of trust; if the community can be trusted to effectively promote the security of their tools, models, and data, then community resources can be hesitantly used. Otherwise, it would be better to avoid exposing systems to unnecessary risk. After all, security problems in widely-used open-source projects have been known to persist for years, and in some cases decades, before the community finally took notice.

Read the rest of the principles here.

ML systems rely on a number of possibly untrusted, external sources for both their data and their computation. Let’s take on data first. Mechanisms used to collect and process data for training and evaluation make an obvious target. Of course, ML engineers need to get their data somehow, and this necessarily invokes the question of trust. How does an ML system know it can trust the data it’s being fed? And, more generally, what can the system do to evaluate the collector’s trustworthiness? Blindly trusting sources of information would expose the system to security risks and must be avoided.

Next, let’s turn to external sources of computation. External tools such as TensorFlow, Kubeflow, and pip can be evaluated based on the security expertise of their engineers, time-proven resilience to attacks, and their own reliance on further external tools, among other metrics. Nonetheless, it would be a mistake to assume that any external tool is infallible. Systems need to extend as little trust as possible, in the spirit of compartmentalization, to minimize the capabilities of threats operating through external tools.

It can help to think of the various components of an ML system as extending trust to one another; dataset assembly could trust the data collectors’ organization of the data, or it could build safeguards to ensure normalization. The inference algorithm could trust the model’s obfuscation of training data, or it could avoid responding to queries that are designed to extract sensitive information. Sometimes it’s more practical to trust certain properties of the data, or various components, but in the interests of secure design only a minimum amount of trust should be afforded. Building more security into each component makes attacks much more difficult to successfully orchestrate.

Read the rest of the principles here.

Security is often about keeping secrets. Users don’t want their personal data leaked. Keys must be kept secret to avoid eavesdropping and tampering. Top-secret algorithms need to be protected from competitors. These kinds of requirements are almost always high on the list, but turn out to be far more difficult to meet than the average user may suspect.

ML system engineers may want to keep the intricacies of their system secret, including the algorithm and model used, hyperparameter and configuration values, and other details concerning how the system trains and performs. Maintaining a level of secrecy is a sound strategy for improving the security of the system, but it should not be the only mechanism.

Past research in transfer learning has demonstrated the ability for new ML systems to be trained from existing ones. If transfer learning is known to have been applied, it may facilitate extraction of the proprietary layers trained “on top” of the base model. Even when the base model is not known, distillation attacks allow an attacker to copy the possibly proprietary behavior of a model using only the ability to query the ML system externally.  As a result, maintaining the secrecy of the system’s design requires more than simply not making the system public knowledge.

A chief concern for ML systems is protecting the confidentiality of training data. Some may attempt to “anonymize” the data used and consider that sufficient. As the government of Australia discovered in 2017, great care must be taken in determining that the data cannot be deanonymized¹. Neural networks similarly provide a layer of anonymization by transforming confidential information into weights, but even those weights can be vulnerable to advanced information extraction techniques. It’s up to system engineers to identify the risks inherent in their system and design protection mechanisms that minimize security exposure. 

Keeping secrets is hard, and it is almost always a source of security risk.

Read the rest of the principles here.

1. Culnane, Chris, Benjamin Rubinstein, Vanessa Teague. “Understanding the Maths is Crucial for Protecting Privacy.” Technical Report from Department of Computing and Information Systems, University of Melbourne. (Published Sept 29, 2016; Accessed Oct 28, 2019.)

Privacy is tricky even when ML is not involved. ML makes things ever trickier by in some sense re-representing sensitive and/or confidential data inside of the machine.  This makes the original data “invisible” (at least to some users), but remember that the data are still in some sense “in there somewhere.”  So, for example, if you train a classifier on sensitive medical data and you don’t consider what will happen when an attacker tries to get those data back out through a set of sophisticated queries, you may not be doing your job.

When it comes to sensitive data, one promising approach in privacy-preserving ML is differential privacy. The idea behind differential privacy is to set up privacy restrictions that, for example, guarantee that an individual patient’s private medical data never has too much influence on a dataset or on a trained ML system.  The idea is to in some sense “hide in plain sight” with a goal of ensuring that anything that can be learned about an individual from the released information, can also be learned without that individual’s data being included.  An algorithm is differentially private if an observer examining the output is not able to determine whether a specific individual’s information was used in the computation.  Differential privacy is achieved through the use of random noise that is generated according to a chosen distribution and is used to perturb a true answer.  Somewhat counterintuitively, because of its use of noise, differential privacy can also be used to combat overfitting in some ML situations.  Differential privacy is a reasonably promising line of research that can in some cases provide for privacy protection.

      Privacy also applies to the behavior of a trained-up ML system in operation.  We’ve discussed the tradeoffs associated with providing (or not providing) confidence scores.  Sometimes that’s a great idea, and sometimes it’s not. Figuring out the impact on system security that providing confidence scores will have is another decision that should be explicitly considered and documented.

      In short, you will do well to spend some cycles thinking about privacy in your ML system while you’re thinking about security.

Read the rest of the principles here.