An important part of BIML’s mission as an institute is to spread the word about our understanding of machine learning security risk throughout the world. We recently decided to take on three college and high school interns to provide a bridge to academia and to inculcate young minds early in the intricacies of machine learning security. We introduce them here in a series of blog entries.
We are very pleased to introduce Trinity Stroud who is a BIML University Scholar.
Trinity is a senior at the University of South Alabama in Mobile, Alabama. She has been programming since middle school, where she cut her teeth on the ROBOTC language. Later, in high school, she learned to code in Python and Java. In college she became passionately interested in the area of computer security.
Trinity participates in her university’s DayZero Cyber Competition Team and represents the USA School of Computing as a Student Government Association Senator. She enjoys participating in security competitions such as National Cyber League and Cyber FastTrack, the latter during which she was named a national finalist and awarded a full scholarship for the Undergraduate Certificate Program in Applied Cybersecurity with the SANS Technology Institute.
In her free time, Trinity reads science fiction novels written by such authors as Robert A. Heinlein, Orson Scott Card, and Anne McCaffrey.
As BIML University Scholar, Trinity will:
Examine and document University of South Alabama’s ML security research interests and activity
Examine and document BIML’s ML security research interests and activity
Create a cross reference for joint research interests and activity between University of South Alabama and BIML
Create a short list (10-15 items) of prospective joint SoC BIML research projects
Be jointly supervised by a University of South Alabama faculty member and BIML research staff member
A $2000 BIML scholarship has been allocated to pay for these activities.
An important part of BIML’s mission as an institute is to spread the word about our understanding of machine learning security risk throughout the world. We recently decided to take on three college and high school interns to provide a bridge to academia and to inculcate young minds early in the intricacies of machine learning security. We introduce them here in a series of blog entries.
We are very pleased to introduce Nikil Shyamsunder who is the first BIML High School Scholar.
Nikil is a sophomore at John Handley High School in Winchester, VA. He has been programming for most of his (short) life and has become keenly interested in Machine Learning.
Nikil organizes and teaches coding camps and also offers private coding classes to his peers. He enjoys participating in a philosophy-based style of debate called Lincoln-Douglass. He is currently competing in the International Public Policy Forum on the pros and cons of AI. He and his team recently reached the Top 16 and continue to compete. Nikil is fascinated by linguistics and has advanced through the Spelling Bee to Nationals twice.
In his free time, Nikil plays the violin, an Indian tonal percussion instrument called Mridangam, and enjoys producing music.
As BIML High School Scholar, Nikil will help to curate the BIML annotated bibliography. This bibliography has become an important resource for researchers working in the field of Machine Learning security as it provides an opinionated overview of work in the field, including a top 5 papers section.
For his efforts on behalf of BIML, Nikil will receive a scholarship of $500 to put towards expenses at the college of his choice.
BERRYVILLE INSTITUTE OF MACHINE LEARNING (BIML) GETS $150,000 OPEN PHILANTHROPY GRANT
Berryville Institute of Machine Learning (BIML) Gets $150,000 Open Philanthropy Grant. Funding will advance ethical AI research
Online PR News – 27-January-2021 – BERRYVILLE, VA – The Berryville Institute of Machine Learning (BIML), a research think tank dedicated to safe, secure and ethical development of AI technologies, announced today that it is the recipient of a $150,000 grant from Open Philanthropy.
BIML, which is already well known in ML circles for its pioneering document, “Architectural Risk Analysis of Machine Learning Systems: Toward More Secure Machine Learning,” will use the Open Philanthropy grant to further its scientific research on Machine Learning risk and get the word out more widely through talks, tutorials, and publications.“In what is by now an all too familiar pattern our embrace of advanced ML technology is outpacing an understanding of the security risks its use drags along with it. AI and ML automation continues to accelerate at an alarming pace. At BIML we’re dedicated to exposing and elucidating security risk in ML systems. We are pleased as punch that Open Philanthropy is pouring accelerant on our spark.”
“In a future where machine learning shapes the trajectory of humanity, we’ll need to see substantially more attention on thoroughly analyzing ML systems from a security and safety standpoint,” said Catherine Olsson, Senior Program Associate for Potential Risks from Advanced Artificial Intelligence at Open Philanthropy. “We are excited to see that BIML is taking a holistic, security-engineering inspired view, that considers both accidental risk and intentional misuse risk. We hope this funding will support the growth of a strong community of ML security practitioners at the intersection of real-world systems and basic research.”
Early work on ML security focuses on specific failures, including systems that learn to be sexist, racist and xenophobic, and systems that can be manipulated by attackers. The BIML ML Security Risk Framework details the top 10 security risks in ML systems today. It is designed for use by developers, engineers, designers and others who are creating applications and services that use ML technologies, and can be practically applied in the early design and development phases of any ML project.
“In what is by now an all too familiar pattern, our embrace of advanced ML technology is outpacing an understanding of the security risks its use drags along with it. AI and ML automation continues to accelerate at an alarming pace,” said Dr. Gary McGraw, co-founder of BIML and world renowned software security pioneer. “At BIML, we’re dedicated to exposing and elucidating security risk in ML systems. We are pleased as punch that Open Philanthropy is pouring accelerant on our spark.”
About BIML
The Berryville Institute of Machine Learning was created in 2019 to address security issues with ML and AI. The organization was founded by Gary McGraw, author, long-time security expert and CTO of Cigital (acquired by Synopsys); Harold Figueroa, director of Machine Intelligence Research and Applications (MIRA) Lab at Ntrepid; Victor Shepardson, an artist and research engineer at Ntrepid; and Richie Bonett, a systems engineer at Verisign. BIML is headquartered in Berryville, Virginia. For more information, visit http://berryvilleiml.com/.
About Open Philanthropy
Open Philanthropy identifies outstanding giving opportunities, makes grants, follows the results, and publishes its findings. Its mission is to give as effectively as it can and share the findings openly so that anyone can build on them.
IEEE Computer Article Distills BIML Top Ten ML Security Risks
BERRYVILLE, Va., Feb. 13, 2020 – The Berryville Institute of Machine Learning (BIML), a research think tank dedicated to safe, secure and ethical development of AI technologies, today released the first-ever risk framework to guide development of secure ML. The “Architectural Risk Analysis of Machine Learning Systems: Toward More Secure Machine Learning” is designed for use by developers, engineers, designers and others who are creating applications and services that use ML technologies.
Early work on ML security focuses on specific failures, including systems that learn to be sexist, racist and xenophobic like Microsoft’s Tay, or systems that can be manipulated into seeing a STOP sign as a speed limit sign using a few pieces of tape. The BIML ML Security Risk Framework details the top 10 security risks in ML systems today. A total of 78 risks have been identified by BIML using a generic ML system as an organizing concept. The BIML ML Security Risk Framework can be practically applied in the early design and development phases of any ML project.
“The tech industry is racing ahead with AI and ML with little to no consideration for the security risks that automated machine learning poses,” says Dr. Gary McGraw, co-founder of BIML. “We saw with the development of the internet the consequences of security as an afterthought. But with AI we have the chance now to do it right.”
For more information about An Architectural Risk Analysis of Machine Learning Systems: Toward More Secure Machine Learning, visit https://berryvilleiml.com/results/.
The first talk on BIML’s new Architectural Risk Analysis of Machine Learning Systems was delivered this Wednesday at Lord Fairfax Community College. The talk was well attended and included a remote audience attending virtually. The Winchester Star published a short article about the talk.
Berryville Institute of Machine Learning (BIML) is located in Clarke County, Virginia, an area served by Lord Fairfax Community College.
Early work in security and privacy of ML has taken an “operations security” tack focused on securing an existing ML system and maintaining its data integrity. For example, Nicolas Papernot uses Salzter and Schroeder’s famous security principles to provide an operational perspective on ML security1. In our view, this work does not go far enough into ML design to satisfy our goals. Following Papernot, we directly address Salzter and Schroeder’s security principles as adapted in the book Building Secure Software by Viega and McGraw. Our treatment is more directly tied to security engineering than to security operations.
Security Principles and
Machine Learning
In security engineering
it is not practical to protect against every type of possible attack. Security engineering is an exercise in risk management. One approach that works very well is to make use
of a set of guiding principles when designing and building systems. Good guiding
principles tend to improve the security outlook even in the face of unknown
future attacks. This strategy helps to alleviate the “attack-of-the-day”
problem so common in early days of software security
(and also sadly common in early approaches to ML security).
In this series of blog entries we present ten principles for ML security lifted directly from Building Secure Software and adapted for ML. The goal of these principles is to identify and to highlight the most important objectives you should keep in mind when designing and building a secure ML system. Following these principles should help you avoid lots of common security problems. Of course, this set of principles will not be able to cover every possible new flaw lurking in the future.
Some
caveats are in order. No list of principles like the one presented here is ever perfect.
There is no guarantee that if you follow these principles your ML system will be
secure. Not only do our principles present an incomplete picture, but they also
sometimes conflict with each other. As with any complex set of principles,
there are often subtle tradeoffs involved.
Clearly,
application of these ten principles must be sensitive to context. A mature risk
management approach to ML provides the sort of data required to apply these principles
intelligently.
What will follow in the next few blog entries is a treatment of each of the ten principles from an ML systems engineering perspective.
We’ll start with the first two tonight
1. N. Papernot, “A Marauder’s Map of Security and Privacy in Machine Learning,” arXiv:1811.01134, Nov. 2018. (see https://berryvilleiml.com/references/ for more)
The exceptionally tasteful BIML logo was designed by Jackie McGraw. The logo incorporates both a yin/yang concept (huh, wonder where that comes from?) and a glyph that incorporates a B, and M, and an L in a clever way.
Here is the glyph:
The BIML glyph
Here is my personal logo (seen all over, but most famously on the cover of Software Security:
Gary McGraw’s logo (as seen on the cover of Software Security among other places)
Here is the combined glyph plus yin/yang which makes up the official BIML logo.
Last, but not least, there is the “bonus” cow, which secretly includes a picture of Clarke county in its spots. Clarke county is where metropolitan Berryville is situated in Virginia.
Welcome to the BIML blog where we will (informally) write about MLsec, otherwise known as Machine Learning security. BIML is short for the Berryville Institute of Machine Learning. For what it’s worth, we think it is pretty amusing to have a “Berryville Institute” just like Santa Fe has the “Santa Fe Institute.” You go, Berryville!
BIML was born when I retired from my job of 24 years in January 2019. Many years ago as a graduate student at Indiana University, I did lots of work in machine learning and AI as part of my Ph.D. program in Cognitive Science. As a student of Doug Hofstadter’s I was deeply interested in emergent computation, sub-symbolic representation, error making, analogy, and low-level perceptual systems. I was fortunate to be surrounded by lots of fellow students interested in building things and finding out how the systems we were learning about by reading papers actually worked. Long story short, we built lots of real systems and published a bunch of papers about what we learned in the scientific literature.
The official BIML logo, designed by Jackie McGraw
Our mission at BIML is to explore the security implications built into ML systems. We’re starting with neural networks, which are all the rage at the moment, but we intend to think and write about genetic algorithms, sparse distributed memory, and other ML systems. Just to make this perfectly clear, we’re not really thinking much about using ML for security, rather we are focused on the security of ML systems themselves.
Fast forward 24 years. As one of the fathers of software security and security engineering at the design level, I have been professionally interested in how systems break, what kinds of risks are inherent in system design, and how to design systems that are more secure. At BIML we are applying these techniques and approaches directly to ML systems.
Through a series of small world phenomenon, the BIML group coalesced, sparked first when I met Harold Figueroa at an Ntrepid Technical Advisory Board meeting in the Fall of 2018 (I am the Chair of Ntrepid’s TAB, and Harold leads Ntrepid’s machine learning research group). Harold and I had a great initial discussion over dinner about representation, ML progress, and other issues. We decided that continuing those discussions and digging into some research was in order. Victor Shepardson, who did lots of ML work at Dartmouth as a Masters student, was present for our first meeting in January. We quickly added Richie Bonett, a Berryville local like me (!!) and a Ph.D. student at William and Mary, to the group. And BIML was officially born.
We started with a systematic and in depth review of the MLsec literature. You can see the results of our work in the annotated bibliography that we will continue to curate as we read and discuss papers.
Our second task was to develop an Attack Taxonomy that makes sense at a meta-level of the burgeoning ML attack literature. These days, lots of energy is being expended to attack certain ML systems. Some of the attacks are quite famous (stop sign recognition, and seeing cats as tanks both come to mind), and the popular press has made much of both ML progress and amusing attacks against ML systems. You can review the (ongoing) Attack Taxonomy work elsewhere on our website.
We’re now in the midst of an Architectural Risk Analysis (ARA) of a generic ML system. Our approach follows the ARA process introduced in my book Software Security and applied professionally for many years at Cigital. We plan to publish our work here as we make progress.
We’re really having fun with this group, and we hope you will get as much of a kick out of our results as we’re getting. We welcome contact and collaboration. Please let us know what you think.
