An important part of BIML’s mission as an institute is to spread the word about our understanding of machine learning security risk throughout the world. We recently decided to take on three college and high school interns to provide a bridge to academia and to inculcate young minds early in the intricacies of machine learning security. We introduce them here in a series of blog entries.
We are very pleased to introduce Aishwarya Seth who is a BIML University Scholar.
Aishwarya is a graduate student at North Carolina State University in Raleigh, North Carolina. An ardent fan of crime thrillers since early childhood, she has always been passionate about security. When Aishwarya was introduced to Java programming in high school, her interest in security took a turn towards computer security.
The rise of Machine Learning coincides directly with Aishwarya’s study of security and cryptography, the confluence of which fascinate her. After earning her undergraduate degree in Computer Science, Aishwarya worked as a team member of the Clari5 AI/ML team where she focused on reducing the number of false positives detected for potentially fraudulent transactions online.
Apart from pondering different ways to secure the world, Aishwarya likes to read novels, scribble, travel, and explore.
As BIML University Scholar, Aishwarya will:
Examine and document North Carolina State University’s ML security research interests and activity
Examine and document BIML’s ML security research interests and activity
Create a cross reference for joint research interests and activity between NCSU and BIML
Be jointly supervised by Dr. Lauri Williams and a member and BIML research staff member
A $2000 BIML scholarship has been allocated to pay for these activities.
Berryville resident Gary McGraw is founder of the Berryville Institute of Machine Learning, which is a think tank. BIML’s small group of researchers tries to find ways to make technology safer so hackers cannot breach vital — or even secret — information. The institute has received a $150,000 grant from the Open Philanthropy foundation to help further its work.
BERRYVILLE — When
thinking about Clarke County, farms and rolling hills generally come to mind,
not sophisticated gadgets or high-tech wizardry.
In fact, many parts of the county
still lack high-speed internet service.
But hidden away in the
countryside is a small group of researchers trying to find ways to make
technology safer so hackers cannot breach vital — or even secret — information.
The Berryville Institute of
Machine Learning (BIML) was established in 2019 to address security issues
associated with machine learning (ML) and artificial intelligence (AI).
Recently, the institute received a $150,000 grant from the Open Philanthropy foundation
to help further its work.
BIML, a think tank, was founded
by software security expert Gary McGraw plus Richie Bonett, a computer
scientist from Berryville; Harold Figueroa, director of Machine Intelligence
Research and Applications Lab at Ntrepid, a Herndon-based cybersecurity firm,
and Victor Shepardson, an artist and research engineer at Ntrepid.
Artificial intelligence is
brainpower demonstrated by emotionless machines, in contrast to that of humans
and animals which involves consciousness and, in certain instances,
sensitivity.
Machine learning, on the other
hand, involves developing computer programs that help machines access data and
use it for their own benefit. The intent is to help computer systems develop
the ability to automatically learn and improve their functions from experience
without being specially programmed along that line.
“Usually, computers are
programmed with a bunch of rules telling them what to do,” McGraw said.
“Machine learning involves enabling machines to recognize certain inputs and
outputs so they can do certain tasks themselves.”
An example of such a machine, he
mentioned, is Alexa, a device developed by Amazon that uses speech recognition
abilities in performing tasks.
“When you’re talking to Alexa,
you’re interacting with a machine learning system,” McGraw noted.
Automatic banking machines are
another example of the technology, he pointed out. So are some types of video
games.
Technology is ever-evolving. And,
“when technologies catch on fast, people forget to secure them properly,”
McGraw said.
That can lead to trouble.
“A bad person may intentionally
trick a system into doing the wrong thing” for personal gain or harm, said
McGraw. “What we’re trying to do at BIML is to make it harder for bad people to
misuse systems.”
Each computer system is unique,
“so they learn in unique ways,” he said. As a result, unique solutions must be
created to prevent potential problems with them.
BIML’s research and
recommendations are placed into the “creative common” so people have free access
to them, McGraw said.
McGraw said the Open Philanthropy
grant will be used for various purposes, including research, recruiting interns
and making presentations on cybersecurity issues at colleges and universities
nationwide.
The institute already has recruited its first High School Scholar: Nikil Shyamsunder, a sophomore at Handley High School in Winchester. He will be involved in curating the “BIML Annotated Biography,” a resource for ML security workers providing an overview of research in that field, including a “Top 5 Papers” section.
As part of his internship,
Shyamsunder will receive a $500 college scholarship.
BIML is based in the Berryville
area largely because McGraw lives there — much of its work is based at his home
— and Bonett is from there.
“It doesn’t really matter where
this type of work is done,” McGraw said. “You don’t have to be physically
present somewhere with people to get the work done. The majority of the work is
done over the internet,” consulting with researchers and AI and ML
practitioners.
As technology evolves, “it’s hard
to anticipate” what BIML will be doing in the future, he said. But the machine
learning field is growing, so demand for services that the institute provides
is increasing, he asserted. Therefore, he expects the institute to be around
for many years to come.
More information about the
institute is online at berryvilleiml.com.
An important part of BIML’s mission as an institute is to spread the word about our understanding of machine learning security risk throughout the world. We recently decided to take on three college and high school interns to provide a bridge to academia and to inculcate young minds early in the intricacies of machine learning security. We introduce them here in a series of blog entries.
We are very pleased to introduce Trinity Stroud who is a BIML University Scholar.
Trinity is a senior at the University of South Alabama in Mobile, Alabama. She has been programming since middle school, where she cut her teeth on the ROBOTC language. Later, in high school, she learned to code in Python and Java. In college she became passionately interested in the area of computer security.
Trinity participates in her university’s DayZero Cyber Competition Team and represents the USA School of Computing as a Student Government Association Senator. She enjoys participating in security competitions such as National Cyber League and Cyber FastTrack, the latter during which she was named a national finalist and awarded a full scholarship for the Undergraduate Certificate Program in Applied Cybersecurity with the SANS Technology Institute.
In her free time, Trinity reads science fiction novels written by such authors as Robert A. Heinlein, Orson Scott Card, and Anne McCaffrey.
As BIML University Scholar, Trinity will:
Examine and document University of South Alabama’s ML security research interests and activity
Examine and document BIML’s ML security research interests and activity
Create a cross reference for joint research interests and activity between University of South Alabama and BIML
Create a short list (10-15 items) of prospective joint SoC BIML research projects
Be jointly supervised by a University of South Alabama faculty member and BIML research staff member
A $2000 BIML scholarship has been allocated to pay for these activities.
An important part of BIML’s mission as an institute is to spread the word about our understanding of machine learning security risk throughout the world. We recently decided to take on three college and high school interns to provide a bridge to academia and to inculcate young minds early in the intricacies of machine learning security. We introduce them here in a series of blog entries.
We are very pleased to introduce Nikil Shyamsunder who is the first BIML High School Scholar.
Nikil is a sophomore at John Handley High School in Winchester, VA. He has been programming for most of his (short) life and has become keenly interested in Machine Learning.
Nikil organizes and teaches coding camps and also offers private coding classes to his peers. He enjoys participating in a philosophy-based style of debate called Lincoln-Douglass. He is currently competing in the International Public Policy Forum on the pros and cons of AI. He and his team recently reached the Top 16 and continue to compete. Nikil is fascinated by linguistics and has advanced through the Spelling Bee to Nationals twice.
In his free time, Nikil plays the violin, an Indian tonal percussion instrument called Mridangam, and enjoys producing music.
As BIML High School Scholar, Nikil will help to curate the BIML annotated bibliography. This bibliography has become an important resource for researchers working in the field of Machine Learning security as it provides an opinionated overview of work in the field, including a top 5 papers section.
For his efforts on behalf of BIML, Nikil will receive a scholarship of $500 to put towards expenses at the college of his choice.
BERRYVILLE INSTITUTE OF MACHINE LEARNING (BIML) GETS $150,000 OPEN PHILANTHROPY GRANT
Berryville Institute of Machine Learning (BIML) Gets $150,000 Open Philanthropy Grant. Funding will advance ethical AI research
Online PR News – 27-January-2021 – BERRYVILLE, VA – The Berryville Institute of Machine Learning (BIML), a research think tank dedicated to safe, secure and ethical development of AI technologies, announced today that it is the recipient of a $150,000 grant from Open Philanthropy.
BIML, which is already well known in ML circles for its pioneering document, “Architectural Risk Analysis of Machine Learning Systems: Toward More Secure Machine Learning,” will use the Open Philanthropy grant to further its scientific research on Machine Learning risk and get the word out more widely through talks, tutorials, and publications.“In what is by now an all too familiar pattern our embrace of advanced ML technology is outpacing an understanding of the security risks its use drags along with it. AI and ML automation continues to accelerate at an alarming pace. At BIML we’re dedicated to exposing and elucidating security risk in ML systems. We are pleased as punch that Open Philanthropy is pouring accelerant on our spark.”
“In a future where machine learning shapes the trajectory of humanity, we’ll need to see substantially more attention on thoroughly analyzing ML systems from a security and safety standpoint,” said Catherine Olsson, Senior Program Associate for Potential Risks from Advanced Artificial Intelligence at Open Philanthropy. “We are excited to see that BIML is taking a holistic, security-engineering inspired view, that considers both accidental risk and intentional misuse risk. We hope this funding will support the growth of a strong community of ML security practitioners at the intersection of real-world systems and basic research.”
Early work on ML security focuses on specific failures, including systems that learn to be sexist, racist and xenophobic, and systems that can be manipulated by attackers. The BIML ML Security Risk Framework details the top 10 security risks in ML systems today. It is designed for use by developers, engineers, designers and others who are creating applications and services that use ML technologies, and can be practically applied in the early design and development phases of any ML project.
“In what is by now an all too familiar pattern, our embrace of advanced ML technology is outpacing an understanding of the security risks its use drags along with it. AI and ML automation continues to accelerate at an alarming pace,” said Dr. Gary McGraw, co-founder of BIML and world renowned software security pioneer. “At BIML, we’re dedicated to exposing and elucidating security risk in ML systems. We are pleased as punch that Open Philanthropy is pouring accelerant on our spark.”
About BIML
The Berryville Institute of Machine Learning was created in 2019 to address security issues with ML and AI. The organization was founded by Gary McGraw, author, long-time security expert and CTO of Cigital (acquired by Synopsys); Harold Figueroa, director of Machine Intelligence Research and Applications (MIRA) Lab at Ntrepid; Victor Shepardson, an artist and research engineer at Ntrepid; and Richie Bonett, a systems engineer at Verisign. BIML is headquartered in Berryville, Virginia. For more information, visit http://berryvilleiml.com/.
About Open Philanthropy
Open Philanthropy identifies outstanding giving opportunities, makes grants, follows the results, and publishes its findings. Its mission is to give as effectively as it can and share the findings openly so that anyone can build on them.
IEEE Computer Article Distills BIML Top Ten ML Security Risks
BERRYVILLE, Va., Feb. 13, 2020 – The Berryville Institute of Machine Learning (BIML), a research think tank dedicated to safe, secure and ethical development of AI technologies, today released the first-ever risk framework to guide development of secure ML. The “Architectural Risk Analysis of Machine Learning Systems: Toward More Secure Machine Learning” is designed for use by developers, engineers, designers and others who are creating applications and services that use ML technologies.
Early work on ML security focuses on specific failures, including systems that learn to be sexist, racist and xenophobic like Microsoft’s Tay, or systems that can be manipulated into seeing a STOP sign as a speed limit sign using a few pieces of tape. The BIML ML Security Risk Framework details the top 10 security risks in ML systems today. A total of 78 risks have been identified by BIML using a generic ML system as an organizing concept. The BIML ML Security Risk Framework can be practically applied in the early design and development phases of any ML project.
“The tech industry is racing ahead with AI and ML with little to no consideration for the security risks that automated machine learning poses,” says Dr. Gary McGraw, co-founder of BIML. “We saw with the development of the internet the consequences of security as an afterthought. But with AI we have the chance now to do it right.”
For more information about An Architectural Risk Analysis of Machine Learning Systems: Toward More Secure Machine Learning, visit https://berryvilleiml.com/results/.
The first talk on BIML’s new Architectural Risk Analysis of Machine Learning Systems was delivered this Wednesday at Lord Fairfax Community College. The talk was well attended and included a remote audience attending virtually. The Winchester Star published a short article about the talk.
Berryville Institute of Machine Learning (BIML) is located in Clarke County, Virginia, an area served by Lord Fairfax Community College.
On recent Microsoft and NIST ML security documents
Recently there have been several documents published as
guides to security in machine learning. In October 2019, NIST published a draft
called “A
Taxonomy and Terminology of Adversarial Machine Learning”. Then in
November, Microsoft published several
interrelated webpages laying out a threat model for AI/ML systems and tying
it to MS’s existing Software Development Lifecycle. We took a look at these
documents to find out what they are trying to do, what they do well, and what
they lack.
The NIST document is a tool for navigating MLsec literature,
somewhat in the vein of an academic survey paper but accessible to those
outside the field. The focus is explicitly “adversarial ML”, i.e. the failures
a motivated attacker can induce in an ML system through input. They present a
taxonomy of concepts in the literature rather than covering specific attacks
or risks. Also included is a technical terminology with definitions, synonyms
and references to the originating papers. The taxonomy at first appeared
conceptually bizarre to us, but we came to see it as a powerful tool for a
particular task: working backward from an unfamiliar technical term to its root
concept and related ideas. In this way the NIST document may be very helpful to
non-ML experts concerned with security attempting to wrangle the ML security
literature.
The Microsoft effort is a three-headed beast:
“Failure
Modes in Machine Learning”, a brief taxonomy of 16 intentional and
unintentional failures. It supposedly meets “the need to equip software
developers, security incident responders, lawyers, and policy makers with a
common vernacular to talk about this problem”. To this end the authors avoid
technical language where possible. Each threat is classified using the somewhat
dated and quaint Confidentiality/Integrity/Availability security model. This is
easy enough to understand, though we find the distinction between Integrity and
Availability attacks unclear for most ML scenarios. The unintentional
failures are oddly fixated on Reinforcement Learning, and several seem to boil
down to the same thing. For example #16 “Common Corruption” appears to be a
subcategory of #14 “Distributional Shifts.”
“AI/ML Pivots to
the Security Development Lifecycle Bug Bar”, similar to the above but aimed
at a different audience, “as a reference for the triage of AI/ML-related
security issues”. This section presents materials for use while applying some
of the standard Microsoft SDL processes.
Of interest is the fact that threat modeling is emphasized in its own
section. We approve of that move.
“Threat
Modeling AI/ML Systems and Dependencies” is the most detailed component, containing the
meat of the Microsoft MLsec effort. Here you can find security review
checklists and a survey paper-style elaboration of each major risk with an
emphasis on mitigations. The same eleven categories of “intentional failures”
are used as in the other documents. However, (at the time of writing) the unintentional
failures are left out. We found the highlighting of risk #6 “Neural Net Reprogramming”
particularly interesting, as it had been unknown to us before. This work shows
how adversarial examples can be used to do a kind of arbitrage where a service
provided at cost (say, automatically tagging photos in a cloud storage account)
can be repurposed to a similar task like breaking CAPTCHAs.
The Microsoft documents function as practical tools for securing
software, including checklists for a security review and lists of potential
mitigations. However, we find their categorizations confusing or redundant in
places. Laudably, they move beyond adversarial ML to the concept of
“unintentional failures”. But unfortunately, these failure modes remain mostly
unelaborated in the more detailed documents.
Adversarial/intentional failures are important, but we
shouldn’t neglect the unintentional ones. Faulty evaluation, unexpected distributional
shifts, mis-specified models, and unintentional reproduction of data biases can
all threaten the efficacy, safety and fairness of every ML system. Both the Microsoft
and NIST documents are tools for an organization seeking to secure itself
against external threats. But equally important to secure against is the misuse
of AI/ML.
