As our MLsec work makes abundantly clear, data play a huge role in security of an ML system. Our estimation is that somewhere around 60% of all security risk in ML can be directly associated with data. And data are biased in ways that lead to serious social justice problems including racism, sexism, classism, and xenophobia. We’ve read a few ML bias papers (see the BIML Anotated Bibliography for our commentary). Turns out that social justice in ML is a thorny and difficult subject.
We were joined this week by Martiza Johnson, a Computer Scientist and the inaugural director of a new center for data science, AI, and society at the University of San Diego. Maritza assigned us some homework (reading Chapter One and Chapter Four of Data Feminism, this blog entry, and watching Coded Bias), and then led us in a very interesting and far ranging conversation on bias in ML.
We recorded our conversation with Maritza which you can listen to. A video of our conversation is below.
Maritza Johnson leads a conversation on bias with BIML
An important part of BIML’s mission as an institute is to spread the word about our understanding of machine learning security risk throughout the world. We recently decided to take on three college and high school interns to provide a bridge to academia and to inculcate young minds early in the intricacies of machine learning security. We introduce them here in a series of blog entries.
We are very pleased to introduce Aishwarya Seth who is a BIML University Scholar.
Aishwarya is a graduate student at North Carolina State University in Raleigh, North Carolina. An ardent fan of crime thrillers since early childhood, she has always been passionate about security. When Aishwarya was introduced to Java programming in high school, her interest in security took a turn towards computer security.
The rise of Machine Learning coincides directly with Aishwarya’s study of security and cryptography, the confluence of which fascinate her. After earning her undergraduate degree in Computer Science, Aishwarya worked as a team member of the Clari5 AI/ML team where she focused on reducing the number of false positives detected for potentially fraudulent transactions online.
Apart from pondering different ways to secure the world, Aishwarya likes to read novels, scribble, travel, and explore.
As BIML University Scholar, Aishwarya will:
Examine and document North Carolina State University’s ML security research interests and activity
Examine and document BIML’s ML security research interests and activity
Create a cross reference for joint research interests and activity between NCSU and BIML
Be jointly supervised by Dr. Lauri Williams and a member and BIML research staff member
A $2000 BIML scholarship has been allocated to pay for these activities.
Berryville resident Gary McGraw is founder of the Berryville Institute of Machine Learning, which is a think tank. BIML’s small group of researchers tries to find ways to make technology safer so hackers cannot breach vital — or even secret — information. The institute has received a $150,000 grant from the Open Philanthropy foundation to help further its work.
BERRYVILLE — When
thinking about Clarke County, farms and rolling hills generally come to mind,
not sophisticated gadgets or high-tech wizardry.
In fact, many parts of the county
still lack high-speed internet service.
But hidden away in the
countryside is a small group of researchers trying to find ways to make
technology safer so hackers cannot breach vital — or even secret — information.
The Berryville Institute of
Machine Learning (BIML) was established in 2019 to address security issues
associated with machine learning (ML) and artificial intelligence (AI).
Recently, the institute received a $150,000 grant from the Open Philanthropy foundation
to help further its work.
BIML, a think tank, was founded
by software security expert Gary McGraw plus Richie Bonett, a computer
scientist from Berryville; Harold Figueroa, director of Machine Intelligence
Research and Applications Lab at Ntrepid, a Herndon-based cybersecurity firm,
and Victor Shepardson, an artist and research engineer at Ntrepid.
Artificial intelligence is
brainpower demonstrated by emotionless machines, in contrast to that of humans
and animals which involves consciousness and, in certain instances,
sensitivity.
Machine learning, on the other
hand, involves developing computer programs that help machines access data and
use it for their own benefit. The intent is to help computer systems develop
the ability to automatically learn and improve their functions from experience
without being specially programmed along that line.
“Usually, computers are
programmed with a bunch of rules telling them what to do,” McGraw said.
“Machine learning involves enabling machines to recognize certain inputs and
outputs so they can do certain tasks themselves.”
An example of such a machine, he
mentioned, is Alexa, a device developed by Amazon that uses speech recognition
abilities in performing tasks.
“When you’re talking to Alexa,
you’re interacting with a machine learning system,” McGraw noted.
Automatic banking machines are
another example of the technology, he pointed out. So are some types of video
games.
Technology is ever-evolving. And,
“when technologies catch on fast, people forget to secure them properly,”
McGraw said.
That can lead to trouble.
“A bad person may intentionally
trick a system into doing the wrong thing” for personal gain or harm, said
McGraw. “What we’re trying to do at BIML is to make it harder for bad people to
misuse systems.”
Each computer system is unique,
“so they learn in unique ways,” he said. As a result, unique solutions must be
created to prevent potential problems with them.
BIML’s research and
recommendations are placed into the “creative common” so people have free access
to them, McGraw said.
McGraw said the Open Philanthropy
grant will be used for various purposes, including research, recruiting interns
and making presentations on cybersecurity issues at colleges and universities
nationwide.
The institute already has recruited its first High School Scholar: Nikil Shyamsunder, a sophomore at Handley High School in Winchester. He will be involved in curating the “BIML Annotated Biography,” a resource for ML security workers providing an overview of research in that field, including a “Top 5 Papers” section.
As part of his internship,
Shyamsunder will receive a $500 college scholarship.
BIML is based in the Berryville
area largely because McGraw lives there — much of its work is based at his home
— and Bonett is from there.
“It doesn’t really matter where
this type of work is done,” McGraw said. “You don’t have to be physically
present somewhere with people to get the work done. The majority of the work is
done over the internet,” consulting with researchers and AI and ML
practitioners.
As technology evolves, “it’s hard
to anticipate” what BIML will be doing in the future, he said. But the machine
learning field is growing, so demand for services that the institute provides
is increasing, he asserted. Therefore, he expects the institute to be around
for many years to come.
More information about the
institute is online at berryvilleiml.com.
An important part of BIML’s mission as an institute is to spread the word about our understanding of machine learning security risk throughout the world. We recently decided to take on three college and high school interns to provide a bridge to academia and to inculcate young minds early in the intricacies of machine learning security. We introduce them here in a series of blog entries.
We are very pleased to introduce Trinity Stroud who is a BIML University Scholar.
Trinity is a senior at the University of South Alabama in Mobile, Alabama. She has been programming since middle school, where she cut her teeth on the ROBOTC language. Later, in high school, she learned to code in Python and Java. In college she became passionately interested in the area of computer security.
Trinity participates in her university’s DayZero Cyber Competition Team and represents the USA School of Computing as a Student Government Association Senator. She enjoys participating in security competitions such as National Cyber League and Cyber FastTrack, the latter during which she was named a national finalist and awarded a full scholarship for the Undergraduate Certificate Program in Applied Cybersecurity with the SANS Technology Institute.
In her free time, Trinity reads science fiction novels written by such authors as Robert A. Heinlein, Orson Scott Card, and Anne McCaffrey.
As BIML University Scholar, Trinity will:
Examine and document University of South Alabama’s ML security research interests and activity
Examine and document BIML’s ML security research interests and activity
Create a cross reference for joint research interests and activity between University of South Alabama and BIML
Create a short list (10-15 items) of prospective joint SoC BIML research projects
Be jointly supervised by a University of South Alabama faculty member and BIML research staff member
A $2000 BIML scholarship has been allocated to pay for these activities.
An important part of BIML’s mission as an institute is to spread the word about our understanding of machine learning security risk throughout the world. We recently decided to take on three college and high school interns to provide a bridge to academia and to inculcate young minds early in the intricacies of machine learning security. We introduce them here in a series of blog entries.
We are very pleased to introduce Nikil Shyamsunder who is the first BIML High School Scholar.
Nikil is a sophomore at John Handley High School in Winchester, VA. He has been programming for most of his (short) life and has become keenly interested in Machine Learning.
Nikil organizes and teaches coding camps and also offers private coding classes to his peers. He enjoys participating in a philosophy-based style of debate called Lincoln-Douglass. He is currently competing in the International Public Policy Forum on the pros and cons of AI. He and his team recently reached the Top 16 and continue to compete. Nikil is fascinated by linguistics and has advanced through the Spelling Bee to Nationals twice.
In his free time, Nikil plays the violin, an Indian tonal percussion instrument called Mridangam, and enjoys producing music.
As BIML High School Scholar, Nikil will help to curate the BIML annotated bibliography. This bibliography has become an important resource for researchers working in the field of Machine Learning security as it provides an opinionated overview of work in the field, including a top 5 papers section.
For his efforts on behalf of BIML, Nikil will receive a scholarship of $500 to put towards expenses at the college of his choice.
BERRYVILLE INSTITUTE OF MACHINE LEARNING (BIML) GETS $150,000 OPEN PHILANTHROPY GRANT
Berryville Institute of Machine Learning (BIML) Gets $150,000 Open Philanthropy Grant. Funding will advance ethical AI research
Online PR News – 27-January-2021 – BERRYVILLE, VA – The Berryville Institute of Machine Learning (BIML), a research think tank dedicated to safe, secure and ethical development of AI technologies, announced today that it is the recipient of a $150,000 grant from Open Philanthropy.
BIML, which is already well known in ML circles for its pioneering document, “Architectural Risk Analysis of Machine Learning Systems: Toward More Secure Machine Learning,” will use the Open Philanthropy grant to further its scientific research on Machine Learning risk and get the word out more widely through talks, tutorials, and publications.“In what is by now an all too familiar pattern our embrace of advanced ML technology is outpacing an understanding of the security risks its use drags along with it. AI and ML automation continues to accelerate at an alarming pace. At BIML we’re dedicated to exposing and elucidating security risk in ML systems. We are pleased as punch that Open Philanthropy is pouring accelerant on our spark.”
“In a future where machine learning shapes the trajectory of humanity, we’ll need to see substantially more attention on thoroughly analyzing ML systems from a security and safety standpoint,” said Catherine Olsson, Senior Program Associate for Potential Risks from Advanced Artificial Intelligence at Open Philanthropy. “We are excited to see that BIML is taking a holistic, security-engineering inspired view, that considers both accidental risk and intentional misuse risk. We hope this funding will support the growth of a strong community of ML security practitioners at the intersection of real-world systems and basic research.”
Early work on ML security focuses on specific failures, including systems that learn to be sexist, racist and xenophobic, and systems that can be manipulated by attackers. The BIML ML Security Risk Framework details the top 10 security risks in ML systems today. It is designed for use by developers, engineers, designers and others who are creating applications and services that use ML technologies, and can be practically applied in the early design and development phases of any ML project.
“In what is by now an all too familiar pattern, our embrace of advanced ML technology is outpacing an understanding of the security risks its use drags along with it. AI and ML automation continues to accelerate at an alarming pace,” said Dr. Gary McGraw, co-founder of BIML and world renowned software security pioneer. “At BIML, we’re dedicated to exposing and elucidating security risk in ML systems. We are pleased as punch that Open Philanthropy is pouring accelerant on our spark.”
About BIML
The Berryville Institute of Machine Learning was created in 2019 to address security issues with ML and AI. The organization was founded by Gary McGraw, author, long-time security expert and CTO of Cigital (acquired by Synopsys); Harold Figueroa, director of Machine Intelligence Research and Applications (MIRA) Lab at Ntrepid; Victor Shepardson, an artist and research engineer at Ntrepid; and Richie Bonett, a systems engineer at Verisign. BIML is headquartered in Berryville, Virginia. For more information, visit http://berryvilleiml.com/.
About Open Philanthropy
Open Philanthropy identifies outstanding giving opportunities, makes grants, follows the results, and publishes its findings. Its mission is to give as effectively as it can and share the findings openly so that anyone can build on them.
IEEE Computer Article Distills BIML Top Ten ML Security Risks
BERRYVILLE, Va., Feb. 13, 2020 – The Berryville Institute of Machine Learning (BIML), a research think tank dedicated to safe, secure and ethical development of AI technologies, today released the first-ever risk framework to guide development of secure ML. The “Architectural Risk Analysis of Machine Learning Systems: Toward More Secure Machine Learning” is designed for use by developers, engineers, designers and others who are creating applications and services that use ML technologies.
Early work on ML security focuses on specific failures, including systems that learn to be sexist, racist and xenophobic like Microsoft’s Tay, or systems that can be manipulated into seeing a STOP sign as a speed limit sign using a few pieces of tape. The BIML ML Security Risk Framework details the top 10 security risks in ML systems today. A total of 78 risks have been identified by BIML using a generic ML system as an organizing concept. The BIML ML Security Risk Framework can be practically applied in the early design and development phases of any ML project.
“The tech industry is racing ahead with AI and ML with little to no consideration for the security risks that automated machine learning poses,” says Dr. Gary McGraw, co-founder of BIML. “We saw with the development of the internet the consequences of security as an afterthought. But with AI we have the chance now to do it right.”
For more information about An Architectural Risk Analysis of Machine Learning Systems: Toward More Secure Machine Learning, visit https://berryvilleiml.com/results/.
