Machine Learning 102: Attacking Facial Authentication with Poisoned Data

This blog post is the second in a series related to machine learning, and demonstrates exactly how a data poisoning attack might work to insert a backdoor into a facial authentication system. The simplified system has similarities to that which the TSA is running a proof of concept trial at the Detroit and Atlanta airports. As background, … Continue reading Machine Learning 102: Attacking Facial Authentication with Poisoned Data

Using Semgrep with Jupyter Notebook files

If you frequently deliver source code review assessments of products, including machine learning components, I'm sure you are used to reviewing Jupyter Notebook files (usually python). Although I spend most of my time reviewing the source code manually, I also use static analysis tools such as semgrep, using both public and private rules. This tool … Continue reading Using Semgrep with Jupyter Notebook files

Project Bishop: Clustering Web Pages

Written by Jose Selvi and Thomas Atkinson If you are a Machine Learning (ML) enthusiast like us, you may recall our blogpost series from 2019 regarding Project Ava, which documented our experiments in using ML techniques to automate web application security testing tasks. In February 2020 we set out to build on Project Ava with … Continue reading Project Bishop: Clustering Web Pages

Machine Learning 101: The Integrity of Image (Mis)Classification?

Professor Ron Rivest observed the close relationship between cryptography and machine learning at the ASIACRYPT conference back in 1991. Cross-fertilization of common notions, such as integrity, privacy, confidentiality and authenticity, have only grown in the following three decades as these fields have become more central to our everyday lives. This blog post is the first in … Continue reading Machine Learning 101: The Integrity of Image (Mis)Classification?

Exploring Prompt Injection Attacks

Have you ever heard about Prompt Injection Attacks[1]? Prompt Injection is a new vulnerability that is affecting some AI/ML models and, in particular, certain types of language models using prompt-based learning.  This vulnerability was initially reported to OpenAI by Jon Cefalu (May 2022)[2] but it was kept in a responsible disclosure status until it was … Continue reading Exploring Prompt Injection Attacks

Five Essential Machine Learning Security Papers

We recently published "Practical Attacks on Machine Learning Systems", which has a very large references section - possibly too large - so we've boiled down the list to five papers that are absolutely essential in this area. If you're beginning your journey in ML security, and have the very basics down, these papers are a … Continue reading Five Essential Machine Learning Security Papers

Whitepaper – Practical Attacks on Machine Learning Systems

This paper collects a set of notes and research projects conducted by NCC Group on the topic of the security of Machine Learning (ML) systems. The objective is to provide some industry perspective to the academic community, while collating helpful references for security practitioners, to enable more effective security auditing and security-focused code review of ML systems. Details of specific practical attacks and common security problems are described. Some general background information on the broader subject of ML is also included, mostly for context, to ensure that explanations of attack scenarios are clear, and some notes on frameworks and development processes are provided.

Machine Learning for Static Analysis of Malware – Expansion of Research Scope

Introduction The work presented in this blog post is that of Ewan Alexander Miles (former UCL MSci student) and explores the expansion of scope for using machine learning models on PE (portable executable) header files to identify and classify malware. It is built on work previously presented by NCC Group, in conjunction with UCL’S Centre … Continue reading Machine Learning for Static Analysis of Malware – Expansion of Research Scope

Cracking Random Number Generators using Machine Learning – Part 2: Mersenne Twister

Outline 1. Introduction2. How does MT19937 PRNG work?3. Using Neural Networks to model the MT19937 PRNG3.1 Using NN for State Twisting3.1.1 Data Preparation3.1.2 Neural Network Model Design3.1.3 Optimizing the NN Inputs3.1.4 Model Results3.1.5 Model Deep Dive3.1.5.1 Model First Layer Connections3.1.5.2 The Logic Closed-Form from the State Twisting Model Output3.2 Using NN for State Tempering3.2.1 Data Preparation3.2.2 … Continue reading Cracking Random Number Generators using Machine Learning – Part 2: Mersenne Twister