Offensive Security & Artificial Intelligence
Non-Deterministic Nature of Prompt Injection
As we explained in a previous blogpost, exploiting a prompt injection attack is conceptually easy to understand: There are previous instructions in the prompt, and we include additional instructions within the user input, which is merged together with the legitimate instructions in a way that the underlying model cannot distinguish…
Machine Learning 104: Breaking AES With Power Side-Channels
This executable blog post is the fourth in a series related to machine learning and is a fascinating trifecta involving hardened cryptography software, embedded IoT-type hardware, and deep machine learning techniques. While the AES algorithm is designed such that a brute-force secret key guessing attack would likely finish ‘sometime near…
Exploring Overfitting Risks in Large Language Models
In the following blog post, we explore how overfitting can affect Large Language Models (LLMs) in particular, since this technology is used in the most promising AI technologies we see today (chatGPT, LLaMa, Bard, etc). Furthermore, by exploring the likelihood of inferring data from the dataset, we will determine how…
Security Code Review With ChatGPT
TL;DR: Don’t use ChatGPT for security code review. It’s not meant to be used that way, it doesn’t really work (although you might be fooled into thinking it does), and there are some other major problems that make it impractical. Also, both the CEO of OpenAI and ChatGPT itself say…
Machine Learning 102: Attacking Facial Authentication with Poisoned Data
This blog post is the second in a series related to machine learning, and demonstrates exactly how a data poisoning attack might work to insert a backdoor into a facial authentication system. The simplified system has similarities to that which the TSA is running a proof of concept trial at the Detroit…
Using Semgrep with Jupyter Notebook files
If you frequently deliver source code review assessments of products, including machine learning components, I’m sure you are used to reviewing Jupyter Notebook files (usually python). Although I spend most of my time reviewing the source code manually, I also use static analysis tools such as semgrep, using both public…
Impersonating Gamers With GPT-2
In this blog post, I’m going to recount the story of my quest to train OpenAI’s large language model, GPT-2, to create a virtual doppelganger of myself and my peers. Machine learning is one of those buzzwords that, sometimes, lives up to its reputation. As an information security professional, my…
Five Essential Machine Learning Security Papers
We recently published “Practical Attacks on Machine Learning Systems”, which has a very large references section – possibly too large – so we’ve boiled down the list to five papers that are absolutely essential in this area. If you’re beginning your journey in ML security, and have the very basics…
Whitepaper – Practical Attacks on Machine Learning Systems
This paper collects a set of notes and research projects conducted by NCC Group on the topic of the security of Machine Learning (ML) systems. The objective is to provide some industry perspective to the academic community, while collating helpful references for security practitioners, to enable more effective security auditing…
Machine Learning for Static Analysis of Malware – Expansion of Research Scope
Introduction The work presented in this blog post is that of Ewan Alexander Miles (former UCL MSci student) and explores the expansion of scope for using machine learning models on PE (portable executable) header files to identify and classify malware. It is built on work previously presented by NCC Group,…
On the malicious use of large language models like GPT-3
(Or, “Can large language models generate exploits?”) While attacking machine learning systems is a hot topic for which attacks have begun to be demonstrated, I believe that there are a number of entirely novel, yet-unexplored attack-types and security risks that are specific to large language models (LMs), that may be…
Cracking Random Number Generators using Machine Learning – Part 2: Mersenne Twister
Outline 1. Introduction 2. How does MT19937 PRNG work? 3. Using Neural Networks to model the MT19937 PRNG 3.1 Using NN for State Twisting 3.1.1 Data Preparation 3.1.2 Neural Network Model Design 3.1.3 Optimizing the NN Inputs 3.1.4 Model Results 3.1.5 Model Deep Dive 3.1.5.1 Model First Layer Connections 3.1.5.2 The…
Cracking Random Number Generators using Machine Learning – Part 1: xorshift128
Outline 1. Introduction 2. How does xorshift128 PRNG work? 3. Neural Networks and XOR gates 4. Using Neural Networks to model the xorshift128 PRNG 4.1 Neural Network Model Design 4.2 Model Results 4.3 Model Deep Dive 5. Creating a machine-learning-resistant version of xorshift128 6. Conclusion 1. Introduction This blog post proposes…
Research Paper – Machine Learning for Static Malware Analysis, with University College London
For the past few years, NCC Group has been an industry partner to the Centre for Doctoral Training in Data Intensive Science (CDT in DIS) at University College London (UCL). CDT is composed of a group of over 80 academics from across UCL in areas such as High Energy Physics,…