Breaking into Security Research at NCC Group
Overview
This blog post is a slightly modified version of an internal document recently produced at NCC Group. The aim of the original document was to introduce security research to our consultants, particularly those new to the business or to security research in general, and it summarised the different areas of focus within security research. We hope that others wanting to know more about breaking into security research, and how they can get involved at NCC Group, find this blog post useful.
Security research and its history
Security research typically involves looking for flaws in products or systems. For security professionals within NCC Group this also involves working within legal frameworks and working with affected vendors through responsible disclosure; providing advice on mitigations and fixes that they can implement in order to address any vulnerabilities found.
Historically (in the late 1990s and early 2000s) it was standard to disclose vulnerabilities (often zero-day) found from security research on public mailing lists such as full disclosure. Looking back, this reflects on what was then a fairly immature security research industry [1]. Back then, many vulnerabilities that were exposed related to memory corruption flaws (such as stack overflows) in software. These were usually trivial to exploit, since operating systems at the time lacked many of the modern memory and exploit protection mechanisms that exist today. Note that security immaturity also existed on the vendor side – this resulted in multiple flaws found in systems such as Microsoft Windows. This prompted companies like Microsoft to implement and follow secure development lifecycles; since then Microsoft has become an organisation with one the world’s best and most mature security programmes.
Today, the technology landscape is vast, and security research can include everything from IoT devices to web applications and everything in-between. Anything that is electronic and has some form of data processing or critical security function warrants research, in order to ensure that inherent vulnerabilities do not exist and cannot be exploited.
The way that vulnerabilities, flaws, and bugs are reported and acknowledged has also changed in recent times. Bug bounties now exist to reward security researchers; many of the large technology companies now have established and mature bug bounty schemes. These schemes reward researchers financially for any vulnerabilities submitted that have been verified as genuine and which are confirmed to pose a risk to the confidentiality, integrity, or availability of the target product or system or its underlying data.[2]
There are however some grey areas here; many markets now exist in which third parties such as Zerodium[3] will buy zero-day vulnerabilities from security researchers, often at a much higher monetary value than that offered by the vendor through their own legitimate bug bounty programme. The reason for such high bounties (for example a million dollars, for a series of zero-days that result in remote access to a device with the latest version of iOS [4]) is usually because the resale value of such vulnerabilities to some parties is high. Buyers of such vulnerabilities might therefore be nation states or nefarious groups seeking zero-day exploits in common platforms for some form of mass unauthorised access (e.g. spying or botnet propagation).
This potential for receipt of large sums of money in exchange for critical vulnerabilities in well-known and well-used products and systems has led to a major shift in motivation for security research.
People’s motivation for wanting to engage in security research will differ; it could be general curiosity about how things work and how they might be insecure, wanting to improve understanding and technical skills, or simply financial reward or notoriety through the submission of vulnerabilities to bug bounty schemes.
NCC Group has an internal policy on bug bounties to which all our consultants must adhere.
Where do I start?
To become a competent security researcher is by no means easy. It requires a lot of hard work, usually on one’s own time. Curiosity and persistence are necessary traits for good researchers; to a degree these traits can be imparted, however some people will be more predisposed to these behaviours than others.
“Learning this stuff takes a long time, so you have to have an interest to pursue things outside of a 9-5 job.”
Pete Beck, Exploit Development Group
A very important realisation when embarking on security research on a product or system is that flaws might not actually be found. This may be because no flaws exist, or because the researcher has failed to identify a vulnerability or realise its impact through a working proof of concept (PoC).
On the other end of the spectrum, some products or systems may be riddled with security flaws; in these scenarios it becomes a task of prioritising the myriad systemic issues found.
At NCC Group, the secure development lifecycle and threat modelling course is a great internal training course to attend when starting out in security research. This course teaches use of threat modelling as a tool to understand where security issues might lie in products and systems, helping to understand trust boundaries and data flows. Threat modelling in this way provides a much more structured approach to security research, as opposed to launching directly into tasks such as fuzzing without consideration of how the product or system is composed and where its critical security functions lie.
Reverse engineering and exploit development
If finding low-level bugs and exploiting these to gain elevated privilege or command control is of interest to you then you will need to learn:
- How to reverse-engineer software
- How to debug software
- How various memory corruption flaws manifest themselves within software
- How flaws can be exploited
- How to bypass modern memory and anti-exploitation mechanisms
It is likely that you will start with gaining an understanding of assembly code, and x86 instruction sets in particular as these typically underpin Windows and *nix operating systems.
At NCC Group, the x86 Foundations course provides exactly this initial grounding in machine instructions and memory management. Disassembly and debugging tools, and how to use them, are introduced.
Building on these x86 foundations, one can then look to attend the exploit development course, which will teach understanding of bugs, their causes, and the consequences of their exploitation.
It is important to understand that reverse engineering and vulnerability exploitation can be incredibly laborious and time-consuming. Even the best security researchers will often spend weeks or more working on finding flaws and then weaponising them as working exploits. These efforts might not even lead to any results or reward. Note however that the value here is in the process followed, what methods were used, any new scripts or tools that were developed in the process that might be used to great effect in your next security research venture. Depending on research goals, a working PoC may not even need to be produced. Just finding the vulnerability and attack vector may be sufficient.
Fuzzing
There are different approaches to finding bugs in software. Some researchers favour disassembling code and then manually tracing through it looking for flaws. Another common method is fuzzing, where automated tools are used to send malformed data to program inputs with the intention of making the target program crash or behave in unexpected ways. Crashes are usually a result of flaws, which can then be triaged for their potential to be turned into working exploits.
Thousands of fuzzing tools exist in the security industry; some commercial, many free. NCC Group employees have developed fuzzers and fuzzing frameworks to help find potential software flaws. Great examples are Zulu [5] and FuzzLabs [6].
Some people might enjoy the act of finding bugs more than actually exploiting them. If this is the case then research and development of fuzzing tools for protocols, file types, and so on is likely to be of interest to you.
Internally at NCC Group there is a fuzzing interest group mailing list for anybody interested in fuzzing. We use this forum to share information on fuzzing and associated techniques.
Instrumentation, triaging, and debugging
Instrumentation involves monitoring a product or system being tested or fuzzed, so if and when it does crash or behave strangely, the security researcher can be notified of this with supporting evidence. This is very important when thousands or even millions of different test cases are being run against a target; instrumentation is invaluable in helping to triage where the exact vulnerability lies and under which conditions it is triggered.
Triaging is the process of confirming the existence of software flaws and the location of the flaw in the code. For example, if a crash is experienced while fuzzing, triaging will involve re-running the test case which crashed the software to understand whether it’s repeatable. Once confirmed, various crash and memory dumps can be examined, to understand where in code the actual issue exists.
“I keep working on a problem no matter how many times I fail at it, even if it means hundreds of hours and thousands of failed attempts.”
Aaron Adams, Exploit Development Group
Debugging in security research will typically involve setting various break points in programs when triaging potential bugs. The researcher will eventually get to a point where a piece of code in memory is identified as vulnerable. Debugging will therefore help understand how inputs are being managed in the code and at which specific instructions they are being incorrectly written to or managed within memory.
It is important to remember that security research is not just about production of exploits or the discovery of vulnerabilities. Many security researchers spend their time developing tools which assist in instrumentation, triaging and debugging. For them, this is the challenge that they enjoy more than weaponising an actual exploit.
Exploitation
If after attending the exploit development course you are interested in further developing your exploitation skills, then the advanced exploit development course is the next step.
“You need creative thinking, not assuming you know everything; perseverance; and an unhealthy obsession with hex.”
Paul Collett, Exploit Development Group
To become proficient in exploitation you will need to understand everything from assembly and instruction sets to modern operating systems and how they work.
At advanced level you might also be interested in identifying new classes of vulnerability (a class being a type of vulnerability such as stack overflow, memory leak or null pointer dereference) – this will not be easy, but classes of vulnerability that are yet to be discovered might still exist.
The Exploit Development Group (EDG) at NCC Group is available to assist with development of exploits for vulnerabilities or to help with exploitation techniques.
“Exploit simple bugs such as stack overflows without mitigations like DEP, ASLR, and SMEP. Then add one mitigation at a time.”
Cedric Halbronn, Senior Researcher
Programming
Programming skills are not strictly required for security research; however to be fully adept as a security researcher one does need to be familiar with one or more languages. At a minimum, being familiar with syntax and semantics can serve you well; however often understanding the nuances of specific languages can go a long way.
Useful languages for people with little to no coding background include Python and Ruby.
If a lot of reverse engineering is going to be required within your security research then knowledge of C and C++ will be paramount.
There are a myriad of books available for learning different programming languages, in addition to many free online resources such as Code Academy[7].
As a security researcher it will also often be useful to be able to write your own tools or scripts to automate certain tasks. This capability might help you significantly reduce the time spent on operations such as fuzzing, triaging and debugging. Coding actual exploits as proofs of concept or to demonstrate risk will also require programming skills, likely in assembler.
Don’t reinvent the wheel – repurpose and modify existing code that already does a large percentage of what you need, but doesn’t yet include, for example, the security testing element (make sure you credit the original developers though).
“Write more exploits, either for contrived or real examples. Nothing beats experience.”
Nick Coffey, Exploit Development Group
Hardware
As a result of IoT, and the prevalence of electronics and software across devices from Smart TVs to modern and driverless vehicles, there now exists a wide attack surface involving hardware. From a research perspective the hardware aspect brings a whole new dimension and requires additional skills; examples include chip teardown, flashing firmware, and logic analysis. Knowledge of at least basic electronics is a required skillset in this domain. The core challenges in hardware research are getting access to the actual underlying code and being able to debug the hardware through dynamic analysis.
“Hardware hacking is all about privilege escalation… The interesting data is always in the software, we just go through the hardware to get at it.”
Rob Wood, NCC Group Global Hardware Lead
A new hardware hacking course is now available internally at NCC Group for those wanting to be introduced to the basic tools, techniques, and methodology of hardware hacking. Additionally, internal training is being developed around use of our GSM/GPRS interception capability and for automotive vehicle testing, which will provide training around modern vehicle control systems and how to research these for vulnerabilities.
Web applications and security
Security research also traverses web technologies. While most penetration testers will be familiar with application-layer flaws such as the OWASP Top 10, research can include finding flaws across commonly used frameworks such as content management systems (CMS). At a lower level, researchers might want to look for vulnerabilities in web server technologies.
“Learning by trial and error and being curious about understanding how a technology works under the hood drove me to research on web security for a number of years, and the result of it was most of the technology that now makes sqlmap the tool it is.”
Bernardo Damele, Creator of sqlmap
Because of the rich landscape of web applications and a constant stream of new frameworks that appear on the market (both open and closed source) there is a potential opportunity to find new classes of web vulnerability during research in this field. A good example from 2015 is the server-side template injection class of vulnerability, as discovered and reported by PortSwigger web security [8]. Finding a new class of vulnerability like this that can affect multiple technologies typically has a large impact across the security industry. NCC Group’s Hacking and Securing Web Applications course has been developed and refined over a number of years, and provides a thorough overview of the common security flaws in web applications and how to test for them.
Mobile
Mobile also provides a wide potential attack surface and therefore research potential. This includes research of the device security from a hardware and software perspective. Research here might include extracting sensitive data from the chipsets on the device, the potential for jailbreaking the device, or gaining full OS-level access. Additionally, exposed interfaces such as wireless and Bluetooth protocols might be fuzzed to find remotely-exploitable vulnerabilities in those protocol stacks. Today’s mobile devices have software similar to desktop computers, however exploitation of their software flaws usually requires knowledge of ARM assembly.
At the mobile app layer, research can be performed on application sandboxing, up to and including how apps store data and interact with remote web servers and services. Interestingly, mobile research traverses most disciplines across security research, from hardware right up to web services. The variation across different mobile manufacturers and operating systems (iOS, Android, Windows, etc.) also presents many interesting security research challenges due to their design differences – examples include finding ways of bypassing certificate pinning and other security controls offered by most mobile OS APIs.
Two NCC Group courses exist to provide consultants with the skills required to perform mobile assessments across Android and iOS. These are great courses to introduce those new to mobile security to the methodology and toolsets required to test and research these mobile platforms.
Cryptography
Cryptography is a discipline in its own right and presents a huge field of research. Cryptography will surface in various guises across most research tasks performed across most technologies, whether through use of hashing, encryption, or digital signing. Having at least a basic understanding of cryptography is essential to be effective in security research.
A course exists at NCC Group, which has been developed internally and delivered at Black Hat, named “Beyond the beast: deep dives into crypto vulnerabilities”. This course will provide those interested in cryptography with an understanding of the types of vulnerability that can arise within cryptographic implementations.
Additionally, NCC Group has a Cryptography Services (CS) practice which provides niche consultancy around cryptography in terms of reviews and assessments of design and implementation.
Malware and network forensics
Many people enjoy researching malware and understanding how Trojans and backdoors are delivered and used to compromise systems. The techniques involved in understanding malware are the same as in exploit development: reverse engineering, debugging, and so on. Cryptography usually plays a big part here, as malware might be packed and implement custom cryptography to protect itself and the network traffic it generates. This complexity is why many researchers enjoy the challenges in understanding malware behaviour.
“On x86 my tools of choice are IDA for reversing, IDA or OllyDbg for debugging. Ten years later there are still things I am learning about IDA.”
David Cannings, Cyber Defence Operations (CDO)
For those wanting to learn more in this field, the NCC Group Introduction to Malware Analysis course provides this grounding. There are also various malware challenges available on our internal wiki. NCC Group’s malware handling policy must be adhered to for any research performed on real-world malware.
Conclusion – and what next?
If you are new to security research, then hopefully this blog post has provided some ideas as to the areas in which you might get involved and how you can go about improving your skills in specific areas. A number of training courses exist internally at NCC Group across most disciplines to help people develop skills across these different research areas.
Get yourself a mentor. A number of senior members of staff at NCC Group have many years’ experience in security research and are happy to help with questions or direction in your chosen fields of research. Never be afraid to ask questions.
“Keep notes! So many times I come across stuff that I’m sure I once knew but can’t remember any more.”
Phillip Langlois, Exploit Development Group
Build yourself a research environment. Virtual machines have made security research significantly easier as they provide a way to separate research tasks from host builds, while their snapshotting facility is invaluable. In cases where research tasks crash or corrupt software, the VM can simply be reset to a known good state.
Go ‘old skool’. Look at old exploits and research from the past, including publicised exploits. See if you can find, triage, and exploit the same bugs in old software and compare your approach to those publicised.
To stay up to date with current research, follow blogs and Twitter feeds of security researchers within and outside of NCC Group. A list of 87 security experts who can be followed on Twitter is here [9].
Most importantly, have fun – learn, share knowledge, be ethical, and disclose responsibly.
“If we knew what it was we were doing, it would not be called research, would it?”
Albert Einstein
Happy bug hunting!
Acknowledgements
Many thanks to the Exploit Development Group for their input, comments and feedback on this document.
Thanks also to Andy Davis, David Cannings, Rob Wood, Tom Ritter, Aaron Grattafiori and Bernardo Damele for their valuable contributions.
- https://en.wikipedia.org/wiki/Full_disclosure_(mailing_list)
- https://bugcrowd.com/list-of-bug-bounty-programs
- https://www.zerodium.com/
- http://www.digitaltrends.com/mobile/zerodium-1-million-bug-bounty-claimed-news/
- https://github.com/nccgroup/Zulu
- https://github.com/keymandll/FuzzLabs
- https://www.codecademy.com/
- http://blog.portswigger.net/2015/08/server-side-template-injection.html
- http://www.securityinnovationeurope.com/blog/87-security-experts-you-need-to-be-following-on-twitter
Published date: 18 February 2016
Written by: Matt Lewis