Helping Engineering Teams Tackle Security Debt in Embedded Systems: U-Boot Configuration Auditing Introduced in Depthcharge v0.2.0

Depthcharge v0.2.0 is now available on GitHub and PyPi. This release introduces new “configuration checker” functionality and includes some major updates intended to improve usability. A tl;dr summary can be found in the CHANGELOG file. This blog post dives a bit more into the motivations for the changes, envisioned use-cases, and how this update fits into the longer-term vision for the project.

Motivation for Configuration Checker Functionality

The launch of Depthcharge v0.2.0 coincided with my Open Source Firmware Conference (OSFC) talk entitled, Guiding Engineering Teams Toward a More Secure Usage of U-Boot. (Click here for all OSFC 2020 talks!) While some of the talk will be familiar to those who tuned in for the Hardwear.io webinar, this presentation focused more on how inherited security debt can become security vulnerabilities.  In many instances, security risks emerge due to inconsistencies between the functionality enabled in the boot loader and either the intended security objectives of the product vendor, or the expectations of customers deploying the product within their own infrastructure.  Although products that ship with software left in a fully permissive state (with configurations borrowed from reference design platforms) represent one a disappointingly common extreme, bear in mind that hardened platforms leveraging secure boot flows do indeed exist, along with an entire spectrum of security postures in between.  As we looked at in the previous Depthcharge blog post – all it takes is one extraneous enabled feature in a U-Boot configuration to undermine all the hard work involved in properly extending a hardware-backed root of trust.

“How could ${product_vendor} possibly have included functionality that obviously undermines their security objectives?” you might ask.  It’s a fair question and one I’ve pondered over the course of numerous product security assessments. As I’ve come to appreciate, modern software supply chains can be very complex. It is not always the product vendor that is responsible for the boot loader shipped with a product – it may very well be the case that the engineering teams at the company whose name is on the product have little no knowledge about the inner workings of the product’s software. Instead, software and configurations may be accumulated as various stakeholders contribute different goods and services within the supply chain, each with their own key focuses and priorities.  This is illustrated in the following diagram, which lists a few possible stakeholders in the software supply chain, along with some of their corresponding focuses. 

A platform developed by an OEM or a silicon vendor’s third-party partner may be highly generic so that (derivations of) it can be adapted to meet the needs of their own diverse customer base. For a common base platform, the boot loader and kernel will likely have a significant number of features enabled, often in a highly permissive nature.  As we work left to right in this diagram, we move closer toward a place in the supply chain where a platform and its software stacks serve specific users, fulfill well-defined functional requirements, and can have sufficiently detailed threat models.  While silicon vendors and OEMs may not concern themselves with attack surface reduction and evaluating the cost of defending against physical threats, these very much should be concerns for product development teams. In some unfortunate situations the burden may be entirely left to the end-customer, who then needs to engage security professionals to audit products in order determine the degree to which they do or do not fulfill their needs, with respect to their threat model and security requirements.

With the above in mind, it is a bit easier to see how what is viewed as functionality, working as intended for some stakeholders can constitute a security vulnerability for others.  The context surrounding a product’s real-world usage shapes the threat model and ultimately establishes the way we define and evaluate potential risks.  As software and configurations propagate through a supply chain, accumulated technical debt can quickly become security debt if built-in functionality is not accounted for and evaluated within the context of verifiable security requirements.  The ability to quickly review software (configurations) and identify inconsistencies with respect to security objectives is invaluable in tackling security debt.  The earlier teams can do this within their product life cycle, the better.

The Nuts and Bolts of Depthcharge's Config Checker

Much of what was provided in the initial release of Depthcharge was aimed for an audience of security researchers and professionals; the framework and tools did not immediately serve engineering teams who were looking to improve the security posture of their product.  The new depthcharge.checker API and depthcharge-audit-config script intend to put more immediately useful tooling into the hands of development teams, not just security professionals.  The functionality is similar in spirit to that of kconfig-hardened-check – one can inspect their configuration to determine if functionality presenting risk is enabled. Using a tool such as this earlier in the product development phases can help identify features that inadvertently creep in and prevent products from shipping with “risky” functionality. This also means that time invested into security testing can be better spent focusing on new functionality or customizations, rather than playing U-Boot configuration whack-a-mole.

Depthcharge’s configuration checker functionality can currently parse two different types of configurations: .config files and configuration headers. The former item (implemented by the depthcharge.checker.UBootConfigChecker class) refers to the files produced by running make <board>_defconfig in newer U-Boot version that uses Kconfig-based infrastructure.  The latter item (implemented by depthcharge.checker.UBootHeaderChecker) parses the board-specific header files (and #included’d files) used by older versions of U-Boot, located in its include/configs/ directory. 

Adding support for additional input types is possible by implementing a new depthcharge.checker.ConfigChecker subclass that implement the load() method responsible for parsing the input and returning the configuration state in the documented format.  Alternate formats to parse could include the Depthcharge “device configuration” files produced by depthcharge-inspect, or even U-Boot binaries themselves.  If you can inspect it and infer configuration setting states – it’s fair game!

Based upon the configuration state and corresponding U-Boot version, the depthcharge-audit-config tool can emit potential risks in a few formats – HTML, CSV, and Markdown.  By default, the first two provide terse summaries more appropriate for the table-based formats, whereas the Markdown output provides more detailed descriptions and high-level recommendations.  Below is an example usage of the tool, using a configuration from a reference design.

$ git clone https://gitlab.denx.de/u-boot/u-boot.git
$ cd u-boot; git checkout v2018.07
$ export CROSS_COMPILE=arm-none-eabi-
$ make mx6sxsabreauto_defconfig
$ depthcharge-audit-config -V 2018.07 -u .config -o ~/example-output.md
 ...<snip>...
 27 potential security risks identified.
 Results written to /home/jynik/example.md

The resulting output contains a mix of both console commands that may pose risk in a production product, as well as CVEs affecting the v2018.07 code in the current configuration.  Two example items in the output are shown below. The “Source” entry notes where the relevant configuration setting was found, such as the CONFIG_CMD_NFS entry in the first excerpt, and the CONFIG_CMD_CRC32 in the latter.

By default, the HTML and CSV outputs forego the Description and Recommendation fields for brevity. This allows them to better serve as an “at a glance” table. Additionally, the Impact items shown in CSV and HTML output are abbreviated according to the scheme shown in the output of depthcharge-audit-config --impact-legend.

Working with older version of U-Boot can be a little bit more tedious, given that Depthcharge leverages the C preprocessor to collect all the relevant macro definitions. However, control over include paths and the ability to insert “dummy” header on-demand makes life a little bit easier.  In the below example, we inspect the configuration header for another development kit.

# Still in the U-Boot directory...
$ git checkout v2011.12
$ depthcharge-audit-config -V 2011.12 -H ./include/configs/mx35pdk.h 
                            -I include/ -I arch/arm/include 
                            --dummy-hdr asm/arch/imx-regs.h 
                            -o ~/example2.md
 [*] Running preprocessor to collect macro definitions
 17 potential security risks identified.
 Results written to /home/jynik/example2.md

In the case of i.MX-based platforms, the imx-regs.h file does not exist until it is symlinked at build time to a corresponding asm/arch-$(SOC)/imx-regs.h file.  Given that this file does not provide any information relevant to our configuration checking we can simply request that Depthcharge stage a “dummy” header in path with a higher precedence in the include search paths. In the above example, we use the --dummy-hdr option to specify that we want to create an empty asm/arch/imx-regs.h file (which will be later be deleted). Depthcharge will take care of creating the intermediate paths as needed and will take care of providing the include path to “dummy headers” to the C preprocessor.

So how does Depthcharge know about these potential security risks? With the current release there are a few handfuls of built-in depthcharge.checker.SecurityRisk definitions. We’ll plan to add more over time, but with thousands* of Kconfig items – hundreds** of which are associated with instances of U_BOOT_CMD() – it will take more time and effort to cover the most common subset of configuration options observed “in the wild.”  (The built-in definitions are an area where I’d love to see some pull requests, based upon your own experiences with different devices!)

# *thousands > Based upon the following silversearcher-ag invocation:
$ ag '^config [A-Z0-9_]+' -G Kconfig --no-heading --no-filename | 
    sed '/^$/d' | sort | uniq | wc -l

# **hundreds > Per the following semgrep query:
$ semgrep --lang c -e 'U_BOOT_CMD(...)'

However, this is where the “framework first, toolkit second” theme of the Depthcharge project shines.  Rather than only rely upon upstream definitions, one can build and maintain their own SecurityRisk definitions and load them into the configuration checkers at runtime via the register_handle() method.  This allows SecurityRisk objects to be reported based upon the U-Boot version in conjunction with the values of various configuration settings, as determined by their boolean state, string literal, regular expression match, or return value from a custom function that evaluates a configuration setting with respect to the values of a number of other settings. In this sense, the depthcharge-audit-config script represents a starting point and a functioning example of how one could build security-focused U-Boot configuration checker.  Contributions in the way of adding additional built-in SecurityRisk definitions will help make the tool more useful over time, while the ability to write custom scripts atop of the depthcharge.checker API allows security and engineering teams to tailor the tool to their specific needs.  Whether it’s for pruning undesirable functionality from a reference design configuration or supplementing existing continuous integration workflows to help catch security regressions – I hope you’ll find this helpful!

Usability Improvements & Other Major Updates

The version 0.2.0 release also comes with some big changes, intended to improve the usability of both the framework and tools built atop of it. While introducing API-breaking changes is not something we’ll want to make a habit of, in this case it was for the better. As you’ll note in the CHANGELOG, such changes are explicitly outlined, detailing what you need to know when crossing the v0.1.x to v0.2.x boundary.

In the interest of making Depthcharge’s device inspection behavior more intuitive, v0.2.0 transitions away from requiring users to opt-out of operations that are more likely to cause a target device to spiral out of control in a cloud of cryptic error messages.  Instead, a more conservative set of operations are enabled by default and one must explicitly opt-in to leverage additional functionality.  For example, when running depthcharge-inspect on an ARM target, one will now need to specify --arch arm -AR in order to leverage all available functionality.  The -A/--allow-deploy option permits Depthcharge to deploy and execute binary payloads as-needed. The -R/--allow-reboot argument permits the use functionality such as  DataAbortRegisterReader implementations, which leverages crash-and-reboot behavior in order to locate the address of U-Boot’s global data structure.  If Depthchage recognizes that some functionality is available on the target device, but the operator has not opted to use it, a warming similar to the following will be printed.

[!] Excluded: CpMemoryWriter - Operation requires crash or reboot, but opt-in not specified.

In moving to toward a paradigm of opt-in semantics, it became clear that a subset of available functionality could be used on architectures that were otherwise not yet technically supported. As such, the depthcharge.arch was refactored and now contains a collection of “generic” architectures that define only memory endianness, word sizes, and access-alignment requirements. By introducing these “generic” architecture variants, it means that higher-level console functionality, such as SetexprMemoryReader or CpMemoryWriter, can still be used on a target even if support for the relevant architecture is not yet complete enough support more invasive operations.  As a bonus, this simplifies the path moving forward for adding support for various architectures (e.g. MIPS, Power, AARCH64).  With this change, 32-bit ARM is no longer the default architecture, “generic” is. This new default should hopefully be much more resilient to platform-specific quirks, making for a gentler experience for those getting just started with Depthcharge.

Once the above changes were introduced, I was delighted to hear that the students in Matthew Alt’s hardware hacking course were able to use a bleeding-edge version of Depthcharge (always available in the next branch) to not only write their own custom scripts to dump code and data from a MIPS-based target, but also implement their own support for re-flashing the device using its vendor-specific spi command!

The Future of Depthcharge

The issue tracker contains a couple bugs worth squashing, as well as a list of features that I’d like to see integrated at some point. In the coming months, I hope to sort these into a subset for a v0.3.0 milestone, perhaps with some v0.2.x patch releases as bugs are reported and squashed.

The future of this project will largely be driven by the needs of the NCC Group Hardware and Embedded Security Services team, my own needs in personal device hacking adventures, and of course – the greater community of security-focused minds that wish to participate in the project.  Whether it be through bug reports, pull requests, or just brainstorming – I look forward to seeing how folks will put Depthcharge to use and help shape its future.

P.S. - I love collecting Depthcharge device configuration files and U-Boot binaries from various platforms – feel free to hit me up on Twitter or via jon.szymaniak.foss(a)gmail.com!