Cisco ASA series part three: Debugging Cisco ASA firmware

This article is part of a series of blog posts. We recommend that you start at the beginning. Alternatively, scroll to the bottom of this article to navigate through the whole series.

We have developed a small framework of tools to automate the debugging of most Cisco ASA firmware files using gdb, while supporting both real ASA devices and emulated (using GNS3). These tools can allow us to automatically boot specific firmware on a device, set definite breakpoints in gdb at startup, pre-load certain analysis tools, and more. If you want a summary of these tools, you can go to the end of this article.

Debugging with gdb

Most firmware already contain a gdbserver binary. Whether the ASA is real or emulated needs to be taken into consideration while debugging. The main difference between debugging a real ASA and an emulated ASA is the port gdb connects to. A real device will be debugged using a serial line while a GNS3 instance will be debugged over TCP/IP (telnet).

Suppose we extracted the firmware as detailed in part two and that the ASA firmware is patched to attach gdbserver at boot. We get the following at boot:

Debugging emulated ASA

We start gdb and indicate that we are going to attach to the lina executable and set the path to the libraries it depends on. Then we attach to the remote gdbserver:

$ gdb --quiet
(gdb) file _asav941-200.qcow2.extracted/rootfs/asa/bin/lina
Reading symbols from _asav941-200.qcow2.extracted/rootfs/asa/bin/lina...(no debugging symbols found)...done.
(gdb) set solib-absolute-prefix _asav941-200.qcow2.extracted/rootfs/
(gdb) target remote 192.168.5.1:12005
Remote debugging using 192.168.5.1:12005
Reading symbols from _asav941-200.qcow2.extracted/rootfs/lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
0x0000003655201190 in ?? () from _asav941-200.qcow2.extracted/rootfs/lib64/ld-linux-x86-64.so.2

By default, the lina process won't let us debug it and will trigger signals like this due to a watchdog mechanism:

(gdb) c
Continuing.

Thread 2 received signal SIG38, Real-time event 38.
[Switching to Thread 1512]
0x00000036556e7da7 in epoll_pwait () from _asav941-200.qcow2.extracted/rootfs/lib64/libc.so.6
(gdb) c
Continuing.

Thread 8 received signal SIG38, Real-time event 38.
[Switching to Thread 1513]
0x00000036556e7da7 in epoll_pwait () from _asav941-200.qcow2.extracted/rootfs/lib64/libc.so.6

In order to debug it and avoid any similarly triggered signals, we need to set the internal firmware variable clock_interval to zero at boot before continuing execution, as detailed by Exodus Intel [1]:

(gdb) set *0x03cc1d00 = 0
(gdb) c
Continuing.

Synchronisation with IDA

As discussed in part two, most ASA firmware files don't have symbols in lina , which is the main executable we are interested in. This is confirmed in the gdb trace above when it loads lina. In order to make debugging easier, we used ret-sync, which is a framework used to synchronise a debugger (gdb in our case) with IDA Pro. We added a bunch of features that were merged in the main repository [2].

(gdb) source asa_sync.py 
[sync] HOST, PORT: 192.168.5.1, 9100
[sync] BIN_NAME: asav941-200.qcow2
[sync] MAPPING: [[4194304, 128455000, 124260696, 'asav941-200.qcow2|lina']]
[sync] commands added
(gdb) sync
[sync] Tunnel to IDA initializing...
[sync] sync is now enabled with host 192.168.5.1
[sync] pid: 200

We added some gdb commands to ret-sync such as:

• cc: continue to the cursor position in IDA Pro
• bbt: beautiful backtrace after requesting symbols from IDA Pro
• patch: patch some bytes in IDA in order to reflect live information

For instance, the difference between bt and bbt is shown below:

(gdb) bt
#0 0x0000000000a91a73 in ?? ()
#1 0x0000000000a6d994 in ?? ()
#2 0x0000000000a89125 in ?? ()
#3 0x0000000000a8a574 in ?? ()
#4 0x000000000044f83b in ?? ()
#5 0x0000000000000000 in ?? ()
(gdb) bbt
#0 0x0000000000a91a73 in IKE_GetAssembledPkt ()
#1 0x0000000000a6d994 in catcher ()
#2 0x0000000000a89125 in IKEProcessMsg ()
#3 0x0000000000a8a574 in IkeDaemon ()
#4 0x000000000044f83b in sub_44F7D0 ()
#5 0x0000000000000000 in ()

We decided not to use pwndbg [3] even though it is a good gdb plug-in that enhances the gdb experience. Unfortunately, since we rely on the serial line for debugging real hardware, it is way too slow as pwndbg needs lots of reads/writes in memory to enhance the gdb experience and add all the additional commands/output.

Hunting for addresses

Logging functions

Our analysis was originally focused on ASA 9.2.4, which does not have symbols. However, as pointed out by Exodus Intel [1], you can use debugging strings and an IDA Python script to rename several functions, as is often the case when analysing embedded firmware. The idea here is that you identify all cross-references to logging function(s) and automatically find the debug string parameter. This string will often follow some pattern that indicates the name of the calling function. This won't give you all the functions, though.

Logging function

Note that the name ikev2_log_exit_path comes from a firmware with symbols that we will detail below. In the example above, we determine that the calling function is ikev2_parse_config_payload.

Symbols from other firmware

We noticed that some recent 64-bit firmware files include symbols. This allows porting function names to older firmware files by matching the unique strings referenced by the functions. For example, we identify the following functions in asav941-200.qcow2:

• sub_15864D0 = c_aaa
• sub_237A050 = lic_ct_proxy_restricted
• sub_238ABB0 = cp_printf

Comparison of two firmware

The manual process of renaming functions is also always possible and we used additional tools to help us do this, as described below.

We modified an IDA Python script based on @Heurs’ MACHOC algorithm [4] to match functions based on their Control Flow Graph (CFG). It takes less than ten minutes to generate the list of hashes for source/destination and do the comparison/apply the new names. To illustrate the results, we use the following firmware:

• asa924-k8.bin: 32-bit
• asa924-smp-k8.bin: 64-bit
• asav941-200.qcow2: GNS3 64-bit
• asav962-7.qcow2: GNS3 64-bit with symbols

We ported the symbols from asav962-7.qcow2 to the other firmware. The results are detailed below.

Source	Destination	Src hashes	Dest hashes	Imported names
asav9627	asav941200	103449	70637	18932
asav9627	asa924-smp	103449	71231	4455
asa924	asa924-smp	57636	71231	434

It is possible to port symbols using the BinDiff tool [14], however in our experience the analysis can take up to six hours to complete and during that time it will appear to be hung.

Automated idb symbol renaming & lookup

We wrote a tool named idahunt [5] to help us automatically generate databases for the various symbols we need for analysis and exploitation. This tool executes IDA Pro in batch mode and supports execution of external IDA Python scripts. It was useful when we analysed 170+ lina binaries from various firmware files, and has been made in such a way that it can be used for projects other than ASA firmware.

To make idahunt work for your needs, you must define a filter in Python that dictates which files in a directory hierarchy will be analysed by IDA, and whether or not the 32-bit or 64-bit version of IDA should be used for analysis.

You would normally start by generating idb files for all of the files you want to run scripts against, and this also lets you test your filter. For example, a command like ./idahunt --inputdir /path/to/asa_firmware --analyse --filter ciscoasa.py would parse all of the ASA firmware, identify the lina files in each firmware that we want to analyse, determine if it is 32-bit or 64-bit, and then generate an idb for each. Refer to idahunt [5] repository for more information.

Once we have our idbs, we can use a rename IDA Python script [6] to find and mark as many symbols as possible. We leverage the logging functionality inside lina that was mentioned earlier to rename functions. We look for static string references, function cross-references and code/register patterns, and then parse the idbs for patterns and rename the appropriate sections.

After the renaming stage, a separate IDA Python script [7] can be used to look up specific symbols, some of which shown below, that are needed for our ASA database to populate it automatically. We use the clock_interval value in a gdb script to patch the watchdog timeout to allow debugging any firmware. We use the aaa_admin_authenticate and socks_proxy_server_start values to support a special debug shell. We will describe it in more detail later in this blog post.

The following is the output of hunting for symbols for firmware asa924-k8.bin (referred to as asadb.json):

{
   "addresses": {
        "clock_interval": 38760808, 
        "mempool_array": 57678848, 
        "socks_proxy_server_start": 19283008, 
        "aaa_admin_authenticate": 252672 }, 
   "fw": "asa924-k8.bin", 
   "version": "9.2.4", 
   "imagebase": 134512640,
}

In this way, we can easily data mine information across hundreds of firmware versions with very little manual involvement. We can add new symbols and update the ASA database with minimal effort. This approach is useful with any large corpus of binaries or firmware and we have done similar automated analysis on Android firmware in the past, as part of our libstagefright exploitation research [15].

asadbg

Our framework automates booting a specific firmware when several firmware files are preloaded on the flash (such as a CF card inserted into the ASA device). It supports patching clock_interval at boot and loading additional gdb scripts to add breakpoints, etc.

For debugging a GNS3 emulator, we define an asadbg.cfg configuration file similar to below. It contains the GNS3 IP/port for the configured firewall that we obtain from GSN3 itself.

[GLOBAL]
gns3_host=192.168.5.1
asadb_file=asadb.json

[asav941200]
version=941200
arch=gns3
rootfs_path= /home/user/_asav941-200.qcow2.extracted/rootfs
gns3_port=12005
attach_gdb=yes

Note that we indicate we want to attach gdb. The asadb_file entry points to the ASA database asadb.json file, which contains the addresses required. What is important here is that the version=941200 and arch=gns3 specified in asadbg.cfg allow us to match the firmware in asadb.json below:

{
   "ASLR": false, 
   "addresses": {
       "clock_interval": 59514112, 
       "socks_proxy_server_start": 22139744, 
       "aaa_admin_authenticate": 418288
   }, 
   "fw": "asav941-200.qcow2", 
   "imagebase": 4194304,
   "version": "9.4.1.200",
   "arch": 64
}

Now we start debugging and clock_interval will be automatically patched:

asadbg$ ./asadbg.py --name asav941200 --asadbg-config asadbg.cfg
[asadbg] Using config file: asadbg.cfg
[asadbg] Found section: 'asav941200' in config
[asadbg] Using gdb: '/usr/bin/gdb'
[asadbg] Using architecture: gns3
[asadbg] Trying lina: /home/user/_asav941-200.qcow2.extracted/rootfs/asa/bin/lina
[asadbg] Going to debug...
[asadbg] Using GNS3 emulator 192.168.5.1:12005
[asadbg] Starting gdb now...
[gdbinit_941200] Configuring paths...
[gdbinit_941200] Disabling pagination...
[gdbinit_941200] Connecting over TCP/IP...
0x0000003655201190 in ?? () from /home/user/_asav941-200.qcow2.extracted/rootfs/lib64/ld-linux-x86-64.so.2
[gdbinit_941200] Connected.
[gdbinit_941200] Watchdog disabled
[gdbinit_941200] heap debugging plugins loaded
[gdbinit_941200] Additional gdb scripts loaded
[gdbinit_941200] Done.
(gdb) c
Continuing.

For debugging real hardware, we define a similar configuration, except it uses a serial port:

[GLOBAL]
serial_port=/dev/ttyUSB0
asadb_file=asadb.json

[asa924-gdb]
version=924
arch=32
rootfs_path=/home/user/_asa924-k8.bin.extracted/rootfs
firmware=asa924-k8-debugshell-gdbserver.bin
config=config-924
firmware_type=gdb
attach_gdb=yes

We need to specify a firmware_type which indicates if the firmware is unmodified, rooted or if gdb has been enabled at boot. This is because we will do different things at boot depending on the format. Here we can see that we use a modified firmware asa924-k8-debugshell-gdbserver.bin which has gdbserver enabled at boot and contains a debug shell (more on this later). We see below that when the bootrom starts, asadbg automatically interrupts the sequence in order to load the firmware and the config files already on the CF card.

asadbg$ ./asadbg.py --name asa924-gdb --asadbg-config asadbg.cfg
[asadbg] Using config file: asadbg.cfg
[asadbg] Found section: 'asa924-gdb' in config
[asadbg] Using gdb: '/usr/bin/gdb'
[asadbg] Using architecture: 32
[asadbg] Trying lina: /home/user/_asa924-k8.bin.extracted/rootfs/asa/bin/lina
[asadbg] Going to debug...
[asadbg] Using serial port: /dev/ttyUSB0
[asadbg] Loading 'asa924-k8-debugshell-gdbserver.bin' with 'config-924'...
[comm] Waiting boot...
[SNIP]
Platform ASA5505

Use BREAK or ESC to interrupt boot.
Use SPACE to begin boot immediately.
Boot interrupted.

[SNIP]
rommon #0> boot asa924-k8-debugshell-gdbserver.bin cfg=config-924
Launching BootLoader...
Boot configuration file contains 1 entry.

Loading asa924-k8-debugshell-gdbserver.bin........ Booting...
Platform ASA5505
[SNIP]
SMFW PID: 514, Starting /asa/bin/lina under gdbserver /dev/ttyS0
SMFW PID: 512, started gdbserver on member: 514//asa/bin/lina
SMFW PID: 512, created member ASA BLOB, PID=514
Process /asa/bin/lina created; pid = 517
Remote debugging using /dev/ttyS0
[comm] gdb detected - boot finished.
[comm] Boot should be finished now?
[asadbg] Starting gdb now...
[gdbinit_924] Configuring paths...
[gdbinit_924] Disabling pagination...
[gdbinit_924] Connecting over USB...
0xdc7e2820 in ?? () from /home/user/_asa924-k8.bin.extracted/rootfs/lib/ld-linux.so.2
[gdbinit_924] Connected.
[gdbinit_924] Watchdog disabled
[gdbinit_924] heap debugging plugins loaded
[gdbinit_924] Additional gdb scripts loaded
[gdbinit_924] Done.
(gdb) c
Continuing.

Debugging different versions on real hardware is made possible by dropping all the asa*.bin firmware on to the flash (the CF card). Then asadbg.py can automatically load the right version at boot.

Debug shell

One problem when debugging real ASA devices with gdb over serial is that CTRL^C does not seem to work. In other words, it is impossible to set breakpoints after lina is running. A possible reason is that the CTRL^C character is not sent correctly on the serial line.

To work around that, we patch lina to execute a reverse shell every time we connect over SSH. We do this by patching aaa_admin_authenticate() in lina with connect-back shellcode based on Exodus Intel research [8] that we also ported to 64-bit. This allows us to have a Linux root shell in addition to the usual Cisco ASA CLI.

Any attempt to authenticate over SSH will trigger the reverse shell:

$ ssh user@192.168.1.77
user@192.168.1.77's password: 
Type help or '?' for a list of available commands.
ciscoasa>

Our listening connection gets a connection and initialises the debug shell. We get lina's, process ID and send a SIGTRAP signal to lina:

$ nc -lvp 4444
Listening on [0.0.0.0] (family 0, port 4444)
Connection from [192.168.1.77] port 4444 [tcp/*] accepted (family 2, sport 22488)
/bin/sh: can't access tty; job control turned off
# ps | grep lina
 512 root      /asa/bin/lina_monitor -l -g -s /dev/ttyS0 -d
 514 root      gdbserver /dev/ttyS0 /asa/bin/lina -p 512 -t -g -l
 517 root      /asa/bin/lina -p 512 -t -g -l
# kill -5 517

As expected, gdb catches the signal and returns us a gdb prompt, without having to use CTRL^C.

(gdb) c
Continuing.

Thread 1 received signal SIGTRAP, Trace/breakpoint trap.
[Switching to Thread 517]
0xffffe430 in __kernel_vsyscall ()
(gdb) bt
#0 0xffffe430 in __kernel_vsyscall ()
#1 0xdc6f2e84 in pthread_cond_wait () from _asa924-k8.bin.extracted/rootfs/lib/libpthread.so.0
#2 0x090fa424 in ?? ()
#3 0x090f4729 in ?? ()
#4 0x090eaec2 in ?? ()
#5 0x090eb582 in ?? ()
#6 0xdbf6e6b5 in __libc_start_main () from /_asa924-k8.bin.extracted/rootfs/lib/libc.so.6
#7 0x0804ea21 in ?? ()

Even though the slides are not public, it is interesting to note that huku presented a similar workaround at Warcon 2017 when he had to deal with gdb debugging on Android devices.

Bypassing boot process verification on GNS3

As Alec detailed in his presentation’s “Uploading the Firmware” slides [9], there is a checksum verification for firmware when uploading over FTP/TFTP/SCP. This is almost certainly used to detect transmission errors. Alec also pointed out that there is no boot process verification when loading the firmware, except on new hardware.

You may have noticed that, while modifying the lina ELF in the asa*.bin to add a debug shell, we found there is no such verification check at boot. Despite our changes, we still managed to load it on a real ASA5505 hardware. Moreover, we experienced a similar behaviour for the ASA5512-X hardware (64-bit). Alec mentioned that ASA 5506-X, 5508-X, and 5516-X perform 'secure boot' verification, but we have no indication of what that is.

If we try to do the same with firmware for GNS3 (e.g. asav962-7.qcow2), we get an error at boot:

Error message for a patched lina

We locate the error message in lina_monitor.

.text:3939 loc_3939:                             ; CODE XREF: main+293
.text:3939    call    sub_A7B0
.text:393E    test    eax, eax
.text:3940    jz      loc_39D4
.text:3946    lea     rdi, aLoading___ ; "nnLoading...n"
.text:394D    xor     eax, eax
.text:394F    call    _printf
.text:3954    lea     rdi, aAsaBinLina ; "/asa/bin/lina"
.text:395B    call    code_sign_verify_signature_image
.text:3960    test    eax, eax
.text:3962    mov     ebx, eax
.text:3964    jz      short loc_39B6
.text:3966    call    sub_A6E0
.text:396B    mov     edi, ebx
.text:396D    test    eax, eax
.text:396F    jnz     loc_3BFB
.text:3975    call    sub_A5E0
.text:397A    lea     rdi, aTheDigitalSign ; "nThe digital signature of the booted image file did not verify successfully. %d (%s)n"
.text:3981    mov     esi, ebx
.text:3983    mov     rdx, rax
.text:3986    xor     eax, eax
.text:3988    call    _printf
.text:398D
.text:398D loc_398D:                             ; CODE XREF: main+5E3
.text:398D    lea     rdi, aRebootingNow__ ; "Rebooting now ...nn"
.text:3994    xor     eax, eax
.text:3996    call    _printf
.text:399B    mov     edi, 5           ; seconds
.text:39A0    call    _sleep
.text:39A5    jmp     loc_38C5
.text:39AA
.text:39AA loc_39AA:                             ; CODE XREF: main+2B5
.text:39AA    call    sub_185F0
.text:39AF    mov     edi, ebx          ; status
.text:39B1    call    _exit
.text:39B6 loc_39B6:                             ; CODE XREF: main+334
.text:39B6    call    sub_A6E0
.text:39BB    test    eax, eax
.text:39BD    jnz     loc_3BE8
.text:39C3    lea     rdi, aTheDigitalSi_0 ; "The digital signature of the running image verified successfullyn"
[SNIP]

As you see above, if the signature check fails at 395B, the instruction at 3964 does not jump to 39B6 and instead it makes the device reboot. We can patch the jz short loc_39B6 to be jmp short loc_39B6 in lina_monitor to force it to boot. Since there is no other check, it boots correctly.

Automating asadbg

One of the really useful aspects of having the automation that asadbg provides is that you can then, in turn, automate using asadbg itself.

Consider the scenario where you want to exploit a bug and test whether your exploit works reliably across as many lina versions as possible. You can preload a CF card with all of the target firmware files and then loop over a list, directing asadbg to boot each one.

All of the steps can be automatically performed. After the current firmware is loaded, you can run your exploit against the target firmware, generate whatever data you may need using scripts loaded by asadbg and then reboot the device to load the next firmware (which you can do by using comm.py to issue a reboot command).

Similarly, you can use these scripts to aid in building fuzzers that can collect enough information about live device crashes to aid with bug triage and automatically reset them upon entering certain states.

asadbg at a glance

As demonstrated earlier, asadbg is quite useful for automating the debugging of ASA firmware files:

• It leverages the data mining from asafw detailed in a previous blog post.
• It supports an asadbg.cfg configuration file to enable easy debugging of different versions.
• It supports both real hardware and GNS3.
• Optionally, it uses ret-sync to allow a better debugging experience from IDA Pro.
• It supports executing additional gdb scripts at boot.
• It provides libraries that can be useful for automating tests across multiple firmware versions on real hardware.

Conclusion

We detailed asadbg, which is a framework to debug Cisco ASA devices and emulators using GNS3. You can access the tool in our GitHub [10] repo.

In order to understand the IKE fragmentation bug (CVE-2016-1287), as well as carry out general analysis of Cisco ASA behavior, we wanted a better way to look at heap activity on the device. We decided to build gdb Python plugins, along the lines of other heap tools [11][12][13]. These can be plugged into asadbg for use and we will detail them in future blog posts.

We would appreciate any feedback or corrections. If you would like to contact us we can be reached by email or twitter: aaron(dot)adams(at)nccgroup(dot)trust / @fidgetingbits and cedric(dot)halbronn(at)nccgroup(dot)trust / @saidelike.

---

Read all posts in the Cisco ASA series

• Cisco ASA series part one: Intro to the Cisco ASA
• Cisco ASA series part two: Static analysis & datamining of Cisco ASA firmware
• Cisco ASA series part three: Debugging Cisco ASA firmware
• Cisco ASA series part four: dlmalloc-2.8.x, libdlmalloc, & dlmalloc on Cisco ASA
• Cisco ASA series part five: libptmalloc gdb plugin
• Cisco ASA series part six: Cisco ASA mempools
• Cisco ASA series part seven: Checkheaps
• Cisco ASA series part eight: Exploiting the CVE-2016-1287 heap overflow over IKEv1

References

[1] https://www.slideshare.net/CanSecWest/csw2016-wheeler-barksdalegruskovnjakexecutemypacket

[2] https://github.com/bootleg/ret-sync/commit/55a3fd62708b3b9e24d3b44ed160343a2d5126c0

[3] https://github.com/pwndbg/pwndbg

[4] https://github.com/0x00ach/idadiff/

[5] https://github.com/nccgroup/idahunt

[6] https://github.com/nccgroup/asadbg/blob/master/asadbg_rename.py 

[7] https://github.com/nccgroup/asadbg/blob/master/asadbg_hunt.py

[8] https://blog.exodusintel.com/2016/02/10/firewall-hacking/

[9] https://github.com/alec-stuart/BreakingBricks/blob/master/Breaking Bricks SEC-T15.pdf

[10] https://github.com/nccgroup/asadbg

[11] https://github.com/nccgroup/libtalloc

[12] https://github.com/argp/unmask_jemalloc

[13] https://github.com/cloudburst/libheap

[14] https://www.zynamics.com/bindiff.html

[15] https://research.nccgroup.com/wp-content/uploads/2020/07/libstagefright-exploit-notes.pdf

Published date:  02 October 2017

Written by:  Aaron Adams and Cedric Halbronn