Testing HTTP/2 only web services

Many web servers are using HTTP/2 but few current web application penetration testing tools support it. In most cases, the common workaround is simple - perform most of the testing of the application and its logic using HTTP/1.x and then perform additional testing for HTTP/2 specific vulnerabilities and requests that are handled differently if HTTP/2 is used as an additional task.

This is all fairly straightforward if the target supports HTTP/1.x, as most HTTP/2 speaking targets do, but what if it doesn’t? This was the case when we were asked to look at a third-party Internet of Things (IoT) device for a client who was curious as to what the device was sending back to its vendor.

HTTP/2 is a relatively recent major revision of HTTP which originated from the Google-developed SPDY protocol [1], adopted by the IETF in RFC 7540 [2]. It is targeted at superseding HTTP/1.1 in applications that require increased speed, higher compression and lower latency. It breaks from the plaintext-based request/response model of HTTP/1.x, hence current tooling has relatively little support for it due to the design changes which would be required. The adoption of HTTP/2 is gradually increasing, but remains relatively limited due to the scale of the web. Several major websites have adopted support for it however, increasing the amount of HTTP/2 traffic on the Internet [3].

The vendor was a large Silicon Valley tech company with a mature cybersecurity posture and an open bug bounty programme with a scope including the device. There was an SDK available for adding third-party functionality to the device, but the device itself and the vendor’s web services were proprietary and undocumented. They used HTTPS with certificate pinning; however, when this was bypassed we found that both endpoints only communicated via HTTP/2, likely due to them being a closed system of recent design, which was not designed for third party interaction and required no backwards compatibility.

Diverting traffic

As the device did not have a user interface where a proxy could be set, it was necessary to use invisible proxying. A fake DNS server was set up to divert all requests to the attacker’s laptop. DHCP was used to provide this DNS server to the device. Wireshark was used to determine which ports the device tried to hit and Burp Suite, OWASP ZAP or other alternatives could be used if preferred) was used to open HTTPS capable invisible proxy ports on all relevant ports. The device connected to the proxy ports but did not successfully negotiate SSL/TLS connections due to certificate failure.

Certificate pinning

Firstly, the vendor was using certificate pinning (as we would recommend any vendor building such a device do) so this had to be bypassed. Bypassing the certificate pinning check would be possible by changing the pinned SSL certificate to one that we had generated or getting the device to skip the verification. This would require modifying the flash of the device.

Helpfully the vendor had included a reference device implementation with the SDK. Due to time constraints we chose to attack this as it was at least very similar to the implementation on the device and communicated with the same web service. Even if it wasn’t identical, attacking the reference implementation which uses the same web service would be a very useful first step. Analyzing device firmware updates revealed the reference implementation to be at least very similar and a teardown revealed that removing the flash so that we could write to it would be fiddly and time consuming but probably not too difficult.

Others had completed this step, confirming that the flash wasn’t encrypted, but they had not publicly shared their firmware dumps. Bypassing the certificate pinning in the reference implementation was relatively easy and it was hard to imagine that the actual device would be much different, providing the ability to write files to its memory was available.

Interception

Now that we could see the unencrypted contents of the traffic, our next challenge was to view and analyze it with the goal of progressing to interception and modification. Burp Suite displayed requests, but they all consisted of the device trying to upgrade the connection to HTTP/2 as per the HTTP/2 RFC. We attempted to make HTTP/1.1, HTTP/1.0 and even HTTP/0.9 requests to the server and responses to the device but this failed. None of the intercepting web proxy tools available to penetration testers at the time (or at this time of writing) properly support HTTP/2 despite users requesting the feature for several years. This is probably due to HTTP/2 departing from the traditional request/response model of HTTP/1.1 and before, requiring a significant design change in intercepting proxy tools and most applications supporting HTTP/1.1 anyway, rendering support for HTTP/2 a low priority.

Proxying HTTP/2

There is however, a way to convert traffic between HTTP/2 and previous HTTP protocol implementations using nghttpx, a tool normally intended as a forward or reverse proxy for load-balancing applications.

Proxy Chains

Normally, an intercepting proxy is used in the following arrangement during penetration tests:

CLIENT:[HTTP1] <–> [HTTP1]:ATTACKER PROXY:[HTTP1] <–> [HTTP1]:SERVER

Using nghttpx as an intermediate server, we can convert HTTP/1.1 to HTTP/2 traffic and vice versa. We therefore place an nghttpx proxy server in between device and attacker, then a second one between server and attacker, chaining the proxy servers together. For simplicity, in this post we refer to the proxy servers in relation to ourselves, the attacker. The proxy server between the device and the attacker is named the downstream proxy server. The proxy server between the attacker and the server is named the upstream proxy server

The attacker therefore receives a series of HTTP/1.1 requests from the device via the downstream proxy server and must forward these to the vendor server via the upstream proxy server. In return, the attacker receives HTTP/1.1 requests from the vendor server via the upstream server and forwards them to the device via the downstream proxy server. In reality, the device and vendor server only speak HTTP/2, but nghttpx is used as a translation layer to allow the attacker to make use of existing tooling.

Our intercepting proxy arrangement is therefore changed to a chain of proxies:

CLIENT:[HTTP2] <–> [HTTP2]:nghttpx:[HTTP1] <–> [HTTP1]:ATTACKER 
PROXY:[HTTP1] <–> [HTTP1]:nghttpx:[HTTP2] <-> [HTTP2]:SERVER

Issues

This is not a perfect solution, as it appeared to break when proprietary commands were sent, or HTTP/2 data streams exceeded a certain size or time. An amount of lag was experienced too. Nevertheless, it provided a large amount of valuable information (from an attacker’s perspective) and in this instance we were able to make a large number of successful commands. With further time, a fork of nghttpx could probably be modified to fix most of these issues.

We also faced the issue of our device contacting multiple servers belonging to the vendor, without providing host headers in the requests. This was solved in an inelegant but effective way of building up multiple chains of proxies and resolving device requests to different DNS names to downstream proxies in different chains. For simplicity during configuration and as resource usage was not a priority, each nghttpx instance was hosted within its own Debian VM. If resource usage is a concern then several proxy servers could probably be hosted on the same VM or the use of containerization could be explored.

Proxy Configuration

The upstream and downstream proxies in each chain were configured with slightly different settings due to their differing roles. We successfully used the configuration settings in the following section.

Nghttpx [4] can be installed on debian derived distributions (including Kali Linux) using ‘apt-get install nghttp2’ as root or via sudo.

The default configuration file for nghttpx is /etc/nghttpx/nghttpx.conf on a Debian system. We configured the file as below and restarted the nghttpx service using ‘service nghttpx restart’ as root or via sudo. We recommend reading the nghttpx manpage for additional options that may be useful for other situations.

Downstream

The configuration file on the downstream server was set up similar to the following example. In particular, the instance was configured to have long timeouts / large window sizes due to what it was used for.

We used the following file on a downstream proxy VM in one of our chains and explain some of the statements in greater detail below.

frontend=192.168.140.10,443;
frontend=192.168.140.10,80;no-tls
backend=192.168.41.1,9000;;proto=http/1.1
backend-keep-alive-timeout=5m
frontend-http2-read-timeout=5m
frontend-read-timeout=5m
frontend-write-timeout=5m
stream-read-timeout=5m
stream-write-timeout=5m
backend-read-timeout=5m
backend-write-timeout=5m
backend-connect-timeout=5m
listener-disable-timeout=5m
backend-http2-window-size=134217720
backend-http2-connection-window-size=1073741760
frontend-http2-window-size=134217720
frontend-http2-connection-window-size=1073741760
http2-proxy=no
private-key-file=/etc/nghttpx/target.key
certificate-file=/etc/nghttpx/target.crt
cacert=/etc/nghttpx/cacert.crt
accesslog-syslog=yes
errorlog-syslog=yes
workers=5

We create the listeners for requests from the device using ‘frontend’ statement lines in the configuration file. The ‘backend’ statement line points upstream to the intercepting proxy and tells the instance it should use HTTP/1.1 (note the ‘proto’ option) with no TLS, as we are not worried about third party eavesdroppers in this case and have no desire to complicate matters further with TLS configuration issues.

The statements ‘accesslog-syslog=yes’ and ‘errorlog-syslog=yes’ tell the instance to log for debug purposes. The service can also be called with -L INFO or modify the service startup script to do the same. This produces useful output in the syslog which can be used to help determine why something isn’t working.

TLS certificates were generated for the downstream proxy server so that we could support HTTPS connections from the client device. The statement ‘private-key-file=/etc/nghttpx/target.key’ sets the server’s private key, ‘certificate-file=/etc/nghttpx/target.crt’ configures the corresponding TLS server certificate and cacert=/etc/nghttpx/cacert.crt configures the TLS CA certificate. The client must be configured to accept these if HTTPS is used.

Upstream

Many of the settings for the upstream proxy server are similar to the downstream server with a few notable exceptions. Most of the relevant configuration flags are therefore explained in the section for the downstream server.

We set the configuration file as follows:

frontend=192.168.41.10,8080;no-tls
backend=vendor.example.com,443;;tls;proto=h2
backend-keep-alive-timeout=5m
frontend-http2-read-timeout=5m
frontend-read-timeout=5m
frontend-write-timeout=5m
stream-read-timeout=5m
stream-write-timeout=5m
backend-read-timeout=5m
backend-write-timeout=5m
backend-connect-timeout=5m
listener-disable-timeout=5m
http2-proxy=no
accesslog-syslog=yes
errorlog-syslog=yes
workers=5

This time, we use the ‘frontend’ statement to create the listeners that the attacker’s intercepting proxy can connect to via its upstream proxy settings. The ‘backend’ statement points upstream to the service on the real server and tells the instance it should use HTTP/2 (note the ‘proto’ option) with TLS to connect to the vendor server when it is pretending to be the device.

There is no need to configure a TLS certificate as the device vendor did not require the client to authenticate with one in this case and the intercepting proxy doesn’t care what it connects to, so can connect using plaintext HTTP for simplicity.

Interception

The configuration above will enable most traffic to be captured, recorded and manipulated by a HTTP/1.1 supporting intercepting proxy such as Burp Suite or OWASP ZAP.

Final notes

Using the methods detailed in this post we were able to intercept and provide a reasonable level of assurance to a client with concerns about the possibilities of data egress from a third party device. The methods described are rather cumbersome and far from ideal yet were still effective in showing that the device was not routinely communicating sensitive information back to the vendor in its present state. Further hardware testing and firmware could have been performed to provide a greater level of assurance, though the client did not require this.

The solution above was able to make currently available tools work with web services that only support HTTP/2. The techniques could be further improved by configuring multiple nghttpx proxy servers on the same VM in order to reduce the resources required. Patches could be applied to nghttpx to help resolve the errors which caused some requests to be dropped. Alternatively, the attacker could probably forgo the use of any proxy servers, strip the SSL/TLS from the client requests and modify them using ‘netsed’ or similar before piping the result to an OpenSSL connection to the server, though this method would be slower and a lot less user friendly.

Ultimately, modifying existing intercepting proxies and tooling to support HTTP/2 or building dedicated replacements designed around HTTP/2 would be better than this solution, though as these are not currently available and are non-trivial to build, the solution above was seen as a useful way to achieve the required functionality within a short timescale.

References

[1] https://www.chromium.org/spdy/spdy-protocol

[2] https://tools.ietf.org/html/rfc7540

[3] http://isthewebhttp2yet.com/measurements/adoption.html

[4] https://nghttp2.org/documentation/nghttpx-howto.html

First Published date:  09 May 2018

Written by:  Stuart Walker