Properly Signed Certificates on CPE Devices

During late January 2020, a hot topic surfaced between security professionals on an issue that has historically had different proposed solutions. This blog post seeks to explore these solutions and identify pragmatic approaches to risk reduction on this specific issue concerning Customer Premises Equipment (CPE) security.

Two security researchers (Tom Pohl and Nick Starke) analysed a Netgear firmware image and found that the publicly available image contains two properly signed and valid TLS certificates that contained private keys. Because these keys were leaked, it would be trivial to carry out attacks such as Man-In-The-Middle exploits.

The advisory that was published was relatively brief and had no recommended remediation advice, therefore we decided to publish this blog post on recommended ways to mitigate this hard problem without exposing users to additional risks that might be inherent in Netgear’s solution.

First, some background…

What is Netgear’s method of allowing its customers to manage their devices?

When the client machine connects to the network, the DCHP server on the Netgear router will push down a configuration to the client that lists itself as the DNS server. The client is advised to use the www.routerlogin.net webpage to administer the router. Because the DNS server is the router itself, it will tell the user that the domain name resolves to the router’s IP address (usually 192.168.0.1 by default). When the connection is made a TLS key exchange is performed, which includes certificate validation. Since the certificate provided by the router is signed and issued for the exact same domain name, validation will succeed. The connection between the router and the user is now considered secure.

But is it?

Why did Netgear put two signed certificates into the firmware?

Having a secure connection between the client’s PC and a Customer Premises Equipment (CPE) (for example a router) is considered a hard problem, because of many reasons:

• The device is only accessible from the internal network, it has no external name (no public DNS record)
• The user might change its IP address, therefore the address cannot be used as a fixed reference
• TLS certificates usually need a hostname to be validated against (a DNS record that points to the server)
• Private keys and other secrets must not be shared among multiple devices to avoid a break-once-break-everywhere scenario

Self-signed certificates could partly solve this issue, but those are not trusted by the browser, so the user would see a certificate-related error in their browser. We want to avoid the user having to grant exceptions to such errors, as it trains users to ignore, possibly much more serious, errors.

Netgear attempted to resolve the issue by spoofing DNS on the internal network and using hard-coded signed certificates.

What is the issue with Netgear’s approach?

In short, the certificates are unique for the hostname but not unique for the device. The private keys are shared across many devices, which makes it easy to compromise.

If an attacker manages to get the private key (see the advisory) then they can reuse the certificates in a MITM attack. It is possible to decrypt the communication or to setup a website that is served over a secure TLS channel with the signed certificate. The victim only needs to be redirected to this site somehow. There are plenty of attack scenarios for this, including:

• Spoofing DNS records over Wi-Fi
• DNS takeover
• Spoofed DHCP responses
• DNS rebinding

If the attack is successful, then it would be possible for example to exploit the user’s browser and steal credentials.

How to solve this issue?

There is no perfect way to solve this issue without increasing the manufacturing or maintenance cost, but we believe that one of the following solutions would significantly improve the security with minimal extra cost.

Solution 1: Factory provisioning

During the manufacturing process all devices can be assigned a unique hostname and a corresponding certificate which is properly signed by a Certificate Authority (CA) that is trusted by common browsers. This may increase the manufacturing costs, as additional provisioning steps may need to be added to the process. The user experience may also be affected since the user will need to open a more complex URL than a simple short hostname or IP address. One major drawback of this approach is that it cannot be made to apply to existing in-field devices.

Solution 2: Online provisioning

Factory provisioning allows the device certificates to be provisioned in a somewhat controlled environment (notably not perfect as manufacturing is complicated). Applying such a system to in-field devices requires exposing the certificate signing server to the devices after they are deployed. Authenticating devices properly would still ideally require some sort of device-unique secret that the device and server can share. However, even without factory-installed certificates, most consumer devices will have some form of unique (not necessarily secret) identity available to them, and these can be whitelisted on the certificate signing server to help limit unexpected devices from getting certificates signed.

 When the device first gets connected to the Internet, the provisioning could start:

• The device generates a private key and a certificate signing request using the unique device identity
• The device requests a signed certificate from a provisioning server over a secure channel. This could be simply HTTPS with proper certificate checks and certificate pinning
• Store the signed certificate and private key in the device’s non-volatile storage

Whenever a client connects to the device and HTTP is used, it redirects the user to the secure HTTPS URL that corresponds only with that device (using the new signed certificate):

• A DNS request for the hostname will be sent that could be either answered by the device itself or the vendor’s DNS server by recursion
• The DNS record should be resolved to the device’s internal address
  • Some devices have fixed addresses like modems use 192.168.100.1 that cannot be changed
  • If the device supports a user configured IP address, then that needs to be updated in the vendor’s or the device’s DNS server at every change
• Since the hostname will be correct and the corresponding certificate is signed the connection will be secured

For initial out-of-box configuration, when there might not yet be an available Internet connection, the access would be provided over insecure HTTP.

Obviously, in both cases where HTTP is in the picture (offline mode and redirection) the user can be interfered with, but even still this provides a much smaller attack surface than most other currently used solutions (HTTP only or hardcoded signed certificate).

Solution 3: The “Plex” solution

This approach was first described by Plex. It works only when a device has Internet access and requires that the user provision their own device. This might be too much burden for many less-technical users.

The vendor sets up a webpage that is accessible over the Internet. The user must register on that webpage and set the credentials on the device as well so that the provisioning can start:

• The provisioning server will generate and sign a certificate and set up a DNS record that is unique for that device. The name of the record can be any unique value assigned to that user’s device, such as a hash of the certificate or a hash of an agreed random value
• The device will download the corresponding certificate from the provisioning server and store it in non-volatile storage
• In case the user wants to change or view the configuration, the vendor’s website will be opened, the credentials will be submitted for login. This identifies the user. The unique domain name that was generated previously will be used
• The browser now can send XMLHttpRequest (XHR) or Asynchronous JavaScript and XML (AJAX) requests in the background, Cross-Origin Resource Sharing (CORS) will not be an issue if everything is properly configured.

This configuration allows any device to use a properly signed, unique certificate on a per-user basis. In our opinion this solution is the most elegant.

Why did the researchers publish their findings?

A big part of the reason why this issue was prominent in the media was the way in which the issue was disclosed. It seems that the two researchers were not happy with the disclosure policies that Bugcrowd and Netgear follow. It was not possible to report the bug and make it public at the same time, so they decided to publish to encourage the vendor to change their solution, or to let users protect themselves by switching vendors.

The debate around ethics of various disclosure methods is not for this blog post, however an interesting fact relevant to this field is that the same vulnerability was found by another researcher who reported it in 2017 on Bugcrowd. There it was marked as a duplicate, which means that multiple parties were already aware of this issue, the vendor was notified, yet did not implement a fix. Users were left vulnerable. In publishing, the researchers succeeded at least in getting the certificates revoked within 2 days of the advisory, which helps prevent them from being used to attack victims.

One thing can be said for sure; having a practical recommendation in an advisory or disclosure is more useful to end users than not having one, even if this recommendation is to switch equipment vendors.

Conclusion

None of the solutions are perfect. Some of them either suffer practical concerns, like only supporting newly manufactured devices or requiring Internet access. Others have weak points like insecure initial connections that are easier to attack. Nevertheless, all of these solutions are better than the common practice of using self-signed, hard-coded, or no certificates.

In any of the three cases discussed here, the connection between the device and the user’s browser is eventually secured. Moreover, if a private key gets stolen from the device, the attacker will be only able to attack one specific user, and not all users of that device model.

Written by Balazs Bucsay.