Search

Varnish HTTP Proxy Protocol V2 Denial of Service Security

Published February 4, 2020.

Overview

An assert can be triggered in Varnish Cache and Varnish Enterprise when using Varnish with a TLS termination proxy, and the proxy and Varnish use the PROXY version 2 protocol to communicate connection details. Depending on the type of proxy used and the details it include in the proxy payload, it may be possible for remote clients to cause Varnish to assert and restart, making it a denial of service attack.

This was discovered by Varnish Software.

Impact

The potential impact is reduced or lost availability.

The cache will be empty after the restart unless Massive Storage Engine is used in persistence mode. An empty cache after restart will reduce overall performance due to an increased number of cache misses, and may cause higher load on the backend servers.

There is no potential for remote code execution or data leaks related to this vulnerability.

Status

Affected software versions

  • Varnish Enterprise 6.0 series up to and including 6.0.5r2.
  • Varnish Cache 6.0 LTS series up to and including 6.0.5.
  • Varnish Cache releases 6.0.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.1.0, 6.1.1, 6.2.0, 6.2.1, 6.2.2, 6.3.0, 6.3.1.

Versions not affected

  • All versions of Varnish Enterprise 4.1.
  • All versions of Varnish Cache LTS 4.1 series.
  • All versions of Varnish Cache prior to 6.0.0.

Resolved in

  • Varnish Enterprise 6.0.5r3 (released 2020-01-07).
  • Varnish Cache LTS 6.0.6 (released 2020-02-04).
  • Varnish Cache 6.2.3 and 6.3.2 (released 2020-02-04).

Solution

The recommended solution is to upgrade Varnish to one of the versions where this issue has been resolved, and then ensure that Varnish is restarted.

Upgrading on RedHat and derivatives

You should already have configured the Varnish Enterprise repository, so a normal upgrade will be enough:

sudo yum update varnish-plus
sudo systemctl restart varnish

Verify that the version that is installed is recent:

rpm -q varnish-plus
varnish-plus-6.0.5r3-1.el7.x86_64

Upgrading on Ubuntu and Debian

You should already have configured the Varnish Enterprise repository, so a normal upgrade should be enough:

sudo apt-get update
sudo apt-get install --only-upgrade varnish-plus
sudo systemctl restart varnish

Verify that the version that is installed is recent:

dpkg -l varnish-plus
[...]
ii  varnish-plus                6.0.5r3-1~xenial   amd64              A supercharged version of the popular web cache, Varnish Cache

Workaround

It is possible to mitigate the problem in several ways.

Switch to proxy protocol version 1

Setups using proxy protocol version 1 are not affected, so if the TLS termination proxy used supports both versions, switching to version 1 will mitigate the problem. This is the recommended mitigation method.

Note that version 1 of the proxy protocol will communicate less details about the connection than what is possible when using version 2. For example the SNI server name used by the connecting client will not be transferred, and can then not be queried using the Proxy VMOD in VCL.

When using the Hitch TLS proxy, version 1 of the proxy protocol can be selected by replacing any write-proxy or write-proxy-v2 options with write-proxy-v1, and restarting Hitch.

Disallow non-matching SNI names in Hitch

When using Hitch as the TLS termination proxy, one can work around the problem by disallowing client connects connecting using a SNI server name that does not match any of the configured certificates. This can be done by adding the sni-nomatch-abort option to the Hitch configuration.

Please note that this mitigation strategy is only effective if none of the configured certificates allow wildcard domain names.

Increase the session workspace

By increasing the session workspace, one can make sure that an attacker can not successfully exhaust the space available, thus not triggering the assert. The size it needs to be set at depends on the TLS proxy in use, and what fields it include in the proxy payload and how a remote client can influence its contents.

By default the session workspace is set to 512 bytes. The session workspace can be changed by setting the workspace_session Varnish parameter, and restarting the Varnish daemon.

When using Hitch as the TLS proxy, setting the session workspace to 34k will mitigate the problem completely. Add “-p workspace_session=34k” to the varnishd command line to set this value.

Note that increasing the session workspace will increase the amount of memory Varnish holds per connection, and you may want to decrease the memory cache size to compensate.

Identification

If a Varnish server is attacked using this vulnerability it will manifest as a series of assertion panics followed by child process restarts. These asserts will be logged to syslog. The assertion itself may come from multiple places in the source code.

The command line tool varnishadm can be used to indicate if this assert has been triggered or is currently being triggered on an instance of Varnish. The following command will output the latest panic message, if any:

sudo varnishadm panic.show

Under normal and healthy circumstances it outputs:

Child has not panicked or panic has been cleared

Timeline

2019-12-06

  • An anomaly is found during internal testing at Varnish Software.

2019-12-11

  • Problem is analyzed and severity acknowledged.

2019-12-19

  • Preliminary patch set to address the issue developed. Workaround documentation finished.

2020-01-07

  • Varnish Plus 6.0.5r3 released with patch for the issue.

2020-02-04

  • Public disclosure and updated open source packages.