Varnish HTTP/2 Request Smuggling Security

Published July 13, 2021.

Overview

A request smuggling attack can be performed on Varnish Cache and Varnish Cache Plus servers that have the HTTP/2 protocol enabled. The smuggled requests do not go through normal VCL processing, and any authorization steps implemented in VCL would be bypassed.

The responses to the smuggled requests can under some circumstances also be obtained by the attacker. Also, it may be possible for an attacker to use this for cache poisoning, where the response to a smuggled request is inserted as the cached content.

This problem was discovered by Martin Blix Grydeland of Varnish Software.

Impact

The potential impact is information disclosure and cache poisoning.

Status

Affected software versions

Varnish Enterprise 6.0 series up to and including 6.0.8r2
Varnish Cache 6.0 LTS series up to and including 6.0.7
Varnish Cache releases 6.0.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.1.0, 6.1.1, 6.2.0, 6.2.1, 6.2.2, 6.2.3, 6.3.0, 6.3.1, 6.3.2, 6.4.0, 6.5.0, 6.5.1, 6.6.0.
Varnish Cache releases 5.x.x. Notice that the experimental HTTP/2 support in these releases are known to have several issues, and enabling HTTP/2 is not recommended.

Versions not affected

All versions of Varnish Enterprise 4.1
All versions of Varnish Cache LTS 4.1 series.
All versions of Varnish Cache prior to 5.0.0

Resolved in

Varnish Enterprise 6.0.8r3 (released )
Varnish Cache LTS 6.0.8 (released )
Varnish Cache 6.6.1 and 6.5.2 (released )

The recommended solution is to upgrade Varnish to one of the versions where this issue has been resolved, and then ensure that Varnish is restarted.

Upgrading on RedHat and derivatives

You should already have configured the Varnish Enterprise repository, so a normal upgrade will be enough:

sudo yum update varnish-plus
sudo systemctl restart varnish

Verify that the version that is installed is recent:

rpm -q varnish-plus
varnish-plus-6.0.8r3-1.el7.x86_64

Upgrading on Ubuntu and Debian

You should already have configured the Varnish Enterprise repository, so a normal upgrade should be enough:

sudo apt-get update
sudo apt-get install --only-upgrade varnish-plus
sudo systemctl restart varnish

Verify that the version that is installed is recent:

dpkg -l varnish-plus
[...]
ii  varnish-plus                6.0.8r3-1~xenial   amd64              A supercharged version of the popular web cache, Varnish Cache

Workaround

Turning off support for HTTP/2

The problem only affects servers that have HTTP/2 support enabled. This support can be turned off at runtime. To disable HTTP/2 on a server do:

sudo varnishadm param.set feature -http2

To verify that HTTP/2 is disabled on a server, execute this command and make sure the current value does not list http2:

sudo varnishadm param.show feature

When using in-core TLS termination, this will also remove h2 as a supported protocol advertised through ALPN.

When using Hitch (or any other TLS termination proxy) in front of Varnish to handle TLS termination, it is also necessary unlist the h2 token as a possible protocol in the ALPN advertisement sent to connecting clients.

To unlist h2 as a support protocol in Hitch, remove or comment out the line stating alpn-protos = "h2, http/1.1" in your Hitch configuration file. Then restart the Hitch service (reload is not sufficient).

Preventing connection reuse

On compliant backends it is possible to prevent the execution of smuggled requests by disabling connection reuse of backend requests. Note that for this workaround to be effective, it relies on the backend to refuse any additional requests after seeing a Connection: close header.

To disable backend connection reuse, add a Connection: close header on the outgoing backend requests:

sub vcl_backend_fetch {
	set bereq.http.Connection = "close";
}

Identifying smuggled requests

Smuggled requests will not show in any logs generated by Varnish, but would show in the backend logs. It may be possible to identify the smuggled requests in the backend logs by missing Varnish inserted artifacts, like the X-Varnish header. Though a determined attacker may spoof these artifacts in the smuggled requests.

Timeline

2021-05-31

The problem is identified during internal testing at Varnish Software.

2021-06-11

Problem is analyzed and severity acknowledged.

2021-06-15

Varnish Enterprise 6.0.8r3 released with patch for the issue.

2021-07-13

Public disclosure and updated open source packages.