Varnish HTTP Workspace information leak vulnerability Security

Published October 21, 2019.

Overview

A bug has been discovered in Varnish Cache and Varnish Plus where we fail to clear a pointer between the handling of one client requests and the next on the same connection. This can under specific circumstances lead to information being leaked from the connection workspace.

The vulnerability was discovered by Trygve T√łnnesland of VG/Schibsted, a Varnish Software customer. We wish to thank them for their responsible disclosure.

Impact

The impact is an information leak, limited to the data in the workspace used for handling the connection. This will typically be data structures and stale header data from previous requests on the same connection, but could also be temporary headers set during processing of VCL.

For the leak to occur, Varnish needs to switch to synthetic handling due to an internal error, in order to generate an error response to the client request. Note that it will not be a problem when synthetic response handling is reached explicitly through VCL statements.

The leaked data will be pre populated in the resp.reason variable on entering vcl_synth, and will unless overridden show up in the response field of the HTTP response sent.

The following error situations will cause Varnish to go directly to synthetic handling for the current client request, and can cause data to be leaked:

  • Reaching max_restarts. There is a maximum limit on how many times the VCL is allowed to do a return(restart), and when this limit is reached the request is forced to synthetic handling to send an error response.

  • Requests initiating backend fetches, there is no grace candidate, and the backend responds, but fails to send valid HTTP headers in return. In this situation, specifically for the single request initiating the backend fetch, the request is forced to synthetic handling to send an error response.

  • Attempting to switch to another VCL label after a restart. Switching to another VCL is only allowed on the first run through vcl_recv, and attempts to do it later will force the request to synthetic handling to send an error response.

To successfully attack a vulnerable Varnish server, the system must either be vulnerable from its VCL configuration (max restarts or erroneous VCL switching), or the attacker must be able to cause backend response issues.

Mitigation is possible from VCL or by updating to a fixed version of Varnish Cache.

Status

Affected software versions

  • Varnish Cache Plus 6.0 series up to and including 6.0.4r2.
  • Varnish Cache 6.0 LTS series up to and including 6.0.4.
  • Varnish Cache releases from 5.0 up to and including 6.3.

Resolved in

  • Varnish Cache Plus 6.0.4r3 (released 2019-09-27).
  • Varnish Cache LTS 6.0.5 (released 2019-10-21).
  • Varnish Cache 6.2.2 and 6.3.1 (released 2019-10-21).

Solution

The recommended solution is to upgrade Varnish to one of the versions where this issue has been resolved, and then ensure that Varnish is restarted.

Upgrading on RedHat and derivatives

You should already have configured the Varnish Cache Plus repository, so a normal upgrade will be enough:

sudo yum update varnish-plus
sudo systemctl restart varnish

Verify that the version that is installed is recent:

rpm -q varnish-plus
varnish-plus-6.0.4r3-1.el7.x86_64

Upgrading on Ubuntu and Debian

You should already have configured the Varnish Cache Plus repository, so a normal upgrade should be enough:

sudo apt-get update
sudo apt-get install --only-upgrade varnish-plus
sudo systemctl restart varnish

Verify that the version that is installed is recent:

dpkg -l varnish-plus
[...]
ii  varnish-plus                6.0.4r3-1~xenial   amd64              A supercharged version of the popular web cache, Varnish Cache

Workaround

It is possible to mitigate the problem through VCL, by making sure that the resp.reason field is overridden in the vcl_synth function. That will update the stale pointer value to a known value, eliminating the problem.

The following does this by setting resp.status to itself, updating resp.reason to the generic value for that status code in the process:

sub vcl_synth {
	# This line should be added last in vcl_synth,
	# or just before any return statements
	set resp.status = resp.status;
}

Note that this will override the reason field of the HTTP reply. If your setup relies on information in that field, the workaround may need to be adjusted. Contact support if you need help to apply the workaround.

Identification

Information leaked would typically show as strange and non-related strings appearing on the reason field of HTTP replies being sent. This will show under the RespReason varnishlog tag.

Timeline

2019-09-16

  • Varnish Software customer reports strange behaviour through support.

2019-09-20

  • Problem is analyzed and severity acknowledged in cooperation with the reporting customer.

2019-09-27

  • Varnish Software released patched Varnish Cache Plus packages for supported versions and platforms to customer repositories.

2019-10-21

  • Source code patches pushed to the Varnish Cache git repository at Github.
  • Varnish Cache packages released to the official Varnish Cache package repositories.
  • Public announcement.

References