Varnish HTTP Cache DoS vulnerability Security

Published September 3, 2019.

Overview

An assert can be triggered in Varnish Cache and Varnish Enterprise by a remote client sending specially crafted HTTP requests. Varnish will panic and restart when the assert is triggered, which constitutes a denial of service attack.

This was discovered by Varnish Software.

Impact

The potential impact is reduced or completely lost availability.

The cache will be empty after the restart unless Massive Storage Engine is used in persistence mode. An empty cache after restart will reduce overall performance due to an increased number of cache misses, and may cause higher load on the backend servers.

There is no potential for remote code execution or data leaks related to this vulnerability.

Status

Affected software versions

Varnish Enterprise 6.0 series up to and including 6.0.3r8.
Varnish Enterprise 4.1 series up to and including 4.1.11r3.
Varnish Cache 6.0 series up to and including 6.0.3.

Resolved in

Varnish Enterprise 6.0.3r9 (released 2019-08-19).
Varnish Enterprise 4.1.11r4 (released 2019-08-19).
Varnish Cache 6.0.4 (released 2019-09-02).

Solution

The solution is to upgrade Varnish to one of the versions where this issue has been resolved, and then ensure that Varnish is restarted.

Verify that the installed version is current

Verify that the varnish binary /usr/sbin/varnishd is one of the correct versions:

$ sudo varnishd -V
varnishd (varnish-plus-6.0.3r9 revision 6c4fbaf520e2a9528a3dfc61373a8b2fd53d5a0a)
Copyright (c) 2006 Verdens Gang AS
Copyright (c) 2006-2019 Varnish Software AS

Verify that varnishadm connects to a varnishd which is one of the correct versions:

$ sudo varnishadm banner
-----------------------------
Varnish Cache CLI 1.0
-----------------------------
Linux,3.10.0-957.27.2.el7.x86_64,x86_64,-junix,-smse,-sdefault,-hcritbit
varnish-plus-6.0.3r9 revision 6c4fbaf520e2a9528a3dfc61373a8b2fd53d5a0a

Type 'help' for command list.
Type 'quit' to close CLI session.

Workaround

There is no ideal VCL workaround for this vulnerability.

In many environments Varnish is one of several components in a stack. If another HTTP component sits in front of Varnish in the stack it is likely that this other HTTP component will reject the invalid HTTP requests that are needed to exploit the vulnerability.

Identification

The command line tool varnishadm can be used to indicate if this assert has been triggered or is currently being triggered on an instance of Varnish. The following command will output the latest panic message, if any:

sudo varnishadm panic.show

Under normal and healthy circumstances it outputs:

Child has not panicked or panic has been cleared

Example panic output

This is what the panic message looks like on Varnish Enterprise 6.0.3r8, when someone has successfully used this vulnerability to restart the cache server.

Assert error in HTC_RxPipeline(), cache/cache_session.c line 236:
  Condition((char*)p < htc->rxbuf_e) not true.
version = varnish-plus-6.0.3r8 revision 364f3e955779ebd880221c6a17d02431fcd00da8, vrt api = 7.1
ident = Linux,3.10.0-957.10.1.el7.x86_64,x86_64,-junix,-sdefault,-sdefault,-hcritbit,epoll
now = 357.185075 (mono), 1567364174.777856 (real)
Backtrace:
  0x446b30: varnishd() [0x446b30]
  0x4c8148: varnishd(VAS_Fail_Dump+0x48) [0x4c8148]
  0x4c821f: varnishd() [0x4c821f]
  0x4500d5: varnishd(HTC_RxPipeline+0xb5) [0x4500d5]
  0x472d28: varnishd() [0x472d28]
  0x473247: varnishd(HTTP1_DissectRequest+0x57) [0x473247]
  0x470cb3: varnishd() [0x470cb3]
  0x466953: varnishd() [0x466953]
  0x466e20: varnishd() [0x466e20]
  0x7f9d848ffdd5: /usr/lib64/libpthread.so.0(+0x7dd5) [0x7f9d848ffdd5]
...

Timeline

2019-08-07

One of the fuzzers used by Varnish Software indicated that it had successfully crashed an instance of Varnish running the development branch of Varnish Enterprise 6.0.

2019-08-09

First successful test case to trigger the assert was completed.

2017-08-16

Final patch set was completed and verified with the fuzzer.

2017-08-19

Varnish Software released patched Varnish Enterprise packages for supported versions and platforms to customer repositories.

2017-09-03

Source code patches pushed to the Varnish Cache git repository at Github.
Varnish Cache packages released to the official Varnish Cache package repositories.
Public announcement.