Varnish HTTP Cache DoS vulnerability Security

Published August 2, 2017.

Overview

An assert can be triggered in Varnish Cache and Varnish Cache Plus by a remote client sending a specially crafted HTTP request. Varnish will panic and restart when the assert is triggered, which constitutes a denial of service attack.

This was discovered by StackPath.

Impact

The potential impact is reduced or completely lost availability.

The cache will be empty after the restart unless Massive Storage Engine is used in persistence mode. An empty cache after restart will reduce overall performance due to an increased number of cache misses, and may cause higher load on the backend servers.

There is no potential for remote code execution or data leaks related to this vulnerability.

Status

Affected software versions

  • Varnish Cache Plus 4.0.2r0, 4.0.3r1, 4.0.3r2, 4.0.3r3, 4.0.3r4, 4.0.3r5, 4.0.3r6 and 4.0.4r1.
  • Varnish Cache Plus 4.1.2r1, 4.1.3r1, 4.1.4r1, 4.1.4r2, 4.1.4r3, 4.1.4r4, 4.1.4r5, 4.1.5r1, 4.1.5r2, 4.1.6r1, 4.1.6r2, 4.1.7r1 and 4.1.7r2.
  • Varnish Cache 4.0.1, 4.0.2, 4.0.3 and 4.0.4.
  • Varnish Cache 4.1.0, 4.1.1, 4.1.2, 4.1.3, 4.1.4, 4.1.5, 4.1.6 and 4.1.7.
  • Varnish Cache 5.0.0, 5.1.0, 5.1.1 and 5.1.2.

Not affected software versions

  • All releases up to and including 4.0.0.

Resolved in

  • Varnish Cache Plus 4.0.4r2.
  • Varnish Cache Plus 4.1.7r3.
  • Varnish Cache 4.0.5.
  • Varnish Cache 4.1.8.
  • Varnish Cache 5.1.3.

Solution

The solution is to upgrade Varnish to one of the versions where this issue has been resolved, and then ensure that Varnish is restarted.

The procedures below explain how to upgrade Varnish Cache Plus and Varnish Cache:

Redhat and derivatives

Upgrade to latest version

You should already have configured the Varnish Cache Plus repository, so a normal upgrade should be enough:

sudo yum update varnish-plus
sudo service varnish restart

Verify that the installed version is current

We recommend doing the three following steps to verify that the upgrade worked as intended. Versions starting with 4.1.7r3 and 4.0.4r2 are safe.

Verify that the correct version of the package is installed:

# yum list varnish-plus
Installed Packages
varnish-plus.x86_64     4.1.7r3-1.el6     @varnish-4.1-plus

Verify that the varnish binary /usr/sbin/varnishd reports the correct version:

# /usr/sbin/varnishd -V
varnishd (varnish-plus-4.1.7r3 revision ba23f33)
Copyright (c) 2006 Verdens Gang AS
Copyright (c) 2006-2015 Varnish Software AS

Verify that varnishadm connects to a varnishd with the correct version:

# sudo varnishadm banner
-----------------------------
Varnish Cache CLI 1.0
-----------------------------
Linux,4.4.0-87-generic,x86_64,-junix,-smalloc,-smalloc,-hcritbit
varnish-plus-4.1.7r3 revision ba23f33

Type 'help' for command list.
Type 'quit' to close CLI session.

Ubuntu

Upgrade to latest version

You should already have configured the Varnish Cache Plus repository, so a normal upgrade should be enough:

sudo apt-get update
sudo apt-get install --only-upgrade varnish-plus
sudo service varnish restart

Verify that the installed version is current

We recommend doing the three following steps to verify that the upgrade worked as intended. Versions starting with 4.1.7r3 and 4.0.4r2 are safe.

Verify that the correct version of the package is installed:

# dpkg -l | grep varnish
ii  varnish-plus                      4.1.7r3-1~xenial                           amd64
ii  varnish-plus-addon-ssl            1.4.6-1~xenial                             amd64        Hitch TLS proxy

Verify that the varnish binary /usr/sbin/varnishd reports the correct version:

# /usr/sbin/varnishd -V
varnishd (varnish-plus-4.1.7r3 revision ba23f33)
Copyright (c) 2006 Verdens Gang AS
Copyright (c) 2006-2015 Varnish Software AS

Verify that varnishadm connects to a varnishd with the correct version:

# varnishadm banner
-----------------------------
Varnish Cache CLI 1.0
-----------------------------
Linux,4.4.0-87-generic,x86_64,-junix,-smalloc,-smalloc,-hcritbit
varnish-plus-4.1.7r3 revision ba23f33

Type 'help' for command list.
Type 'quit' to close CLI session.

Workaround

Organizations that are not able to upgrade their Varnish environment on short notice can deploy a workaround in VCL to close the denial of service vector.

This workaround will deliberately block and close the connections from clients attempting to make POST requests through the Varnish server data using non-standard transfer-encodings (e.g. chunked encoding). The workaround should not affect regular traffic as these types transfer-encodings in POST requests are not common.

To check if your Varnish is receiving POSTS with non-standard transfer-encodings, run this varnishlog command:

varnishlog -cq ReqHeader:Transfer-Encoding -i ReqMethod -i ReqURL

If this check shows legitimate requests using chunked encoding, the workarounds will most likely break the respective functionality. If that is a concern, consider incorporating an ACL in the workaround VCL to selectively apply it only to known attacking addresses.

Enabling inline-C support in the Varnish VCL compiler

In order for these workarounds to compile on your Varnish server, you will need to enable support for inline-C. This feature is disabled by default.

To enable inline-C support at the Varnish daemon startup, add -p vcc_allow_inline_c=true to the varnishd command line options. This will however not enable it for your currently running Varnish instance. To enable it at runtime, execute the next step as well.

To enable inline-C support at runtime execute the following varnishadm command:

varnishadm "param.set vcc_allow_inline_c true"

To verify that the option is on execute:

varnishadm "param.show vcc_allow_inline_c"

Varnish Cache Plus 4.0.x (and Varnish Cache 4.0.x)

Note: The following instructions are for Varnish Cache Plus and Varnish Cache versions 4.0.x only. They will not work for any other versions, and will, if deployed on the wrong version, cause undefined behavior likely causing crashes.

Note2: Inline-C support must be enabled on the Varnish instance for this VCL code to compile. See the previous chapter for instructions.

The workaround is presented as a piece of VCL code to integrate in your VCL configuration. There are two parts to the configuration.

  1. A VCL subroutine called exploit_workaround_4_0. This is the actual workaround that will trigger the block and close off the connection for the malign requests described above. An early return() statement is executed to block further VCL processing.

  2. A call to the exploit_workaround_4_0 VCL subroutine from vcl_recv.

The following code section should be copied verbatim into your VCL configuration at the very start, just after the line stating vcl 4.0;.

sub exploit_workaround_4_0 {
	# This needs to come before your vcl_recv function
	# The following code is only valid for Varnish Cache and
	# Varnish Cache Plus versions 4.0.x
	if (req.http.transfer-encoding ~ "(?i)chunked") {
		C{
		struct dummy_req {
			unsigned magic;
			int restarts;
			int esi_level;
			int disable_esi;
			char hash_ignore_busy;
			char hash_always_miss;
			void *sp;
			void *wrk;
			int req_step;
			struct {
				void *a;
				void *b;
			};
			int req_body_status;
		};
		((struct dummy_req *)ctx->req)->req_body_status = 6;
		}C

		return (synth(503, "Bad request"));
	}
}

sub vcl_recv {
	# Call this early in your vcl_recv function
	call exploit_workaround_4_0;
}

Varnish Cache Plus 4.1.x (and Varnish Cache 4.1.x / 5.0.x)

Note: The following instructions are for Varnish Cache Plus and Varnish Cache versions 4.1.x and 5.0.0 only. They will not work for any other versions, and will, if deployed on the wrong version, cause undefined behavior likely causing crashes.

Note2: Inline-C support must be enabled on the Varnish instance for this VCL code to compile. See the previous chapter for instructions.

The workaround is presented as a piece of VCL code to integrate in your VCL configuration. There are two parts to the configuration.

  1. A VCL subroutine called exploit_workaround_4_1. This is the actual workaround that will trigger the block and close off the connection for the malign requests described above. An early return() statement is executed to block further VCL processing.

  2. A call to the exploit_workaround_4_1 VCL subroutine from vcl_recv.

The following code section should be copied verbatim into your VCL configuration at the very start, just after the line stating vcl 4.0;.

sub exploit_workaround_4_1 {
	# This needs to come before your vcl_recv function
	# The following code is only valid for Varnish Cache and
	# Varnish Cache Plus versions 4.1.x and 5.0.0
	if (req.http.transfer-encoding ~ "(?i)chunked") {
		C{
		struct dummy_req {
			unsigned magic;
			int step;
			int req_body_status;
		};
		((struct dummy_req *)ctx->req)->req_body_status = 5;
		}C

		return (synth(503, "Bad request"));
	}
}

sub vcl_recv {
	# Call this early in your vcl_recv function
	call exploit_workaround_4_1;
}

Varnish Cache 5.1.x (and current master)

Note: The following instructions are for Varnish Cache versions 5.1.x and current master only. They will not work for any other versions.

The workaround is presented as a piece of VCL code to integrate in your VCL configuration. The following should be copied verbatim into your VCL configuration at the very start, just after the line stating vcl 4.0;.

sub vcl_recv {
	# The following code is only valid for Varnish Cache version 5.1.x
	# and current master
	if (req.http.transfer-encoding ~ "(?i)chunked") {
		return (fail);
	}
}

Identification

The command line tool varnishadm can be used to indicate if this assert has been triggered or is currently being triggered on an instance of Varnish. The following command will output the latest panic message, if any:

sudo varnishadm panic.show

Under normal and healthy circumstances it outputs:

Child has not panicked or panic has been cleared

Example panic output on Varnish Cache Plus 4.0.x

This is what the panic message looks like on Varnish Cache Plus 4.0.x, when someone has successfully used this vulnerability to restart the cache server.

Last panic at: Sat, 29 Jul 2017 18:31:26 GMT
Assert error in HTTP1_Chunked(), cache/cache_http1_proto.c line 614:
  Condition((*priv) == 0) not true.
thread = (cache-worker)
version = varnish-plus-4.0.4r1 revision 4d6456a
ident = Linux,3.16.0-4-amd64,x86_64,-smalloc,-smalloc,-hcritbit,epoll
Backtrace:
  0x45d229: pan_backtrace+0x19
  0x45d0a9: pan_ic+0x339
  0x45d204: nodump_pan_ic+0x54
  0x456d3a: HTTP1_Chunked+0x84a
  0x453208: http1_iter_req_body+0x3d8
  0x452d0e: HTTP1_IterateReqBody+0x4be
  0x44f08e: V1F_fetch_hdr+0x4fe
  0x43766d: vbf_stp_startfetch+0x60d
  0x4358e7: vbf_fetch_thread+0x757
  0x46215c: Pool_Work_Thread+0x6dc
...

Example panic output on Varnish Cache Plus 4.1.x

This is what the panic message looks like on Varnish Cache Plus 4.1.x, when someone has successfully used this vulnerability to restart the cache server.

Last panic at: Sat, 29 Jul 2017 18:28:31 GMT
"Assert error in v1f_pull_chunked(), http1/cache_http1_vfp.c line 175:
  Condition((vfe->priv2) == 0) not true.
thread = (cache-worker)
version = varnish-plus-4.1.7r3 revision cd3268f
ident = Linux,3.16.0-4-amd64,x86_64,-jnone,-smalloc,-smalloc,-hcritbit,epoll
now = 22603.115991 (mono), 1501352911.055518 (real)
Backtrace:
  0x461de0: pan_backtrace+0x20
  0x461ba1: pan_ic+0x3c1
  0x461cf0: ccf_panic
  0x4a0985: v1f_pull_chunked+0x815
  0x446dc9: VFP_Suck+0x269
  0x466e5e: VRB_Iterate+0x67e
  0x4995b2: V1F_SendReq+0x4c2
  0x4168d4: vbe_dir_gethdrs+0x424
  0x42e821: VDI_GetHdr+0x1f1
  0x441c66: vbf_stp_startfetch+0x3c6
...

Timeline

2017-07-20

  • StackPath reported the discovery to Varnish Software.
  • Varnish Software patched the vulnerability in the specific version and distributed snapshot packages to StackPath.
  • Varnish Software started coordination with core members in the Varnish Cache project for responsible disclosure.

2017-07-27

  • Varnish Software released patched Varnish Cache Plus packages for supported versions and platforms to customer repositories.

2017-07-29

  • The security teams in Ubuntu, Debian, RedHat and FreeBSD were notified.

2017-07-31

  • Patches distributed to the security teams and package maintainers of Varnish Cache.

2017-08-02

  • Source code patches pushed to the Varnish Cache git repository at Github.
  • Varnish Cache packages released to the official Varnish Cache package repositories.
  • Public announcement.

Acknowledgement

Varnish Software would like to thank StackPath for discovering and reporting this issue.

References