SearchAn assert can be triggered in Varnish Cache and Varnish Enterprise by a remote client sending a specially crafted HTTP request. Varnish will panic and restart when the assert is triggered, which constitutes a denial of service attack.
This was discovered by StackPath.
The potential impact is reduced or completely lost availability.
The cache will be empty after the restart unless Massive Storage Engine is used in persistence mode. An empty cache after restart will reduce overall performance due to an increased number of cache misses, and may cause higher load on the backend servers.
There is no potential for remote code execution or data leaks related to this vulnerability.
The solution is to upgrade Varnish to one of the versions where this issue has been resolved, and then ensure that Varnish is restarted.
The procedures below explain how to upgrade Varnish Enterprise and Varnish Cache:
You should already have configured the Varnish Enterprise repository, so a normal upgrade should be enough:
sudo yum update varnish-plus
sudo service varnish restart
We recommend doing the three following steps to verify that the upgrade worked as intended. Versions starting with 4.1.7r3 and 4.0.4r2 are safe.
Verify that the correct version of the package is installed:
# yum list varnish-plus
Installed Packages
varnish-plus.x86_64 4.1.7r3-1.el6 @varnish-4.1-plus
Verify that the varnish binary /usr/sbin/varnishd reports the correct version:
# /usr/sbin/varnishd -V
varnishd (varnish-plus-4.1.7r3 revision ba23f33)
Copyright (c) 2006 Verdens Gang AS
Copyright (c) 2006-2015 Varnish Software AS
Verify that varnishadm connects to a varnishd with the correct version:
# sudo varnishadm banner
-----------------------------
Varnish Cache CLI 1.0
-----------------------------
Linux,4.4.0-87-generic,x86_64,-junix,-smalloc,-smalloc,-hcritbit
varnish-plus-4.1.7r3 revision ba23f33
Type 'help' for command list.
Type 'quit' to close CLI session.
You should already have configured the Varnish Enterprise repository, so a normal upgrade should be enough:
sudo apt-get update
sudo apt-get install --only-upgrade varnish-plus
sudo service varnish restart
We recommend doing the three following steps to verify that the upgrade worked as intended. Versions starting with 4.1.7r3 and 4.0.4r2 are safe.
Verify that the correct version of the package is installed:
# dpkg -l | grep varnish
ii varnish-plus 4.1.7r3-1~xenial amd64
ii varnish-plus-addon-ssl 1.4.6-1~xenial amd64 Hitch TLS proxy
Verify that the varnish binary /usr/sbin/varnishd reports the correct version:
# /usr/sbin/varnishd -V
varnishd (varnish-plus-4.1.7r3 revision ba23f33)
Copyright (c) 2006 Verdens Gang AS
Copyright (c) 2006-2015 Varnish Software AS
Verify that varnishadm connects to a varnishd with the correct version:
# varnishadm banner
-----------------------------
Varnish Cache CLI 1.0
-----------------------------
Linux,4.4.0-87-generic,x86_64,-junix,-smalloc,-smalloc,-hcritbit
varnish-plus-4.1.7r3 revision ba23f33
Type 'help' for command list.
Type 'quit' to close CLI session.
Organizations that are not able to upgrade their Varnish environment on short notice can deploy a workaround in VCL to close the denial of service vector.
This workaround will deliberately block and close the connections from clients attempting to make POST requests through the Varnish server data using non-standard transfer-encodings (e.g. chunked encoding). The workaround should not affect regular traffic as these types transfer-encodings in POST requests are not common.
To check if your Varnish is receiving POSTS with non-standard transfer-encodings, run this varnishlog command:
varnishlog -cq ReqHeader:Transfer-Encoding -i ReqMethod -i ReqURL
If this check shows legitimate requests using chunked encoding, the workarounds will most likely break the respective functionality. If that is a concern, consider incorporating an ACL in the workaround VCL to selectively apply it only to known attacking addresses.
In order for these workarounds to compile on your Varnish server, you will need to enable support for inline-C. This feature is disabled by default.
To enable inline-C support at the Varnish daemon startup, add -p vcc_allow_inline_c=true to the varnishd command line
options. This will however not enable it for your currently running
Varnish instance. To enable it at runtime, execute the next step as
well.
To enable inline-C support at runtime execute the following varnishadm command:
varnishadm "param.set vcc_allow_inline_c true"
To verify that the option is on execute:
varnishadm "param.show vcc_allow_inline_c"
Note: The following instructions are for Varnish Enterprise and Varnish Cache versions 4.0.x only. They will not work for any other versions, and will, if deployed on the wrong version, cause undefined behavior likely causing crashes.
Note2: Inline-C support must be enabled on the Varnish instance for this VCL code to compile. See the previous chapter for instructions.
The workaround is presented as a piece of VCL code to integrate in your VCL configuration. There are two parts to the configuration.
A VCL subroutine called exploit_workaround_4_0. This is the
actual workaround that will trigger the block and close off the
connection for the malign requests described above. An early return()
statement is executed to block further VCL processing.
A call to the exploit_workaround_4_0 VCL subroutine from vcl_recv.
The following code section should be copied verbatim into your VCL
configuration at the very start, just after the line stating vcl 4.0;.
sub exploit_workaround_4_0 {
# This needs to come before your vcl_recv function
# The following code is only valid for Varnish Cache and
# Varnish Enterprise versions 4.0.x
if (req.http.transfer-encoding ~ "(?i)chunked") {
C{
struct dummy_req {
unsigned magic;
int restarts;
int esi_level;
int disable_esi;
char hash_ignore_busy;
char hash_always_miss;
void *sp;
void *wrk;
int req_step;
struct {
void *a;
void *b;
};
int req_body_status;
};
((struct dummy_req *)ctx->req)->req_body_status = 6;
}C
return (synth(503, "Bad request"));
}
}
sub vcl_recv {
# Call this early in your vcl_recv function
call exploit_workaround_4_0;
}
Note: The following instructions are for Varnish Enterprise and Varnish Cache versions 4.1.x and 5.0.0 only. They will not work for any other versions, and will, if deployed on the wrong version, cause undefined behavior likely causing crashes.
Note2: Inline-C support must be enabled on the Varnish instance for this VCL code to compile. See the previous chapter for instructions.
The workaround is presented as a piece of VCL code to integrate in your VCL configuration. There are two parts to the configuration.
A VCL subroutine called exploit_workaround_4_1. This is the
actual workaround that will trigger the block and close off the
connection for the malign requests described above. An early return()
statement is executed to block further VCL processing.
A call to the exploit_workaround_4_1 VCL subroutine from
vcl_recv.
The following code section should be copied verbatim into your VCL
configuration at the very start, just after the line stating vcl 4.0;.
sub exploit_workaround_4_1 {
# This needs to come before your vcl_recv function
# The following code is only valid for Varnish Cache and
# Varnish Enterprise versions 4.1.x and 5.0.0
if (req.http.transfer-encoding ~ "(?i)chunked") {
C{
struct dummy_req {
unsigned magic;
int step;
int req_body_status;
};
((struct dummy_req *)ctx->req)->req_body_status = 5;
}C
return (synth(503, "Bad request"));
}
}
sub vcl_recv {
# Call this early in your vcl_recv function
call exploit_workaround_4_1;
}
Note: The following instructions are for Varnish Cache versions 5.1.x and current master only. They will not work for any other versions.
The workaround is presented as a piece of VCL code to integrate in
your VCL configuration. The following should be copied verbatim into
your VCL configuration at the very start, just after the line stating
vcl 4.0;.
sub vcl_recv {
# The following code is only valid for Varnish Cache version 5.1.x
# and current master
if (req.http.transfer-encoding ~ "(?i)chunked") {
return (fail);
}
}
The command line tool varnishadm can be used to indicate if this assert has been triggered or is currently being triggered on an instance of Varnish. The following command will output the latest panic message, if any:
sudo varnishadm panic.show
Under normal and healthy circumstances it outputs:
Child has not panicked or panic has been cleared
This is what the panic message looks like on Varnish Enterprise 4.0.x, when someone has successfully used this vulnerability to restart the cache server.
Last panic at: Sat, 29 Jul 2017 18:31:26 GMT
Assert error in HTTP1_Chunked(), cache/cache_http1_proto.c line 614:
Condition((*priv) == 0) not true.
thread = (cache-worker)
version = varnish-plus-4.0.4r1 revision 4d6456a
ident = Linux,3.16.0-4-amd64,x86_64,-smalloc,-smalloc,-hcritbit,epoll
Backtrace:
0x45d229: pan_backtrace+0x19
0x45d0a9: pan_ic+0x339
0x45d204: nodump_pan_ic+0x54
0x456d3a: HTTP1_Chunked+0x84a
0x453208: http1_iter_req_body+0x3d8
0x452d0e: HTTP1_IterateReqBody+0x4be
0x44f08e: V1F_fetch_hdr+0x4fe
0x43766d: vbf_stp_startfetch+0x60d
0x4358e7: vbf_fetch_thread+0x757
0x46215c: Pool_Work_Thread+0x6dc
...
This is what the panic message looks like on Varnish Enterprise 4.1.x, when someone has successfully used this vulnerability to restart the cache server.
Last panic at: Sat, 29 Jul 2017 18:28:31 GMT
"Assert error in v1f_pull_chunked(), http1/cache_http1_vfp.c line 175:
Condition((vfe->priv2) == 0) not true.
thread = (cache-worker)
version = varnish-plus-4.1.7r3 revision cd3268f
ident = Linux,3.16.0-4-amd64,x86_64,-jnone,-smalloc,-smalloc,-hcritbit,epoll
now = 22603.115991 (mono), 1501352911.055518 (real)
Backtrace:
0x461de0: pan_backtrace+0x20
0x461ba1: pan_ic+0x3c1
0x461cf0: ccf_panic
0x4a0985: v1f_pull_chunked+0x815
0x446dc9: VFP_Suck+0x269
0x466e5e: VRB_Iterate+0x67e
0x4995b2: V1F_SendReq+0x4c2
0x4168d4: vbe_dir_gethdrs+0x424
0x42e821: VDI_GetHdr+0x1f1
0x441c66: vbf_stp_startfetch+0x3c6
...
2017-07-20
2017-07-27
2017-07-29
2017-07-31
2017-08-02
Varnish Software would like to thank StackPath for discovering and reporting this issue.