SearchAn assert can be triggered in Varnish Cache and Varnish Enterprise when using Varnish with a TLS termination proxy, and the proxy and Varnish use the PROXY version 2 protocol to communicate connection details. Depending on the type of proxy used and the details it include in the proxy payload, it may be possible for remote clients to cause Varnish to assert and restart, making it a denial of service attack.
This was discovered by Varnish Software.
The potential impact is reduced or lost availability.
The cache will be empty after the restart unless Massive Storage Engine is used in persistence mode. An empty cache after restart will reduce overall performance due to an increased number of cache misses, and may cause higher load on the backend servers.
There is no potential for remote code execution or data leaks related to this vulnerability.
The recommended solution is to upgrade Varnish to one of the versions where this issue has been resolved, and then ensure that Varnish is restarted.
You should already have configured the Varnish Enterprise repository, so a normal upgrade will be enough:
sudo yum update varnish-plus
sudo systemctl restart varnish
Verify that the version that is installed is recent:
rpm -q varnish-plus
varnish-plus-6.0.5r3-1.el7.x86_64
You should already have configured the Varnish Enterprise repository, so a normal upgrade should be enough:
sudo apt-get update
sudo apt-get install --only-upgrade varnish-plus
sudo systemctl restart varnish
Verify that the version that is installed is recent:
dpkg -l varnish-plus
[...]
ii varnish-plus 6.0.5r3-1~xenial amd64 A supercharged version of the popular web cache, Varnish Cache
It is possible to mitigate the problem in several ways.
Setups using proxy protocol version 1 are not affected, so if the TLS termination proxy used supports both versions, switching to version 1 will mitigate the problem. This is the recommended mitigation method.
Note that version 1 of the proxy protocol will communicate less details about the connection than what is possible when using version 2. For example the SNI server name used by the connecting client will not be transferred, and can then not be queried using the Proxy VMOD in VCL.
When using the Hitch TLS proxy, version 1 of the proxy protocol can be
selected by replacing any write-proxy or write-proxy-v2 options
with write-proxy-v1, and restarting Hitch.
When using Hitch as the TLS termination proxy, one can work around the
problem by disallowing client connects connecting using a SNI server name
that does not match any of the configured certificates. This can be done
by adding the sni-nomatch-abort option to the Hitch configuration.
Please note that this mitigation strategy is only effective if none of the configured certificates allow wildcard domain names.
By increasing the session workspace, one can make sure that an attacker can not successfully exhaust the space available, thus not triggering the assert. The size it needs to be set at depends on the TLS proxy in use, and what fields it include in the proxy payload and how a remote client can influence its contents.
By default the session workspace is set to 512 bytes. The session
workspace can be changed by setting the workspace_session Varnish
parameter, and restarting the Varnish daemon.
When using Hitch as the TLS proxy, setting the session workspace to 34k
will mitigate the problem completely. Add “-p workspace_session=34k” to
the varnishd command line to set this value.
Note that increasing the session workspace will increase the amount of memory Varnish holds per connection, and you may want to decrease the memory cache size to compensate.
If a Varnish server is attacked using this vulnerability it will manifest as a series of assertion panics followed by child process restarts. These asserts will be logged to syslog. The assertion itself may come from multiple places in the source code.
The command line tool varnishadm can be used to indicate if this
assert has been triggered or is currently being triggered on an instance
of Varnish. The following command will output the latest panic message, if
any:
sudo varnishadm panic.show
Under normal and healthy circumstances it outputs:
Child has not panicked or panic has been cleared
2019-12-06
2019-12-11
2019-12-19
2020-01-07
2020-02-04