Varnish Cache data leak when using file stevedore with synthetic objects Security

Published November 15, 2017.

Overview

A data leak can be triggered in Varnish Cache by a remote client if Varnish Cache is using the file stevedore for synthetic error objects and the backend does not respond properly or times out.

This was discovered and patched respectively by Carlo Cannas of Altervista.org and @shamger.

Impact

Potential impact is data leak and segmentation fault:

The data leaked will be an excerpt from the anonymously mapped memory of the varnishd process, limited upwards to 4k bytes and it will be present in the body of the synthesized error object created in place of the failed backend fetch.
Some requests will cause varnishd to segfault. This happens when the buffer overflow causes a read of unmapped memory.

Repeated attempts to fetch leaked memory will eventually produce a segmentation fault which can be detected by the administrator.

Prerequisites

All of the following conditions are required to trigger the problem:

A file stevedore must be configured using the -s [<name>=]file,... option to varnishd.
A synthetic object must be created in vcl_backend_error{}. This happens e.g. when Varnish is unable to get a properly formatted HTTP response from the backend, or if requested from VCL by doing return (fail) or return (abandon).

(Note: The problem also applies in the same way for uses of the deprecated and unsupported stevedore persistent.)

Status

Affected software versions

Varnish Cache: 4.1.0 - 5.2.0

Not affected software versions

Varnish Cache: Earlier than 4.1.0
Varnish Enterprise: All versions

Resolved in

Varnish Cache: 4.1.9
Varnish Cache: 5.2.1

Solution

The solution is to upgrade Varnish to one of the versions where this issue has been resolved, and then ensure that Varnish is restarted.

Workaround

Organizations that are not able to upgrade their Varnish environment on short notice can switch from the file stevedore to the malloc stevedore.

The stevedore to use is specified using the -s option to the varnishd process:

# Existing configuration specifying file
varnishd -s file,1G,/var/lib/varnish/varnish_storage.bin

# New configuration specifying malloc
varnishd -s malloc,1G

Please note that the malloc stevedore uses system memory, and the cache size should be adjusted accordingly.

This is configured in:

/etc/sysconfig/varnish.params in Redhat and derivatives.
/etc/default/varnish in Debian and derivatives.

Identification

The vulnerability may trigger a segmentation fault, which will appear in the system logs.

Timeline

2017-09-18

@shamger opened pull request #2429 that fixed the segmentation fault.

2017-09-19

Varnish Software accepted the pull request and backported the patch to the Varnish Cache 4.1 branch.

2017-10-31

Carlo Cannas of Altervista.org reported to PHK that the issue can also cause data leaks in certain situations.

2017-11-08

The security teams in Ubuntu, Debian, RedHat and FreeBSD were notified.

2017-11-15

Varnish Cache packages released to the official Varnish Cache package repositories.
Public announcement.

2017-11-22

Update the Prerequisites section removing the last clause as this was found to be inaccurate.

Acknowledgement

Varnish Software would like to thank Carlo Cannas of Altervista.org and @shamger for discovering and patching the vulnerability.