Securing multi tier Varnish environments Tutorial

Introduction

Some environments consist of multiple tiers of Varnish instances. Typically these have an edge tier located close to clients and an origin protect tier located close to the origin. Sometimes one or more tiers sit between the edge tier and origin protect tier, but this is outside the scope of this tutorial.

The purpose of the edge tier is to serve external clients, and the purpose of the origin protect tier is to serve the edge tier and reduce the amount of requests to the origin. The origin protect tier should not under any circumstances be accessible directly by external clients, especially if certain logic is handled in the edge tier - such as authentication and authorization.

This tutorial shows how to add request signatures in the edge tier that are verified in the origin protect tier, in order to make sure that the origin protect tier is only accessible from the edge tier.

Logic

Prerequisites

In order to complete this guide, you will need the following:

A fully functional environment consisting of two tiers of Varnish instances and one or more backends. The approach works fine with more tiers, but for the sake of simplicity this tutorial addresses two tiers of Varnish instances specifically.
The hosts need to keep their system clocks reasonably synchronized. NTP is recommended.

Step 1 - Install VMOD digest

VMOD digest provides functionality to compute the HMAC of strings, and we will use this functionality to generate and verify the signatures. This means that it will have to be installed on all instances running Varnish in the various tiers.

In Varnish Enterprise, the VMOD digest is shipped in the package varnish-plus-vmods-extra, which is available from the Varnish Enterprise repository. Install it in Redhat and CentOS using:

sudo yum install varnish-plus-vmods-extra

Install it in Ubuntu and Debian using:

sudo apt-get install varnish-plus-vmods-extra

If using Varnish Cache, the VMOD digest needs to be installed from source.

Step 2 - Generate the signatures in the edge tier

The signatures will have to be generated and added to the backend requests in the edge tier. The backend request header X-Timestamp will contain the current timestamp, which we will use to avoid replay attacks, and the backend request header X-Signature will contain the signature, which is a SHA256 HMAC. In this tutorial the signature will be generated from the Host header, URL and the current timestamp, but feel free to add more headers, such as the Cookie header.

Add the following VCL configuration to the Varnish instances in the edge tier:

vcl 4.0;
import digest;

sub generate_signature {
    unset bereq.http.X-Timestamp;
    unset bereq.http.X-Signature;
    set bereq.http.X-Timestamp = now;

    # Using changeme as the secret. This should be be changed.
    set bereq.http.X-Signature = digest.hmac_sha256("changeme", bereq.http.host + bereq.url + bereq.http.X-Timestamp);
}

sub vcl_backend_fetch {
    # Call this late in vcl_backend_fetch, but before any return statements.
    call generate_signature;
}

Remember to change the secret in the VCL configuration above!

Step 3 - Verify the signature in the origin protect tier

All backend requests from the edge tier toward the origin protect tier will now contain a timestamp and a signature. These signatures need to be verified in the origin protect tier, and any requests that do not contain a valid signature will be rejected.

Add the following VCL configuration to the Varnish instances in the origin protect tier:

vcl 4.0;
import std;
import digest;

sub verify_signature {
    if (!req.http.X-Signature) {
        std.log("Signature is missing in the request");
        return(synth(403));
    }
    if (!req.http.X-Timestamp) {
        std.log("Timestamp is missing in the request");
        return(synth(403));
    }

    # Allow certain clock skew between the tiers. 30 seconds in this case.
    if (std.time(req.http.X-Timestamp, now + 1d) > now + 30s) {
        std.log("Timestamp is in the future");
        return(synth(403));
    }
    if (std.time(req.http.X-Timestamp, now - 1d) < now - 30s) {
        std.log("Timestamp is in the past");
        return(synth(403));
    }

    # Verify HMAC signature
    if (req.http.X-Signature != digest.hmac_sha256("changeme", req.http.host + req.url + req.http.X-Timestamp)) {
        std.log("Signature not valid");
        return(synth(403));
    }

    # Clean up to avoid passing these headers to the next tier.
    unset req.http.X-Timestamp;
    unset req.http.X-Signature;
}

sub vcl_recv {
    # Call this early in vcl_recv.
    call verify_signature;
}

Remember to change the secret in the VCL configuration above!

Step 4 - Review using varnishlog

Run the varnishlog command below in the edge tier while executing an HTTP request to verify that the X-Timestamp and X-Signature headers are successfully added in vcl_backend_fetch as specified in step 2.

edge-tier$ sudo varnishlog -i VCL_call,ReqMethod,BereqMethod,ReqURL,BereqURL,BerespStatus,RespStatus -I 'X-Timestamp' -I 'X-Signature'
*   << BeReq    >> 98319
-   BereqMethod    GET
-   BereqURL       /
-   VCL_call       BACKEND_FETCH
-   BereqHeader    X-Timestamp: Sat, 14 Oct 2017 11:30:19 GMT
-   BereqHeader    X-Signature: 0xdd3c3788c643b94030f9154a696c13f06e1451d305b4879a183551f7f1db6f68
-   BerespStatus   200
-   VCL_call       BACKEND_RESPONSE

*   << Request  >> 98318
-   ReqMethod      GET
-   ReqURL         /
-   VCL_call       RECV
-   VCL_call       HASH
-   VCL_call       PASS
-   RespStatus     200
-   VCL_call       DELIVER

Run the same varnishlog command in the origin protect tier to verify that the X-Timestamp and X-Signature headers are received in the request from the edge tier, and also that the headers are unset before the backend request is sent to the web application in the origin tier.

origin-protect-tier$ sudo varnishlog -i VCL_call,ReqMethod,BereqMethod,ReqURL,BereqURL,BerespStatus,RespStatus -I 'X-Timestamp' -I 'X-Signature'
*   << BeReq    >> 26
-   BereqMethod    GET
-   BereqURL       /
-   VCL_call       BACKEND_FETCH
-   BerespStatus   200
-   VCL_call       BACKEND_RESPONSE

*   << Request  >> 25
-   ReqMethod      GET
-   ReqURL         /
-   BereqHeader    X-Timestamp: Sat, 14 Oct 2017 11:30:19 GMT
-   BereqHeader    X-Signature: 0xdd3c3788c643b94030f9154a696c13f06e1451d305b4879a183551f7f1db6f68
-   VCL_call       RECV
-   ReqUnset       X-Timestamp: Sat, 14 Oct 2017 11:30:19 GMT
-   ReqUnset       X-Signature: 0xdd3c3788c643b94030f9154a696c13f06e1451d305b4879a183551f7f1db6f68
-   VCL_call       HASH
-   VCL_call       PASS
-   RespStatus     200
-   VCL_call       DELIVER

External clients sending HTTP requests directly to the origin protect tier will now be rejected.

client$ curl -iI https://origin.protect.tier/
HTTP/1.1 403 Forbidden
Date: Sat, 14 Oct 2017 15:36:37 GMT
Server: Varnish
X-Varnish: 32774
Content-Type: text/html; charset=utf-8
Retry-After: 5
Content-Length: 249
Connection: keep-alive

Figuring out why a request fails is done using varnishlog:

$ sudo varnishlog -i VCL_Log,VCL_call,ReqMethod,ReqURL,RespStatus -I 'X-Timestamp' -I 'X-Signature'
*   << Request  >> 32774
-   ReqMethod      HEAD
-   ReqURL         /
-   VCL_call       RECV
-   VCL_Log        Signature is missing in the request
-   VCL_call       HASH
-   RespStatus     403
-   VCL_call       SYNTH

The following is a simple bash script to generate HMAC signatures on the command line:

#!/bin/bash

# usage: token_checker.sh KEY [STRING....]

key="$1"
shift
s=
for i in "$@"; do
	s="$s$i"
done
echo $s
echo -n "$s" | openssl sha256 -hmac "$1"

Conclusion

HMAC-based signature verification is now implemented between the two tiers of Varnish instances, and this has been done transparently without affecting client request/responses or origin requests/responses.

Clients are now able to access the edge tier as before, but can no longer access the origin protect tier directly.

A complete test case for this setup is available here.