Persistent caching to increase the cache capacity Tutorial

Introduction

This tutorial shows how to configure Varnish Enterprise to cache objects from Amazon S3 using the local filesystem. This makes it possible to increase the cache capacity beyond the amount of memory available, and also to restart Varnish without dropping the cached content.

Persistent caching is provided using Massive Storage Engine (MSE) stevedore, which is included in Varnish Enterprise. Refer to the Massive Storage Engine documentation for more information.

Prerequisites

• One or more servers running Varnish Enterprise with Amazon S3 as a backend. The basic setup tutorials explain how this can be done.

Step 1 - Create an MSE configuration file

Create the file /var/lib/mse/mse.conf with the following contents, which will allow Varnish to use 100 GB to cache contents in /var/lib/mse/store.dat and 2 GB to store metadata in /var/lib/mse/book. Adjust filesystem paths and sizes to your environment:

env: {
    id = "mse";
    # how much RAM can be used, "auto" defaults to 80% of the total memory
    # see the "Memory Governor" section for more information
    memcache_size = "auto";

    # one caching set, containing one book (for metadata) and one or more
    # stores (for the actual cache content)
    books = ( {
            id = "book";
            # the location of the book
            directory = "/var/lib/mse/book";
            # how large the book should be, a cautious rule of thumb is
            # 2KB per object in the stores
            database_size = "2G";

            stores = ( {
                    id = "store";
                    # the store is actually a large contiguous file, note that
                    # it does NOT have to be on the same directory or disk as
                    # the book
                    filename = "/var/lib/mse/store.dat";
                    # how large the store should be.
                    size = "100G";
                    } );
            } );
};

We recommend putting one book and one store on each drive if caching on multiple drives. Refer to the Massive Storage Engine documentation for more information.

Step 2 - mkfs.mse

Initialize the book and store for caching:

sudo mkfs.mse -c /var/lib/mse/mse.conf

Step 3 - Enable MSE in Varnish

Edit the varnish systemd service file:

sudo systemctl edit --full varnish

Replace the existing stevedore configuration, specified using the -s argument, with:

-s mse,/var/lib/varnish/mse.conf

Step 4 - Use Memory Governor to control memory usage

Edit the varnish systemd service file:

sudo systemctl edit --full varnish

Add a new parameter to specify the memory target. By default it is 80% of the available memory on the server, and it can be specified both in percent and in bytes. The following will allow Varnish to use 45 GB of memory:

-p memory_target=45G

Next steps

The configuration above can be expanded with more functionality, as covered by the other Varnish tutorials for Amazon S3.