Partial requests to slice large objects Tutorial

Introduction

This tutorial shows how to configure Varnish Enterprise to slice large objects fetched from Amazon S3. This to achieve better cache capacity utilization and reduce network transfer when caching and distributing large objects via Varnish.

Varnish supports partial requests coming from clients by default, but will strip away the range header and fetch the entire object from the backend before serving the partial request. This is efficient in certain situations where multiple clients are requesting all parts of an object multiple times. However, sometimes it is more beneficial to slice the object and fetch it using multiple partial requests that are cached individually. This is particularly efficient when serving large files such as ISO files and non-segmented video files.

Prerequisites

• One or more servers running Varnish Enterprise with Amazon S3 as a backend. The basic setup tutorials explain how this can be done.

Step 1 - Enable partial requests

Partial request caching is provided using the slicer module, which is included in Varnish Enterprise. The configuration below will automatically slice backend requests into chunks that are maximum 25 MB in size, and cache them individually according to the cache policy set in the VCL or by the origin.

import slicer;

sub vcl_backend_response {
    if (!slicer.enable()) {
        set beresp.transit_buffer = 25M;
        return (pass);
    }
}

Refer to the slicer module documentation for more information.

Next steps

The configuration above can be expanded with more functionality, as covered by the other Varnish tutorials for Amazon S3.