Migrating from MSE 3

Introduction

This document describes how to migrate an existing Varnish Enterprise installation from the Massive Storage Engine 3 (MSE3) to MSE4.

Both storage engines ship in the same Varnish Enterprise package. The engine is selected with the -s argument to varnishd:

# MSE3
-s mse,/etc/varnish/mse.conf

# MSE4, persisted
-s mse4,/etc/varnish/mse4.conf

# MSE4, ephemeral (memory-only cache)
-s mse4

The two engines are mutually exclusive; only one can be used at a time. MSE4 requires Varnish Enterprise 6.0.13r2 or newer.

Constraints that do not exist in MSE3

MSE4 adds two constraints that MSE3 did not have, and both have to be resolved before migrating.

The first is that MSE4 must be the only stevedore. MSE3 allowed multiple -s mse instances, and it could coexist with other stevedores such as malloc unless the memory governor was enabled. MSE4 refuses multiple MSE4 instances and cannot be combined with any other stevedore, so a multi-stevedore setup has to be consolidated into a single MSE4 configuration first.

The second is that the memory governor is always enabled. MSE3’s memcache_size setting is gone and there is no opt-out. Memory is sized exclusively with the memory_target parameter, which defaults to 80% of system memory. Transient storage becomes governed, memory-only objects. MSE4 also requires readable kernel memory statistics at startup.

Cache content does not survive the migration

The on-disk formats of the two engines are incompatible. MSE3 uses an LMDB-based book directory together with “MSE3 STORAGE” store files, while MSE4 uses single preallocated book and store files in its own format. There is no conversion tool and no import path, so new MSE4 devices start empty. Running mkfs.mse4 configure -f on a path that still holds an old MSE3 file warns Deleting invalid file device and recreates the file empty.

Plan for a cold cache after the migration. On a fleet, warm the nodes by replicating content with cluster.vcl, VHA, or by using a staged rollout.

Initializing the data files

Persisted MSE4 requires a one-time initialization of the data files before the first start:

$ mkfs.mse4 -c /etc/varnish/mse4.conf configure

The mkfs.mse4 utility handles all offline maintenance of the MSE4 data files. Its subcommands are:

• check-config: validate the configuration file.
• configure: create and initialize the book and store files.
• delete: remove the data files.
• resize: resize the data files. MSE3 could only grow them, while MSE4 can both grow and shrink them.
• headers: inspect the file headers.

MSE4 also maintains a statelog file recording device state changes, by default at /var/lib/mse/<hostname>.mse4_statelog.

Configuration file mapping

Both engines use libconfig env: {...} syntax, but the keys differ. The MSE3 configuration keys map to MSE4 as follows:

See the Configuration and Configuration Reference pages for the full MSE4 configuration syntax.

VCL migration

The MSE3 VCL interface (vmod_mse) maps to vmod_mse4 as follows:

Running a single VCL on a mixed fleet

During a staged rollout a fleet will usually run a mix of MSE3 and MSE4 nodes, and a single VCL can serve both. Importing vmod_mse4 on an MSE3 node, or vmod_mse on an MSE4 node, does not fail, and the mse.is_mse3() and mse4.is_mse4() functions are there to gate the engine-specific parts:

import mse;
import mse4;

sub vcl_backend_response {
    if (mse.is_mse3()) {
        if (bereq.url ~ "^/video/") { mse.set_stores("fast"); }
    } else if (mse4.is_mse4()) {
        if (bereq.url ~ "^/video/") { mse4.set_category("video"); }
    }
}

Calls into the vmod that does not match the running engine degrade instead of failing the VCL:

• vmod_mse4 on an MSE3 node: set_category() returns false, set_storage() does nothing, and get_location() returns NULL.
• vmod_mse on an MSE4 node: set_stores() returns false, set_weighting() does nothing, and is_mse3() returns false. Note that under MSE3 an unremedied false return from set_stores() causes a 503, and that this does not happen under MSE4.

Behavioral changes

The following differences affect running systems rather than configuration.

degradable is off by default, so MSE3 panics on a disk I/O error unless you have enabled it. MSE4 has no such setting: it always fails the affected device at runtime and keeps serving traffic from the remaining devices. Failed devices are inspected and managed with the CLI commands mse4.status, mse4.fail and mse4.reset, which are described on the Operations page.

Checksumming is on by default, as both write_checksum and verify_checksum default to true. This costs some CPU, and a checksum failure causes the object to be evicted.

lru_interval still applies. It continues to govern how often an object may be moved back to the head of the LRU list of its persisted store, so that a busy object is not reordered on every single hit. What is new is the per-category *reorder_interval* setting, which does the equivalent job for the memory cache LRU lists. The two settings cover different lists and are configured in different places: lru_interval is a varnishd parameter, *reorder_interval* is a category key in the MSE4 configuration file.

Book sizing is slot-based. Plan for about a 5G book per 27 million objects, or per 2TB of store, and note that a book holds at most 16 stores. See Persisted caching for details.

Further reading

• Basic Configuration / Getting Started
• Configuration
• Persisted caching
• Operations