Search
Varnish Cache Plus

MSE 1.0

Varnish 4.0

Massive Storage Engine 1.0

Description

Varnish Massive Storage Engine 1.0 (MSE or MSE1) is an improved storage backend for Varnish. It is a replacement for the original malloc and file storage backends. Main improvements are decreased disk IO load and lower storage fragmentation.

MSE is designed and tested with storage sizes up to 10 TB, but expected to work on far larger workloads.

Availability

MSE1 is available in Varnish Cache Plus 4.0 from 4.0.2r1 and newer. In Varnish Cache Plus 4.1 MSE1 has been replaced by MSE2.

Installation

MSE is built into supported Varnish Cache Plus versions, and does not need additional installation steps.

Configuration

MSE is configured in the same way that the file storage backend is:

varnishd -s mse,<file>,<size>[,<segments>]

It expects a data file path on disk and what size it should be. Number of storage segments can be left out.

MSE introduces a set of new parameters:

mse_bigalloc               1M [bytes] (default)
mse_delay_writes           on [bool] (default)
mse_membuf_size            4 [pages] (default)
mse_minextfree             4k [bytes] (default)
mse_nuke_limit             10 (default)
mse_pad_writes             on [bool] (default)
mse_prune_factor           2 (default)
mse_prune_loop             10 (default)
mse_sendfile_min           0b [bytes] (default)

It is recommended to keep these at the default values, with the exception of mse_bigalloc. This parameter should be as large as the bigger objects cached, for example 5MB for HTTP Live Streaming setups.

Advanced configuration

Using multiple stores

Using multiple storage segments is possible. For example it can be beneficial to use malloc storage (standard) for normal objects, and MSE for special objects.

This can be done through setting explicit storage in vcl_backend_response, and giving each storage defined on the command line a name:

varnishd -s memory=malloc,1G -s msestorage=mse,/var/lib/varnish/mse,100G

Example VCL:

# Note: advanced usage
sub vcl_backend_response {
	set beresp.storage_hint = "memory";
	if (bereq.url ~ ".mp4$") {
		set beresp.storage_hint = "msestorage";
	}
}

Number of MSE segments

Number of storage segments within MSE can be set on the command line.

Increasing it reduces lock contention when doing memory allocations, especially when forcefully expunging content is necessary.

See the varnishd manual pages for more information on tuning this parameter for big cache sizes.