SearchVarnish Enterprise
Varnish Massive Storage Engine 1.0 (MSE or MSE1) is an improved storage
backend for Varnish. It is a replacement for the original malloc and file
storage backends. Main improvements are decreased disk IO load and lower
storage fragmentation.
MSE is designed and tested with storage sizes up to 10 TB, but expected to work on far larger workloads.
MSE1 is available in Varnish Enterprise 4.0 from 4.0.2r1 and newer. In Varnish
Cache Plus 4.1 MSE1 has been replaced by MSE2.
MSE is built into supported Varnish Enterprise versions, and does not need additional installation steps.
MSE is configured in the same way that the file storage backend is:
varnishd -s mse,<file>,<size>[,<segments>]
It expects a data file path on disk and what size it should be. Number of storage segments can be left out.
MSE introduces a set of new parameters:
mse_bigalloc 1M [bytes] (default)
mse_delay_writes on [bool] (default)
mse_membuf_size 4 [pages] (default)
mse_minextfree 4k [bytes] (default)
mse_nuke_limit 10 (default)
mse_pad_writes on [bool] (default)
mse_prune_factor 2 (default)
mse_prune_loop 10 (default)
mse_sendfile_min 0b [bytes] (default)
It is recommended to keep these at the default values, with the exception of
mse_bigalloc. This parameter should be as large as the bigger objects cached,
for example 5MB for HTTP Live Streaming setups.
Using multiple storage segments is possible. For example it can be beneficial to use malloc storage (standard) for normal objects, and MSE for special objects.
This can be done through setting explicit storage in vcl_backend_response, and giving
each storage defined on the command line a name:
varnishd -s memory=malloc,1G -s msestorage=mse,/var/lib/varnish/mse,100G
Example VCL:
# Note: advanced usage
sub vcl_backend_response {
set beresp.storage_hint = "memory";
if (bereq.url ~ ".mp4$") {
set beresp.storage_hint = "msestorage";
}
}
Number of storage segments within MSE can be set on the command line.
Increasing it reduces lock contention when doing memory allocations, especially when forcefully expunging content is necessary.
See the varnishd manual pages for more information on tuning this parameter for
big cache sizes.