SearchVarnish Enterprise
The kvstore vmod allows for high performance key value storage with optional TTLs.
This API applies to Varnish Enterprise 6.0 and higher.
For the older kvstore API please refer to the kvstore 4.x API.
init_file() and init_conf() can be used to regularly sync data
from file into a kvstore. These functions can be safely used while threads
are accessing the kvstore being initialized. When used,
threads will not block and will have access
to the previous instance of the kvstore while the new kvstore is being built
from file. When ready, threads will be directed to the new kvstore and the
previous instance will be safely cleaned up.
Note that data from the previous instance will not carry over to the new instance.
kvstore has a memory overhead of 100 bytes per bucket and 100 bytes per
key plus the actual key and value size. So for 1 million keys with a average 1KB
key and value size stored in 100,000 buckets, the memory usage is approximately:
100000 buckets * 100 bytes = 10MB
1000000 keys * 100 bytes = 100MB
1000000 keys * 1KB size = 1GB
TOTAL = 1.11GB
On a 4 core Intel Core i7 with an SSD, the above kvstore takes 1.25s to load
from file and has an average key read time of 600ns.
On the same system, a kvstore with 1 million keys, 100k buckets, 70% readers,
and 30% writers has a throughput of 3 millions operations per second.
Key length, value length, and overall kvstore size is only bounded by
available memory.
Buckets are implemented using red-black trees. The performance complexity of
kvstore is:
O(log(size/buckets))
When using TTLs, expired keys are only removed when they are landed on during a
get() or set() operation or when explicitly deleted. If you have a highly
dynamic key set, expired keys may never be landed on and your kvstore size can
grow unchecked. This is due to the way keys are hashed and the fact that each
bucket stores keys in a red black tree. If this is the case, it’s recommended to
periodically call compact() to remove expired keys. You can use the size()
function to gauge kvstore growth.
Storing variables into kvstore:
vcl 4.0;
import kvstore;
sub vcl_init
{
new kvs = kvstore.init();
kvs.set("alpha", "one");
kvs.set("beta", "two");
}
sub vcl_recv
{
set req.http.alpha = kvs.get("alpha", "error");
set req.http.beta = kvs.get("beta", "error");
}
Loading a CSV file into a kvstore:
vcl 4.0;
import kvstore;
import std;
sub vcl_init
{
new csv = kvstore.init();
// Load plain comma separated data
if (csv.init_file("/some/path/data.csv", ",") < 0) {
std.syslog(3, "Varnish kvstore ERROR: couldn't load /some/path/data.csv");
return(fail);
}
}
sub vcl_recv
{
// Reload data
if (req.method == "REFRESH") {
set req.http.kvinit = csv.init_file("/some/path/data.csv", ",");
if (req.http.kvinit == "-1") {
return(synth(500, "-1"));
} else {
return(synth(200, req.http.kvinit));
}
}
set req.http.somekey = csv.get("somekey", "error");
}
Using kvstore to cache a value:
vcl 4.0;
import kvstore;
sub vcl_init
{
new kvcache = kvstore.init();
}
sub vcl_recv
{
// Lookup "somekey" in cache
set req.http.cachevalue = kvcache.get("somekey", "");
if(req.http.cachevalue == "") {
// Store "somevalue" for 10 seconds
set req.http.cachevalue = "somevalue";
kvcache.set("somekey", req.http.cachevalue, 10s);
}
}
Using kvstore to synchronize an if statement:
vcl 4.0;
import kvstore;
import std;
sub vcl_init
{
new sync = kvstore.init();
}
sub vcl_recv
{
// This block will only be taken once per url every 10 minutes
if (sync.counter(req.url, 1, 10m) == 1) {
std.log("URL called: " + req.url);
}
}
Logging kvstore counter values to varnishstat:
vcl 4.0;
import kvstore;
sub vcl_init
{
new vsc = kvstore.init();
}
sub vcl_deliver
{
// Count all media responses and log it in varnishstat
if (resp.http.Content-Type ~ "audio|image|video") {
vsc.counter("media_responses", 1, varnishstat = true);
}
}
OBJECT init(INT buckets = 250, ENUM {STATIC, GLOBAL, REQUEST} scope = GLOBAL)
Initializes a kvstore. Can be used to re-initialize an active kvstore.
The resulting kvstore is empty.
This can be used as both a constructor and a method.
Arguments:
buckets accepts type INT with a default value of 250 optional
scope is an ENUM that accepts values of STATIC, GLOBAL, and REQUEST with a default value of GLOBAL optional
Type: Object
Returns: Object.
VOID .init(INT buckets = 0, BOOL scrub = 0)
Initializes the kvstore vmod.
Arguments:
buckets accepts type INT with a default value of 0 optional
scrub accepts type BOOL with a default value of 0 optional
Type: Method
Returns: None
INT .init_file(STRING path, STRING delimiter, INT buckets = 0)
Equivalent to init(), but initializes from a file. Can be used to rebuild
an active kvstore. Modifications to the kvstore are not synced back to
the file.
The delimiter is string used to separate the key and the value. Only the first occurrence of the delimiter is used as the key value separator.
For example:
foo==bar
If the delimiter is =, then there will be a key named foo with the value =bar.
This is because only the first = is interpreted as a delimiter.
If the delimiter is not found or is empty, the whole line is treated as a key with an empty value.
Arguments:
path accepts type STRING
delimiter accepts type STRING
buckets accepts type INT with a default value of 0 optional
Type: Method
Returns: Int
INT .init_conf(STRING path, INT buckets = 0)
Equivalent to init(), but initializes from a configuration file. Can be
used to rebuild an active kvstore. Modifications to the kvstore are not
synced back to the file. A configuration file uses ini file
format. Comments start with #, whitespace is stripped, keys and values are
separated with an = (equal) sign, and section names are surrounded with [ ]
and are prepended to key names with _ as a separator.
Arguments:
path accepts type STRING
buckets accepts type INT with a default value of 0 optional
Type: Method
Returns: Int
STRING .get(STRING key, STRING default = 0)
Get a key from the kvstore. If it’s not found, default is returned.
Arguments:
key accepts type STRING
default accepts type STRING with a default value of 0 optional
Type: Method
Returns: String
VOID .set(STRING key, STRING value, DURATION ttl = 0)
Sets a key in the kvstore with an optional ttl.
Arguments:
key accepts type STRING
value accepts type STRING
ttl accepts type DURATION with a default value of 0 optional
Type: Method
Returns: None
INT .counter(STRING key, INT count, DURATION ttl = 0, BOOL varnishstat = 0, STRING comment = "", BOOL auto = 0)
Create a counter for key and add count to it. The initial value of a counter is always 0. If a counter expires or is deleted, it’s value will be reset to 0. Operations on this counter are atomic, and it can be safely used for synchronization.
Note: If ttl is greater than 0s, this is the ttl for the counter (nearest
second). After the ttl duration, the counter will reset. If 0s, the
counter is stored forever.
If varnishstat is true, the value of this counter will be logged in
varnishstat in the format of KVSTORE.object_name.vcl_name.key.
The comment parameter will dictate the description provided to varnishstat.
Arguments:
key accepts type STRING
count accepts type INT
ttl accepts type DURATION with a default value of 0 optional
varnishstat accepts type BOOL with a default value of 0 optional
comment accepts type STRING with a default value of empty. optional
auto accepts type BOOL with a default value of 0 optional
Type: Method
Returns: Int
BOOL .limit(STRING key, INT max, INT count = 1)
Increment a key and check if the value is more than max.
Arguments:
key accepts type STRING
max accepts type INT
count accepts type INT with a default value of 1 optional
Type: Method
Returns: Bool
BACKEND .get_backend(STRING key, BACKEND default = 0)
Get a backend from the kvstore. If it’s not found, default is returned.
Arguments:
key accepts type STRING
default accepts type BACKEND with a default value of 0 optional
Type: Method
Returns: Backend
VOID .set_backend(STRING key, BACKEND value, DURATION ttl = 0)
Sets a backend in the kvstore with an optional ttl. Not safe to use with
dynamic backends.
Arguments:
key accepts type STRING
value accepts type BACKEND
ttl accepts type DURATION with a default value of 0 optional
Type: Method
Returns: None
VOID .lock(STRING key)
Lock a key.
Arguments:
key accepts type STRINGType: Method
Returns: None
VOID .unlock(STRING key)
Unlock a key.
Arguments:
key accepts type STRINGType: Method
Returns: None
VOID .delete(STRING key)
Delete a key from the kvstore.
Arguments:
key accepts type STRINGType: Method
Returns: None
INT .size()
Get the current size of the kvstore.
Arguments: None
Type: Method
Returns: Int
INT .compact()
Removes expired keys from the kvstore and returns the number of keys removed.
A kvstore can be configured with one of three scopes. Each of
these scopes define different lifetimes and sharing rules.
Values in a GLOBAL kvstore will have a lifetime equal to that of the VCL it was loaded in. Access to values is shared across request transactions, i.e. you will see the same value when referenced from separate request transactions and edits will be visible to other requests.
This is the default scope.
Changes to values in a REQUEST kvstore will be cleaned up when the request terminates. Access is limited to that single request.
If a value is set in vcl_init this will be treated as a default, visible to all requests. Any edits will have its visibility restricted to the request that did the edit.
Values in a STATIC kvstore will have a lifetime equal to the loading/unloading of the VMOD. All access is shared across VCLs and request transactions.
Arguments: None
Type: Method
Returns: Int
The kvstore VMOD is available in Varnish Enterprise version 6.0.0r0 and later.