Key value storage (kvstore)

Varnish 6.0

Description

High performance key value storage with optional TTLs. This API applies to Varnish Cache Plus 6.0 and higher.

Example VCL

Variables

Storing variables into kvstore:

vcl 4.0;

import kvstore;

sub vcl_init
{
    new kvs = kvstore.init();

    kvs.set("alpha", "one");
    kvs.set("beta", "two");
}

sub vcl_recv
{
    set req.http.alpha = kvs.get("alpha", "error");
    set req.http.beta = kvs.get("beta", "error");
}

CSV File

Loading a CSV file into a kvstore:

vcl 4.0;

import kvstore;
import std;

sub vcl_init
{
    new csv = kvstore.init();

    // Load plain comma separated data
    if (csv.init_file("/some/path/data.csv", ",") < 0) {
        std.syslog(3, "Varnish kvstore ERROR: couldn't load /some/path/data.csv");
        return(fail);
    }
}

sub vcl_recv
{
    // Reload data
    if (req.method == "REFRESH") {
        req.http.kvinit = csv.init_file("/some/path/data.csv", ",");
        if (req.http.kvinit == "-1") {
            return(synth(500, "-1"));
        } else {
            return(synth(200, req.http.kvinit));
        }
    }

    set req.http.somekey = csv.get("somekey", "error");
}

Value Caching

Using kvstore to cache a value:

vcl 4.0;

import kvstore;

sub vcl_init
{
    new kvcache = kvstore.init();
}

sub vcl_recv
{
    // Lookup "somekey" in cache
    set req.http.cachevalue = kvcache.get("somekey", "");

    if(req.http.cachevalue == "") {
        // Store "somevalue" for 10 seconds
        set req.http.cachevalue = "somevalue";
        kvcache.set("somekey", req.http.cachevalue, 10s);
    }
}

Synchronization

Using kvstore to synchronize an if statement:

vcl 4.0;

import kvstore;
import std;

sub vcl_init
{
    new sync = kvstore.init();
}

sub vcl_recv
{
    // This block will only be taken once per url every 10 minutes
    if (sync.counter(req.url, 1, 10m) == 1) {
        std.log("URL called: " + req.url);
    }
}

Varnishstat

Logging kvstore counter values to varnishstat:

vcl 4.0;

import kvstore;

sub vcl_init
{
    new vsc = kvstore.init();
}

sub vcl_deliver
{
    // Count all media responses and log it in varnishstat
    if (resp.http.Content-Type ~ "audio|image|video") {
        vsc.counter("media_responses", 1, varnishstat = true);
    }
}

Functions

init

OBJECT kvstore.init(INT buckets, ENUM scope, BOOLEAN scrub)

Description

Initializes a kvstore. Can be used to re-initialize an active kvstore. The resulting kvstore is empty. This can be used as both a constructor and a method.
Return value

The kvstore object.
- buckets
  
  The number of hash buckets to create (roughly 1 per key). Defaults to 250 buckets.
- scope
  
  GLOBAL or REQUEST scope. If REQUEST, kvstore is local to the request (frontend or backend). Defaults to GLOBAL. When using REQUEST scope, keep buckets to a minimum for the best performance.
- scrub
  
  Zero out any previous data inside of the kvstore being initialized. Defaults to false.

init_file

INT .init_file(STRING path, STRING delimiter, INT buckets)

Description

Equivalent to init(), but initializes from a file. Can be used to rebuild an active kvstore. Modifications to the kvstore are not synced back to the file.
Return value

The numbers of keys loaded. -1 if the file cannot be read (resulting kvstore will be empty).
- path
  
  The path to the file with the key value data. 1 key and value per line.
- delimiter
  
  The delimiter string used to separate the key and the value. Only the first occurrence of the delimiter is used as the key value separator. If the delimiter is not found or is empty, the whole line is treated as a key with an empty value.
- buckets
  
  The number of hash buckets to create (roughly 1 per key). Defaults to the bucket value used in init().

init_conf

INT .init_conf(STRING path, INT buckets)

Description

Equivalent to init(), but initializes from a configuration file. Can be used to rebuild an active kvstore. Modifications to the kvstore are not synced back to the file. A configuration file uses ini file format. Comments start with #, whitespace is stripped, keys and values are separated with an = (equal) sign, and section names are surrounded with [ ] and are prepended to key names with _ as a separator.
Return value

The numbers of keys loaded. -1 if the file cannot be read (resulting kvstore will be empty).
- path
  
  The path to the file with the conf data. 1 key and value per line.
- buckets
  
  The number of hash buckets to create (roughly 1 per key). Defaults to the bucket value used in init().

get

STRING .get(STRING key, STRING default)

Description

Get a key from the kvstore. If its not found, default is returned.
Return value

The key value.
- key
  
  The key name.
- default
  
  The default value if not found. Optional.

set

VOID .set(STRING key, STRING value, DURATION ttl)

Description

Sets a key in the kvstore with an optional ttl.
Return value

None
- key
  
  The key name.
- value
  
  They key value.
- ttl
  
  Optional. If greater than 0s, this is the ttl for the key (nearest second). After the ttl duration, the key is deleted. If 0s, the key is stored forever.

counter

INT .counter(STRING key, INT count, DURATION ttl, BOOL varnishstat, STRING comment, BOOL auto)

Description

Create a counter for key and add count to it. The initial value of a counter is always 0. If a counter expires or is deleted, its value will be reset to 0. Operations on this counter are atomic and it can be safely used for synchronization.
Return value

The new counter value.
- key
  
  The key name.
- count
  
  The value to add to the counter. Use 0 to read the current counter value.
- ttl
  
  Optional. If greater than 0s, this is the ttl for the counter (nearest second). After the ttl duration, the counter will reset. If 0s, the counter is stored forever.
- varnishstat
  
  Optional. If true, the value of this counter will be logged in varnishstat in the format of KVSTORE.object_name.vcl_name.key. Defaults to false.
- comment
  
  Optional. If varnishstat is true, this string will be used as the description of the varnishstat counter.
- auto
  
  Optional. Automatically decrement the counter after finishing the request. Defaults to false.

limit

BOOl .limit(STRING key, INT max, INT count)

Description

Increment a key and check if the value is more than max.
Return value

If the value of the key is greater than max false is returned. Otherwise true is returned.
- key
  
  The key name.
- key
  
  The maximum value the key can be.
- count
  
  Optional. The value to add to the counter. Use 0 to read the current counter value. Defaults to 1.

get_backend

BACKEND .get_backend(STRING key, BACKEND default)

Description

Get a backend from the kvstore. If its not found, default is returned.
Return value

The backend value.
- key
  
  The key name.
- default
  
  The default backend if not found. Optional.

set_backend

VOID .set_backend(STRING key, BACKEND value, DURATION ttl)

Description

Sets a backend in the kvstore with an optional ttl. Not safe to use with dynamic backends.
Return value

None
- key
  
  The key name.
- value
  
  The backend value.
- ttl Optional. If greater than 0s, this is the ttl for the backend (nearest second). After the ttl duration, the backend is removed. If 0s, the backend is stored forever.

delete

VOID .delete(STRING key)

Description

Delete a key from the kvstore.
Return value

None
- key
  
  The key name.

size

INT .size()

Description

Get the current size of the kvstore.
Return value

The size.

compact

INT .compact()

Description

Removed expired keys from the kvstore.
Return value

The number of keys removed.

Reloading from file

init_file() and init_conf() can be used to regularly sync data from file into a kvstore. These functions can be safely used while threads are accessing the kvstore being initialized. When used, threads will not block and will have access to the previous instance of the kvstore while the new kvstore is being built from file. When ready, threads will be directed to the new kvstore and the previous instance will be safely cleaned up.

Note that data from the previous instance will not carry over to the new instance.

Memory and performance

kvstore has a memory overhead of 100 bytes per bucket and 100 bytes per key plus the actual key and value size. So for 1 million keys with a average 1KB key and value size stored in 100,000 buckets, the memory usage is approximately:

    100000 buckets * 100 bytes = 10MB
    1000000 keys * 100 bytes = 100MB
    1000000 keys * 1KB size = 1GB

    TOTAL = 1.11GB

On a 4 core Intel Core i7 with an SSD, the above kvstore takes 1.25s to load from file and has an average key read time of 600ns.

On the same system, a kvstore with 1 million keys, 100k buckets, 70% readers, and 30% writers has a throughput of 3 millions operations per second.

Key length, value length, and overall kvstore size is only bounded by available memory.

Buckets are implemented using red-black trees. The performance complexity of kvstore is:

O(log(size/buckets))

When using TTLs, expired keys are only removed when they are landed on during a get() or set() operation or when explicitly deleted. If you have a highly dynamic key set, expired keys may never be landed on and your kvstore size can grow unchecked. This is due to the way keys are hashed and the fact that each bucket stores keys in a red black tree. If this is the case, its recommended to periodically call compact() to remove expired keys. You can use the size() function to gauge kvstore growth.