Varnish Cache Plus

Key value storage (kvstore)

Varnish 6.0

Description

High performance key value storage with optional TTLs. This API applies to Varnish Cache Plus 6.0 and higher.

kvstore 4.x API

Example VCL

Variables

Storing variables into kvstore:

vcl 4.0;

import kvstore;

sub vcl_init
{
    new kvs = kvstore.init();

    kvs.set("alpha", "one");
    kvs.set("beta", "two");
}

sub vcl_recv
{
    set req.http.alpha = kvs.get("alpha", "error");
    set req.http.beta = kvs.get("beta", "error");
}

CSV File

Loading a CSV file into a kvstore:

vcl 4.0;

import kvstore;
import std;

sub vcl_init
{
    new csv = kvstore.init();

    // Load plain comma separated data
    if (csv.init_file("/some/path/data.csv", ",") < 0) {
        std.syslog(3, "Varnish kvstore ERROR: couldn't load /some/path/data.csv");
        return(fail);
    }
}

sub vcl_recv
{
    // Reload data
    if (req.method == "REFRESH") {
        req.http.kvinit = csv.init_file("/some/path/data.csv", ",");
        if (req.http.kvinit == "-1") {
            return(synth(500, "-1"));
        } else {
            return(synth(200, req.http.kvinit));
        }
    }

    set req.http.somekey = csv.get("somekey", "error");
}

Value Caching

Using kvstore to cache a value:

vcl 4.0;

import kvstore;

sub vcl_init
{
    new kvcache = kvstore.init();
}

sub vcl_recv
{
    // Lookup "somekey" in cache
    set req.http.cachevalue = kvcache.get("somekey", "");

    if(req.http.cachevalue == "") {
        // Store "somevalue" for 10 seconds
        set req.http.cachevalue = "somevalue";
        kvcache.set("somekey", req.http.cachevalue, 10s);
    }
}

Synchronization

Using kvstore to synchronize an if statement:

vcl 4.0;

import kvstore;
import std;

sub vcl_init
{
    new sync = kvstore.init();
}

sub vcl_recv
{
    // This block will only be taken once per url every 10 minutes
    if (sync.counter(req.url, 1, 10m) == 1) {
        std.log("URL called: " + req.url);
    }
}

Functions

init

OBJECT kvstore.init(INT buckets, ENUM scope, BOOLEAN scrub)

  • Description

    Initializes a kvstore. Can be used to re-initialize an active kvstore. The resulting kvstore is empty. This can be used as both a constructor and a method.

  • Return value

    The kvstore object.

    • buckets

      The number of hash buckets to create (roughly 1 per key). Defaults to 250 buckets.

    • scope

      GLOBAL or REQUEST scope. If REQUEST, kvstore is local to the request (frontend or backend). Defaults to GLOBAL. When using REQUEST scope, keep buckets to a minimum for the best performance.

    • scrub

      Zero out any previous data inside of the kvstore being initialized. Defaults to false.

init_file

INT .init_file(STRING path, STRING delimiter, INT buckets)

  • Description

    Equivalent to init(), but initializes from a file. Can be used to rebuild an active kvstore. Modifications to the kvstore are not synced back to the file.

  • Return value

    The numbers of keys loaded. -1 if the file cannot be read (resulting kvstore will be empty).

    • path

      The path to the file with the key value data. 1 key and value per line.

    • delimiter

      The delimiter string used to separate the key and the value. Only the first occurrence of the delimiter is used as the key value separator. If the delimiter is not found or is empty, the whole line is treated as a key with an empty value.

    • buckets

      The number of hash buckets to create (roughly 1 per key). Defaults to the bucket value used in init().

init_conf

INT .init_conf(STRING path, INT buckets)

  • Description

    Equivalent to init(), but initializes from a configuration file. Can be used to rebuild an active kvstore. Modifications to the kvstore are not synced back to the file. A configuration file uses ini file format. Comments start with #, whitespace is stripped, keys and values are separated with an = (equal) sign, and section names are surrounded with [ ] and are prepended to key names.

  • Return value

    The numbers of keys loaded. -1 if the file cannot be read (resulting kvstore will be empty).

    • path

      The path to the file with the conf data. 1 key and value per line.

    • buckets

      The number of hash buckets to create (roughly 1 per key). Defaults to the bucket value used in init().

get

STRING .get(STRING key, STRING default)

  • Description

    Get a key from the kvstore. If its not found, default is returned.

  • Return value

    The key value.

    • key

      The key name.

    • default

      The default value if not found. Optional.

set

VOID .set(STRING key, STRING value, DURATION ttl)

  • Description

    Sets a key in the kvstore with an optional ttl.

  • Return value

    None

    • key

      The key name.

    • value

      They key value.

    • ttl

      Optional. If greater than 0s, this is the ttl for the key (nearest second). After the ttl duration, the key is deleted. If 0s, the key is stored forever.

counter

INT .counter(STRING key, INT count, DURATION ttl)

  • Description

    Create a counter for key and add count to it. The initial value of a counter is always 0. If a counter expires or is deleted, its value will be reset to 0. Operations on this counter are atomic and it can be safely used for synchronization.

  • Return value

    The new counter value.

    • key

      The key name.

    • count

      The value to add to the counter. Use 0 to read the current counter value.

    • ttl

      Optional. If greater than 0s, this is the ttl for the counter (nearest second). After the ttl duration, the counter will reset. If 0s, the counter is stored forever.

get_backend

BACKEND .get_backend(STRING key, BACKEND default)

  • Description

    Get a backend from the kvstore. If its not found, default is returned.

  • Return value

    The backend value.

    • key

      The key name.

    • default

      The default backend if not found. Optional.

set_backend

VOID .set_backend(STRING key, BACKEND value, DURATION ttl)

  • Description

    Sets a backend in the kvstore with an optional ttl. Not safe to use with dynamic backends.

  • Return value

    None

    • key

      The key name.

    • value

      The backend value.

    • ttl Optional. If greater than 0s, this is the ttl for the backend (nearest second). After the ttl duration, the backend is removed. If 0s, the backend is stored forever.

delete

VOID .delete(STRING key)

  • Description

    Delete a key from the kvstore.

  • Return value

    None

    • key

      The key name.

size

INT .size()

  • Description

    Get the current size of the kvstore.

  • Return value

    The size.

compact

INT .compact()

  • Description

    Removed expired keys from the kvstore.

  • Return value

    The number of keys removed.

Reloading from file

init_file() and init_conf() can be used to regularly sync data from file into a kvstore. These functions can be safely used while threads are accessing the kvstore being initialized. When used, threads will not block and will have access to the previous instance of the kvstore while the new kvstore is being built from file. When ready, threads will be directed to the new kvstore and the previous instance will be safely cleaned up.

Note that data from the previous instance will not carry over to the new instance.

Memory and performance

kvstore has a memory overhead of 100 bytes per bucket and 100 bytes per key plus the actual key and value size. So for 1 million keys with a average 1KB key and value size stored in 100,000 buckets, the memory usage is approximately:

    100000 buckets * 100 bytes = 10MB
    1000000 keys * 100 bytes = 100MB
    1000000 keys * 1KB size = 1GB

    TOTAL = 1.11GB

On a 4 core Intel Core i7 with an SSD, the above kvstore takes 1.25s to load from file and has an average key read time of 600ns.

On the same system, a kvstore with 1 million keys, 100k buckets, 70% readers, and 30% writers has a throughput of 3 millions operations per second.

Key length, value length, and overall kvstore size is only bounded by available memory.

Buckets are implemented using red-black trees. The performance complexity of kvstore is:

O(log(size/buckets))

When using TTLs, expired keys are only removed when they are landed on during a get() or set() operation or when explicitly deleted. If you have a highly dynamic key set, expired keys may never be landed on and your kvstore size can grow unchecked. This is due to the way keys are hashed and the fact that each bucket stores keys in a red black tree. If this is the case, its recommended to periodically call compact() to remove expired keys. You can use the size() function to gauge kvstore growth.