Search
Varnish Enterprise

Key value storage (kvstore)

Description

The kvstore vmod allows for high performance key value storage with optional TTLs.

This API applies to Varnish Enterprise 6.0 and higher.

For the older kvstore API please refer to the kvstore 4.x API.

Reloading from file

init_file() and init_conf() can be used to regularly sync data from file into a kvstore. These functions can be safely used while threads are accessing the kvstore being initialized. When used, threads will not block and will have access to the previous instance of the kvstore while the new kvstore is being built from file. When ready, threads will be directed to the new kvstore and the previous instance will be safely cleaned up.

Note that data from the previous instance will not carry over to the new instance.

Memory and performance

kvstore has a memory overhead of 100 bytes per bucket and 100 bytes per key plus the actual key and value size. So for 1 million keys with a average 1KB key and value size stored in 100,000 buckets, the memory usage is approximately:

100000 buckets * 100 bytes = 10MB
1000000 keys * 100 bytes = 100MB
1000000 keys * 1KB size = 1GB

TOTAL = 1.11GB

On a 4 core Intel Core i7 with an SSD, the above kvstore takes 1.25s to load from file and has an average key read time of 600ns.

On the same system, a kvstore with 1 million keys, 100k buckets, 70% readers, and 30% writers has a throughput of 3 millions operations per second.

Key length, value length, and overall kvstore size is only bounded by available memory.

Buckets are implemented using red-black trees. The performance complexity of kvstore is:

O(log(size/buckets))

When using TTLs, expired keys are only removed when they are landed on during a get() or set() operation or when explicitly deleted. If you have a highly dynamic key set, expired keys may never be landed on and your kvstore size can grow unchecked. This is due to the way keys are hashed and the fact that each bucket stores keys in a red black tree. If this is the case, it’s recommended to periodically call compact() to remove expired keys. You can use the size() function to gauge kvstore growth.

Examples

Variables

Storing variables into kvstore:

vcl 4.0;

import kvstore;

sub vcl_init
{
  new kvs = kvstore.init();

  kvs.set("alpha", "one");
  kvs.set("beta", "two");
}

sub vcl_recv
{
  set req.http.alpha = kvs.get("alpha", "error");
  set req.http.beta = kvs.get("beta", "error");
}

CSV File

Loading a CSV file into a kvstore:

vcl 4.0;

import kvstore;
import std;

sub vcl_init
{
  new csv = kvstore.init();

  // Load plain comma separated data
  if (csv.init_file("/some/path/data.csv", ",") < 0) {
    std.syslog(3, "Varnish kvstore ERROR: couldn't load /some/path/data.csv");
    return(fail);
  }
}

sub vcl_recv
{
  // Reload data
  if (req.method == "REFRESH") {
    set req.http.kvinit = csv.init_file("/some/path/data.csv", ",");
    if (req.http.kvinit == "-1") {
      return(synth(500, "-1"));
    } else {
      return(synth(200, req.http.kvinit));
    }
  }

  set req.http.somekey = csv.get("somekey", "error");
}

Value Caching

Using kvstore to cache a value:

vcl 4.0;

import kvstore;

sub vcl_init
{
  new kvcache = kvstore.init();
}

sub vcl_recv
{
  // Lookup "somekey" in cache
  set req.http.cachevalue = kvcache.get("somekey", "");

  if(req.http.cachevalue == "") {
    // Store "somevalue" for 10 seconds
    set req.http.cachevalue = "somevalue";
    kvcache.set("somekey", req.http.cachevalue, 10s);
  }
}

Synchronization

Using kvstore to synchronize an if statement:

vcl 4.0;

import kvstore;
import std;

sub vcl_init
{
  new sync = kvstore.init();
}

sub vcl_recv
{
  // This block will only be taken once per url every 10 minutes
  if (sync.counter(req.url, 1, 10m) == 1) {
    std.log("URL called: " + req.url);
  }
}

Varnishstat

Logging kvstore counter values to varnishstat:

vcl 4.0;

import kvstore;

sub vcl_init
{
  new vsc = kvstore.init();
}

sub vcl_deliver
{
  // Count all media responses and log it in varnishstat
  if (resp.http.Content-Type ~ "audio|image|video") {
    vsc.counter("media_responses", 1, varnishstat = true);
  }
}

API

init

OBJECT init(INT buckets = 250, ENUM {STATIC, GLOBAL, REQUEST} scope = GLOBAL)

Initializes a kvstore. Can be used to re-initialize an active kvstore. The resulting kvstore is empty. This can be used as both a constructor and a method.

Arguments:

  • buckets accepts type INT with a default value of 250 optional

  • scope is an ENUM that accepts values of STATIC, GLOBAL, and REQUEST with a default value of GLOBAL optional

Type: Object

Returns: Object.

.init

VOID .init(INT buckets = 0, BOOL scrub = 0)

Initializes the kvstore vmod.

Arguments:

  • buckets accepts type INT with a default value of 0 optional

  • scrub accepts type BOOL with a default value of 0 optional

Type: Method

Returns: None

.init_file

INT .init_file(STRING path, STRING delimiter, INT buckets = 0)

Equivalent to init(), but initializes from a file. Can be used to rebuild an active kvstore. Modifications to the kvstore are not synced back to the file.

The delimiter is string used to separate the key and the value. Only the first occurrence of the delimiter is used as the key value separator.

For example:

foo==bar

If the delimiter is =, then there will be a key named foo with the value =bar. This is because only the first = is interpreted as a delimiter.

If the delimiter is not found or is empty, the whole line is treated as a key with an empty value.

Arguments:

  • path accepts type STRING

  • delimiter accepts type STRING

  • buckets accepts type INT with a default value of 0 optional

Type: Method

Returns: Int

.init_conf

INT .init_conf(STRING path, INT buckets = 0)

Equivalent to init(), but initializes from a configuration file. Can be used to rebuild an active kvstore. Modifications to the kvstore are not synced back to the file. A configuration file uses ini file format. Comments start with #, whitespace is stripped, keys and values are separated with an = (equal) sign, and section names are surrounded with [ ] and are prepended to key names with _ as a separator.

Arguments:

  • path accepts type STRING

  • buckets accepts type INT with a default value of 0 optional

Type: Method

Returns: Int

.get

STRING .get(STRING key, STRING default = 0)

Get a key from the kvstore. If it’s not found, default is returned.

Arguments:

  • key accepts type STRING

  • default accepts type STRING with a default value of 0 optional

Type: Method

Returns: String

.set

VOID .set(STRING key, STRING value, DURATION ttl = 0)

Sets a key in the kvstore with an optional ttl.

Arguments:

  • key accepts type STRING

  • value accepts type STRING

  • ttl accepts type DURATION with a default value of 0 optional

Type: Method

Returns: None

.gauge

INT .gauge(STRING key, INT value, DURATION ttl = 0, BOOL varnishstat = 0, STRING comment = "")

Create a key and set its value, or overwrite if key exists.

If varnishstat is true, the value of this counter will be logged in varnishstat in the format of KVSTORE.object_name.vcl_name.key.

Arguments:

  • key accepts type STRING

  • value accepts type INT

  • ttl accepts type DURATION with a default value of 0 optional

  • varnishstat accepts type BOOL with a default value of 0 optional

  • comment accepts type STRING with a default value of empty. optional

Type: Method

Returns: Int

.counter

INT .counter(STRING key, INT count, DURATION ttl = 0, BOOL varnishstat = 0, STRING comment = "", BOOL auto = 0)

Create a counter for key and add count to it. The initial value of a counter is always 0. If a counter expires or is deleted, it’s value will be reset to 0. Operations on this counter are atomic, and it can be safely used for synchronization.

Note: If ttl is greater than 0s, this is the ttl for the counter (nearest second). After the ttl duration, the counter will reset. If 0s, the counter is stored forever.

If varnishstat is true, the value of this counter will be logged in varnishstat in the format of KVSTORE.object_name.vcl_name.key. The comment parameter will dictate the description provided to varnishstat.

Arguments:

  • key accepts type STRING

  • count accepts type INT

  • ttl accepts type DURATION with a default value of 0 optional

  • varnishstat accepts type BOOL with a default value of 0 optional

  • comment accepts type STRING with a default value of empty. optional

  • auto accepts type BOOL with a default value of 0 optional

Type: Method

Returns: Int

.limit

BOOL .limit(STRING key, INT max, INT count = 1)

Increment a key and check if the value is more than max.

Arguments:

  • key accepts type STRING

  • max accepts type INT

  • count accepts type INT with a default value of 1 optional

Type: Method

Returns: Bool

.get_backend

BACKEND .get_backend(STRING key, BACKEND default = 0)

Get a backend from the kvstore. If it’s not found, default is returned.

Arguments:

  • key accepts type STRING

  • default accepts type BACKEND with a default value of 0 optional

Type: Method

Returns: Backend

.set_backend

VOID .set_backend(STRING key, BACKEND value, DURATION ttl = 0)

Sets a backend in the kvstore with an optional ttl. Not safe to use with dynamic backends.

Arguments:

  • key accepts type STRING

  • value accepts type BACKEND

  • ttl accepts type DURATION with a default value of 0 optional

Type: Method

Returns: None

.lock

VOID .lock(STRING key)

Lock a key.

Arguments:

  • key accepts type STRING

Type: Method

Returns: None

.unlock

VOID .unlock(STRING key)

Unlock a key.

Arguments:

  • key accepts type STRING

Type: Method

Returns: None

.delete

VOID .delete(STRING key)

Delete a key from the kvstore.

Arguments:

  • key accepts type STRING

Type: Method

Returns: None

.size

INT .size()

Get the current size of the kvstore.

Arguments: None

Type: Method

Returns: Int

.compact

INT .compact()

Removes expired keys from the kvstore and returns the number of keys removed.

Arguments: None

Type: Method

Returns: Int

.contains

BOOL .contains(STRING key)

Returns true if the key exists and is not expired. Returns false if the key does not exist or is expired. Note that if the key is expired, it is deleted from the kvstore.

Scopes and lifetimes

A kvstore can be configured with one of three scopes. Each of these scopes define different lifetimes and sharing rules.

  • GLOBAL

Values in a GLOBAL kvstore will have a lifetime equal to that of the VCL it was loaded in. Access to values is shared across request transactions, i.e. you will see the same value when referenced from separate request transactions and edits will be visible to other requests.

This is the default scope.

  • REQUEST

Changes to values in a REQUEST kvstore will be cleaned up when the request terminates. Access is limited to that single request.

If a value is set in vcl_init this will be treated as a default, visible to all requests. Any edits will have its visibility restricted to the request that did the edit.

  • STATIC

Values in a STATIC kvstore will have a lifetime equal to the loading/unloading of the VMOD. All access is shared across VCLs and request transactions.

Arguments:

  • key accepts type STRING

Type: Method

Returns: Bool

Availability

The kvstore VMOD is available in Varnish Enterprise version 6.0.0r0 and later.