High performance key value storage with optional TTLs. This API applies to Varnish Cache Plus 6.0 and higher.
Storing variables into kvstore:
vcl 4.0;
import kvstore;
sub vcl_init
{
new kvs = kvstore.init();
kvs.set("alpha", "one");
kvs.set("beta", "two");
}
sub vcl_recv
{
set req.http.alpha = kvs.get("alpha", "error");
set req.http.beta = kvs.get("beta", "error");
}
Loading a CSV file into a kvstore:
vcl 4.0;
import kvstore;
import std;
sub vcl_init
{
new csv = kvstore.init();
// Load plain comma separated data
if (csv.init_file("/some/path/data.csv", ",") < 0) {
std.syslog(3, "Varnish kvstore ERROR: couldn't load /some/path/data.csv");
return(fail);
}
}
sub vcl_recv
{
// Reload data
if (req.method == "REFRESH") {
req.http.kvinit = csv.init_file("/some/path/data.csv", ",");
if (req.http.kvinit == "-1") {
return(synth(500, "-1"));
} else {
return(synth(200, req.http.kvinit));
}
}
set req.http.somekey = csv.get("somekey", "error");
}
Using kvstore to cache a value:
vcl 4.0;
import kvstore;
sub vcl_init
{
new kvcache = kvstore.init();
}
sub vcl_recv
{
// Lookup "somekey" in cache
set req.http.cachevalue = kvcache.get("somekey", "");
if(req.http.cachevalue == "") {
// Store "somevalue" for 10 seconds
set req.http.cachevalue = "somevalue";
kvcache.set("somekey", req.http.cachevalue, 10s);
}
}
Using kvstore to synchronize an if statement:
vcl 4.0;
import kvstore;
import std;
sub vcl_init
{
new sync = kvstore.init();
}
sub vcl_recv
{
// This block will only be taken once per url every 10 minutes
if (sync.counter(req.url, 1, 10m) == 1) {
std.log("URL called: " + req.url);
}
}
Logging kvstore counter values to varnishstat:
vcl 4.0;
import kvstore;
sub vcl_init
{
new vsc = kvstore.init();
}
sub vcl_deliver
{
// Count all media responses and log it in varnishstat
if (resp.http.Content-Type ~ "audio|image|video") {
vsc.counter("media_responses", 1, varnishstat = true);
}
}
OBJECT kvstore.init(INT buckets, ENUM scope, BOOLEAN scrub)
Description
Initializes a kvstore. Can be used to re-initialize an active kvstore.
The resulting kvstore is empty.
This can be used as both a constructor and a method.
Return value
The kvstore object.
buckets
The number of hash buckets to create (roughly 1 per key). Defaults to 250 buckets.
scope
GLOBAL or REQUEST scope. If REQUEST, kvstore is local to the
request (frontend or backend).
Defaults to GLOBAL.
When using REQUEST scope, keep buckets to a minimum for the best performance.
scrub
Zero out any previous data inside of the kvstore being initialized.
Defaults to false.
INT .init_file(STRING path, STRING delimiter, INT buckets)
Description
Equivalent to init(), but initializes from a file. Can be used to rebuild
an active kvstore. Modifications to the kvstore are not synced back to
the file.
Return value
The numbers of keys loaded. -1 if the file cannot be read (resulting
kvstore will be empty).
path
The path to the file with the key value data. 1 key and value per line.
delimiter
The delimiter string used to separate the key and the value. Only the first occurrence of the delimiter is used as the key value separator. If the delimiter is not found or is empty, the whole line is treated as a key with an empty value.
buckets
The number of hash buckets to create (roughly 1 per key).
Defaults to the bucket value used in init().
INT .init_conf(STRING path, INT buckets)
Description
Equivalent to init(), but initializes from a configuration file. Can be
used to rebuild an active kvstore. Modifications to the kvstore are not
synced back to the file. A configuration file uses ini file
format. Comments start with #, whitespace is stripped, keys and values are
separated with an = (equal) sign, and section names are surrounded with [ ]
and are prepended to key names with _ as a separator.
Return value
The numbers of keys loaded. -1 if the file cannot be read (resulting
kvstore will be empty).
path
The path to the file with the conf data. 1 key and value per line.
buckets
The number of hash buckets to create (roughly 1 per key).
Defaults to the bucket value used in init().
STRING .get(STRING key, STRING default)
Description
Get a key from the kvstore. If its not found, default is returned.
Return value
The key value.
key
The key name.
default
The default value if not found. Optional.
VOID .set(STRING key, STRING value, DURATION ttl)
Description
Sets a key in the kvstore with an optional ttl.
Return value
None
key
The key name.
value
They key value.
ttl
Optional. If greater than 0s, this is the ttl for the key (nearest second). After the ttl duration, the key is deleted. If 0s, the key is stored forever.
INT .counter(STRING key, INT count, DURATION ttl, BOOL varnishstat, STRING comment, BOOL auto)
Description
Create a counter for key and add count to it. The initial value of a counter is always 0. If a counter expires or is deleted, its value will be reset to 0. Operations on this counter are atomic and it can be safely used for synchronization.
Return value
The new counter value.
key
The key name.
count
The value to add to the counter. Use 0 to read the current counter value.
ttl
Optional. If greater than 0s, this is the ttl for the counter (nearest second). After the ttl duration, the counter will reset. If 0s, the counter is stored forever.
varnishstat
Optional. If true, the value of this counter will be logged in
varnishstat in the format of KVSTORE.object_name.vcl_name.key.
Defaults to false.
comment
Optional. If varnishstat is true, this string will be used as the
description of the varnishstat counter.
auto
Optional. Automatically decrement the counter after finishing the request. Defaults to false.
BOOl .limit(STRING key, INT max, INT count)
Description
Increment a key and check if the value is more than max.
Return value
If the value of the key is greater than max false is returned. Otherwise true is returned.
key
The key name.
key
The maximum value the key can be.
count
Optional. The value to add to the counter. Use 0 to read the current counter value. Defaults to 1.
BACKEND .get_backend(STRING key, BACKEND default)
Description
Get a backend from the kvstore. If its not found, default is returned.
Return value
The backend value.
key
The key name.
default
The default backend if not found. Optional.
VOID .set_backend(STRING key, BACKEND value, DURATION ttl)
Description
Sets a backend in the kvstore with an optional ttl. Not safe to use with
dynamic backends.
Return value
None
key
The key name.
value
The backend value.
ttl Optional. If greater than 0s, this is the ttl for the backend (nearest second). After the ttl duration, the backend is removed. If 0s, the backend is stored forever.
VOID .delete(STRING key)
Description
Delete a key from the kvstore.
Return value
None
key
The key name.
INT .size()
Description
Get the current size of the kvstore.
Return value
The size.
INT .compact()
Description
Removed expired keys from the kvstore.
Return value
The number of keys removed.
init_file() and init_conf() can be used to regularly sync data
from file into a kvstore. These functions can be safely used while threads
are accessing the kvstore being initialized. When used,
threads will not block and will have access
to the previous instance of the kvstore while the new kvstore is being built
from file. When ready, threads will be directed to the new kvstore and the
previous instance will be safely cleaned up.
Note that data from the previous instance will not carry over to the new instance.
kvstore has a memory overhead of 100 bytes per bucket and 100 bytes per
key plus the actual key and value size. So for 1 million keys with a average 1KB
key and value size stored in 100,000 buckets, the memory usage is approximately:
100000 buckets * 100 bytes = 10MB
1000000 keys * 100 bytes = 100MB
1000000 keys * 1KB size = 1GB
TOTAL = 1.11GB
On a 4 core Intel Core i7 with an SSD, the above kvstore takes 1.25s to load
from file and has an average key read time of 600ns.
On the same system, a kvstore with 1 million keys, 100k buckets, 70% readers,
and 30% writers has a throughput of 3 millions operations per second.
Key length, value length, and overall kvstore size is only bounded by
available memory.
Buckets are implemented using red-black trees. The performance complexity of
kvstore is:
O(log(size/buckets))
When using TTLs, expired keys are only removed when they are landed on during a
get() or set() operation or when explicitly deleted. If you have a highly
dynamic key set, expired keys may never be landed on and your kvstore size can
grow unchecked. This is due to the way keys are hashed and the fact that each
bucket stores keys in a red black tree. If this is the case, its recommended to
periodically call compact() to remove expired keys. You can use the size()
function to gauge kvstore growth.