SearchVarnish Enterprise
Varnish 4.0 Varnish 4.1
High performance key value storage with optional TTLs. This API applies to Varnish Enterprise 4.0 and 4.1.
Storing variables into kvstore:
vcl 4.0;
import kvstore;
sub vcl_init
{
kvstore.init(0, 25000);
kvstore.set(0, "alpha", "one");
kvstore.set(0, "beta", "two");
}
sub vcl_recv
{
set req.http.alpha = kvstore.get(0, "alpha", "error");
set req.http.beta = kvstore.get(0, "beta", "error");
}
Loading a CSV file into a kvstore:
vcl 4.0;
import kvstore;
import std;
sub vcl_init
{
// Load plain comma separated data
if (kvstore.init_file(0, 25000, "/some/path/data.csv", ",") < 0) {
std.syslog(3, "Varnish kvstore ERROR: couldn't load /some/path/data.csv");
return(fail);
}
}
sub vcl_recv
{
// Reload data
if (req.method == "REFRESH") {
set req.http.kvinit = kvstore.init_file(0, 25000, "/some/path/data.csv", ",");
if (req.http.kvinit == "-1") {
return(synth(500, "-1"));
} else {
return(synth(200, req.http.kvinit));
}
}
set req.http.somekey = kvstore.get(0, "somekey", "error");
}
Using kvstore to cache a value:
vcl 4.0;
import kvstore;
sub vcl_init
{
kvstore.init(0, 25000);
}
sub vcl_recv
{
// Lookup "somekey" in cache
set req.http.cachevalue = kvstore.get(0, "somekey", "");
if(req.http.cachevalue == "") {
// Store "somevalue" for 10 seconds
set req.http.cachevalue = "somevalue";
kvstore.set(0, "somekey", req.http.cachevalue, 10s);
}
}
Using kvstore to synchronize an if statement:
vcl 4.0;
import kvstore;
import std;
sub vcl_init
{
kvstore.init(0, 25000);
}
sub vcl_recv
{
// This block will only be taken once per url every 10 minutes
if (kvstore.counter(0, req.url, 1, 10m) == 1) {
std.log("URL called: " + req.url);
}
}
VOID init(INT name, INT buckets)
Description
Initializes a kvstore. Can be used to re-initialize an active kvstore.
The resulting kvstore is empty.
Return value
None
name
The name of the `kvstore` instance.
By default, values from 0 to 99 are supported.
The name must be used on all calls which reference this instance.
Referencing an uninitialized instance in a `kvstore` call will result in a Varnish assertion error.
Any previous instance for this name will be safely freed.
buckets
The number of hash buckets to create (roughly 1 per key).
INT init_file(INT name, INT buckets, STRING path, STRING delimiter)
Description
Equivalent to init(), but initializes from a file. Can be used to rebuild
an active kvstore. Modifications to the kvstore are not synced back to
the file.
Return value
The numbers of keys loaded. -1 if the file cannot be read (resulting
kvstore will be empty).
name
The name of the `kvstore` instance. By default, values from 0 to 99 are
supported. The name must be used on all calls which reference this
instance. Referencing an uninitialized instance in a `kvstore` call
will result in a Varnish assertion error. Any previous instance for
this name will be safely freed.
buckets
The number of hash buckets to create (roughly 1 per key).
path
The path to the file with the key value data. 1 key and value per line.
delimiter
The delimiter string used to separate the key and the value. Only the
first occurrence of the delimiter is used as the key value separator. If
the delimiter is not found or is empty, the whole line is treated as a
key with an empty value.
INT init_conf(INT name, INT buckets, STRING path)
Description
Equivalent to init(), but initializes from a configuration file. Can be
used to rebuild an active kvstore. Modifications to the kvstore are not
synced back to the file. A configuration file uses ini file
format. Comments start with #, whitespace is stripped, keys and values are
separated with an = (equal) sign, and section names are surrounded with [ ]
and are prepended to key names.
Return value
The numbers of keys loaded. -1 if the file cannot be read (resulting
kvstore will be empty).
name
The name of the `kvstore` instance. By default, values from 0 to 99 are
supported. The name must be used on all calls which reference this
instance. Referencing an uninitialized instance in a `kvstore` call
will result in a Varnish assertion error. Any previous instance for
this name will be safely freed.
buckets
The number of hash buckets to create (roughly 1 per key).
path
The path to the file with the conf data. 1 key and value per line.
STRING get(INT name, STRING key, STRING default)
Description
Get a key from the kvstore. If its not found, default is returned.
Return value
The key value.
name
The name of the `kvstore` instance.
key
The key name.
default
The default value if not found.
VOID set(INT name, STRING key, STRING value, DURATION ttl)
Description
Sets a key in the kvstore with an optional ttl.
Return value
None
name
The name of the `kvstore` instance.
key
The key name.
value
They key value.
ttl
Optional. If greater than 0s, this is the ttl for the key (nearest
second). After the ttl duration, the key is deleted. If 0s, the key is
stored forever.
INT counter(INT name, STRING key, INT count, DURATION ttl)
Description
Create a counter for key and add count to it. The initial value of a counter is always 0. If a counter expires or is deleted, its value will be reset to 0. Operations on this counter are atomic and it can be safely used for synchronization.
Return value
The new counter value.
name
The name of the `kvstore` instance.
key
The key name.
count
The value to add to the counter. Use 0 to read the current counter
value.
ttl
Optional. If greater than 0s, this is the ttl for the counter (nearest
second). After the ttl duration, the counter will reset. If 0s, the
counter is stored forever.
BACKEND get_backend(INT name, STRING key, BACKEND default)
Description
Get a backend from the kvstore. If its not found, default is returned.
Return value
The backend value.
name
The name of the `kvstore` instance.
key
The key name.
default
The default backend if not found.
VOID set_backend(INT name, STRING key, BACKEND value, DURATION ttl)
Description
Sets a backend in the kvstore with an optional ttl. Not safe to use with
dynamic backends.
Return value
None
name
The name of the `kvstore` instance.
key
The key name.
value
The backend value.
ttl
Optional. If greater than 0s, this is the ttl for the backend (nearest
second). After the ttl duration, the backend is removed. If 0s, the
backend is stored forever.
VOID delete(INT name, STRING key)
Description
Delete a key from the kvstore.
Return value
None
name
The name of the `kvstore` instance.
key
The key name.
INT size(INT name)
Description
Get the current size of the kvstore.
Return value
The size.
name
The name of the `kvstore` instance.
INT compact(INT name)
Description
Removed expired keys from the kvstore.
Return value
The number of keys removed.
name
The name of the `kvstore` instance.
init_file(), init_conf(), and init() can be used to regularly sync data
from file into a kvstore. These functions can be safely used while threads
are accessing the kvstore being initialized. In the case of init_file() and
init_conf(), threads will not block during this process and will have access
to the previous instance of the kvstore while the new kvstore is being built
from file. When ready, threads will be directed to the new kvstore and the
previous instance will be safely cleaned up.
Note that data from the previous instance will not carry over to the new instance.
kvstore has a memory overhead of 100 bytes per bucket and 100 bytes per
key plus the actual key and value size. So for 1 million keys with a average 1KB
key and value size stored in 100,000 buckets, the memory usage is approximately:
100000 buckets * 100 bytes = 10MB
1000000 keys * 100 bytes = 100MB
1000000 keys * 1KB size = 1GB
TOTAL = 1.11GB
On a 4 core Intel Core i7 with an SSD, the above kvstore takes 1.25s to load
from file and has an average key read time of 600ns.
On the same system, a kvstore with 1 million keys, 100k buckets, 70% readers,
and 30% writers has a throughput of 3 millions operations per second.
Key length, value length, and overall kvstore size is only bounded by
available memory.
Buckets are implemented using red-black trees. The performance complexity of
kvstore is:
O(log(size/buckets))
When using TTLs, expired keys are only removed when they are landed on during a
get() or set() operation or when explicitly deleted. If you have a highly
dynamic key set, expired keys may never be landed on and your kvstore size can
grow unchecked. This is due to the way keys are hashed and the fact that each
bucket stores keys in a red black tree. If this is the case, its recommended to
periodically call compact() to remove expired keys. You can use the size()
function to gauge kvstore growth.