Search
Varnish Cache Plus

Tag-based invalidation (Ykey/Xkey)

Ykey and its legacy version, Xkey, implement tag-based invalidation of cached objects. This allows for faster object purging than the usual bans, while also being more maintainable.

The logic for both VMODs is the same: relying on each object being given a list of tags, either by the backend or by the VCL. The user can then ask for a certain tag to be purged, leading to all objects featuring that tag to be invalidated.

Ykey is the newer and current version and should be used as Xkey is deprecated. Both VMODS are available in the varnish-plus package.

Ykey

Ykey is a feature that adds secondary keys to objects, allowing fast purging on all objects with this key. This VMOD provides the user interface for integration into the VCL configuration.

The purge operation may be hard or soft. A hard purge immediately removes the matched objects from the cache completely. A soft purge will expire the objects, but keep the objects around for their configured grace and keep timeouts (grace for stale object delivery to clients while the next fetch is in progress, and keep for conditional fetches).

The keys are managed in-core for efficient handling of many keys, and can safely handle purge operations on keys that span the entire cache. It also interfaces with the MSE stevedore, providing persistence of the Ykey data structure on disk for persisted caches. This makes the Ykey data immediately accessible upon restarts of Varnish, removing the need for the lengthy and I/O intensive re-evaluation of all objects and their associated keys.

To use Ykey, you need to import the ykey VMOD into your VCL configuration. The keys to associate with an object need to be specified specifically by calling one or more of the add key VMOD functions in vcl_backend_response.

The following example adds all keys listed in the backend response header named Ykey, and a custom one for all URLs starting with /content/image/:

import ykey;
sub vcl_backend_response {
	ykey.add_header(beresp.http.Ykey);
	if (bereq.url ~ "^/content/image/") {
		ykey.add_key("IMAGE");
	}
}

To purge objects using Ykey, you will need to map the purge function to your VCL to invoke it. You should use e.g. ACLs to limit the purges to authorized callers. The following example creates a simple purge interface invoked as an HTTP endpoint, limited to localhost. If a header called Ykey-Purge is present, it will purge using Ykey and the keys listed in the header. If not, fall back to regular purge:

import ykey;
acl purgers { "127.0.0.1"; }
sub vcl_recv {
	if (req.method == "PURGE") {
		if (client.ip !~ purgers) {
			return (synth(403, "Forbidden"));
		}
		if (req.http.Ykey-Purge) {
			set req.http.n-gone =
				ykey.purge_header(req.http.Ykey-Purge, sep=" ");
			# or for soft purge:
			#   set req.http.n-gone =
			#	ykey.purge_header(req.http.Ykey-Purge, sep=" ", soft=true);
			return (synth(200, "Invalidated "+req.http.n-gone+" objects"));
		} else {
			return (purge);
		}
	}
}

Transitioning from XKey

The Ykey feature is similar in functionality to the Xkey VMOD, but is better integrated in core Varnish in order to solve several scalability issues that exist with the Xkey VMOD.

The API provided by the Ykey VMOD is not directly backwards compatible with Xkey. Due to technical limitations in the way it integrates with Varnish, Xkey had to make use of an object header with a magic name (xkey, or for historical reasons X-HashTwo) to list the keys to associate with an object. This method was cumbersome to use, especially when needing to amend the list of keys provided from the backend in VCL. Ykey instead requires the keys to associate with an object to be specified in VCL, and provides functions to add each key of a backend header in one go.

Another shortcoming in Xkey that has been addressed, is how to separate individual key strings from headers. Xkey would always split strings on whitespace. In Ykey the way to split strings is configurable, and defaults to splitting on commas and whitespace, which matches better with common headers.

The following VCL example provides an Xkey backwards compatibility snippet that can be integrated into your VCL to quickly start using Ykey by adding the keys from the magic Xkey headers (xkey and X-HashTwo), keeping with XKey’s method of separating strings:

import ykey;

sub vcl_backend_response {
	# Add keys by the xkey backend response header
	ykey.add_header(beresp.http.xkey, sep=" ");
	ykey.add_header(beresp.http.X-HashTwo, sep=" ");
}

Functions

VOID add_key(PRIV_TASK, STRING key)

Adds the key to the list of keys associated with the object being fetched.

VOID add_keys(PRIV_TASK, STRING keys, STRING sep = ", ")

Splits the string keys into individual elements, separated by characters from the string sep, and adds each of them to the list of keys associated with the object being fetched.

VOID add_hashed_keys(STRING keys, STRING sep = ", ")

Splits the string keys into individual elements, separated by characters from the string sep, and adds each of them to the list of keys associated with the object being fetched, with the assumption they are already hashed.

VOID add_header(PRIV_TASK, HEADER hdr, STRING sep = ", ")

Find all headers named hdr, and do an add_keys operation using the specified separator sep on each of them.

VOID add_hashed_header(HEADER hdr, STRING sep = ", ")

Find all headers named hdr, and do an add_hashed_keys operation using the specified separator sep on each of them.

VOID add_blob(PRIV_TASK, BLOB blob)

Adds a key by hashing the bytes described by blob.

INT purge(PRIV_TASK, STRING key, BOOL soft = 0)

Purge the cache of all objects that have the association key key on them. If soft is true, the purge will be a soft purge, setting ttl to zero, but leaving grace and keep as is. The return value is the number of objects that were affected by the operation.

INT purge_keys(PRIV_TASK, STRING keys, STRING sep = ", ", BOOL soft = 0)

Split the string keys using the separator sep, and do a purge on each of them (soft purge if soft is true). The return value is the number of objects that were affected by the operation.

INT purge_header(PRIV_TASK, HEADER hdr, STRING sep = ", ", BOOL soft = 0)

Finds all headers named hdr, and do a purge_keys operation using the separator sep on each of them. The return value is the number of objects affected by the operation.

INT purge_blob(PRIV_TASK, BLOB blob, BOOL soft = 0)

Purge the objects associated with the key described by hashing blob. The return value is the number of objects affected by the operation.

STRING get_hashed_keys(STRING sep = ",")

Gets all the hashed keys added to an object and returns them as string seperated by sep.

VOID namespace(PRIV_TASK, STRING namespace)

Makes all Ykey calls after this to be namespaced to the provided namespace for the duration of the client/backend. It is required that this is called in both the backend request and client request if you want everything to be namespaced.

Xkey

VCL example

import xkey;
sub vcl_recv {
	# object are marked by the backend, using the "xkey" response header.
	if (req.http.x-xkey-purge) {
		if (xkey.purge(req.http.x-xkey-purge) != 0) {
			return(synth(200, "Purged"));
		} else {
			return(synth(404, "Key not found"));
		}
	}
}