Which VMODs are shipped with Varnish Enterprise?

Which VMODs are shipped with Varnish Enterprise?

Varnish Enterprise ships with a whole bunch of VMODs. First of all, all the Varnish Cache VMODs are included except vmod_cookie. On top of that, the varnish-modules VMOD collection is added. Then there are the VMODs maintained by Varnish Software. And finally, there’s a collection of open source VMODs, which were developed by people in the open source community.

Here’s the overview of VMODs that are developed and maintained by Varnish Software. These modules are exclusively part of Varnish Enterprise:

Let’s have a look at a couple of these enterprise VMODs, and see how they add value on the edge.

vmod_accept

No need to go in great detail about vmod_accept because this VMOD was already featured in the previous two chapters. It is used to sanitize the Accept and Accept-Language headers.

Here’s the typical language example:

vcl 4.1;

import accept;

sub vcl_init {
	new lang = accept.rule("en");
	lang.add("nl");
}

sub vcl_recv {
	set req.http.accept-language = lang.filter(req.http.accept-language);
}

If you want to Vary on Accept-Language, you need to make sure the number of variations is limited. vmod_accept makes sure that this is the case.

Imagine the following Accept-Language header:

Accept-language: nl-BE,nl;q=0.9,en-US;q=0.8,en;q=0.7

The VCL example above will turn this header into the following one:

Accept-Language: nl

Any other variant will result in the following:

Accept-Language: en

vmod_aclplus

The typical ACL uses the following syntax:

acl admin {
	"localhost";
	"secure.my-server.com";
	"192.168.0.0/24";
	! "192.168.0.25";   
}

These values are predefined when loading the VCL file and offer limited flexibility.

Advanced ACLs

vmod_aclplus can load ACLs on-the-fly and represents them as a single-line CSV string.

This means that ACLs can be stored pretty much everywhere:

• In a file
• In a database
• In a key-value store
• In a third-party API

Here’s what an ACL looks like in this single-line CSV format:

localhost, secure.my-server.com, 192.168.0.0/24, !192.168.0.25

A key-value store example

Here’s an example where the ACL is stored in a CSV file and loaded into the key-value store for quick access:

vcl 4.1;

import aclplus;
import kvstore;

sub vcl_init {
	new purgers = kvstore.init();
	purgers.init_file("/some/path/data.csv", ",");
}

sub vcl_recv {
	if (req.method == "PURGE") {
		if (aclplus.match(client.ip, purgers.get(req.http.host, "error")) {
			return (purge);
		}
		return (synth(405));
	}
}

The CSV file for this example is as follows:

example.com, localhost, secure.my-server.com, 192.168.0.0/24, !192.168.0.25

And by calling purgers.init_file("/some/path/data.csv", ","), the first part is considered the key, and all the other parts are considered the value.

For requests on the example.com hostname, the ACL can be loaded. You can actually use arbitrary keys, we just happened to use the Host header as the key.

vmod_cookieplus

If you paid attention in chapter 4, you might have seen vmod_cookieplus used there. This is the example that was used:

vcl 4.1;

import cookieplus;

sub vcl_recv {
	cookieplus.keep("language");
	cookieplus.write();   
}

sub vcl_hash {
	if(cookieplus.get("language") ~ "^(nl|en|fr|de|es)$" ) {
		hash_data(cookieplus.get("language"));    
	} else (
		hash_data("en");
	}

}

This example will remove all cookies, except the language cookie. Based on the value of this cookie, language variations are performed.

To some extent, this matches the feature set of vmod_cookie, which was added in Varnish Cache 6.4. What vmod_cookie cannot do, and where vmod_cookieplus shines, is controlling cookies that are sent by the backend through the Set-Cookie header.

Set-Cookie logic

Here’s an example where we will generate a session cookie if it isn’t set:

vcl 4.1;

import cookieplus;
import crypto;

sub vcl_deliver
{
	set req.http.x-sessid = cookieplus.get("sessid", "");

	if (req.http.x-sessid == "") {
		set req.http.x-sessid = crypto.uuid_v4();      
		cookieplus.setcookie_add("sessid", req.http.x-sessid, 30d, 
		req.http.Host, "/");
		cookieplus.setcookie_write();
	}
}

Unless the sessid cookie is set, Varnish will set it itself upon delivery. A unique identifier is generated using crypto.uuid_v4(). This is what the Set-Cookie header will look like when Varnish sets it:

Set-Cookie: sessid=063f2f1a-3752-43d3-b1e3-fcb2c91f3773;
Expires=Wed, 11 Nov 2020 16:43:15 GMT; Domain=localhost:6081; Path=/

vmod_crypto

vmod_crypto contains a collection of cryptographic functions that perform hashing, encryption, and encoding. The previous example already contained the crypto.uuid_v4() function. Let’s have a look at some examples:

Hashing & encoding

vcl 4.1;

import crypto;

sub vcl_deliver {
	set resp.http.x-base64 = crypto.base64_encode(crypto.blob("test"));
	set resp.http.x-md5 = crypto.hex_encode(crypto.hash(md5,"test"));
	set resp.http.x-sha1 = crypto.hex_encode(crypto.hash(sha1,"test"));
}

This example encodes the test string in base64 encoding by leveraging the crypto.base64_encode() function. Because this functions takes a BLOB argument, conversion using the crypto.blob() function is required.

The VCL snippet also contains two hashing examples:

• crypto.hash(md5,"test") creates an md5 hash of the test string
• crypto.hash(sh1,"test") creates a sha1 hash of the test string

In both cases the output is binary and the data type is BLOB. That’s why the crypto.hex_encode() function is required to turn the hashes into strings.

Encryption

Here’s the encryption feature of vmod_crypto that you already saw in chapter 2:

vcl 4.1;
import crypto;

sub vcl_recv {
	crypto.aes_key(crypto.blob("my-16-byte-value"));
	return(synth(200, crypto.hex_encode(crypto.aes_encrypt("password"))));
}

The output that is returned by this VCL example is 60ed8326cfb1ec02359fff4a73fe7e0c. And can be decrypted back into password by running the following code: crypto.aes_decrypt(crypto.hex_decode("60ed8326cfb1ec02359fff4a73fe7e0c"))

In chapter 7 we’ll talk about content encryption, and vmod_crypto will be used for that.

vmod_deviceatlas

vmod_deviceatlas can be used to perform device detection, based on the DeviceAtlas dataset. DeviceAtlas requires a separate subscription though.

Here’s an example in which we will detect mobile users with vmod_deviceatlas:

vcl 4.1;

import deviceatlas;
import std;

sub vcl_init {
	deviceatlas.loadfile("/etc/varnish/da.json");
}

sub vcl_recv {
	if (deviceatlas.lookup(req.http.User-Agent, "isMobilePhone") == "1") {
		std.log("The user-agent is a mobile phone");
	} else if (deviceatlas.lookup(req.http.User-Agent, "isMobilePhone") == "0") {
		std.log("The user-agent is not a mobile phone");
	} else if (deviceatlas.lookup(req.http.User-Agent, "isMobilePhone") == "[unknown]") {
		std.log("The user-agent is unknown");
	} else {
		std.log("Error during lookup");
	}
}

This script logs whether or not the User-Agent corresponds with a mobile device.

vmod_edgestash

Mustache is a popular web templating system that uses curly braces to identify placeholders that can be replaced by actual values.

vmod_edgestash is Varnish Enterprise’s VMOD to handle Mustache syntax, hence the name Edgestash.

This is what your Mustache template could look like when the web server returns the HTTP response for /hello:

Hello {{name}}

The {{name}} placeholder remains unparsed, and vmod_edgestash will pair its value to a JSON dataset. This dataset will be loaded via an internal subrequest to /edgestash.json.

This JSON file could look like this:

{
	"name": "Thijs"
}

And the following VCL code can be used to parse the Mustache handlebars, and pair them with JSON:

vcl 4.1;

import edgestash;

sub vcl_backend_response
{
	if (bereq.url == "/edgestash.json") {
		edgestash.index_json();
	} else if (bereq.url  == "/hello") {
		edgestash.parse_response();
	}
}

sub vcl_deliver
{
	if (req.url == "/hello" && edgestash.is_edgestash()) {
		edgestash.add_json_url("/edgestash.json");
		edgestash.execute();
	}
}

The end result will be:

Hello Thijs

In chapter 8 we’ll have an advanced Edgestash example where the dataset is dynamically loaded.

vmod_file

Although vmod_std has a std.fileread() function to read content from disk, there is still a lot more that can be done with the file system. vmod_file offers a broader API and some cool features.

File backends

Yes, vmod_file can read, write, and delete files. But the coolest feature is the file backend.

The following VCL example will use the file system as a backend. Whatever is returned from the file system is cached in Varnish for subsequent requests:

vcl 4.1;

import file;

backend default none;

sub vcl_init {
	new root = file.init("/var/www/html/");
}

sub vcl_backend_fetch {
	set bereq.backend = root.backend();
}

If properly configured, file backends can eliminate the need for an actual web server: Varnish Enterprise can become the web server.

Command line execution

Another cool vmod_file feature is the fact that you can use it to run programs or scripts on the command line.

Here’s an integrated example where you can use the custom UPTIME request method to retrieve the operating system’s uptime:

vcl 4.1;

import file;

backend default none;

sub vcl_init {
	new fs = file.init();
	fs.allow("/usr/bin/uptime", "x");
}

sub vcl_recv {
	if (req.method == "UPTIME") {
		return (synth(200, "UPTIME"));
	}
}

sub vcl_synth {
	if (resp.reason == "UPTIME") {
		synthetic(fs.exec("/usr/bin/uptime"));
		if (fs.exec_get_errorcode() != 0) {
			set resp.status = 404;
		}
		return (deliver);
	}
}

fs.exec() executes the command on the command line. However, there are some significant security issues involved when you allow random scripts and programs to run. That’s why the .exec() function cannot run unless the allow_exec runtime parameter is set. Commands that are to be executed must also be explicitly allowed through a whitelist, the .allow() function.

In order for this example to work, you need to run the following command:

varnishadm param.set allow_exec true

If you want this parameter to be set at runtime, you should add -p allow_exec=true to your varnishd runtime parameters.

When successfully configured, the uptime service can be called as follows:

curl -XUPTIME localhost

And the output could be the following:

09:53:39 up 2 days, 21:37,  0 users,  load average: 0.79, 0.55, 0.58

If the /usr/bin/uptime program cannot be successfully called, the service will return an HTTP 404 status.

Please be careful not to expose too much control or information of your system by using remote execution. Also, please try to limit external access to these commands through an ACL.

vmod_format

vmod_format is a VMOD that facilitates string formatting, and it does so using the ANSI C printf format.

You already saw this example in chapter 2:

vcl 4.1;

import format;

sub vcl_synth {
	set resp.body = format.quick("ERROR: %s\nREASON: %s\n",
		resp.status, resp.reason);
	return (deliver);
}

The format.quick() function makes it super easy to perform string interpolation. Doing it manually by closing the string, and using the plus-sign can become tedious.

The module also offers functions that allow you to interpolate non-string types. Let’s take the previous example and instead of the format.quick() function, we’ll use the format.set(), format.get(), format.add_string(), and format.add_int() functions:

vcl 4.1;

import format;

sub vcl_synth {
	format.set("ERROR: %d\nREASON: %s\n");
	format.add_int(resp.status);
	format.add_string(resp.reason); 
	set resp.body = format.get();
	return (deliver);
}

As you might have noticed, the %s format for the numeric status code has been replaced with the actual numeric %d format. The format.add_int() can now be used to pass in an integer.

When using format.set(), the corresponding format.add_* functions should be executed in the right order.

vmod_json

The vmod_json module can parse JSON from a string, or directly from the request body.

The VCL example for this VMOD will extract the authorization property from the JSON object that is represented by the request body:

vcl 4.1;

import json;
import std;

sub vcl_recv
{
	std.cache_req_body(100KB);
	json.parse_req_body();

	if (json.is_valid() && json.is_object() &&
			json.get("authorization")) {
		req.http.X-authorization = json.get("authorization");
	} else {
		return(synth(401));
	}
}

If the authorization property is not set, Varnish returns an HTTP 401 Unauthorized status code.

vmod_goto

vmod_goto is a module that allows you to define backends on the fly. Varnish Cache requires you to define all backends upfront. When your backend inventory is dynamic, frequent rewrites and reloads of your VCL file are required, which can become cumbersome.

There are also situations where you want to perform one-off calls to an endpoint and cache that data. Or situations where the endpoint is dynamic and based on other criteria.

The DNS backend

Here’s an example where the goto.dns_backend() method is used to create backends on the fly:

vcl 4.1;

import goto;

backend default none;

sub vcl_backend_fetch {
	set bereq.backend = goto.dns_backend("backends.example.com");
}

This doesn’t look very different from a typical backend definition, but internally there is a big difference: regular backends are static and their respective hostnames are resolved at compile time. If the DNS name changes while a compiled VCL file is running, that DNS change will go unnoticed.

Also, traditional backends don’t support hostnames that resolve to multiple IP addresses. You’d get an Backend host "xyz": resolves to too many addresses error.

However, vmod_goto can handle these types of A-records, and will cycle through the corresponding IP addresses at runtime. If at some point your origin architecture needs to scale, vmod_goto can handle DNS changes without reloading the VCL file.

You can even use the URL as an endpoint:

vcl 4.1;

import goto;

backend default none;

sub vcl_backend_fetch {
	set bereq.backend = goto.dns_backend("https://backends.example.com:8888");
}

This example will connect to backends.example.com, and if multiple IP addresses are assigned to that hostname, it will cycle through them. The connection will be made using HTTPS over port 8888.

The DNS director

Here’s an example where the goto.dns_director() method is used to create a director object that resolves hostnames dynamically:

vcl 4.1;

import goto;

backend default none;

sub vcl_init {
	new dyndir = goto.dns_director("backends.example.com");
}

sub vcl_backend_fetch {
	set bereq.backend = dyndir.backend();
}

Extra options

Both goto.dns_backend() and goto.dns_director() have a bunch of extra optional arguments that allow you to fine-tune some of the connection parameters. They are on par with the capabilities of regular backends, but they also have TTL settings to control the DNS resolution.

Because the arguments are named, it doesn’t really matter in which order you use them. Here’s an example:

vcl 4.1;

import goto;

backend default none;

sub vcl_backend_fetch {
	set bereq.backend = goto.dns_backend("https://backends.example.com:8888", 
		host_header="example.com", connect_timeout=2s, first_byte_timeout=10s, 
		max_connections=10, ssl_verify_peer=false, ip_version=ipv4, 
		ttl=1h, ttl_rule=morethan);
}

This does the following:

• It connects to backends.example.com
• The connection is made over HTTPS on port 8888
• The Host header that is used for the request will be overridden to example.com
• The connection timeout is two seconds
• the first byte timeout is ten seconds
• We allow ten simultaneous connections to the backend
• Although TLS/SSL is used for the connection, peer verification is disabled
• When we resolve the hostname, we only care about IPv4 addresses
• The DNS TTL from the record itself will be used, as long as it is more than the ttl value we set

Dynamic backends example

Although vmod_goto is considered a dynamic backend VMOD, the examples looked a bit static. The real dynamic behavior is behind the scenes, but it’s not very inspiring.

Let’s throw in a cool example where the origin inventory is stored in a JSON file.

Imagine every tenant having its own backend configuration file that is located in /etc/varnish/tenants/xyz.json, where the xyz refers to the hostname of the tenant.

In our case there is a /etc/varnish/tenants/example.com.json file that has the follow content:

{
  "origin": {
	"addr": "https://backends.example.com",
	"connect": "1s"
  }
}

By leveraging vmod_goto, vmod_file, vmod_std, and vmod_json, we can read and parse the file, extract the parameters, and feed them to vmod_goto:

vcl 4.1;

import goto;
import file;
import std;
import json;

backend default none;

sub vcl_init {
	new fs = file.init();
	fs.allow("/etc/varnish/tenants/*.json");
}

sub vcl_backend_fetch {
	json.parse(fs.read("/etc/varnish/tenants/" + bereq.http.host + ".json"));

	if (!json.get("origin.addr")) {
		return(abandon);
	}
	
	set bereq.backend = goto.dns_backend(json.get("origin.addr"), 
	connect_timeout = std.duration(json.get("origin.connect"), 3.5s));
}

Based on /etc/varnish/tenants/example.com.json, the backend would be https://backends.example.com with a connection timeout of one second.

vmod_headerplus

vmod_headerplus was already covered when we discussed the new features of Varnish Enterprise 6 in chapter 2. Let’s make sure we use a different VCL example to show the power of this VMOD:

The following example will add stale-if-error support, which Varnish doesn’t support by default:

vcl 4.1;
import headerplus;

sub vcl_backend_response {
	headerplus.init(beresp);
	set beresp.http.stale-if-error = 
	headerplus.attr_get("Cache-Control", "stale-if-error");
	if (beresp.http.stale-if-error != "") {
		set beresp.grace = std.duration(bereq.http.stale-if-error + "s", 11s);
	}
	unset beresp.http.stale-if-error;
}

The headerplus.attr_get() function allows us to retrieve specific attributes from a header. In this case we’re retrieving the stale-if-error attribute. The value of this attribute is assigned to beresp.grace, which sets the grace mode.

vmod_http

Being able to perform arbitrary HTTP calls from within VCL is a very powerful concept. vmod_http offers us the tools to do this. This module wraps around libcurl, which is an HTTP client library.

There are many ways in which vmod_http can be leveraged, but in the example we’ll use feature content prefetching:

vcl 4.1;

import http;

sub vcl_recv {
	set req.http.X-prefetch = http.varnish_url("/");
}

sub vcl_backend_response {
	if (beresp.http.Link ~ "<.+>; rel=prefetch") {
		set bereq.http.X-link = regsub(beresp.http.Link, "^.*<([^>]*)>.*$", "\1");
		set bereq.http.X-prefetch = regsub(bereq.http.X-prefetch, "/$", bereq.http.X-link);  
   
		http.init(0);
		http.req_copy_headers(0);
		http.req_set_method(0, "HEAD");
		http.req_set_url(0, bereq.http.X-prefetch);
		http.req_send_and_finish(0);
	}
}

This example will prefetch a URL that was mentioned in the Link header.

Imagine the following Link response header:

Link: </style.css>; rel=prefetch

The VCL example above will match the pattern of this header and use regsub to extract the URL, concatenate it with the full Varnish URL, and call this URL in the background.

The http.req_send_and_finish(0) function call ensures that the HTTP request is made in the background, and that we’re not waiting for the response. The fact that the HEAD request method is used also means we don’t need to process the response body. Subrequests that are sent to Varnish using HEAD will automatically be converted into a full GET request. This means the response body will be stored in cache.

When /style.css is eventually called by the client, it is already stored in cache, ready to be served.

vmod_jwt

vmod_jwt is a module for verifying, creating, and manipulating JSON Web Tokens and JSON Web Signatures. It was already covered in chapter 2, when we talked about new features in Varnish Enterprise 6.

Here’s one example we had:

vcl 4.1;

import jwt;

sub vcl_init {
	new jwt_reader = jwt.reader();
}

sub vcl_recv {
	if (!jwt_reader.parse(regsub(req.http.Authorization,"^Bearer ",""))) {
		return (synth(401, "Invalid JWT Token"));
	}

	if (!jwt_reader.set_key("secret")) {
		return (synth(401, "Invalid JWT Token"));
	}

	if (!jwt_reader.verify("HS256")) {
		return (synth(401, "Invalid JWT Token"));
	}
}

This is what one possible JWT could look like:

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.
eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.
XbPfbIHMI6arZ3Y922BhjWgQzWXcXNrz0ogtVhfEd2o

It consists of three distinct parts, separated by a dot, encoded with base64 This is what the decoded version looks like:

The first part is the header:

{
  "alg": "HS256",
  "typ": "JWT"
}

The second part is the actual payload of the token:

{
  "sub": "1234567890",
  "name": "John Doe",
  "iat": 1516239022
}

The third part is the signature, which contains binary data, and is secured by the secret key.

The JWT would be used as follows:

Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.
eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.
XbPfbIHMI6arZ3Y922BhjWgQzWXcXNrz0ogtVhfEd2o

Our VCL example would verify the token based on the format, but the .verify() method would ensure the HMAC signature matches.

In chapter 8 we’ll talk more about authentication, and we’ll feature some of the capabilities of vmod_jwt.

vmod_kvstore

vmod_kvstore offers a built-in key-value store. This is a simple memory database that stores keys with their associated value. The key name is always a string; the value is also a string. By default vmod_kvstore has a global state, so values remain available across requests.

The KVStore has the typical getters, setters, and counters, but it can also be populated at startup from a file.

vmod_kvstore was also part of the example we used for vmod_aclplus. Let’s focus on the KVStore functionality of the example:

vcl 4.1;

import aclplus;
import kvstore;

sub vcl_init {
	new purgers = kvstore.init();
	purgers.init_file("/some/path/data.csv", ",");
}

sub vcl_recv {
	if (req.method == "PURGE") {
		if (aclplus.match(client.ip, purgers.get(req.http.host, "error")) {
			return (purge);
		}
		return (synth(405));
	}
}

The .init_file("/some/path/data.csv", ",") will read /some/path/data.csv and extract the values, using , as a separator. The first value will be used as the key, the rest is considered the value. The .get() method will fetch the key that was passed, and will return the corresponding value.

vmod_mmdb

The mmdb in vmod_mmdb is short for MaxMind DB. MaxMind is a company that provides geoIP databases. This VMOD is a wrapper around libmaxminddb, which contains the API to access and parse the geoIP database.

This VMOD was also part of the new features of Varnish Enterprise 6, and was covered in chapter 2. Here’s the example we used there:

vcl 4.1;

import mmdb;

sub vcl_init {
	new geodb = mmdb.init("/path/to/database");
}

sub vcl_recv {
	return(synth(200, 
		"Country: " + geodb.lookup(client.ip, "country/names/en") + " - " +
		"City: " + geodb.lookup(client.ip, "city/names/en")
	));
}

The mmdb.init() function will fetch the geoIP data from a datafile, and results in an object that contains lookup functionality.

In this example geodb.lookup(client.ip, "country/names/en") is used to extract the country name from the geoIP data that is associated with the client IP address.

vmod_mse

The Massive Storage Engine is one of the key features of Varnish Enterprise. As mentioned before, it is a stevedore that combines memory and file-backed storage.

MSE’s persistence layer consists of books, which are metadatabases, and stores, which are the underlying datastores. Books and stores can be tagged, and vmod_mse has the ability to select a tagged store, and as a result choose where objects get stored.

In chapter 2, vmod_mse was covered when we talked about new features in Varnish Enterprise. The internal configuration of MSE was also showcased.

Here’s an example that was used in that section to illustrate how tags in MSE can be used to select where objects get stored:

vcl 4.1;

import mse;
import std;

sub vcl_backend_response {
	if (beresp.ttl < 120s) {
		mse.set_stores("none");
	} else {
		if (beresp.http.Transfer-Encoding ~ "chunked" || 
		std.integer(beresp.http.Content-Length,0) > std.bytes("1M")) {
			mse.set_stores("sata");
		} else {
			mse.set_stores("ssd");
		}
	}
}

In this case, there are two types of MSE stores: stores that are tagged with sata, and stores that are tagged with ssd. These tags reflect the type of disks that are used for these books.

The VCL example forces objects to be stored on sata tagged books, if the Content-Length header indicates a size bigger than 1 MB, or if we don’t have a Content-Length header at all, and Transfer-Encoding: chunked is used instead.

Please note that it is not recommended to use MSE with spinning disks on servers with high load.

vmod_resolver

vmod_resolver is a module that performs Forwarded Confirmed reverse DNS (FCrDNS). This means that a reverse DNS call is done on the client IP address.

The resulting hostname is then resolved again, and if this matches the original client IP address, the check succeeds.

Here’s the reference example:

vcl 4.1;

import std;
import resolver;

sub vcl_recv {
	if (resolver.resolve()) {
		std.log("Resolver domain: " + resolver.domain());
	} else {
		 std.log("Resolver error: " + resolver.error());
	 }
}

resolver.resolve() performs the FCrDNS and returns true when it succeeds. In that case the resolver.domain() returns the domain that came out of this. When the resolver fails, we can use resolver.error() to retrieve the error message.

vmod_rewrite

URL matching and rewriting is a very common practice in VCL. The language itself offers us the syntax to do this. But the more logic is added, the harder it gets to manage the rules, and the bigger the complexity. It’s basically like any other software project.

Rewrite rules in VCL

Here’s an example of what it could look like:

vcl 4.1;

sub vcl_recv {
	if (req.url ~ "/pay") {
		set req.url = regsub(req.url, "/pay", "/checkout");
	} else if (req.url ~ "(?i)/cart") {
		set req.url = regsub(req.url, "(?i)/cart", "/shopping-cart");
	} else if (req.url ~ "product\-([0-9]+)") {
		set req.url = regsub(req.url, "product\-([0-9]+)", "catalog\-\1");
	}
}

You have to admit that this workflow gets messy really quickly. However, vmod_rewrite helps you organize your URL rewriting logic through rules.

vmod_rewrite rulesets

You can create a rules file that contains all the logic. Here’s what it can look like:

"/pay"             "/checkout"
"(?i)/cart"        "/shopping-cart"
"product-([0-9]+)" "catalog-\1"

This file can then be loaded into VCL using the following code:

vcl 4.1;

import rewrite;

sub vcl_init {
	new rs = rewrite.ruleset("/path/to/file.rules");
}

sub vcl_recv {
	set req.url = rs.replace(req.url);
}

And that’s all it takes. The logic is separated in a different file, it doesn’t pollute your VCL, and is easy to manage.

Rulesets as a string

The same logic can also be loaded as a string in the VCL without compromising too much on readability:

vcl 4.1;

import rewrite;

sub vcl_init {
	new rs = rewrite.ruleset(string = {"
		"/pay"             "/checkout"
		"(?i)/cart"        "/shopping-cart"
		"product-([0-9]+)" "catalog-\1"    
	"});
}

sub vcl_recv {
	set req.url = rs.replace(req.url);
}

Matching URL patterns

So far we’ve been rewriting URLs, but we can also match patterns using the .match``() method. Here’s a quick example:

vcl 4.1;

import rewrite;

sub vcl_init {
	new rs = rewrite.ruleset(string = {"
		"^/admin/"
		"^/purge/"
		"^/private"
	"}, min_fields = 1);
}

sub vcl_recv {
	if (rs.match(req.url)) {
		return (synth(405, "Restricted");
	}
}

The ruleset contains a list of private URLs that cannot be accessed through Varnish. rs.match(req.url) checks whether or not the URL matches those rules.

Extracting ruleset fields

The final example uses rewrite logic, not necessarily to rewrite the URL, but to extract values from a field in the ruleset.

Here’s the code:

vcl 4.1;
import std;
import rewrite;

sub vcl_init {
	new rs = rewrite.ruleset(string = {"
		# pattern       ttl     grace   keep
		"\.(js|css)"    "1m"    "10m"   "1d"
		"\.(jpg|png)"   "1w"    "1w"    "10w"
	"});
}

sub vcl_backend_response {
	# if there's a match, convert text to duration
	if (rs.match(bereq.url)) {
		set beresp.ttl   = std.duration(rs.rewrite(0, mode = only_matching), 0s);
		set beresp.grace = std.duration(rs.rewrite(1, mode = only_matching), 0s);
		set beresp.keep  = std.duration(rs.rewrite(2, mode = only_matching), 0s);
	}
}

If a backend request’s URL matches our ruleset, fields are extracted that represent the corresponding TTL, grace time, and keep time.

If your VCL has specific logic to assign custom TTL values to certain URL patterns, vmod_rewrite can take care of this for you.

vmod_sqlite3

As mentioned in chapter 2 when talking about new features in Varnish Enterprise 6: SQLite is a library that implements a serverless, self-contained relational database system.

vmod_sqlite3 wraps around this library and allows you to write SQL statements to interact with SQLite. The data itself is stored in a single file, as you can see in the example below:

vcl 4.1;

import sqlite3;
import cookieplus;
backend default none;

sub vcl_init {
	sqlite3.open("sqlite.db", "|;");
}

sub vcl_fini {
	sqlite3.close();
}

sub vcl_recv {
	cookieplus.keep("id");
	cookieplus.write();
	if(cookieplus.get("id") ~ "^[0-9]+$") {
		set req.http.userid = cookieplus.get("id");
		set req.http.username = sqlite3.exec("SELECT `name` FROM `users` 
		WHERE rowid=" + sqlite3.escape(req.http.userid));
	}
	if(!req.http.username || req.http.username == "") {
		set req.http.username = "guest";   
	}
	return(synth(200,"Welcome " + req.http.username));
}

This example will fetch the user’s id from the id cookie and fetch the corresponding row from the database. When there’s a match, the username is returned. Otherwise guest is returned as a value.

vmod_stale

vmod_stale is also one of the new VMODS covered in chapter 2. It is a way to revive stale objects that are about to expire.

Varnish has cascading timers to keep track of an object’s lifetime:

• Objects whose TTL hasn’t expired are considered fresh.
• When the TTL of an object has expired, but there is still grace left, the object is considered stale.
• When even the grace has expired, an object can be kept around if there is keep time left.

As long as the object is still around vmod_stale can reset its TTL, grace, and keep value.

Here’s the example that we used in chapter 2:

vcl 4.1;

import stale;

sub stale_if_error {
	if (beresp.status >= 500 && stale.exists()) {
		stale.revive(20m, 1h);
		stale.deliver();
		return (abandon);
	}
}

sub vcl_backend_response {
	set beresp.keep = 1d;
	call stale_if_error;
}

sub vcl_backend_error {
	call stale_if_error;
}

By setting beresp.keep to one day, we make sure the object is kept around long enough, even though its TTL and grace have expired. This allows std.revive() to revive the object and make it fresh again.

In this case, the object is fresh for another 20 minutes, and after that an hour of grace is added so Varnish can perform a background fetch for revalidation while the stale is served.

This revival example only happens when the origin server is responding with HTTP 500-style errors. This is basically a stale-if-error implementation.

vmod_synthbackend

By now, you should be quite familiar with synthetic output. But if you return synth() in VCL, the response happens on the fly, and is not cached.

As explained in chapter 2, vmod_synthbackend will allow you to define a synthetic backend that caches the synthetic output.

Here’s a code example that illustrates how this can be done:

vcl 4.1;

import synthbackend;

backend default none;

sub vcl_backend_fetch {
	set bereq.backend = synthbackend.from_string("URL: " + bereq.url + ", time: " + now);
} 

In this case, the synthetic output will be stored in cache for the duration of the default_ttl runtime parameter. By default this is two minutes.

vmod_tls

In the previous section about VMODs that are shipped with Varnish Cache, we talked about vmod_proxy that retrieves TLS information from a PROXY protocol connection.

Varnish Enterprise offers native TLS support, and when activated, no PROXY connection is needed. In order to support the same functionality as vmod_proxy for local TLS connections, we developed vmod_tls.

The API is identical; it’s just a different module. Here’s the same example as for vmod_proxy, but using vmod_tls:

vcl 4.1;
import tls;

sub vcl_deliver {
	set resp.http.alpn = tls.alpn();
	set resp.http.authority = tls.authority();
	set resp.http.ssl = tls.is_ssl();
	set resp.http.ssl-version = tls.ssl_version();
	set resp.http.ssl-cipher = tls.ssl_cipher();
}

And this is some example output, containing the custom headers:

alpn: h2
authority: example.com
ssl: true
ssl-version: TLSv1.3
ssl-cipher: TLS_AES_256_GCM_SHA384

vmod_urlplus

vmod_urlplus was extensively covered in the New features in Varnish Enterprise 6 section of chapter 2.

We’ll just throw in one of the examples to refresh your memory:

vcl 4.1;

import urlplus;

sub vcl_recv
{
	//Remove all Google Analytics
	urlplus.query_delete_regex("utm_");

	//Sort query string and write URL out to req.url
	urlplus.write();
}

This example will remove all query string parameters that match the utm_ pattern from the URL. This makes a lot of sense, because these parameters add no value from a caching point of view. On the contrary, they cause too much variation to be created and have an detrimental impact on the hit rate.

vmod_xbody

vmod_xbody is a module that provides access to request and response bodies. It is also capable of modifying the request and response bodies.

Imagine having the following static content on your website:

Hello Thijs

Using vmod_xbody, and some other VMODs, you can replace the name with any name you want.

The following example uses the KVStore to store various names, identified by an ID. The name that is displayed depends on the value of the id cookie. The corresponding name is retrieved from the KVStore:

vcl 4.1;

import xbody;
import kvstore;
import cookieplus;

sub vcl_init {
	new people = kvstore.init();
	people.set("1","Thijs");
	people.set("2","Lex");
	people.set("3","Lize");
	people.set("4","Lia");
}

sub vcl_backend_response {
	xbody.regsub("Thijs", people.get(cookieplus.get("id","0"),"guest"));
}

So if the cookie header is Cookie: id=2, the output will be as follows:

Hello Lex

If the cookie header is Cookie: id=4, the output will be:

Hello Lia

If the id cookie is not set, or has an unknown value, the output will be:

Hello guest

vmod_ykey

vmod_ykey is the Varnish Enterprise version of vmod_xkey. The naming is a bit funny, we know.

vmod_xkey is an open source module that adds secondary keys to objects. This allows us to purge objects from cache based on tags rather than the URL. Unfortunately vmod_xkey proved to be incompatible with MSE, so vmod_ykey was built to tackle this issue.

As this VMOD was already featured in chapter 2, we’ll just show you the example:

vcl 4.1;

import ykey;

acl purgers { "127.0.0.1"; }

sub vcl_recv {
	if (req.method == "PURGE") {
		if (client.ip !~ purgers) {
			return (synth(403, "Forbidden"));
		}
		set req.http.n-gone = ykey.purge_header(req.http.Ykey-Purge, sep=" ");
		return (synth(200, "Invalidated "+req.http.n-gone+" objects"));
	}
}

sub vcl_backend_response {
	ykey.add_header(beresp.http.Ykey);
	if (bereq.url ~ "^/content/image/") {
		ykey.add_key("image");
	}
}

An application can add tags by stating them in the Ykey response header, which this VCL script can parse. Content for which the URL matches the /content/image/ pattern will automatically have an image tag assigned to it.

By invalidating a tag, all tagged objects are removed from cache at once.

You just need to send the following HTTP request to Varnish if you want to remove all objects that are tagged with the image tag:

PURGE / HTTP/1.1
Ykey-Purge: image

The next chapter is all about cache invalidation. We’ll talk about vmod_ykey in a lot more detail there.