SearchThe Varnish Cache core contributors and the Varnish Software development team aren’t the only ones building VMODs.
The Varnish Cache website has a section dedicated to third-party VMODs: http://varnish-cache.org/vmods/.
You can also search on GitHub for repositories that start with
libvmod-. Just have a look at the following results:
https://github.com/search?q=libvmod-&type=repositories.
There are some really interesting ones there, but before you install one, make sure it is compatible with Varnish 6.
There are a lot of good third-party VMODs out there, but there’s also some overlap with functionality that can be found in Varnish Enterprise VMODs.
Here are two third-party VMODS I really like:
vmod_basicauthvmod_redisThe vmod_basicauth module reads a typical Apache .htpasswd file
and tries to match it to the incoming Aut``h``orization header.
Here’s an example of this VMOD:
vcl 4.1;
import basicauth;
sub vcl_recv {
if (!basicauth.match("/var/www/.htpasswd",req.http.Authorization)) {
return (synth(401, "Restricted"));
}
}
sub vcl_synth {
if (resp.status == 401) {
set resp.http.WWW-Authenticate = {"Basic realm="Restricted area""};
}
}
If the username and password encoded in the Authorization header
don’t match an entry in .htpasswd, an HTTP 401 status code is
triggered, which will result in a
WWW-Authenticate: Basic realm="Restricted area" response header being
synthetically returned.
This WWW-Authenticate header will trigger a server-side login popup.
For more information about this VMOD, please visit http://man.gnu.org.ua/manpage/?3+vmod-basicauth.
vmod_redis provides a client API to control a Redis server. Redis
is an advanced distributed key-value store and has become somewhat of
an industry standard.
Here’s just a very simple example where we set up a connection and fetch
the value of the foo key:
vcl 4.1;
import redis;
sub vcl_init {
new db = redis.db(
location="192.168.1.100:6379",
connection_timeout=500,
shared_connections=false,
max_connections=2);
}
sub vcl_deliver {
db.command("GET");
db.push("foo");
db.execute();
set resp.http.X-Foo = db.get_string_reply();
}
But the API for this VMOD is extensive and supports some of the following features:
For more information about this VMOD, please visit https://github.com/carlosabalde/libvmod-redis.
Varnish Software also has a GitHub repository with some open source VMODs. You can find the code on https://github.com/varnish/varnish-modules.
These modules are also packaged with Varnish Enterprise. Here is the list of VMODs that are part of this collection:
vmod_bodyaccessvmod_headervmod_saintmodevmod_tcpvmod_varvmod_vsthrottlevmod_xkeyLet’s do a walkthrough, and look at some VCL examples for a couple of these VMODs.
vmod_bodyaccess is a very limited version of vmod_xbody. Whereas
vmod_xbody has read and write access to request and response bodies,
vmod_bodyaccess only has read access to the request body.
Here’s a list of features that this VMOD provides:
Let’s have a look at a VCL example where we’ll use vmod_bodyaccess
in a scenario where we’ll cache POST requests:
vcl 4.1;
import std;
import bodyaccess;
sub vcl_recv {
set req.http.x-method = req.method;
if (req.method == "POST") {
if (std.cache_req_body(110KB)) {
if (bodyaccess.rematch_req_body("id=[0-9]+")) {
return (hash);
}
return (synth(422, "Missing ID");
}
return (synth(413));
}
}
sub vcl_hash {
bodyaccess.hash_req_body();
}
sub vcl_backend_fetch {
set bereq.method = bereq.http.x-method;
}
None of this works unless std.cache_req_body() is called. This starts
caching the request body. bodyaccess.rematch_req_body("id=[0-9]+") is
used to figure out whether or not the id=[0-9]+ pattern is part of the
request body. When the pattern matches, we’ll decide to cache. If the
payload is too large or the ID is missing an error will be generated
instead.
bodyaccess.hash_req_body() is used in vcl_hash to create a cache
variation for each request body value.
This is the HTTP request you can send to trigger this behavior:
POST / HTTP/1.1
Host: localhost
Content-Length: 4
Content-Type: application/x-www-form-urlencoded
id=1
vmod_header allows you to get headers, append values to headers, and
remove values from headers. It’s a very limited VMOD in terms of
functionality, and pales in comparison to vmod_headerplus.
Here’s a quick example, just for the sake of it:
vcl 4.1;
sub vcl_backend_response {
header.remove(beresp.http.foo, "one=1");
}
Imagine the following HTTP response headers:
HTTP/1.1 200 OK
Foo: one=1
Foo: one=2
The response contains two instances of the Foo header, each with
different values. The example above will ensure that the first
occurrence of the header that matches the pattern is removed.
Foo``: one=1 matches that pattern and is removed. Only Foo: one=2
remains.
The TCP VMOD allows you to control certain aspects of the underlying TCP connection that was established.
The first example will enable rate limiting for incoming connections:
vcl 4.1;
import tcp;
sub vcl_recv
{
tcp.set_socket_pace(1024);
}
This example will pace the throughput at a rate of 1024 KB/s.
In the second example the BBR congestion algorithm is used for TCP
connections:
vcl 4.1;
import tcp;
sub vcl_recv {
tcp.congestion_algorithm("bbr");
}
This requires that the BBR congestion controller is both available and
loaded.
VCL clearly lacks the concept of variables. Although headers are commonly used to transport values, the values are usually cast to strings, and one needs to have the discipline to strip off these headers before delivering them to the backend or the client.
vmod_var offers variable support for various data types.
The supported data types are:
Here’s the VCL example to illustrate some of the VMOD’s functions:
vcl 4.1;
import var;
import std;
sub vcl_recv {
var.set_ip("forwarded",std.ip(req.http.X-Forwarded-For,"0.0.0.0"));
var.set_real("start",std.time2real(now,0.0));
}
sub vcl_deliver {
set resp.http.forwarded-ip = var.get_ip("forwarded");
set resp.http.start = std.real2time(var.get_real("start"),now);
}
This example will use the std.ip() function from vmod_std to turn
the X-Forwarded-For header into a valid IP type. The value is stored
using var.set_ip().
The current time is also stored as a real type, using
std.time2real() to convert the type.
In a later stage, we can retrieve the information using the corresponding getter functions.
We’ve already seen rate limiting when we talked about vmod_tcp. But
rate limiting also happens in this VMOD.
In vmod_vsthrottle, we’re restricting the number of requests a client
can send in a given timeframe.
Take for example, the VCL code below:
vcl 4.1;
import vsthrottle;
sub vcl_recv {
if (vsthrottle.is_denied(client.identity, 15, 10s, 30s)) {
return (synth(429, "Too Many Requests. You can retry in "
+ vsthrottle.blocked(client.identity, 15, 10s, 30s)
+ " seconds."));
}
}
This code will only allow clients to perform 15 requests in a 10 second timeframe. If that rate is exceeded, the user gets blocked for 30 seconds.
The vsthrottle.is_denied() function is responsible for that. The
vsthrottle.blocked() is also quite helpful, as it returns the number
of seconds the user is still blocked.
This allows us to set expectations. This is the error messages that users get when they are blocked due to rate limiting:
Too Many Requests. You can retry in 26.873 seconds.
In this case, the user knows they should wait another 26 seconds before attempting to perform another request.
We already mentioned that vmod_xkey is the predecessor of vmod_ykey,
basically the open source version. It doesn’t work when using MSE, but
if you’re using Varnish Cache, that is not a concern.
The API is also more limited, as you are forced to tag objects using
the xkey response header, and you cannot add extra tags in VCL.
Here’s the VCL example to show how you can remove objects from cache
using vmod_xkey:
vcl 4.1;
import xkey;
sub vcl_recv {
if (req.method == "PURGE" && req.http.x-xkey-purge) {
if (xkey.purge(req.http.x-xkey-purge) != 0) {
return(synth(200, "Purged"));
}
return(synth(404, "Key not found"));
}
}
If you’re performing a PURGE call, you can use the x-xkey-purge
request header to specify the keys you want to use for purging. The keys
are space-delimited.
Imagine we want to remove all the objects that are tagged with the js
and css tags. You’d need to send the following HTTP request:
PURGE / HTTP/1.1
x-xkey-purge: js css
The response would be HTTP/1.1 200 Purged if the keys were found or
HTTP/1.1 404 Key not found when there are no corresponding keys.
Please note that you usually want to limit such functionality behind ACL. In the next chapter, we’ll talk about cache invalidation, and we’ll cover the
vmod_xkeyin more detail as well.
We always advise you to install Varnish Cache using our official packages. These are available on https://packagecloud.io/varnishcache. However, we don’t provide packages for the VMOD collection. This means you’ll have to compile them from source.
You can get the source code from https://github.com/varnish/varnish-modules. But for Varnish Cache 6.0 and Varnish Cache 6.0 LTS, you need to download the code from the right branch:
There are some build dependencies that need to be installed. On a Debian or Ubuntu systems, you can install them using the following command:
# apt-get install -y varnish-dev autoconf automake gcc libtool make python3-docutils
On Red Hat, Fedora, and CentOS systems, you can use the following command to install the dependencies:
# yum install -y varnish-devel autoconf automake gcc libtool make python3-docutils
Once all build dependencies are in place, you can compile the VMODs by running the following commands in the directory where the VMOD source files were extracted:
./bootstrap
./configure
make
make install
After having run make install, the corresponding .so files for these
VMODs can be found in the path that was defined by the vmod_dir
runtime parameter in Varnish.
The Debian and Ubuntu distributions also offer Varnish Cache via
their own packages. Some versions still provide the varnish-modules
package, which is Debian and Ubuntu’s version of the VMOD
collection.
Installing these VMODS is done using this very simple command:
# apt-get install varnish-modules
The following version of Debian and Ubuntu offer the
varnish-modules package for Varnish Cache 6:
Although we advise installing Varnish Cache 6.0 LTS from our official
packages, there’s no denying that it’s easy to just install Varnish
using a simple apt-get install varnish command.