Writing your own VMODs

Writing your own VMODs

Anything that can be written in C can become a VMOD. In this chapter we’ve gone through a long list of VMODs: Some of them are managed by the Varnish Cache team, a lot of them are managed by Varnish Software, and then there are VMODs that are managed by individual contributors.

A lot of common scenarios are already covered by a VMOD, but there’s always a chance that you have a use case where VCL cannot solve the issue, and there isn’t a matching VMOD either.

In that case, you can write one yourself. That’s what we’re going to do in this section. Even if you’re not planning to write your own VMOD, it is still interesting to learn how VMODs are composed.

vmod_example

To get started with VMOD development, you should have a look at https://github.com/varnishcache/libvmod-example. This GitHub repository hosts the code and build scripts for vmod_example.

vmod_example is a stripped down VMOD that serves as the boilerplate. It is the ideal starting point for novice VMOD developers.

This is what the directory structure of this repo looks like:

.
|-- Makefile.am
|-- autogen.sh
|-- configure.ac
|-- m4
|   `-- ax_pthread.m4
|-- rename-vmod-script
`-- src
	|-- Makefile.am
	|-- tests
	|   `-- test01.vtc
	|-- vmod_example.c
	`-- vmod_example.vcc

The core of the code is the src folder where the source files are located:

• vmod_example.c contains the source code of this VMOD.
• vmod_example.vcc contains the interface between the code and the VCL compiler (VCC).

A useful script is rename-vmod-script: you’re not going to name your custom VMOD vmod_example. This script is there to rename the VMOD and replace all the occurrences of example with the actual name of the new VMOD.

So, if you wanted to name your VMOD vmod_os, you’d do the following:

./rename-vmod-script os

This means vmod_example is renamed to vmod_os.

The VMOD we’re going to develop will display operating system information, hence the name vmod_os.

Files like Makefile.am, configure.ac, and the m4 directory are there to facilitate the build process. They are used by autoconf, automake, and libtool, and are triggered by the autogen.sh shell script.

Turning vmod_example into vmod_os

Now that we’ve been introduced to vmod_example, it’s time to customize the code, and turn it into your own VMOD.

Dependencies

The first thing we need to do is make sure all the dependencies are in place. Just like in the previous part about the Varnish Software VMOD collection, we need the following dependencies on Debian and Ubuntu systems:

# apt-get install -y varnish-dev autoconf automake gcc libtool make \
python3-docutils git

On Red Hat, Fedora, and CentOS systems, you can use the following command to install the dependencies:

# yum install -y varnish-devel autoconf automake gcc libtool make \
python3-docutils git

Please note that git was also added as a dependency. It is used to retrieve and check out a copy of the repository on your machine.

Downloading these tools allows the autogen.sh to generate the software configuration, and eventually generate the Makefile.

Getting the code

The best way to get the code is by cloning the Git repository, as demonstrated below:

git clone https://github.com/varnishcache/libvmod-example.git

This creates a libvmod-example folder that includes all the code from the repo.

But as explained, we don’t really care about vmod_example; we want to develop vmod_os. That requires some renaming:

mv libvmod-example/ libvmod-os/
cd libvmod-os/
./rename-vmod-script os

These commands will rename the local folder and will make sure all references to example are replaced with os.

Looking at the vmod_os.c

Enough with the directory structure and the build scripts: a VMOD is all about custom code. Let’s take a look at the custom code then.

This is the code we’re going to put inside src/vmod_os.c:

#include "config.h"
#include "cache/cache.h"
#include <sys/utsname.h>
#include "vcc_os_if.h"

VCL_STRING
vmod_uname(VRT_CTX, VCL_BOOL html)
{
	struct utsname uname_data;
	char *uname_str;
	char *br = "";

	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);

	if (uname(&uname_data)) {
		VRT_fail(ctx, "uname() failed");
		return (NULL);
	}

	if (html) {
		br = "<br>";
	}

	uname_str = WS_Printf(ctx->ws,
		"OS: %s%s\n"
		"Release: %s%s\n"
		"Version: %s%s\n"
		"Machine: %s%s\n"
		"Host: %s%s\n",
		uname_data.sysname, br, uname_data.release, br,
		uname_data.version, br, uname_data.machine, br,
		uname_data.nodename, br);

	if (!uname_str) {
		VRT_fail(ctx, "uname() out of workspace");
		return (NULL);
	}

	return (uname_str);
}

After having included the header files of our dependencies, we can define the function we want to expose to VCL: uname.

Conventionally, the C-function is then named vmod_uname(). When we look at the function interface, we notice two arguments:

• VRT_CTX, which is a macro that gets replaced by struct vrt_ctx *ctx at compile time
• VCL_BOOL html, which is an actual argument that will be used in VCL

VRT_CTX refers to the vrt_ctx structure that holds the context of the VMOD.

VCL_BOOL indicates that the input we receive from VCL through this argument will be handled as a boolean with a true or false value. The name of the argument is html.

By enabling this html flag, we return the output in HTML format. Otherwise, we just return plain text data.

Within the function, we see the following variable initialization:

struct utsname uname_data;
char *uname_str;
char *br = "";

• struct utsname uname_data is used to initialize the data structure that will hold the uname data that will be retrieved from the operating system.
• char *uname_str is a string, a char pointer to be precise, that will hold the output of the utsname structure.
• char *br = "" initializes our line break variable as an empty string.

CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC) ensures that the context is correctly set before running the uname() function. It is part of the consistency self-checks that are present all throughout the core Varnish code, and it helps protect against many nasty forms of bugs.

The following part of the source code calls the uname() function and evaluates the output:

if (uname(&uname_data)) {
	VRT_fail(ctx, "uname() failed");
	return (NULL);
}

uname() will store its data in a utsname structure. We named this structure uname_data and passed it by reference. The output of the function reflects its exit code: anything other than zero is an error, according to the documentation for the function, which can be viewed by running the man 2 uname command.

If an error does occur, a VRT_fail() is executed, which will record this call as a failure in Varnish. As a consequence the whole request will fail, and Varnish will return an HTTP 503 error to the client that made the request. Varnish will continue to chug along as if nothing happened. It is not a serious problem, just an indication that the VCL execution failed.

The next step is defining the line break format. If the html variable is true, the br variable will contain <br>, which is the HTML equivalent of a line break:

if (html) {
	br = "<br>";
}

If html is false, br is just an empty string, which is fine for plain text.

Eventually, the output from the uname_data structure is extracted, and turned into a string:

uname_str = WS_Printf(ctx->ws,
	"OS: %s%s\n"
	"Release: %s%s\n"
	"Version: %s%s\n"
	"Machine: %s%s\n"
	"Host: %s%s\n",
	uname_data.sysname, br, uname_data.release, br,
	uname_data.version, br, uname_data.machine, br,
	uname_data.nodename, br);

Using the WS_Printf() function, typical printf() logic is executed, but workspace memory is allocated automatically. The uname_str variable where the results are stored can return a string that looks like this:

OS: Linux
Release: 4.19.76-linuxkit
Version: #1 SMP Tue May 26 11:42:35 UTC 2020
Machine: x86_64
Host: 19ee8d684eea

There is of course no guarantee that the newly formatted string will fit on the workspace, which is quite small on the client side. An extra check is added to the source code to deal with that:

if (!uname_str) {
	VRT_fail(ctx, "uname() out of workspace");
	return (NULL);
}

If the uname_str string is not set, it probably means we ran out of workspace memory. We will once again call VRT_fail, as it is an exceptional condition. It would have been possible to just return NULL without failing the whole request, but we like things being straight and narrow, reducing the complexity our users have to deal with.

So, finally, when all is good and all checks have passed, we can successfully return the uname_string:

return (uname_str);

This will hand over the string value to the VCL code that called os.uname() in the first place.

Looking at the vmod_os.vcc

Another crucial file in the src folder is vmod_os.vcc. This file contains the API that is exposed to the VCL compiler and is called by the vmodtool.py script.

Here’s the vmod_os.vcc code:

$Module os 3 OS VMOD

DESCRIPTION
===========

OS support

$Function STRING uname(BOOL html = 0)

Return the system uname

This file defines the name of the module through the $Module statement, but it also lists the function API through the $Function statement.

The other more verbose information in this file is used to generate the man pages for this module.

What we can determine from this file is pretty straightforward:

• There is a VMOD called os.
• There’s a function called uname().
• The os.uname() function returns a string.
• The os.uname() function takes one argument, which is a boolean called html.
• The default value of the html argument is false.

Building the VMOD

The procedure that is used to build vmod_os is quite similar to the procedure we used to build the Varnish Software VMOD collection. Only the first shell script to initialize the software configuration is different.

Be sure to install build dependencies like autoconf, automake, make, gcc, libtool, and rst2man before proceeding.

Here’s what you have to run:

./autogen.sh
./configure
make
make install

• ./autogen.sh will prepare the software configuration file using tools like autoconf and automake.
• ./configure is a shell script that is generated in the previous step and that prepares the build configuration. It also generates the Makefile.
• make will actually compile the source code using the gcc compiler.
• make install will use libtool to bundle the compiled files into an .so file and will put the library file in the right directory.

Testing the VMOD

Once the VMOD is built, we can test whether it behaves as we would expect. In the src/tests folder, you can edit test01.vtc and put in the following content:

varnishtest "Test os vmod"

server s1 {
} -start

varnish v1 -vcl+backend {
	import ${vmod_os};

	sub vcl_recv {
		return (synth(200, "UNAME"));
	}

	sub vcl_synth {
		synthetic(os.uname());
		return (deliver);
	}
} -start

client c1 {
	txreq
	rxresp
	expect resp.body ~ "OS:"
} -run

This vtc file contains the syntax that is required to perform functional tests in Varnish. This test case has three components:

• A server that acts as the origin
• A varnish that processed the VCL
• A client that sends requests to varnish

In this case, the server does absolutely nothing, because varnish will return synthetic output based on the VMOD that it imports. The ${vmo_os} statement will dynamically parse in the location of the built VMOD. The syntax is composed such that it can be executed before make install is called.

You can run the tests by calling make check. This can be done right after make and before make install if you want.

This is the output that you get:

PASS: tests/test01.vtc
============================================================================
Testsuite summary for libvmod-os 0.1
============================================================================
# TOTAL: 1
# PASS:  1
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================

All is good, and the test has passed. The assertion from the test is that the output from the os.uname() call will contain the OS: string.

test01.vtc can also be called via varnishtest, but it requires ${vmod_os} to be replaced with os.

The following command can then be run:

varnishtest src/tests/test01.vtc

And the output will be the following:

#    top  TEST test01.vtc passed (5.221)

Using the VMOD

Once the build is completed, and make install is executed, libvmod_os.so will be stored in the path that corresponds to the vmod_path runtime parameter.

Then you can safely import the VMOD into your VCL file using import os;. Here’s a VCL file that uses are custom VMOD:

vcl 4.1;

import os;

backend default none;

sub vcl_recv
{
	if (req.url == "/uname") {
		return (synth(200, "UNAME"));
	}
}

sub vcl_synth
{
	if (resp.reason == "UNAME") {
		if (req.http.User-Agent ~ "curl") {
			synthetic(os.uname());
		} else {
			set resp.http.Content-Type = "text/html";
			synthetic(os.uname(html = true));
		}
		return (deliver);
	}
}

The output when /uname is called, could be the following:

OS: Linux
Release: 4.19.76-linuxkit
Version: #1 SMP Tue May 26 11:42:35 UTC 2020
Machine: x86_64
Host: 19ee8d684eea

This VCL example will use plain text output when the curl User-Agent is used. For any other browser, a <br> line break will be added to each line, and the text/html Content-Type header is added.