HTTP calls

Varnish is all about HTTP:

It accepts incoming HTTP requests.
It returns HTTP responses.
It connects to the origin and sends backend HTTP requests.
It stores backend HTTP responses in cache.

The entire flow of Varnish is centered around HTTP, and yet this section is about making HTTP calls using vmod_http.

vmod_http is a Varnish Enterprise module, but there is probably an open source equivalent out there in the community.

Sometimes your use case requires making HTTP calls to endpoints that are not related to the incoming HTTP request that Varnish is currently processing. This could be an API call to fetch stateful data. This could be an internal subrequest to another resource.

Let’s look at some practical use cases.

Prefetching

A very common use case is using vmod_http for prefetching. This means retrieving content before it is actually requested by the client. The assumption is that the client will soon request that content, and having it in cache ahead of time will improve the user experience.

Link prefetching

Link prefetching is where the origin application requests prefetching of certain resources via a Link response header or a <link> HTML tag.

This mechanism is mostly used to load CSS, JavaScript, images, favicons and web fonts that are required by the website theme.

Here’s a VCL example where link prefetching is done by inspecting the Link response header:

vcl 4.1;

import http;

sub vcl_recv {
	set req.http.X-prefetch = http.varnish_url("/");
}

sub vcl_backend_response {
	if(beresp.http.Link ~ "<([^>]+)>; rel=(prefetch|next)") {
		set bereq.http.X-link = regsub(beresp.http.Link, "^.*<([^>]*)>.*$", "\1");
		set bereq.http.X-prefetch = regsub(bereq.http.X-prefetch, "/$", bereq.http.X-link);
		http.init(0);
		http.req_copy_headers(0);
		http.req_set_url(0, bereq.http.X-prefetch);
		http.req_send_and_finish(0);
	}
}

Imagine that an HTTP response contains the following response header:

Link: </style.css>; rel="prefetch"

The VCL example above would extract the URL from this header, and then use it to make an asynchronous HTTP request via vmod_http.

This means that we don’t wait for the response to be returned because frankly we don’t really care about the response. We just want to trigger a subrequest that fetches the required resource. Eventually the client will request /style.css, and we assume that it will be in cache thanks to our prefetching logic.

The same can be done by parsing a <link rel="prefetch" href="/style.css" /> HTML tag in the response body. These tags are designed to trigger prefetching at the browser level, but we might as well benefit from them in Varnish too.

Video prefetching

Varnish can also be used to accelerate video platforms, as you’ll see in the next chapter.

OTT video streaming chops up encoded video files into various segments, each segment representing a couple of seconds of video playback.

A playlist file contains the endpoints of the various video segments. It may look like this:

/vod/video1_1.ts
/vod/video1_2.ts
/vod/video1_3.ts
/vod/video1_4.ts
/vod/video1_5.ts

Because of the sequential naming format, it’s quite easy to guess what the URL of the next segment will be. And that’s exactly what the http.prefetch_next_url() does.

The following VCL example will use this function to preload the next video segment:

vcl 4.1;

import http;

sub vcl_recv {
	if (req.url ~ "^/vod/video[0-9]+_[0-9+]\.ts") {
		http.init(0);
		http.req_copy_headers(0);
		http.req_set_method(0, "HEAD");
		http.req_set_url(0, http.prefetch_next_url());
		http.req_send_and_finish(0);
	}
}

http.prefetch_next_url() looks for numeric sequences and does a standard increment of one. This can be altered through the count argument.

The prefix argument defines a pattern that should be matched in the URL before considering the increment.

The url argument can be used to set the input URL that should be examined, whereas url_prefix allows you to prefix the URL with a scheme or port. If url or url_prefix are not set, http:// is used as the scheme, std.port(server.ip) is used to determine the port, and req.url is used to determine the URL.

There’s also a base argument, which defaults to DECIMAL, which sets the number system that is to be used. This is DECIMAL, HEX or HEX_UPPER.

The function can be run standalone, and here’s an example that contains some arguments:

http.prefetch_next_url(prefix="test",
	url="/test1.txt",
	url_prefix="https://test.com:1234",
	count=4
);

Unsurprisingly, the output goes as follows:

https://test.com:1234/test5.txt

The http.prefetch_next_url() can also be used outside of the video-streaming scope. Paginated web content is also a good use case.

API calls

It is also possible to perform HTTP requests to remote hosts that are not directly related to the origin platform. API calls could be made, and the output could be parsed in VCL.

Here’s an example where we query a RESTful API to get the current weather in London:

vcl 4.1;

import http;

sub vcl_backend_response {
	http.init(0);
	http.req_set_header(0, "Host", "www.metaweather.com");
	http.req_set_url(0, "https://www.metaweather.com/api/location/44418/");
	http.req_send(0);
	http.resp_wait(0);
	if (http.resp_get_status(0) != 200) {
		return (error(500,""));
	}
	json.parse(http.resp_get_body(0));
	xbody.regsub({"(<h1 class="fw-light">The weather in London: )([^<]+)(</h1>)"},
		"\1"+json.get(".consolidated_weather[0].weather_state_name")+"\3");

}

Before the object is stored in cache, a request to https://www.metaweather.com/api/location/44418/ is made, which returns JSON data containing the weather in London.

json.parse(http.resp_get_body(0)) will parse the resulting JSON output, which we can filter using json.get(). In this case we care about the consolidated_weather property. This is an array, and we grab the first item from that array and return the weather_state_name property to get the current weather.

In the end we use xbody.regsub() to inject the actual weather.

Imagine receiving the following HTML tag from the origin:

<h1 class="fw-light">The weather in London: Sunny</h1>

Our vmod_http, vmod_json and vmod_xbody logic will cause the following string to be stored in cache:

<h1 class="fw-light">The weather in London: Light Rain</h1>

Authentication

This chapter has a section dedicated to authentication, so we won’t go into great detail here. We will just throw in one basic example where Varnish can act as an authentication gateway.

Imagine that you want to protect your web application with basic authentication. But if you remember the built-in VCL, a return(pass) will take place when an Authorization header is found in the request.

The idea is to terminate the authentication on the edge and forward the authentication request to an endpoint via vmod_http.

Here’s an example where we offload the authentication to https://auth.example.com/auth:

vcl 4.1;

import kvstore;
import http;

sub vcl_init {
	new auth = kvstore.init();
}

sub vcl_recv {
	if (auth.get(client.ip,"0") == "0" {
		http.init(0);
		http.req_copy_headers(0);
		http.req_set_url(0, "https://auth.example.com/auth");
		http.req_send(0);
		http.resp_wait(0);
		if (http.resp_get_status(0) != 200) {
			return(synth(401,"Authentication required"));
		}
		auth.set(client.ip,"1",3h);
		unset req.http.Authorization;
	}
}

sub vcl_synth {
	if (resp.status == 401) {
		set resp.http.www-authenticate = "Basic";
	}
}

The http.req_copy_headers(0) function ensures that client request headers are forwarded to the authentication endpoint. If the Authorization header is missing, or does not match the expected credentials, an HTTP 401 is triggered, which we return to the client.

If the authentication is successful, we store the authentication state inside a vmod_kvstore instance. We also strip off the Authorization header to ensure that the built-in VCL serves the request from cache.

As you can see, vmod_http offers a lot of options and will be featured again in this chapter. You’ll also see a dedicated authentication section later on in this chapter.