SearchVarnish 6 is not a snapshot, and it doesn’t represent a single release: it’s the major version that groups a set of releases.
These are the seven Varnish versions that have been released so far:
Because Varnish is on a six-month release schedule, the minor versions go up quite quickly.
That doesn’t mean every release adds a lot of functionality. They happen on a regular basis, and when there’s something new, it’s added. Sometimes you end up with feature-heavy releases, sometimes there isn’t a lot to talk about.
There are also occasional maintenance releases. These aren’t really scheduled: they just happen when they happen, and they increase the patch version, e.g. Varnish 6.0.3. These maintenance releases contain bugfixes, enhancements, and sometimes even security fixes.
But in order to answer the question, and to tell you what’s new in Varnish 6, we need to look at the individual minor versions.
Varnish 6.0 isn’t all that feature heavy. The new major version is primarily justified by the work that was done under the hood.
However, there are two features that are quite impactful:
Varnish is a proxy and a cache, but it is often more correct to think of it as a component in an HTTP-based application or application stack, where HTTP is the main communication protocol. Usually, the HTTP calls are made over TCP/IP, where the peer can be very close or somewhere across the globe.
But these services aren’t always located on different servers, so there isn’t always a need for a protocol that can route traffic over the network. TCP/IP is complex, and relatively speaking, there can be a significant overhead and latency when using it, even within the same server.
When the web server is hosted on the same physical machine, you can communicate over UDS instead of TCP/IP. The same applies when you terminate TLS on the same machine: the connection between the TLS terminator and Varnish can be made over UDS.
So instead of connecting to the web server using the following backend definition:
vcl 4.0;
backend default {
.host = "localhost";
.port = "8080";
}
you can remove TCP/IP from the equation and use the .path attribute
to connect to the web server over UDS:
vcl 4.1;
backend default {
.path = "/var/run/webserver.sock";
}
It’s also possible to listen for incoming Varnish connections over UDS
by configuring the -a runtime option accordingly:
varnishd -a /var/run/varnish/varnish.sock,PROXY,user=vcache,group=varnish,mode=660
UDS support introduced a couple of new VCL variables:
local.endpoint: the address of the -a socket the session was
accepted onlocal.socket: the name of the -a socket the session was accepted
onThese variables and the new .path attribute aren’t available in
Varnish versions older than 6.0, so that forced a new VCL version.
When using UDS features in VCL, you have to make sure the VCL file
starts with vcl 4.1;.
There is not a whole lot to say about HTTP/2 support. Yes, it’s stable now, and there are a few new VCL variables related to this feature:
req.protobereq.protoberesp.protoresp.protoThese variables all expose the HTTP protocol version that is used for
the request or response. These are either HTTP/1.1 or HTTP/2.0.
Using these variables in your VCL file also requires the file to start
with vcl 4.1;.
At the moment, Varnish does not support HTTP/2 on the backend side,
but the variables bereq.proto and beresp.proto are reserved for
future versions of Varnish, which might support more protocols than
HTTP/1.1 and HTTP/1.0.
The shard director that is part of vmod_directors received lots of
improvements.
vmod_unix was also added to retrieve the group id (GID), the user
id (UID), the group name, and the user name of incoming connections
over UDS.
vmod_proxy is a useful addition for users who terminate TLS and
connect the TLS proxy to Varnish using the PROXY protocol. This VMOD
has the ability to retrieve TLS information from the PROXY protocol.
Most importantly whether or not the initial connection from the client
was made using TLS.
Varnish 6.0 re-introduced a couple of removed VCL variables:
req.ttl: upper limit on the object age for cache lookups to return
hitreq.grace: upper limit on the object graceThese variables are used to limit the time that objects are served from cache. Here’s an example of how they can be used:
vcl 4.1;
import std;
sub vcl_recv {
if (std.healthy(req.backend_hint)) {
set req.ttl = 1m;
set req.grace = 10s;
}
}
sub vcl_backend_response {
set beresp.ttl = 1h;
set beresp.grace = 10m;
}
Even though the time to live (TTL) is explicitly set to one hour in
vcl_backend_response, setting req.ttl and req.grace will limit the
effective TTL and grace for the current transaction. In other words,
when the backend is healthy, Varnish will only serve objects for one
minute with a ten-second grace. If it turns out that the backend is
not healthy, the original TTL and grace values will be used.
More information about TTL and grace can be found in the next chapter.
Another feature worth mentioning is std.fnmatch, which is part of the
standard VMOD (vmod_std). This function will perform shell-style
pattern matching on a string, whereas PCRE-style pattern matching is
otherwise used in Varnish.
Varnish 6.1 is a periodic release. It’s one of those releases when the release date is due, but apart from bugfixes and enhancements, there’s nothing much more to say.
If you really care about the internals, you can always have a look at the most recent version of the changelog.
Varnish 6.2 introduced two new variables that allow you to check if a
request will bypass the regular caching flow, and immediately fetch the
data from the backend. This happens when the corresponding backend
response is marked as hit-for-miss or hit-for-pass. The respective
variables are req.is_hitmiss and req.is_hitpass.
There was also an adjustment in the flow of Varnish’s finite state
machine, which involved return(miss) being removed from vcl_hit{}.
There are other ways to still trigger a miss, even when a hit took
place, but it is no longer deemed a common scenario.
A couple of new type conversion functions were added to vmod_std.
A new lookup function was added to vmod_directors to look up
individual backends by name.
The varnishadm command line tool can now return output in JSON
format thanks to the -j option.
And logging programs like varnishlog and varnishncsa will perform
better due to internal enhancements and the introduction of the -R
rate-limiting option.
And as always: this version of Varnish features a number of bugfixes and enhancements.
vcl_backend_errorVarnish 6.3 allows you to explicitly return an error from
vcl_backend_fetch and vcl_backend_response. This triggers the
vcl_backend_error state and can be achieved using either of the
statements below:
return (error);
return (error(503));
return (error(503, "Service Unavailable"));
This is the erroneous equivalent of return (synth(200,"OK"));
In Varnish 6.3 it is now possible to import a VMOD multiple times. It’s also possible to run them under a different name, as illustrated below:
vcl 4.1;
import directors as dir;
sub vcl_init {
new rr = dir.round_robin();
}
auto VCL temperature stateWhen loading multiple VCL files using varnishadm, the behavior of
the auto state with regard to the VCL temperature has changed: in
previous releases, the VCLs could cool down and would remain cold. As
of Varnish 6.3 it works in both directions, and these VCLs can
automatically warm up again when required.
std.ip() accepts optional port argumentThe std.ip() function now accepts an optional port argument that
overrides the default port value when it is called through std.port().
The default value is 80.
This is the new definition of std.ip():
IP ip(STRING s, [IP fallback], BOOL resolve=1, [STRING p])
VSL tools like varnishlog and varnishncsa have the ability to
filter output based on VSL queries using the -q option.
Prior to Varnish 6.3, a new line in a -q statement was merely used
as a token separator. As of Varnish 6.3, every new line in a -q
statement is treated as a new query. Unless specified otherwise, the
statements are joined with the or operator.
Here’s an example:
varnishlog -q "
BerespStatus < 400
ReqUrl eq '/'
"
Prior to Varnish 6.3, the use of multiple -q options would result in
the last query being selected. As of Varnish 6.3, multiple queries
will be joined with the or operator, as illustrated below:
varnishlog -q "BerespStatus < 400" -q "ReqUrl eq '/'"
The uppercase -Q is now available and reads stored VSL queries from
a file. The -Q option can also be used multiple times, which just adds
queries with the or operator to any query specified by either -q or
-Q.
Here’s an example:
varnishlog -Q query1.vsl -Q query2.vsl
And finally the varnishncsa program can take an -E option, which
includes ESI transactions in the output.
if-range supportVarnish 6.4 features support for the if-range request header. The
value of that header is either a date or an ETag value.
The result is that range requests are only performed if the value of
the if-range header either matches the Last-Modified date of the
response, or if it matches the value of the ETag response header.
Long story short: the if-range header performs conditional range
requests.
vmod_cookie from varnish_modulesPrior to Varnish 6.4, vmod_cookie had to be installed from the
varnish_modules
repository. You either had to compile it manually, or get it from the
package manager of your operating system.
As of Varnish 6.4, vmod_cookie has been imported into the main tree
and is now available by default.
vmod_cookie makes parsing, fetching, and modifying cookies a lot
easier. Here’s an example:
vcl 4.1;
import cookie;
sub vcl_recv {
if (req.http.cookie) {
cookie.parse(req.http.cookie);
# Either delete the ones you want to get rid of:
cookie.delete("cookie2");
# or delete all but a few:
cookie.keep("SESSIONID,PHPSESSID");
# Store it back into req so it will be passed to the backend.
set req.http.cookie = cookie.get_string();
# If empty, unset so the builtin VCL can consider it for caching.
if (req.http.cookie == "") {
unset req.http.cookie;
}
}
}
In Varnish Enterprise,
vmod_cookieplussolves many of the same use cases in a similar way. The feature set ofvmod_cookieplusis also a bit broader.
none backendsIn Varnish 6.4, it is possible to explicitly state that you do not
want a default backend by writing backend default none;.
The symbol none is similar to NULL, null and nullptr in other
languages and represents the absence of an actual backend.
Any attempt to use the none backend, which is not a backend, will
result in a failure.
In some cases, specifying none as the default makes a lot of sense:
for example, you might use Varnish to just generate synthetic HTTP
responses.
Another use case is when you are using labels to split VCL logic into
more than one VCL, and the main VCL simply jumps to other loaded
VCLs through return (vcl(...)). In these cases, it does not make
sense to define an actual backend in the main VCL.
Finally, vmod_goto, a Varnish Enterprise VMOD, provides dynamic
backends. By using goto.dns_director() or goto.dns_backend(), none
makes a good choice as the default backend.
There are also some smaller VCL changes in 6.4:
std.rollback(header) can be used to roll back the value of that
header to its original state.set resp.http.ok = -std.integer("-200");.+= operator can be used to append data to headers and response
bodies.Access control lists in Varnish support hostnames, individual IP addresses, and also subnets. These subnets use the CIDR notation.
An example is 192.168.0.0/24. As of Varnish Cache 6.5, the use of a
non-zero host part will result in an Address/Netmask mismatch warning
during VCL compilation.
Here’s an example of a such a mismatch:
acl myAcl {
"192.168.0.10"/24;
}
The correct notation that is enforced as of Varnish Cache 6.5, is the following:
acl myAcl {
"192.168.0.0"/24;
}
The vcc_acl_pedantic runtime parameter can turn ACL CIDR mismatch
warnings into actual errors when enabled. As mentioned, these errors
are triggered when the host bits of a CIDR in an ACL aren’t
all-zero.
Varnish Cache 6.5 introduced the obj.can_esi variable, which returns
a boolean. If the response that is stored in the object can be processed
using ESI, it returns true.
As of Varnish Cache 6.5, there is a new .resolve() method for
backends and directors, which immediately resolves a backend.
Explaining exactly what this means requires an understanding of what a
director is. In short, a director organizes a set of backends, for
example for load balancing, and selects one from the set when a backend
is needed to carry out a backend request. The new .resolve() method
forces the director to immediately select a concrete backend.
Note that this method is not related to DNS resolution when a backend is defined using a domain name. In these cases, the DNS lookup happens once, during compilation, and the IP address is constant throughout the lifetime of the VCL.
Read more about different types of directors in chapter 5.
Varnish Cache 6.5 now allows you to explicitly close the connection in VCL, bypassing any potential keep-alive settings that were in place.
Here’s an example of how this can be accomplished in VCL:
sub vcl_backend_response {
if (beresp.backend == faulty_backend) {
if (beresp.http.Connection) {
set beresp.http.Connection += ", close";
} else {
set beresp.http.Connection = "close";
}
}
}
Binary large objects or BLOBs as we call them, now have a new
literal syntax as of Varnish Cache 6.5. The format is :<base64>:.
Here’s an example of such a literal:
:3k0f0yRKtKt7akzkyNsTGSDOJAZOQowTwKWhu5+kIu0=:
Similar to std.fileread(), the new std.blobread() function will read
data from disk, but it will return a BLOB rather than a string. This
is ideal for reading binary files.
When a backend is explicitly set to unhealthy using
varnishadm backend.set_health, Varnish will no longer attempt to
connect to the backend. The unhealthy status is immediately noticed,
and a HTTP 503 error is returned.
If you type h in varnishstat, you’ll get a help page as of Varnish
Cache 6.5, explaining the various controls.
The -b and -f runtime parameters are mutually exclusive. Prior to
Varnish Cache 6.5 it was not possible to start Varnish without
defining the none backend in your VCL file, which required using the
-f option.
In Varnish Cache 6.5 it is now possible to use -b none to start the
varnishd program without having to use the -f parameter.
Headers can now be validated against the rules set by RFC7230. By default header validation doesn’t take place.
By adding the -p feature=+validate_headers runtime parameter to
varnishd, header validation is enabled.
In Varnish Cache 6.6 the number of cache variations is now limited by
the newly introduced vary_notice parameter. The default value is ten.
When the number of cache variations exceeds this value, a Notice
record will be added to the Varnish Shared Memory Log (VSL).
The ban() function is now deprecated because it lacks the ability to
evaluate its success. In Varnish Cache 6.6 two new ban functions were
added:
std.ban(): performs the ban and either returns true or false
depending on its success.std.ban_error(): if an error occurred in std.ban(),
std.ban_error() will display a detailed error message.Here’s a tiny VCL snippet that illustrates its use:
if (std.ban(...)) {
return(synth(200, "Ban added"));
} else {
return(synth(400, std.ban_error()));
}
Varnish Cache 6.6 added the modulus operator to VCL. This
corresponds to % and can be used as follows:
vcl 4.1;
import std;
sub vcl_recv {
set req.http.number = "4";
if(std.integer(req.http.number) % 2 == 0) {
return(synth(200,"Even"));
} else {
return(synth(200,"Odd"));
}
}
In addition to the {" ... "} format for denoting long strings,
Varnish Cache 6.6 has now added the """ ... """ format.
The built-in VCL has been reworked: VCL code has been split into small subroutines, to which custom VCL can prepend custom code.
The client.identity variable is now accessible on the backend side.
Prior to the release of Varnish Cache 6.6, this variable was only
accessible on the client side.
The variables bereq.is_hitpass and bereq.is_hitmiss have been added
to the backend side. They match the corresponding req.is_hitpass and
req.is_hitmiss that operate on the client side.
The bereq.xid variable is now also available in the vcl_pipe
subroutine.
The resp.proto variable is now a read-only variable.
Several of the features described above have been ported to the 6.0 long-term support release series of Varnish Cache, and to the corresponding Varnish Enterprise offering. The changelog for Varnish Cache 6.0 LTS releases can be found via https://varnish-cache.org/, while Varnish Enterprise changes are available at https://docs.varnish-software.com/varnish-cache-plus/changelog/changes/.