SearchHere we will go through some of the typical use cases for VCS. There are endless opportunities for tracking all aspects of website behavior. These examples will hopefully give you a good idea of what is possible and provide you with a bit of inspiration.
Most of these examples can be implemented with a few lines of VCL code in your Varnish setup. They work with both the vanilla Varnish Cache release and the Varnish Enterprise release.
By default, vstatdprobe installs with the -d parameter enabled.
This configuration automatically generates a key for each URL, HOST, and a global ALL key.
Manually tagging a request with a key is done in VCL, by writing an std.log()
line prefixed with the string "vcs-key:". The default key configuration is
equivalent to the following VCL, used as an example:
sub vcl_deliver {
std.log("vcs-key:ALL");
std.log("vcs-key:HOST/" + req.http.Host);
std.log("vcs-key:URL/" + req.http.Host + req.url);
}
In the above example, all requests will be tagged with the following keys:
Host header (example.com)Host + URL (example.com/foo)vcs-key ALLFor std.log() you will also need to include the std VMOD, with
an import std; directive in your VCL.
VCS has a flat namespace. Every key is created in this namespace. So, in order to add a bit of organization to your VCS setup we recommend your split the namespace into various sub-namespaces.
To split a namespace we recommend you use a separator. We recommend
you use / and we’ll be using it in our examples here.
The reasons for splitting the namespace would be to create queries
against VCS that gives you some subset of the data in VCS. Lets say
that you use VCS to track the number of views on your website. If you
prepend those keys with VIEWS you can query VCS to give you a top
list of the views by asking it to show you the top list of every key
beginning with VIEWS.
Then you might have another query that gives you the top list of
caches misses - MISSES and other logical groups.
To omit certain URLs (or requests) from the default keys, remove the
-d parameter from vstatdprobe’s DAEMON_OPTS.
Next, add the following VCL to generate the VCS keys, using an if
statement to skip the requests you do not want to send to VCS:
sub vcl_deliver {
if (req.url != "/healthcheck") {
std.log("vcs-key:ALL");
std.log("vcs-key:HOST/" + req.http.Host);
std.log("vcs-key:URL/" + req.http.Host + req.url);
}
}
In the above example, requests for /healthcheck will not be sent to
VCS.
To track which URLs have the slowest response times, we can make use of VCS’ ability to provide a sorted list of response times for the keys it is tracking. Simply issuing a request for:
/all/top_ttfb
will produce a list of the keys associated with the 10 slowest requests. To further get a breakdown of this, for example to get the actual URLs, we can make use of the default keys and combine this with VCS’ regex matching capabilities:
/match/^URL/top_ttfb
The abbreviation ttfb stands for time to first byte, and is the time
between Varnish first started handling the request until it started
transmitting the first byte to the client.
For a news site there are a few specific things you might want to track. CMS systems typically have unique article IDs that identify one article. Logging the article IDs into VCS gives you easy real time access to what stories are being read right now. We have customers that are embedding this information on their websites generating the what is hot right now lists we often see on a news site.
Logging the article ID and not just the URL make the list ignore different presentations of the same article and makes the list about the articles themselves. It also removes the need to normalize the URL in any way, so query strings that annotate links will not pollute the list itself.
If your CMS can produce an x-artid header you should be all set.
In vcl_deliver you would need to add the following:
sub vcl_deliver {
std.log("vcs-key:ARTICLE_ID/" + resp.http.x-artid);
}
You can expand on the setup in several ways. One might for instance also want to measure the social impact of each article by looking at the referrer header (if set).
In vcl_deliver add the following:
sub vcl_deliver {
if (req.http.referer) {
std.log("vcs-key:ARTREF/" + resp.http.x-artid + "/" + req.http.referer);
}
}
You might also want to expand it further by looking at the user agent
and adding a separate time series for mobile views. In vcl_deliver:
sub vcl_deliver {
if (req.http.user-agent ~ "mobile") {
std.log("vcs-key:MOBILE/" + resp.http.x-artid);
}
}
Many websites want to measure conversions. A conversions might be having a user click a link to sign up, putting an item in the shopping basket. Another use case would be for a paid content site, where the conversion happens with the user clicking the sign up page when reading a specific article.
The first step is to identify the conversion taking place, typically done by looking at the request URL, maybe in combination with the HTTP method used.
In this example our article page might be /news/art/23245. On that
page there is a link pointing to /signup. To register the conversion
in VCS with the article as the main key we would need the following
VCL in vcl_deliver:
sub vcl_deliver {
if (req.url == "/signup") {
set req.http.artid = regsub(...);
std.log("vcs-key:CONVERSION/SIGNUP/" + req.http.artid);
}
}
For a more in depth discussion on using VCS to track conversions, and also a how-to on doing AB testing with Varnish and VCS, please see this blog post: https://info.varnish-software.com/blog/live-ab-testing-varnish-and-vcs
If you are streaming HLS/HDS/Smooth/DASH through Varnish you might want to count the number of users on each Varnish server. This might be useful for statistical reasons but might also be used for directing traffic to your various Varnish Cache clusters.
The tricky part is to uniquely identify a user. In order to do this you need some sort of session cookie to be preset on the client. All the HTTP video clients are suppose to support cookies. If there is a cookie already present we can probably utilize it, if not we have to generate a random one.
We recommend using the cookie VMOD when working with cookies. It will make the VCL much more readable. The following VCL sets a cookie if there is none present.
In vcl_deliver:
import cookie;
sub vcl_deliver
{
cookie.parse(req.http.cookie);
set req.http.X-vcsid = cookie.get("_vcsid");
if (req.http.X-vcsid == "") {
set req.http.X-vcsid = std.random(1, 10000000) + "." + std.random(1, 10000000);
set resp.http.Set-Cookie = "_vcsid=" + req.http.X-vcsid + "; HttpOnly; Path=/";
}
std.log("vcs-key:SESSION/" + req.http.X-vcsid + "/" + req.http.Host + req.url);
}
There is a blog post on the matter that discusses this in some detail: https://info.varnish-software.com/blog/getting-live-statistics-varnish-hlshds
In an e-commerce setting VCS can be used to give stats about how various SKUs behave. A typical use case would be running statistics on which SKUs receive what traffic. In addition there are various other aspects that VCS can help gather data on:
In vcl_deliver:
sub vcl_deliver {
if (req.url ~ "/sku/\d+") {
set req.http.sku = regsub(...);
std.log("vcs-key:VIEWSKU/" + req.http.sku);
if (req.http.referer ~ "facebook.com|twitter.com") {
std.log("vcs-key:SOCIAL/" + req.http.sku);
}
if (req.http.referer ~ "yahoo.com|google.com") {
std.log("vcs-key:ORGANIC/" + req.http.sku);
}
if (req.url ~ "/ajax/put/\d+") {
std.log("vcs-key:PUTBASKET/" + req.http.sku);
}
}
}