Search
Varnish Cache Plus

Accounting

Description

Vmod-accounting is a VMOD that creates customizable statistics that can be used for accounting or any other use cases where you want to create custom summary statistics.

With vmod-accounting you can lock an incoming request to an accounting namespace. Setting a namespace for the request will generate statistics for any actions within the scope of the request (client or backend side). If you want to further scope statistics you can define keys within the namespace.

When accounting is imported for the first time the special ‘std’ namespace is created, together with its two keys ‘total’ and ‘sub’. One of the two keys in ‘std’ (‘total’ or ‘sub’) will always be incremented for all requests regardless of whether you set a specific namespace for the request or not. The ‘total’ key is updated for top level requests and the ‘sub’ key is updated for sub requests.

In order to set a namespace for a request you must first create the namespace in vcl_init. Each namespace will have the ‘total’ and ‘sub’ keys included automatically and space for 100 custom keys by default. If you want to be able to define more keys you can set the max number of keys for the namespace using the optional max_keys parameter when calling create_namespace().

The namespace and key(s) set for a request will persist across any actions within the request handling including labels, restarts and retries. There are however two exceptions to this. Any action taken in backend handling will not transfer to the client handling. I.e setting namespace and/or key(s) in a backend method will not apply for the client side. And secondly, if a request triggers a sub request, the sub request will inherit the parents namespace, but no other accounting information.

If you want to limit the scope when setting the namespace to only be set for client or backend you can do that with the optional ‘scope’ argument.

When calling add_key(), the request will be “tagged” to do accounting to that key. If that key does not yet exist in the set namespace it will be implicitly created. Once the request is completed, the statistics for that request will be accounted to the ‘total’/‘sub’ keys for the set namespace and any keys that the request has been tagged with.

By default, if the current request is a sub request generated by varnish, any call to add_key() will be ignored. If you want your key to track these requests as well you can set the optional parameter ‘sub’ to ‘INCLUDE’ when calling add_key(). Be aware that the data included in the top request from esi subs then will be accounted for twice for that key.

Each individual key has 50 predefined counters. These counters will appear in varnishstat on the form ACCG.<namespace>.<key>.<stat_name> where ACCG is short for accounting.

ACCG.<namespace>.<key>.client_req_count
ACCG.<namespace>.<key>.client_req_hdrbytes
ACCG.<namespace>.<key>.client_req_bodybytes
ACCG.<namespace>.<key>.client_resp_hdrbytes
ACCG.<namespace>.<key>.client_resp_bodybytes
ACCG.<namespace>.<key>.client_hit_count
ACCG.<namespace>.<key>.client_hit_req_hdrbytes
ACCG.<namespace>.<key>.client_hit_req_bodybytes
ACCG.<namespace>.<key>.client_hit_resp_hdrbytes
ACCG.<namespace>.<key>.client_hit_resp_bodybytes
ACCG.<namespace>.<key>.client_miss_count
ACCG.<namespace>.<key>.client_miss_req_hdrbytes
ACCG.<namespace>.<key>.client_miss_req_bodybytes
ACCG.<namespace>.<key>.client_miss_resp_hdrbytes
ACCG.<namespace>.<key>.client_miss_resp_bodybytes
ACCG.<namespace>.<key>.client_pass_count
ACCG.<namespace>.<key>.client_pass_req_hdrbytes
ACCG.<namespace>.<key>.client_pass_req_bodybytes
ACCG.<namespace>.<key>.client_pass_resp_hdrbytes
ACCG.<namespace>.<key>.client_pass_resp_bodybytes
ACCG.<namespace>.<key>.client_synth_count
ACCG.<namespace>.<key>.client_synth_req_hdrbytes
ACCG.<namespace>.<key>.client_synth_req_bodybytes
ACCG.<namespace>.<key>.client_synth_resp_hdrbytes
ACCG.<namespace>.<key>.client_synth_resp_bodybytes
ACCG.<namespace>.<key>.client_pipe_count
ACCG.<namespace>.<key>.client_pipe_req_hdrbytes
ACCG.<namespace>.<key>.client_pipe_req_bodybytes
ACCG.<namespace>.<key>.client_pipe_resp_hdrbytes
ACCG.<namespace>.<key>.client_pipe_resp_bodybytes
ACCG.<namespace>.<key>.client_200_count
ACCG.<namespace>.<key>.client_304_count
ACCG.<namespace>.<key>.client_404_count
ACCG.<namespace>.<key>.client_503_count
ACCG.<namespace>.<key>.client_2xx_count
ACCG.<namespace>.<key>.client_3xx_count
ACCG.<namespace>.<key>.client_4xx_count
ACCG.<namespace>.<key>.client_5xx_count
ACCG.<namespace>.<key>.backend_req_count
ACCG.<namespace>.<key>.backend_req_hdrbytes
ACCG.<namespace>.<key>.backend_req_bodybytes
ACCG.<namespace>.<key>.backend_resp_hdrbytes
ACCG.<namespace>.<key>.backend_resp_bodybytes
ACCG.<namespace>.<key>.backend_200_count
ACCG.<namespace>.<key>.backend_304_count
ACCG.<namespace>.<key>.backend_404_count
ACCG.<namespace>.<key>.backend_503_count
ACCG.<namespace>.<key>.backend_2xx_count
ACCG.<namespace>.<key>.backend_3xx_count
ACCG.<namespace>.<key>.backend_4xx_count
ACCG.<namespace>.<key>.backend_5xx_count

In addition to the stat counters there is also a group for tracking errors in accounting called ACCG_DIAG it defines 5 counters for common errors:

ACCG_DIAG.set_key_failure
ACCG_DIAG.out_of_key_slots
ACCG_DIAG.key_without_namespace
ACCG_DIAG.namespace_already_set
ACCG_DIAG.namespace_undefined
ACCG_DIAG.create_namespace_failure

You can find more detailed information on the predefined counters in the varnish-counters man page.

$ man varnish-counters

VCL Example

This simple Example shows how to leverage namespaces to separate statistics for different tenants while using keys on the tenant side for creating more fine grained statistics:

#Main VCL
vcl 4.1;

import accounting;

sub vcl_init {
	accounting.create_namespace("tenant1");
	accounting.create_namespace("tenant2");
	accounting.create_namespace("generic");
}
sub vcl_recv {
	if (req.http.host ~ "tenant1.com$") {
		accounting.set_namespace("tenant1");
		return(vcl(tenant1));
	} else if (req.http.host ~ "tenant2.com$") {
		accounting.set_namespace("tenant2");
		return(vcl(tenant2));
	} else {
		accounting.set_namespace("generic");
	}
}
# Tenant VCL
vcl 4.1;

import accounting;

sub vcl_backend_response {
	if (beresp.http.Content-Type ~ "^image") {
		accounting.add_keys("images");
	} else {
		accounting.add_keys("generic");
	}
}

API

create_namespace

VOID create_namespace(STRING namespace, INT max_keys = 100)

Creates a namespace to be used for accounting purposes. A namespace name can contain any character, but no more than 100 characters in total. Can only be called from sub vcl_init.

NOTE: For each unique key that is defined in the namespace there is a certain amount of extra work added to the request, defining too many keys in a namespace will eventually have a noticeable effect on performance. Use the optional argument max_keys to set the limit for the number of keys that can be defined in the namespace (default = 100).

set_namespace

VOID set_namespace(STRING namespace, ENUM {LOCAL, FULL} scope = "FULL")

Set the accounting namespace for the request. Only namespaces created during sub vcl_init are valid. Use the optional argument ‘scope’ to control the scope for which the namespace shall be set. If you set the scope to LOCAL the namespace will only be set for the current scope (client/backend). The default scope FULL will force the backend request to inherit the namespace (and keys) from the client request. Setting the namespace from a backend method will not affect the client side of the request. The namespace can only be set once for the set scope, subsequent calls for the same request will be ignored. Note: Sub requests inherit their namespace by the top request that triggered them, they will also inherit the namespace scope of the parent request.

add_keys

INT add_keys(STRING keys, STRING sep = ", ", ENUM {SKIP, INCLUDE} sub = "SKIP")

Add accounting key(s) specified in string ‘keys’, separated by ‘sep’ to the transaction, if no separator is specified the default “, ” will be assumed. Returns the number of key(s) successfully set. A key can contain any character, but not be more than 100 characters long. If the max number of keys for the namespace has already been defined, setting the key(s) will fail. If no namespace has been set for the transaction setting the key(s) will fail. Note that for the client side of the transaction any calls to add_keys() will be ignored if the request is a sub request generated by Varnish due to esi, edgestash or slicer. If you want the key(s) to be set also for sub requests you can set the optional parameter ‘sub’ to ‘INCLUDE’. Be aware that if you do set the same key(s) for both top requests and subs, any data inserted in the top request from the sub will then be double accounted to the key(s). Note: Since sub requests does not pass any VCL executed before a jump to a label, and (in difference to namespaces) keys are not inherited, there is no way to enforce a key to a sub outside of the active VCL.