Search
Varnish Cache Plus

Accounting

Description

The accounting vmod creates customizable statistics that can be used for accounting or any other use cases where you want to create custom summary statistics.

With vmod-accounting you can lock an incoming request to an accounting namespace. Setting a namespace for the request will generate statistics for any actions within the scope of the request (client or backend side). If you want to further scope statistics, you can define keys within the namespace.

When accounting is imported for the first time the special std namespace is created, together with its two keys total and sub. The std namespace is part of all transactions, in parallel to any namespace set in VCL (accounting.set_namespace() can not be used with std as argument and any such calls will be ignored). One of the two keys in std (total or sub) will always be incremented for all requests regardless of whether you set a specific namespace for the request or not. The total key is updated for top level requests and the sub key is updated for sub requests.

Note: It is not possible to set custom keys in the std namespace. Custom keys can only be set in namespaces you have explicitly created.

In order to set a namespace for a request, you must first create the namespace in sub vcl_init. Each namespace will have the total and sub keys included automatically and space for 100 custom keys by default. If you want to be able to define more keys, you can set the max number of keys for the namespace using the optional max_keys parameter when calling create_namespace().

The namespace and key(s) set for a request will persist across any actions within the request handling including labels, restarts and retries. There are however two exceptions to this. Any action taken in backend handling will not transfer to the client handling. I.e setting namespace and/or key(s) in a backend method will not apply for the client side. And secondly, if a request triggers a sub request, the sub request will inherit the parents namespace, but no other accounting information.

You can limit the scope when setting the namespace to only be set for client or backend by using the optional scope argument.

When calling add_key(), the request will be “tagged” to do accounting to that key. If that key does not yet exist in the set namespace, it will be implicitly created. Once the request is completed, the statistics for that request will be accounted to the total/sub keys for the set namespace and any keys that the request has been tagged with.

By default, if the current request is a sub request generated by Varnish, any call to add_key() will be ignored. If you want your key to track these requests as well you can set the optional parameter sub to INCLUDE when calling add_key(). Be aware that the data included in the top request from esi subs then will be accounted for twice for that key.

Each individual key has 50 predefined counters. These counters will appear in varnishstat on the form ACCG.<namespace>.<key>.<stat_name> where ACCG is short for accounting.

ACCG.<namespace>.<key>.client_req_count
ACCG.<namespace>.<key>.client_req_hdrbytes
ACCG.<namespace>.<key>.client_req_bodybytes
ACCG.<namespace>.<key>.client_resp_hdrbytes
ACCG.<namespace>.<key>.client_resp_bodybytes
ACCG.<namespace>.<key>.client_hit_count
ACCG.<namespace>.<key>.client_hit_req_hdrbytes
ACCG.<namespace>.<key>.client_hit_req_bodybytes
ACCG.<namespace>.<key>.client_hit_resp_hdrbytes
ACCG.<namespace>.<key>.client_hit_resp_bodybytes
ACCG.<namespace>.<key>.client_miss_count
ACCG.<namespace>.<key>.client_miss_req_hdrbytes
ACCG.<namespace>.<key>.client_miss_req_bodybytes
ACCG.<namespace>.<key>.client_miss_resp_hdrbytes
ACCG.<namespace>.<key>.client_miss_resp_bodybytes
ACCG.<namespace>.<key>.client_pass_count
ACCG.<namespace>.<key>.client_pass_req_hdrbytes
ACCG.<namespace>.<key>.client_pass_req_bodybytes
ACCG.<namespace>.<key>.client_pass_resp_hdrbytes
ACCG.<namespace>.<key>.client_pass_resp_bodybytes
ACCG.<namespace>.<key>.client_synth_count
ACCG.<namespace>.<key>.client_synth_req_hdrbytes
ACCG.<namespace>.<key>.client_synth_req_bodybytes
ACCG.<namespace>.<key>.client_synth_resp_hdrbytes
ACCG.<namespace>.<key>.client_synth_resp_bodybytes
ACCG.<namespace>.<key>.client_pipe_count
ACCG.<namespace>.<key>.client_pipe_req_hdrbytes
ACCG.<namespace>.<key>.client_pipe_req_bodybytes
ACCG.<namespace>.<key>.client_pipe_resp_hdrbytes
ACCG.<namespace>.<key>.client_pipe_resp_bodybytes
ACCG.<namespace>.<key>.client_200_count
ACCG.<namespace>.<key>.client_304_count
ACCG.<namespace>.<key>.client_404_count
ACCG.<namespace>.<key>.client_503_count
ACCG.<namespace>.<key>.client_2xx_count
ACCG.<namespace>.<key>.client_3xx_count
ACCG.<namespace>.<key>.client_4xx_count
ACCG.<namespace>.<key>.client_5xx_count
ACCG.<namespace>.<key>.backend_req_count
ACCG.<namespace>.<key>.backend_req_hdrbytes
ACCG.<namespace>.<key>.backend_req_bodybytes
ACCG.<namespace>.<key>.backend_resp_hdrbytes
ACCG.<namespace>.<key>.backend_resp_bodybytes
ACCG.<namespace>.<key>.backend_200_count
ACCG.<namespace>.<key>.backend_304_count
ACCG.<namespace>.<key>.backend_404_count
ACCG.<namespace>.<key>.backend_503_count
ACCG.<namespace>.<key>.backend_2xx_count
ACCG.<namespace>.<key>.backend_3xx_count
ACCG.<namespace>.<key>.backend_4xx_count
ACCG.<namespace>.<key>.backend_5xx_count

In addition to the stat counters there is also a group for tracking errors in accounting called ACCG_DIAG it defines 5 counters for common errors:

ACCG_DIAG.set_key_failure
ACCG_DIAG.out_of_key_slots
ACCG_DIAG.key_without_namespace
ACCG_DIAG.namespace_already_set
ACCG_DIAG.namespace_undefined
ACCG_DIAG.create_namespace_failure

Example

This simple Example VCL shows a way to leverage namespaces to separate statistics for different tenants while using keys on the tenant side for creating more fine grained statistics

main vcl:

import accounting;

sub vcl_init {
  accounting.create_namespace("tenant1");
  accounting.create_namespace("tenant2");
  accounting.create_namespace("generic");
}

sub vcl_recv {
  if (req.http.host ~ "tenant1.com$") {
    accounting.set_namespace("tenant1");
    return(vcl(tenant1));
  } else if (req.http.host ~ "tenant2.com$") {
    accounting.set_namespace("tenant2");
    return(vcl(tenant2));
  } else {
    accounting.set_namespace("generic");
  }
}

tenant1 vcl:

import accounting;

sub vcl_backend_response {
  if (beresp.http.Content-Type ~ "^image") {
    accounting.add_keys("images");
  } else {
    accounting.add_keys("generic");
  }
}

API

create_namespace

VOID create_namespace(STRING namespace, INT max_keys = 100)

Creates a namespace to be used for accounting purposes. A namespace name can contain any character, but no more than 100 characters in total. Can only be called from sub vcl_init.

NOTE: For each unique key that is defined in the namespace there is a certain amount of extra work added to the request, defining too many keys in a namespace will eventually have a noticeable effect on performance.

Use the optional argument max_keys to set the limit for the number of keys that can be defined in the namespace (default = 100).

Arguments:

  • namespace accepts type STRING

  • max_keys accepts type INT with a default value of 100 optional

Type: Function

Returns: None

set_namespace

VOID set_namespace(STRING namespace, ENUM {LOCAL, FULL} scope = FULL)

Set the accounting namespace for the request. Only namespaces created during sub vcl_init are valid. Use the optional argument scope to control the scope for which the namespace shall be set. If you set the scope to LOCAL the namespace will only be set for the current scope (client/backend).

The default scope FULL will force the backend request to inherit the namespace (and keys) from the client request. Setting the namespace from a backend method will not affect the client side of the request. The namespace can only be set once for the set scope, subsequent calls for the same request will be ignored.

Note: Sub requests inherit their namespace by the top request that triggered them, they will also inherit the namespace scope of the parent request.

Arguments:

  • namespace accepts type STRING

  • scope is an ENUM that accepts values of LOCAL, and FULL with a default value of FULL optional

Type: Function

Returns: None

add_keys

INT add_keys(STRING keys, STRING sep = ", ", ENUM {SKIP, INCLUDE} sub = SKIP)

Add accounting key(s) specified in string keys, separated by sep to the transaction, if no separator is specified the default “,” will be assumed. Returns the number of key(s) successfully set. A key can contain any character, but not be more than 100 characters long. If the max number of keys for the namespace has already been defined, setting the key(s) will fail. If no namespace has been set for the transaction setting the key(s) will fail. Note that for the client side of the transaction any calls to add_keys() will be ignored if the request is a sub request generated by Varnish due to esi, edgestash or slicer.

For key(s) to be also set for sub requests you can set the optional parameter sub to INCLUDE. Be aware that if you do set the same key(s) for both top requests and subs, any data inserted in the top request from the sub will then be double accounted to the key(s).

Note: Since sub requests do not pass any VCL executed before a jump to a label, and (in difference to namespaces) keys are not inherited, there is no way to enforce a key to a sub outside of the active VCL.

Arguments:

  • keys accepts type STRING

  • sep accepts type STRING with a default value of , optional

  • sub is an ENUM that accepts values of SKIP, and INCLUDE with a default value of SKIP optional

Type: Function

Returns: Int

Availability

The accounting VMOD is available in Varnish Cache Plus version 6.0.8r2 and later.