SearchVarnish Enterprise
The accounting vmod creates customizable statistics that can be used
for accounting or any other use cases where you want to create custom summary
statistics.
With vmod-accounting you can lock an incoming request to an accounting
namespace. Setting a namespace for the request will generate statistics for any
actions within the scope of the request (client or backend side). If you want to
further scope statistics, you can define keys within the namespace.
When accounting is imported for the first time the special std namespace is
created, together with its two keys total and sub. The std namespace is
part of all transactions, in parallel to any namespace set in VCL
(accounting.set_namespace() can not be used with std as argument and any
such calls will be ignored). One of the two keys in std (total or sub)
will always be incremented for all requests regardless of whether you set a
specific namespace for the request or not. The total key is updated for top
level requests and the sub key is updated for sub requests.
Note: It is not possible to set custom keys in the std namespace. Custom keys
can only be set in namespaces you have explicitly created.
In order to set a namespace for a request, you must first create the namespace
in sub vcl_init. Each namespace will have the total and sub keys included
automatically and space for 100 custom keys by default. If you want to be able
to define more keys, you can set the max number of keys for the namespace using
the optional max_keys parameter when calling create_namespace().
The namespace and key(s) set for a request will persist across any actions within the request handling including labels, restarts and retries. There are however two exceptions to this. Any action taken in backend handling will not transfer to the client handling. I.e setting namespace and/or key(s) in a backend method will not apply for the client side. And secondly, if a request triggers a sub request, the sub request will inherit the parents namespace, but no other accounting information.
You can limit the scope when setting the namespace to only be set for
client or backend by using the optional scope argument.
When calling add_key(), the request will be “tagged” to do accounting to that
key. If that key does not yet exist in the set namespace, it will be implicitly
created. Once the request is completed, the statistics for that request will be
accounted to the total/sub keys for the set namespace and any keys that the
request has been tagged with.
By default, if the current request is a sub request generated by Varnish, any
call to add_key() will be ignored. If you want your key to track these requests
as well you can set the optional parameter sub to INCLUDE when calling
add_key(). Be aware that the data included in the top request from esi subs
then will be accounted for twice for that key.
Each individual key has 50 predefined counters. These counters will appear in
varnishstat on the form ACCG.<namespace>.<key>.<stat_name> where ACCG is short
for accounting.
ACCG.<namespace>.<key>.client_req_count
ACCG.<namespace>.<key>.client_req_hdrbytes
ACCG.<namespace>.<key>.client_req_bodybytes
ACCG.<namespace>.<key>.client_resp_hdrbytes
ACCG.<namespace>.<key>.client_resp_bodybytes
ACCG.<namespace>.<key>.client_hit_count
ACCG.<namespace>.<key>.client_grace_hit_count
ACCG.<namespace>.<key>.client_hit_req_hdrbytes
ACCG.<namespace>.<key>.client_hit_req_bodybytes
ACCG.<namespace>.<key>.client_hit_resp_hdrbytes
ACCG.<namespace>.<key>.client_hit_resp_bodybytes
ACCG.<namespace>.<key>.client_miss_count
ACCG.<namespace>.<key>.client_miss_req_hdrbytes
ACCG.<namespace>.<key>.client_miss_req_bodybytes
ACCG.<namespace>.<key>.client_miss_resp_hdrbytes
ACCG.<namespace>.<key>.client_miss_resp_bodybytes
ACCG.<namespace>.<key>.client_pass_count
ACCG.<namespace>.<key>.client_pass_req_hdrbytes
ACCG.<namespace>.<key>.client_pass_req_bodybytes
ACCG.<namespace>.<key>.client_pass_resp_hdrbytes
ACCG.<namespace>.<key>.client_pass_resp_bodybytes
ACCG.<namespace>.<key>.client_synth_count
ACCG.<namespace>.<key>.client_synth_req_hdrbytes
ACCG.<namespace>.<key>.client_synth_req_bodybytes
ACCG.<namespace>.<key>.client_synth_resp_hdrbytes
ACCG.<namespace>.<key>.client_synth_resp_bodybytes
ACCG.<namespace>.<key>.client_pipe_count
ACCG.<namespace>.<key>.client_pipe_req_hdrbytes
ACCG.<namespace>.<key>.client_pipe_req_bodybytes
ACCG.<namespace>.<key>.client_pipe_resp_hdrbytes
ACCG.<namespace>.<key>.client_pipe_resp_bodybytes
ACCG.<namespace>.<key>.client_200_count
ACCG.<namespace>.<key>.client_304_count
ACCG.<namespace>.<key>.client_404_count
ACCG.<namespace>.<key>.client_503_count
ACCG.<namespace>.<key>.client_2xx_count
ACCG.<namespace>.<key>.client_3xx_count
ACCG.<namespace>.<key>.client_4xx_count
ACCG.<namespace>.<key>.client_5xx_count
ACCG.<namespace>.<key>.backend_req_count
ACCG.<namespace>.<key>.backend_req_hdrbytes
ACCG.<namespace>.<key>.backend_req_bodybytes
ACCG.<namespace>.<key>.backend_resp_hdrbytes
ACCG.<namespace>.<key>.backend_resp_bodybytes
ACCG.<namespace>.<key>.backend_200_count
ACCG.<namespace>.<key>.backend_304_count
ACCG.<namespace>.<key>.backend_404_count
ACCG.<namespace>.<key>.backend_503_count
ACCG.<namespace>.<key>.backend_2xx_count
ACCG.<namespace>.<key>.backend_3xx_count
ACCG.<namespace>.<key>.backend_4xx_count
ACCG.<namespace>.<key>.backend_5xx_count
In addition to the stat counters there is also a group for tracking errors in accounting called ACCG_DIAG it defines 5 counters for common errors:
ACCG_DIAG.set_key_failure
ACCG_DIAG.out_of_key_slots
ACCG_DIAG.key_without_namespace
ACCG_DIAG.namespace_already_set
ACCG_DIAG.namespace_undefined
ACCG_DIAG.create_namespace_failure
When a client or backend task is started, there may be no accounting left by the time it ends. This can happen if such a task inherited an accounting namespace, but all VCLs referencing vmod_accounting were discarded. In this case an error message is logged:
accounting: Not updating counters
This can also happen when a task is interrupted before normal completion.
This simple Example VCL shows a way to leverage namespaces to separate statistics for different tenants while using keys on the tenant side for creating more fine grained statistics
main vcl:
import accounting;
sub vcl_init {
accounting.create_namespace("tenant1");
accounting.create_namespace("tenant2");
accounting.create_namespace("generic");
}
sub vcl_recv {
if (req.http.host ~ "tenant1.com$") {
accounting.set_namespace("tenant1");
return(vcl(tenant1));
} else if (req.http.host ~ "tenant2.com$") {
accounting.set_namespace("tenant2");
return(vcl(tenant2));
} else {
accounting.set_namespace("generic");
}
}
tenant1 vcl:
import accounting;
sub vcl_backend_response {
if (beresp.http.Content-Type ~ "^image") {
accounting.add_keys("images");
} else {
accounting.add_keys("generic");
}
}
VOID create_namespace(STRING namespace, INT max_keys = 100)
Creates a namespace to be used for accounting purposes. A namespace name
can contain any character, but no more than 100 characters in total. Can
only be called from sub vcl_init.
NOTE: For each unique key that is defined in the namespace there is a certain amount of extra work added to the request, defining too many keys in a namespace will eventually have a noticeable effect on performance.
Use the optional argument max_keys to set the limit for the number of
keys that can be defined in the namespace (default = 100).
Arguments:
namespace accepts type STRING
max_keys accepts type INT with a default value of 100 optional
Type: Function
Returns: None
Restricted to: vcl_init
VOID set_namespace(STRING namespace, ENUM {LOCAL, FULL} scope = FULL)
Set the accounting namespace for the request. Only namespaces created
during sub vcl_init are valid. Use the optional argument scope to
control the scope for which the namespace shall be set. If you set the
scope to LOCAL the namespace will only be set for the current scope
(client/backend).
The default scope FULL will force the backend request to inherit the namespace (and keys) from the client request. Setting the namespace from a backend method will not affect the client side of the request. The namespace can only be set once for the set scope, subsequent calls for the same request will be ignored.
Note: Sub requests inherit their namespace by the top request that triggered them, they will also inherit the namespace scope of the parent request.
Arguments:
namespace accepts type STRING
scope is an ENUM that accepts values of LOCAL, and FULL with a default value of FULL optional
Type: Function
Returns: None
Restricted to: client, backend
STRING get_namespace()
Get the currently set namespace. Returns NULL if no namespace is set.
Arguments: None
Type: Function
Returns: String
INT add_keys(STRING keys, STRING sep = ", ", ENUM {SKIP, INCLUDE} sub = SKIP)
Add accounting key(s) specified in string keys, separated by sep to
the transaction, if no separator is specified the default “,” will be
assumed. Returns the number of key(s) successfully set. A key can
contain any character, but not be more than 100 characters long. If the
max number of keys for the namespace has already been defined, setting
the key(s) will fail. If no namespace has been set for the transaction
setting the key(s) will fail. Note that for the client side of the
transaction any calls to add_keys() will be ignored if the request is a
sub request generated by Varnish due to esi, edgestash or slicer.
For key(s) to be also set for sub requests you can set the
optional parameter sub to INCLUDE. Be aware that if you do set the
same key(s) for both top requests and subs, any data inserted in the
top request from the sub will then be double accounted to the key(s).
Note: Since sub requests do not pass any VCL executed before a jump to a label, and (in difference to namespaces) keys are not inherited, there is no way to enforce a key to a sub outside of the active VCL.
Arguments:
keys accepts type STRING
sep accepts type STRING with a default value of , optional
sub is an ENUM that accepts values of SKIP, and INCLUDE with a default value of SKIP optional
Type: Function
Returns: Int
Restricted to: client, backend
The accounting VMOD is available in Varnish Enterprise version 6.0.8r2
and later.