Search
Varnish Custom Statistics

vcs

SYNOPSIS

    vcs [-h] [-m period_len] [-b bucket_len] [-a port] [-z port] [-l acl]
        [-P file] [-u dir] [-F] [-g] [-V] [-Z] [-K file] [-d domain] [-H]

DESCRIPTION

The vcs utility consumes and aggregates log records from a set of vcs-agent(8) instances. The resulting data is made available in an API, that is queried over HTTP, presented in JSON format.

The records are aggregated based on keys specified via the vcs-agent(8) utility, and organized in time series buckets.

OPTIONS

The following options are available:

-h Display help. Displays a brief list of vcs’ options, along with default values.

-b seconds Shorthand for setting parameter bucket_len. See description below.

-m num_buckets Shorthand for setting parameter num_buckets. See description below.

-p param=value See the PARAMETERS section

-z port Listening port for vcs-agent(8) instances.

-a port Listening port for the HTTP interface.

-l acl Access control list for the HTTP interface. Specified as a comma-separated list of IP subnets, each prepended by a ‘+’ or ‘-’ sign. Plus means allow, minus means deny. For example, to allow only 192.168/16 subnet to connect, use -0.0.0.0/0,+192.168.0.0/16. Supported only for IPv4 sessions.

-O [host]:port Open a TCP connection to the specified endpoint and transmit finished buckets in JSON format for further handling/storage by 3rd party tools.

-P file Write the process’s PID to the specified file.

-F Run in foreground. Output will be written to stdout/stderr.

-u dir Root directory for vcs ui files.

-g n Output debug information. Defaults to 0.

-V Show version- and copyright information then exit.

-Z Display timestamps in UTC.

-K file Credentials file for HTTP authentication

-d domain htdigest HTTP authentication domain (or realm)

-H [LABEL:]stat_name:{0,a,b,c,... | A,STEP,END | G,BASE,RATIO,END} Specify histogram to be generated for statistic stat_name with optional LABEL. Ranges for histogram can be specified as a sequence of numbers, an arithmetic sequence or a geometric sequence. (For more details, see the “Use cases” section)

PARAMETERS

bucket_len Length of each individual bucket in the time series tracked.

num_buckets Number of buckets in a time series. The total duration of the tracked period will be the number of buckets multiplied by the bucket length (bucket_len).

hash_tbl_len Hash table length. Internally all keys are organized in a hash table. The number here should be roughly the same magnitude as the number of keys you are tracking.

json_out_backlog Maximum backlog for messages waiting to be transmitted to the JSON output endpoint. If this limit is reached, messages will be dropped.

HTTP INTERFACE

The API responds to requests for the following URLs:

Output key filtering

/key/<key> Retrieves stats for a single key. Key name must be URL encoded.

/match/<regex> Retrieves a list of keys matching the URL encoded regular expression. Accepts the query parameter verbose=1, which will display all stats collected for the keys matched.

/all Retrieves a list of all the keys we are currently tracking. Like /match, this also accepts verbose=1 for verbose output.

Top lists

For /match/<r> and /all, vcs can produce sorted lists of keys. The command (which is appended to the URL) defines which sorting criteria should be used:

/all/top Sort based on number of requests.

/all/top_ttfb Sort based on the ttfb_miss field.

/all/top_size Sort based on the n_bodybytes field.

/all/top_miss Sort based on the n_miss field.

/all/top_respbytes Sort based on number of bytes transmitted to clients.

/all/top_reqbytes Sort based on number of bytes received from clients.

/all/top_berespbytes Sort based on number of bytes fetched from backends.

/all/top_bereqbytes Sort based on number of bytes transmitted to backends.

/all/top_restarts Sort based on the avg_restarts field.

/all/top_5xx, /top_4xx, .., /top_1xx Sort based on number of HTTP response codes returned to clients, in buckets for 1xx, 2xx, 3xx, etc.

/all/top_uniq Sort based on the n_req_uniq field.

Note that in the above, you can substitute /all with /match/<r>, to limit the listing to only apply to a specific set of keys.

Further, a /k parameter can be appended, which specifies the number of keys to include in the top list. If no k value is provided the top 10 will be displayed.

Summed window sorting

By default, the sorting is based on the data in the single latest window in the time series. To make it consider multiple windows, a query parameter b=<n> can be specified. Specifying all/top?b=5 will then result in a top list sorted by the combined number of requests over the 5 latest windows.

Format

The API produces JSON output for the data it maintains for each tracked key. When requesting stats for a key, the output is in the following format:

{
    "example.com": [
        {
            "timestamp": "2013-09-18T09:58:30",
            "n_req": 76,
            "n_req_uniq": "NaN",
            "n_miss": 1,
            "avg_restarts": 0.000000,
            "n_bodybytes": 10950,
            "ttfb_miss": 0.000440,
            "ttfb_hit": 0.000054,
            "resp_1xx": 0,
            "resp_2xx": 76,
            "resp_3xx": 0,
            "resp_4xx": 0,
            "resp_5xx": 0
	    ...
        },
        {
            "timestamp": "2013-09-18T09:58:00",
            "n_req": 84,
            "n_req_uniq": "NaN",
            "n_miss": 0,
            "avg_restarts": 0.000000,
            "n_bodybytes": 12264,
            "ttfb_miss": "NaN",
            "ttfb_hit": 0.000048,
            "resp_1xx": 0,
            "resp_2xx": 84,
            "resp_3xx": 0,
            "resp_4xx": 0,
            "resp_5xx": 0
	    ...
        },
	...
	]
   }

For the above example, the key is example.com and the vcs instance was configured with 10 second time buckets.

Each time bucket contains the following fields:

timestamp This is the timestamp for the start of the bucket’s period.

n_req The number of requests.

n_req_uniq The number of “unique” requests, if configured. This counter will increase once for each distinct value of “vcs-unique-id” encountered, configured in VCL. See GETTING LIVE STATISTICS FROM VARNISH WITH HLS/HDS for an example use case for this.

n_miss Number of backend requests (i.e. cache misses). Number of hits can be calculated as n_hit = n_req - n_miss.

avg_restarts The average number of VCL restarts triggered per request.

n_bodybytes The total number of bytes transferred for the response bodies.

ttfb_miss Average time to first byte for requests that ended up with a backend request.

ttfb_hit Average time to first byte for requests that were served directly from varnish cache.

resp_1xx - resp_5xx Counters for response status codes.

reqbytes Number of bytes received from clients.

respbytes Number of bytes transmitted to clients.

berespbytes Number of bytes received from backends.

bereqbytes Number of bytes transmitted to backends.

For top lists, the output is in the following format:

{
   "example.com": 327,
   "MISS": 168,
   "HIT": 159,
   "example.com/img.png": 37,
   ...
}

Status

The URL /status produces a JSON object containing a few simple counters:

{
	"uptime": 2133,
	"n_keys": 358,
	"n_trans": 483,
	"db_mem_usage": 1913328
}
  • uptime is the seconds elapsed since VCS was launched.
  • n_keys is the number of keys we are currently tracking.
  • n_trans is the number of transactions that have been processed by VCS.
  • db_mem_usage is the amount of memory consumed for storage, in bytes.

JSONP

The output can also be presented in JSONP format, with a JavaScript function call wrapped around it. This is done by adding the query parameter ?callback=myFunction to the URL. myFunction has to be a valid ascii JS identifier.

LIMITATIONS

ESI

Due to limitations in the current varnishapi, support for ESI requests are very limited. Until Varnish 4.0, we do not recommend using VCS along with ESI requests. In the case you are using ESI, wrap the key definitions inside of an if (req.esi_level == 0) block, e.g.:

  sub vcl_deliver {
    if (req.esi_level == 0) {
      std.log("vcs-key:" + req.http.host);
    }
  }

Summed window sorting and n_req_uniq

Using the summed window sorting param (b=<n> query param) is not available for the stat n_req_uniq.

EXAMPLES

Retrieve stats for key named example.com:

/key/example.com

Retrieve a list of the top 5 requested keys in the previous window:

/all/top/5

Retrieve a list of the top 5 requested keys, summed over the previous 3 windows, in JSONP format:

/all/top/5?b=3

For keys with names ending with .gif, retrieve a list of the top 10:

/match/(.*)%5C.gif$/top

Find a list of the top 50 slowest backend requests, in JSONP format:

/all/top_ttfb/50?callback=myfunc

HISTORY

This document was written by Dag Haavi Finstad daghf@varnish-software.com.

SEE ALSO

vcs-agent(8)

COPYRIGHT

  • Copyright (c) 2012-2021 Varnish Software AS