SearchVarnish Custom Statistics
vcs [-h] [-m period_len] [-b bucket_len] [-a port] [-z port] [-l acl]
[-P file] [-u dir] [-F] [-g] [-V] [-Z] [-K file] [-d domain] [-H]
The vcs utility consumes and aggregates log records from a set of
vcs-agent(8) instances. The resulting data is made available in an
API, that is queried over HTTP, presented in JSON format.
The records are aggregated based on keys specified via the vcs-agent(8) utility, and organized in time series buckets.
The following options are available:
-h
Display help. Displays a brief list of vcs’ options, along with default values.
-b seconds
Shorthand for setting parameter bucket_len. See description below.
-m num_buckets
Shorthand for setting parameter num_buckets. See description below.
-p param=value See the PARAMETERS section
-z port
Listening port for vcs-agent(8) instances.
-a port
Listening port for the HTTP interface.
-l acl
Access control list for the HTTP interface. Specified as a
comma-separated list of IP subnets, each prepended by a ‘+’ or
‘-’ sign. Plus means allow, minus means deny. For example, to
allow only 192.168/16 subnet to connect, use
-0.0.0.0/0,+192.168.0.0/16. Supported only for IPv4 sessions.
-O [host]:port
Open a TCP connection to the specified endpoint and transmit
finished buckets in JSON format for further handling/storage
by 3rd party tools.
-P file
Write the process’s PID to the specified file.
-F
Run in foreground. Output will be written to stdout/stderr.
-u dir
Root directory for vcs ui files.
-g n
Output debug information. Defaults to 0.
-V
Show version- and copyright information then exit.
-Z
Display timestamps in UTC.
-K file
Credentials file for HTTP authentication
-d domain
htdigest HTTP authentication domain (or realm)
-H [LABEL:]stat_name:{0,a,b,c,... | A,STEP,END | G,BASE,RATIO,END}
Specify histogram to be generated for statistic stat_name
with optional LABEL. Ranges for histogram can be specified
as a sequence of numbers, an arithmetic sequence or a
geometric sequence. (For more details, see the “Use cases” section)
bucket_len
Length of each individual bucket in the time series tracked.
num_buckets
Number of buckets in a time series. The total duration of the
tracked period will be the number of buckets multiplied by the
bucket length (bucket_len).
hash_tbl_len
Hash table length. Internally all keys are organized in a hash
table. The number here should be roughly the same magnitude as
the number of keys you are tracking.
json_out_backlog
Maximum backlog for messages waiting to be transmitted to the
JSON output endpoint. If this limit is reached, messages
will be dropped.
The API responds to requests for the following URLs:
/key/<key> Retrieves stats for a single key. Key name must be URL encoded.
/match/<regex> Retrieves a list of keys matching the URL encoded regular expression. Accepts the query parameter verbose=1, which will display all stats collected for the keys matched.
/all Retrieves a list of all the keys we are currently tracking. Like /match, this also accepts verbose=1 for verbose output.
For /match/<r> and /all, vcs can produce sorted lists of keys. The
command (which is appended to the URL) defines which sorting criteria
should be used:
/all/top
Sort based on number of requests.
/all/top_ttfb
Sort based on the ttfb_miss field.
/all/top_size
Sort based on the n_bodybytes field.
/all/top_miss
Sort based on the n_miss field.
/all/top_respbytes
Sort based on number of bytes transmitted to clients.
/all/top_reqbytes
Sort based on number of bytes received from clients.
/all/top_berespbytes
Sort based on number of bytes fetched from backends.
/all/top_bereqbytes
Sort based on number of bytes transmitted to backends.
/all/top_restarts
Sort based on the avg_restarts field.
/all/top_5xx, /top_4xx, .., /top_1xx
Sort based on number of HTTP response codes returned to clients, in buckets for 1xx, 2xx, 3xx, etc.
/all/top_uniq
Sort based on the n_req_uniq field.
Note that in the above, you can substitute /all with /match/<r>, to
limit the listing to only apply to a specific set of keys.
Further, a /k parameter can be appended, which specifies the number
of keys to include in the top list. If no k value is provided the
top 10 will be displayed.
By default, the sorting is based on the data in the single latest
window in the time series. To make it consider multiple windows, a
query parameter b=<n> can be specified. Specifying all/top?b=5 will
then result in a top list sorted by the combined number of requests
over the 5 latest windows.
The API produces JSON output for the data it maintains for each tracked key. When requesting stats for a key, the output is in the following format:
{
"example.com": [
{
"timestamp": "2013-09-18T09:58:30",
"n_req": 76,
"n_req_uniq": "NaN",
"n_miss": 1,
"avg_restarts": 0.000000,
"n_bodybytes": 10950,
"ttfb_miss": 0.000440,
"ttfb_hit": 0.000054,
"resp_1xx": 0,
"resp_2xx": 76,
"resp_3xx": 0,
"resp_4xx": 0,
"resp_5xx": 0
...
},
{
"timestamp": "2013-09-18T09:58:00",
"n_req": 84,
"n_req_uniq": "NaN",
"n_miss": 0,
"avg_restarts": 0.000000,
"n_bodybytes": 12264,
"ttfb_miss": "NaN",
"ttfb_hit": 0.000048,
"resp_1xx": 0,
"resp_2xx": 84,
"resp_3xx": 0,
"resp_4xx": 0,
"resp_5xx": 0
...
},
...
]
}
For the above example, the key is example.com and the vcs
instance was configured with 10 second time buckets.
Each time bucket contains the following fields:
timestamp
This is the timestamp for the start of the bucket’s period.
n_req
The number of requests.
n_req_uniq
The number of “unique” requests, if configured. This
counter will increase once for each distinct value of
“vcs-unique-id” encountered, configured in VCL. See
GETTING LIVE STATISTICS FROM VARNISH WITH HLS/HDS for an example use case for this.
n_miss
Number of backend requests (i.e. cache misses). Number of hits
can be calculated as n_hit = n_req - n_miss.
avg_restarts
The average number of VCL restarts triggered per request.
n_bodybytes
The total number of bytes transferred for the response bodies.
ttfb_miss
Average time to first byte for requests that ended up with a backend
request.
ttfb_hit
Average time to first byte for requests that were served
directly from varnish cache.
resp_1xx - resp_5xx
Counters for response status codes.
reqbytes
Number of bytes received from clients.
respbytes
Number of bytes transmitted to clients.
berespbytes
Number of bytes received from backends.
bereqbytes
Number of bytes transmitted to backends.
For top lists, the output is in the following format:
{
"example.com": 327,
"MISS": 168,
"HIT": 159,
"example.com/img.png": 37,
...
}
The URL /status produces a JSON object containing a few simple counters:
{
"uptime": 2133,
"n_keys": 358,
"n_trans": 483,
"db_mem_usage": 1913328
}
uptime is the seconds elapsed since VCS was launched.n_keys is the number of keys we are currently tracking.n_trans is the number of transactions that have been processed by VCS.db_mem_usage is the amount of memory consumed for storage, in bytes.The output can also be presented in JSONP format, with a JavaScript
function call wrapped around it. This is done by adding the query
parameter ?callback=myFunction to the URL. myFunction has to be a
valid ascii JS identifier.
Due to limitations in the current varnishapi, support for ESI requests
are very limited. Until Varnish 4.0, we do not recommend using VCS
along with ESI requests. In the case you are using ESI, wrap the key
definitions inside of an if (req.esi_level == 0) block, e.g.:
sub vcl_deliver {
if (req.esi_level == 0) {
std.log("vcs-key:" + req.http.host);
}
}
n_req_uniqUsing the summed window sorting param (b=<n> query param) is not
available for the stat n_req_uniq.
Retrieve stats for key named example.com:
/key/example.com
Retrieve a list of the top 5 requested keys in the previous window:
/all/top/5
Retrieve a list of the top 5 requested keys, summed over the previous 3 windows, in JSONP format:
/all/top/5?b=3
For keys with names ending with .gif, retrieve a list of the top 10:
/match/(.*)%5C.gif$/top
Find a list of the top 50 slowest backend requests, in JSONP format:
/all/top_ttfb/50?callback=myfunc
This document was written by Dag Haavi Finstad daghf@varnish-software.com.
vcs-agent(8)