SearchVarnish Controller
Agents are required to have unique names in the Varnish Controller system. The names must be alphanumeric but can also include a dash (-).
They can have the same name in version 5.0+, if different private tokens are used.
Each Varnish should have a corresponding agent running if it should be included in the Varnish Controller system. Running multiple agents toward the same Varnish is not supported and can result in unpredicted behavior.
Restarting an agent will not affect the Varnish instance as long as the Varnish instance has not been manually changed with regard to loaded VCL.
Agents will start by waiting for connection towards NATS, once connected it will continue.
# Ubuntu/Debian
sudo apt install varnish-controller-agent
# RPM-based
sudo dnf install varnish-controller-agent
The agent will communicate with Varnish using several interfaces:
-varnish-admin-port intVARNISH_CONTROLLER_VARNISH_ADMIN_PORT=int
Varnish administration port, must match the -T parameter of Varnish. This is used by the agent to load VCL. The default is 6082, since varnish is by default started with -T localhost:6082.-varnish-host stringVARNISH_CONTROLLER_VARNISH_HOST=string
Host and port where varnish is located. Format should be example.com:80.
The default is 127.0.0.1:6081 and must match one of the -a parameters of Varnish. The default is 127.0.0.1:6081, since varnish by default is started with -a :6081.-varnish-invalidation-host stringVARNISH_CONTROLLER_VARNISH_INVALIDATION_HOST=string
Host and port where varnish is located. Format should be the same as varnish-host, and must also match one of the -a parameters of Varnish. This is the connection used by the agent to do invalidations. The default is the same as varnish-host, so if those are the same, only varnish-host need to be specified.
If varnish is only listening on TLS, please look at the varnish-invalidation-tls-parameter below.-varnish-secret pathVARNISH_CONTROLLER_VARNISH_SECRET=path
Varnish secret file for varnishadm. Varnish-agent must have permissions to read this file, and this must match the -S parameter of Varnish. The default is /etc/varnish/secret, which is also the default for Varnish.Agents must be able to communicate with Varnish on the Varnish administration port. The default
port is 6082. That means that Varnish must be started with -T :6082 for example. The agent must also be able to
read the Varnish secret file (defaults to /etc/varnish/secret). The agent can be configured with the Varnish administration port
using the flag -varnish-admin-port(VARNISH_CONTROLLER_VARNISH_ADMIN_PORT). The host used for the Varnish administration interface
is the same as configured by -varnish-host (VARNISH_CONTROLLER_VARNISH_HOST).
Since the agent is responsible for reading/writing VCL files to disk and loading them into Varnish, Varnish must be able to read the
files as well. This is not a problem when running on the same server, as long as permissions are set correctly. But when running
in a container environment, they both need to be able to read the same shared directory.
This is default /var/lib/varnish, but can be changed if needed, see Varnish Name.
If Varnish is listening on a different host/port, this must be reflected in the agents configuration variables -varnish-invalidation-host(VARNISH_CONTROLLER_VARNISH_INVALIDATION_HOST)
and -varnish-host (VARNISH_CONTROLLER_VARNISH_HOST). Otherwise, the invalidation will not work. If no varnish-invalidation-host is specified, the agent will use
varnish-host as invalidation host and port.
See the containers for more information on running in containers.
NOTE: When a Varnish instance is operated by the Varnish Controller Agent, the agent will remove any loaded VCL/Label that has not been loaded through the Varnish Controller.
The agent will use varnishstat to fetch statistics from Varnish.
To be able to fetch statistics, the varnishstat binary is required on the same server as the agent and it will need to have read permissions to the varnish shared memory area (VSM).
This is by default /var/lib/varnish, but can be changed with the -n parameter of varnish (see Varnish Name).
Then varnishstat also needs to know which Varnish VSM (Varnish Shared Memory) is used.
This can be specified with the -varnish-name parameter, that will be the same as the -n argument to varnishd.
Example:
# Varnish is started with the argument -n /tmp/myvarnish
varnishd -a 0.0.0.0:8091 -S /tmp/secret -T localhost:6082 -f '' -n /tmp/myvarnish
# The agent then needs to know the VSM directory /tmp/myvarnish
varnish-controller-agent -varnish-name /tmp/myvarnish
To identify the agents in the Varnish Controller, the agent name is used. Agents are required to have unique names in the Varnish Controller system. The names must be alphanumeric but can also include a dash (-). Set the agent name as an argument -agent-name agent01 or environment variable VARNISH_CONTROLLER_AGENT_NAME=agent01.
Note that they can have the same name in version 5.0+, if different private tokens are used. See private/shared tokens chapter
An agent can be started in read-only mode using the -read-only flag. This makes the agent not usable in Varnish Controller with regard to deployments, but it
will read information from Varnish and present it to Varnish Controller. It will read loaded VCLs, Varnish version, etc. and present via the REST API.
The agent must be restarted without the -read-only flag in order to be used for deployment via Varnish Controller.
The agent generates a file called cmds.cli every time a new configuration is applied to the agent. This file can be passed to varnishd with the flag -I /var/lib/varnish-controller/varnish-controller-agent/<agent_name>/cmds.cli.
The file contains all the commands required to load the last configuration into Varnish when Varnish starts.
When the agent starts after this has been loaded, any changes that the agent retrieves from Brainz will be applied. If no VCL changes has been made, then nothing will be redeployed.
The agent uses the default system temporary directory to test compiling the VCL files before deploying them. This requires execution permission on the directory. If the temporary directory,
such as /tmp, is mounted with noexec this will not work. Hence, we recommend to pointing out a different temporary directory for the agent via the environment variable TMPDIR. This can be
added to the systemd service file.
Example:
sudo systemctl edit varnish-controller-agent
# Add the following
[Service]
Environment="TMPDIR=/var/lib/varnish-controller/varnish-controller-agent/<agentname>/"
NOTE: from version 5.0 of Varnish Controller. The agent will use the agent base-dir as temporary directory. This means that the above information is not
applicable for version higher or equal to 5.0 release of Varnish Controller.
From version 6.2.0 of the controller, the agents will create backups of its last deployments.
Number of backups can be changed with the configuration parameter -max-backups or via configurable components. The default is last 20 deployments.
A backup is created as soon as anything changes, such as the root VCL in a shared deployment, or a new VCL is loaded or removed etc.
The backups are stored in the base-dir/backups/ directory. Each backup is stored within a directory named with the current timestamp. The directories contains
all files loaded for each VCLGroup, the temp VCL, the root VCL, cmds.cli file and a script to load the files into Varnish.
Note that when restoring from a backup, the agent must be stopped or else it will override the loaded configuration if it differs from the backup.
Example dir of a backup:
/agent1/backup/20240613121756
/agent1/backup/20240613121756/restore.sh
/agent1/backup/20240613121756/cmds.cli
/agent1/backup/20240613121756/root.vcl
/agent1/backup/20240613121756/temp_root
/agent1/backup/20240613121756/1
/agent1/backup/20240613121756/1/main1.vcl
The script restore.sh is a helper script that will load the files in the directory into Varnish via the cmds.cli file. The cmds.cli file contains all the commands that will be executed via Varnish CLI.
There are some configuration parameters that could be tuned to change how the agent operates. These might not be self-explanatory and are described below.
Note: Do not change these parameters if you are not sure about the implications.
-load-from-disk <bool>VARNISH_CONTROLLER_LOAD_FROM_DISK=<bool>Load previous deployments (if any) from disk, even if NATS or Brainz is not reachable. The agent saves manifest files containing the deployments (VCLs) to disk. These are then loaded into the agent and Varnish if this option is turned on (by default true).
-default-vcl stringVARNISH_CONTROLLER_DEFAULT_VCL=<string>The default VCL is the VCL that will be loaded into Varnish if no other VCL has been loaded in the system.
This VCL can be replaced with any other VCL and can be an error page of some kind. Default is a synth 503.
The default VCL can be viewed:
varnish-controller-agent -default-vcl-show
-hb-rate <duration>VARNISH_CONTROLLER_HB_RATE=<duration>The heartbeat rate is the rate of heartbeats being sent by the agent to brainz to tell brainz of its presence and state. The lower heartbeat the faster response time for changes. Note that brainz might require some extra configuration if this value is changed to identify an agent that is down based on missed heartbeats. (default 3s)
-nats-server stringVARNISH_CONTROLLER_NATS_SERVER=<string>
The agent needs to communicate with brainz via NATS bus. The NATS server connection string is in the format of host:port or with username/password user:pass@host:port. (default “localhost:4222”)-nats-reconnect-time <duration>VARNISH_CONTROLLER_NATS_RECONNECT_TIME=<duration>Time between each reconnect to NATS server upon lost connection. (default 5s)
-nats-timeout <duration>VARNISH_CONTROLLER_NATS_TIMEOUT=<duration>Timeout for sending a message on the NATS message-bus. (default 10s)
-root-vcl stringVARNISH_CONTROLLER_ROOT_VCL=<string>This should usually not be modified. The default is a template used to handle multiple domains. This VCL is only used when running shared/cloud deployments with shared Varnish instances between multiple domains.
The default root VCL can be viewed:
varnish-controller-agent -root-vcl-show
-tags tag1,tag2VARNISH_CONTROLLER_TAGS=<string>,<string>Starts the agent with pre-defined tags that will be marked static in the database. If someone removes a static tag it will be automatically added again if the agent has it defined at startup.
If multiple agents are started with the same tag names, only one instance of the tag will be created. Hence, it is safe to start multiple agents with same tag names.
-stats-filter stringVARNISH_CONTROLLER_STATS_FILTER=<string>This is a comma-separated string of statistics that will be sampled by the agent. These are the same names as the output from varnishstat -j.
Anything other than the specified statistics will be discarded. Wildcard (*) usage is supported.
Example configuration:
./varnish-controller-agent -stats-filter MAIN.client_req,MAIN.cache_hit,ACCG.*.total.client_*
-agent-stats-filter stringVARNISH_CONTROLLER_AGENT_STATS_FILTER=<string>From version 6.4.0 please use the -stats-filter configuration. This is a comma-separated string of statistics that will be sampled by the agent. These are the same names as the output from varnishstat -j.
Anything other than the specified statistics will be discarded.
Example configuration:
./varnish-controller-agent -agent-stats-filter MAIN.client_req,MAIN.cache_hit
-accounting boolVARNISH_CONTROLLER_ACCOUNTING=<bool>This can be explicitly set to allow/deny the agent to sample accounting statistics.
Example configuration:
./varnish-controller-agent -accounting true
-accounting-stats-filter stringVARNISH_CONTROLLER_ACCOUNTING_STATS_FILTER=<string>From version 6.4.0 please use the -stats-filter configuration. This is a comma-separated string of accounting statistics that will be sampled by the agent.
Example configuration:
./varnish-controller-agent -accounting-stats-filter is_healthy,client_hit_count,client_miss_count
-accounting-stats-keys stringVARNISH_CONTROLLER_ACCOUNTING_STATS_KEYS=<string>From version 6.4.0 please use the -stats-filter configuration. This is a comma-separated string of accounting statistics keys that will be sampled by the agent.
Example configuration:
./varnish-controller-agent -accounting-stats-keys total,sub
-vclgroup-stats-filter stringVARNISH_CONTROLLER_VCLGROUP_STATS_FILTER=<string>From version 6.4.0 please use the -stats-filter configuration. This is almost the same as -agent-stats-filter except that this filters out statistics for VCLGroups (loaded VCLs). It can only
filter out parts of the VCL that are presented by varnishstat for VCL backends. The filter names take the last part of the statistics name
after the loaded VCL name.
As an example the varnishstat output VBE.vg1_129621b78f1ea3696eb116cbae3abc85bf89ef4ae0d6b642b873c9c978537117_1.req shows
both VBE and the VCL name (which will be different depending on both content and reloads). Hence, we only specify the last part as filter. In this example req as
the statistics that we want to gather.
Example configuration:
./varnish-controller-agent -vclgroup-stats-filter conn,req
(default “req,conn,unhealthy,is_healthy,bereq_hdrbytes,bereq_bodybytes,beresp_hdrbytes,beresp_bodybytes”)
-ipv4 string and -ipv6 stringVARNISH_CONTROLLER_IPV4=<string> and VARNISH_CONTROLLER_IPV6=<string>The IPv4 and IPv6 options are used by routers in DNS routing towards the agents. These IP addresses will be specified in the DNS records if the routers act as DNS backends to PowerDNS. Meaning that a domain lookup is made and the agent is healthy, these IP addresses will be returned back in response to the client DNS request.
-base-url stringVARNISH_CONTROLLER_BASE_URL=<string>The base URL is the full URL to the agent’s Varnish server. This is used when routers are used for traffic routing towards the Varnish servers.
The routers will forward requests in the form of <base-url>/<domain>/<path>. The base URL should be a valid URL such as https://example.com.
-latitude float and -longitude floatVARNISH_CONTROLLER_LATITUDE=<float> and VARNISH_CONTROLLER_LONGITUDE=<float>The latitude and longitude is the geographical position of the Varnish server. This is used for calculating distance between client IP and the Varnish servers when performing traffic routing via the router.
-nics stringVARNISH_CONTROLLER_UTIL_NICS=<string>This is a comma separated list of Network Interface Card (NIC) names on the system. The given NICs will be used to calculate bandwidth
that will be sent as utilization to all traffic routers. If multiple NICs are specified, the bandwidth sent as utilization will be an
additive value of all the NICs bandwidth. TxBytes are only included in the bandwidth, not RxBytes.
In order for this to work, the agent must have access to the same NICs as the Varnish server uses.
-max-mbps-hard floatVARNISH_CONTROLLER_MAX_MBPS_HARD=<float>This is used for request routing when traffic routers are used. This is the maximum bandwidth that the server can deliver before considered over utilized and not taken into account for traffic routing by the router’s built-in rules. Plugins see servers above this limit and are responsible for applying this limit as per their policies, hard or otherwise.
Note that since only TxBytes are sampled (see -nics), this should be taken
into consideration when specifying an appropriate Max Mbps Hard waterlevel.
-stats-interval <duration>VARNISH_CONTROLLER_STATS_INTERVAL=<duration>Specify how often Varnish statistics should be gathered from the Varnish instance. (default 1m0s)
-varnish-invalidation-tls <bool>VARNISH_CONTROLLER_VARNISH_INVALIDATION_TLS=<bool>When set to true (default false), this makes the agent send invalidation requests to varnish using HTTPS instead of HTTP. If verification of the
TLS certificate should be skipped, use the varnish-invalidation-tls-verify.
-varnish-invalidation-tls-verify <bool>VARNISH_CONTROLLER_VARNISH_INVALIDATION_TLS_VERIFY=<bool>When Varnish is using TLS (HTTPS) for invalidation requests (see: varnish-invalidation-tls), this flag is used to enable/disable TLS
certificate verifications. (default true)
The certificates used to verify TLS are the ca-certificates provided by the OS.
-varnish-nameVARNISH_CONTROLLER_VARNISH_NAMEVarnish instance name, should match the varnishd -n argument, should only be specified if varnishd is started with -n.
This is by default /var/lib/varnish
--helpThe varnish-controller-agent provides a full list of parameters with --help and then exit.
Given the following default ExecStart of varnishd.service:
ExecStart=/usr/sbin/varnishd \
-a :6081 \
-a localhost:8443,proxy \
-T localhost:6082 \
-S /etc/varnish/secret \
-p feature=+http2 \
-r vcc_allow_inline_c \
-r allow_exec \
-f /etc/varnish/default.vcl \
-s mse
the agent does not need additional configuration.
Given the following non-default varnishd.service:
ExecStart=/usr/sbin/varnishd \
-a :80 \
-a localhost:6081,proxy \
-T localhost:6000 \
-S /secrets/varnish \
-n /srv/varnish
the agent need the following configuration added:
[Service]
Environment="VARNISH_CONTROLLER_VARNISH_SECRET=/secrets/varnish"
Environment="VARNISH_CONTROLLER_VARNISH_HOST=localhost:80"
Environment="VARNISH_CONTROLLER_VARNISH_ADMIN_PORT=6000"
Environment="VARNISH_CONTROLLER_VARNISH_NAME=/srv/varnish"
It is strongly advised to keep the configuration to defaults, but it is sometimes necessary to make adjustments for certain environments.