Agents

Agents are required to have unique names in the Varnish Controller system. The names must be alphanumeric but can also include a dash (-). They can have the same name in version 5.0+, if different private tokens are used.

Each Varnish should have a corresponding agent running if it should be included in the Varnish Controller system. Running multiple agents toward the same Varnish is not supported and can result in unpredicted behavior.

Restarting an agent will not affect the Varnish instance as long as the Varnish instance has not been manually changed with regard to loaded VCL.

Agents will start by waiting for connection towards NATS, once connected it will continue.

Installation

# Ubuntu/Debian
sudo apt install varnish-controller-agent

# RPM-based
sudo dnf install varnish-controller-agent

Varnish Interaction

The agent will communicate with Varnish using several interfaces:

Administration port

Arg: -varnish-admin-port int
Env: VARNISH_CONTROLLER_VARNISH_ADMIN_PORT=int Varnish administration port, must match the -T parameter of Varnish. This is used by the agent to load VCL. The default is 6082, since varnish is by default started with -T localhost:6082.

Varnish Host

Arg: -varnish-host string
Env: VARNISH_CONTROLLER_VARNISH_HOST=string Host and port where varnish is located. Format should be example.com:80. The default is 127.0.0.1:6081 and must match one of the -a parameters of Varnish. The default is 127.0.0.1:6081, since varnish by default is started with -a :6081.

Varnish Invalidation Host

Arg: -varnish-invalidation-host string
Env: VARNISH_CONTROLLER_VARNISH_INVALIDATION_HOST=string Host and port where varnish is located. Format should be the same as varnish-host, and must also match one of the -a parameters of Varnish. This is the connection used by the agent to do invalidations. The default is the same as varnish-host, so if those are the same, only varnish-host need to be specified. If varnish is only listening on TLS, please look at the varnish-invalidation-tls-parameter below.

Varnish Secret

Arg: -varnish-secret path
Env: VARNISH_CONTROLLER_VARNISH_SECRET=path Varnish secret file for varnishadm. Varnish-agent must have permissions to read this file, and this must match the -S parameter of Varnish. The default is /etc/varnish/secret, which is also the default for Varnish.

Agents must be able to communicate with Varnish on the Varnish administration port. The default port is 6082. That means that Varnish must be started with -T :6082 for example. The agent must also be able to read the Varnish secret file (defaults to /etc/varnish/secret). The agent can be configured with the Varnish administration port using the flag -varnish-admin-port(VARNISH_CONTROLLER_VARNISH_ADMIN_PORT). The host used for the Varnish administration interface is the same as configured by -varnish-host (VARNISH_CONTROLLER_VARNISH_HOST).

Since the agent is responsible for reading/writing VCL files to disk and loading them into Varnish, Varnish must be able to read the files as well. This is not a problem when running on the same server, as long as permissions are set correctly. But when running in a container environment, they both need to be able to read the same shared directory. This is default /var/lib/varnish, but can be changed if needed, see Varnish Name.

If Varnish is listening on a different host/port, this must be reflected in the agents configuration variables -varnish-invalidation-host(VARNISH_CONTROLLER_VARNISH_INVALIDATION_HOST) and -varnish-host (VARNISH_CONTROLLER_VARNISH_HOST). Otherwise, the invalidation will not work. If no varnish-invalidation-host is specified, the agent will use varnish-host as invalidation host and port.

See the containers for more information on running in containers.

NOTE: When a Varnish instance is operated by the Varnish Controller Agent, the agent will remove any loaded VCL/Label that has not been loaded through the Varnish Controller.

Varnish Statistics

The agent will use varnishstat to fetch statistics from Varnish. To be able to fetch statistics, the varnishstat binary is required on the same server as the agent and it will need to have read permissions to the varnish shared memory area (VSM). This is by default /var/lib/varnish, but can be changed with the -n parameter of varnish (see Varnish Name). Then varnishstat also needs to know which Varnish VSM (Varnish Shared Memory) is used. This can be specified with the -varnish-name parameter, that will be the same as the -n argument to varnishd.

Example:

# Varnish is started with the argument -n /tmp/myvarnish
varnishd -a 0.0.0.0:8091 -S /tmp/secret -T localhost:6082 -f '' -n /tmp/myvarnish

# The agent then needs to know the VSM directory /tmp/myvarnish
varnish-controller-agent -varnish-name /tmp/myvarnish

Setting the Agent Name

To identify the agents in the Varnish Controller, the agent name is used. Agents are required to have unique names in the Varnish Controller system. The names must be alphanumeric but can also include a dash (-). Set the agent name as an argument -agent-name agent01 or environment variable VARNISH_CONTROLLER_AGENT_NAME=agent01.

Note that they can have the same name in version 5.0+, if different private tokens are used. See private/shared tokens chapter

Read Only

An agent can be started in read-only mode using the -read-only flag. This makes the agent not usable in Varnish Controller with regard to deployments, but it will read information from Varnish and present it to Varnish Controller. It will read loaded VCLs, Varnish version, etc. and present via the REST API.

The agent must be restarted without the -read-only flag in order to be used for deployment via Varnish Controller.

Varnishadm CLI Command File

The agent generates a file called cmds.cli every time a new configuration is applied to the agent. This file can be passed to varnishd with the flag -I /var/lib/varnish-controller/varnish-controller-agent/<agent_name>/cmds.cli. The file contains all the commands required to load the last configuration into Varnish when Varnish starts.

When the agent starts after this has been loaded, any changes that the agent retrieves from Brainz will be applied. If no VCL changes has been made, then nothing will be redeployed.

Agent Temp Dir

The agent uses the default system temporary directory to test compiling the VCL files before deploying them. This requires execution permission on the directory. If the temporary directory, such as /tmp, is mounted with noexec this will not work. Hence, we recommend to pointing out a different temporary directory for the agent via the environment variable TMPDIR. This can be added to the systemd service file.

Example:

sudo systemctl edit varnish-controller-agent
# Add the following
[Service]
Environment="TMPDIR=/var/lib/varnish-controller/varnish-controller-agent/<agentname>/"

NOTE: from version 5.0 of Varnish Controller. The agent will use the agent base-dir as temporary directory. This means that the above information is not applicable for version higher or equal to 5.0 release of Varnish Controller.

Deployment Backups

From version 6.2.0 of the controller, the agents will create backups of its last deployments. Number of backups can be changed with the configuration parameter -max-backups or via configurable components. The default is last 20 deployments.

A backup is created as soon as anything changes, such as the root VCL in a shared deployment, or a new VCL is loaded or removed etc.

The backups are stored in the base-dir/backups/ directory. Each backup is stored within a directory named with the current timestamp. The directories contains all files loaded for each VCLGroup, the temp VCL, the root VCL, cmds.cli file and a script to load the files into Varnish.

Note that when restoring from a backup, the agent must be stopped or else it will override the loaded configuration if it differs from the backup.

Example dir of a backup:

/agent1/backup/20240613121756
/agent1/backup/20240613121756/restore.sh
/agent1/backup/20240613121756/cmds.cli
/agent1/backup/20240613121756/root.vcl
/agent1/backup/20240613121756/temp_root
/agent1/backup/20240613121756/1
/agent1/backup/20240613121756/1/main1.vcl

The script restore.sh is a helper script that will load the files in the directory into Varnish via the cmds.cli file. The cmds.cli file contains all the commands that will be executed via Varnish CLI.

Advanced Configuration

There are some configuration parameters that could be tuned to change how the agent operates. These might not be self-explanatory and are described below.

Note: Do not change these parameters if you are not sure about the implications.

Load From Disk

Arg： -load-from-disk <bool>
Env: VARNISH_CONTROLLER_LOAD_FROM_DISK=<bool>

Load previous deployments (if any) from disk, even if NATS or Brainz is not reachable. The agent saves manifest files containing the deployments (VCLs) to disk. These are then loaded into the agent and Varnish if this option is turned on (by default true).

Default VCL

Arg: -default-vcl string
Env: VARNISH_CONTROLLER_DEFAULT_VCL=<string>

The default VCL is the VCL that will be loaded into Varnish if no other VCL has been loaded in the system. This VCL can be replaced with any other VCL and can be an error page of some kind. Default is a synth 503.

The default VCL can be viewed:

varnish-controller-agent -default-vcl-show

Heartbeat Rate

Arg: -hb-rate <duration>
Env: VARNISH_CONTROLLER_HB_RATE=<duration>

The heartbeat rate is the rate of heartbeats being sent by the agent to brainz to tell brainz of its presence and state. The lower heartbeat the faster response time for changes. Note that brainz might require some extra configuration if this value is changed to identify an agent that is down based on missed heartbeats. (default 3s)

NATS Server

Arg: -nats-server string
Env: VARNISH_CONTROLLER_NATS_SERVER=<string> The agent needs to communicate with brainz via NATS bus. The NATS server connection string is in the format of host:port or with username/password user:pass@host:port. (default “localhost:4222”)

NATS Reconnect Time

Arg: -nats-reconnect-time <duration>
Env: VARNISH_CONTROLLER_NATS_RECONNECT_TIME=<duration>

Time between each reconnect to NATS server upon lost connection. (default 5s)

NATS Timeout

Arg: -nats-timeout <duration>
Env: VARNISH_CONTROLLER_NATS_TIMEOUT=<duration>

Timeout for sending a message on the NATS message-bus. (default 10s)

Root VCL

Arg: -root-vcl string
Env: VARNISH_CONTROLLER_ROOT_VCL=<string>

This should usually not be modified. The default is a template used to handle multiple domains. This VCL is only used when running shared/cloud deployments with shared Varnish instances between multiple domains.

The default root VCL can be viewed:

varnish-controller-agent -root-vcl-show

Static Tags

Arg: -tags tag1,tag2
Env: VARNISH_CONTROLLER_TAGS=<string>,<string>

Starts the agent with pre-defined tags that will be marked static in the database. If someone removes a static tag it will be automatically added again if the agent has it defined at startup.

If multiple agents are started with the same tag names, only one instance of the tag will be created. Hence, it is safe to start multiple agents with same tag names.

Stats Filter (From: 6.4.0)

Arg: -stats-filter string
Env: VARNISH_CONTROLLER_STATS_FILTER=<string>

This is a comma-separated string of statistics that will be sampled by the agent. These are the same names as the output from varnishstat -j. Anything other than the specified statistics will be discarded. Wildcard (*) usage is supported.

Example configuration:

./varnish-controller-agent -stats-filter MAIN.client_req,MAIN.cache_hit,ACCG.*.total.client_*

Agent Stats Filter (Deprecated from: 6.4.0)

Arg: -agent-stats-filter string
Env: VARNISH_CONTROLLER_AGENT_STATS_FILTER=<string>

From version 6.4.0 please use the -stats-filter configuration. This is a comma-separated string of statistics that will be sampled by the agent. These are the same names as the output from varnishstat -j. Anything other than the specified statistics will be discarded.

Example configuration:

./varnish-controller-agent -agent-stats-filter MAIN.client_req,MAIN.cache_hit

Agent Accounting

Arg: -accounting bool
Env: VARNISH_CONTROLLER_ACCOUNTING=<bool>

This can be explicitly set to allow/deny the agent to sample accounting statistics.

Example configuration:

./varnish-controller-agent -accounting true

Agent Accounting Stats Filter (Deprecated from: 6.4.0)

Arg: -accounting-stats-filter string
Env: VARNISH_CONTROLLER_ACCOUNTING_STATS_FILTER=<string>

From version 6.4.0 please use the -stats-filter configuration. This is a comma-separated string of accounting statistics that will be sampled by the agent.

Example configuration:

./varnish-controller-agent -accounting-stats-filter is_healthy,client_hit_count,client_miss_count

Agent Accounting Stats Keys (Deprecated from: 6.4.0)

Arg: -accounting-stats-keys string
Env: VARNISH_CONTROLLER_ACCOUNTING_STATS_KEYS=<string>

From version 6.4.0 please use the -stats-filter configuration. This is a comma-separated string of accounting statistics keys that will be sampled by the agent.

Example configuration:

./varnish-controller-agent -accounting-stats-keys total,sub

VCLGroup Stats Filter (Deprecated from: 6.4.0)

Arg: -vclgroup-stats-filter string
Env: VARNISH_CONTROLLER_VCLGROUP_STATS_FILTER=<string>

From version 6.4.0 please use the -stats-filter configuration. This is almost the same as -agent-stats-filter except that this filters out statistics for VCLGroups (loaded VCLs). It can only filter out parts of the VCL that are presented by varnishstat for VCL backends. The filter names take the last part of the statistics name after the loaded VCL name.

As an example the varnishstat output VBE.vg1_129621b78f1ea3696eb116cbae3abc85bf89ef4ae0d6b642b873c9c978537117_1.req shows both VBE and the VCL name (which will be different depending on both content and reloads). Hence, we only specify the last part as filter. In this example req as the statistics that we want to gather.

Example configuration:

./varnish-controller-agent -vclgroup-stats-filter conn,req

(default “req,conn,unhealthy,is_healthy,bereq_hdrbytes,bereq_bodybytes,beresp_hdrbytes,beresp_bodybytes”)

IPv4 and IPv6

Arg: -ipv4 string and -ipv6 string
Env VARNISH_CONTROLLER_IPV4=<string> and VARNISH_CONTROLLER_IPV6=<string>

The IPv4 and IPv6 options are used by routers in DNS routing towards the agents. These IP addresses will be specified in the DNS records if the routers act as DNS backends to PowerDNS. Meaning that a domain lookup is made and the agent is healthy, these IP addresses will be returned back in response to the client DNS request.

Base URL

Arg: -base-url string
Env: VARNISH_CONTROLLER_BASE_URL=<string>

The base URL is the full URL to the agent’s Varnish server. This is used when routers are used for traffic routing towards the Varnish servers. The routers will forward requests in the form of <base-url>/<domain>/<path>. The base URL should be a valid URL such as https://example.com.

Latitude and Longitude

Arg: -latitude float and -longitude float
Env: VARNISH_CONTROLLER_LATITUDE=<float> and VARNISH_CONTROLLER_LONGITUDE=<float>

The latitude and longitude is the geographical position of the Varnish server. This is used for calculating distance between client IP and the Varnish servers when performing traffic routing via the router.

Network Interface Cards

Arg: -nics string
Env: VARNISH_CONTROLLER_UTIL_NICS=<string>

This is a comma separated list of Network Interface Card (NIC) names on the system. The given NICs will be used to calculate bandwidth that will be sent as utilization to all traffic routers. If multiple NICs are specified, the bandwidth sent as utilization will be an additive value of all the NICs bandwidth. TxBytes are only included in the bandwidth, not RxBytes.

In order for this to work, the agent must have access to the same NICs as the Varnish server uses.

Max Mbps Hard

Arg: -max-mbps-hard float
Env: VARNISH_CONTROLLER_MAX_MBPS_HARD=<float>

This is used for request routing when traffic routers are used. This is the maximum bandwidth that the server can deliver before considered over utilized and not taken into account for traffic routing by the router’s built-in rules. Plugins see servers above this limit and are responsible for applying this limit as per their policies, hard or otherwise.

Note that since only TxBytes are sampled (see -nics), this should be taken into consideration when specifying an appropriate Max Mbps Hard waterlevel.

Stats Interval

Arg： -stats-interval <duration>
Env: VARNISH_CONTROLLER_STATS_INTERVAL=<duration>

Specify how often Varnish statistics should be gathered from the Varnish instance. (default 1m0s)

Varnish Invalidation TLS

Arg: -varnish-invalidation-tls <bool>
Env: VARNISH_CONTROLLER_VARNISH_INVALIDATION_TLS=<bool>

When set to true (default false), this makes the agent send invalidation requests to varnish using HTTPS instead of HTTP. If verification of the TLS certificate should be skipped, use the varnish-invalidation-tls-verify.

Varnish Invalidation TLS verification

Arg: -varnish-invalidation-tls-verify <bool>
Env: VARNISH_CONTROLLER_VARNISH_INVALIDATION_TLS_VERIFY=<bool>

When Varnish is using TLS (HTTPS) for invalidation requests (see: varnish-invalidation-tls), this flag is used to enable/disable TLS certificate verifications. (default true)

The certificates used to verify TLS are the ca-certificates provided by the OS.

Varnish Name

Arg: -varnish-name
Env: VARNISH_CONTROLLER_VARNISH_NAME

Varnish instance name, should match the varnishd -n argument, should only be specified if varnishd is started with -n.

This is by default /var/lib/varnish

Using Help

Arg: --help

The varnish-controller-agent provides a full list of parameters with --help and then exit.

Example systemd configurations:

Given the following default ExecStart of varnishd.service:

ExecStart=/usr/sbin/varnishd \
          -a :6081 \
          -a localhost:8443,proxy \
          -T localhost:6082 \
          -S /etc/varnish/secret \
          -p feature=+http2 \
          -r vcc_allow_inline_c \
          -r allow_exec \
          -f /etc/varnish/default.vcl \
          -s mse

the agent does not need additional configuration.

Given the following non-default varnishd.service:

ExecStart=/usr/sbin/varnishd \
          -a :80 \
          -a localhost:6081,proxy \
          -T localhost:6000 \
          -S /secrets/varnish \
          -n /srv/varnish

the agent need the following configuration added:

[Service]
Environment="VARNISH_CONTROLLER_VARNISH_SECRET=/secrets/varnish"
Environment="VARNISH_CONTROLLER_VARNISH_HOST=localhost:80"
Environment="VARNISH_CONTROLLER_VARNISH_ADMIN_PORT=6000"
Environment="VARNISH_CONTROLLER_VARNISH_NAME=/srv/varnish"

It is strongly advised to keep the configuration to defaults, but it is sometimes necessary to make adjustments for certain environments.