Settings

This page describes the settings that are supported when using Varnish High Availability 6.

Global

Global settings must be configured in vcl_init. Each setting is shown with its default value.

`vha6_opts.set("token", "[secret]");`

HMAC signing key used in all transactions. This key is used for signing and is never exposed in an actual transaction. It must be identical across all nodes.
After setting the key, call vha6_token_init; must be used. Not setting a token or calling the init function will prevent Varnish from successfully loading the VCL.

`vha6_opts.set("broadcaster_scheme", "http");`

Scheme to use when connecting to Broadcaster
Can be http or https
If using https, make sure Broadcaster is configured for HTTPS.

`vha6_opts.set("broadcaster_host", "localhost");`

Host to use when connecting to Broadcaster

`vha6_opts.set("broadcaster_port", "8088");`

Port to use when connecting to Broadcaster

`vha6_opts.set("broadcaster_group", "");`

Group to broadcast to
Defined in the nodes.conf
Defaults to all nodes in the root configuration

`vha6_opts.set("broadcaster_ssl_verify_peer", "true");`

If using SSL, validate Broadcaster node’s certificate chain.

`vha6_opts.set("broadcaster_ssl_verify_host", "false");`

If using SSL, verify Broadcaster node’s identity in the SSL certificate.

`vha6_opts.set("broadcast_limit", "200");`

Note: available since 6.0.13r14.

Maximum number of concurrent outgoing VHA_BROADCAST requests.
If unset, broadcast_rate_limit is used as a fallback.

See limits for more information.

`vha6_opts.set("broadcast_rate_limit", "200");`

Note: available since 6.0.13r14.

Maximum number of outgoing VHA_BROADCAST requests per second.
If unset, max_requests_sec is used as a fallback.

See limits for more information.

`vha6_opts.set("request_rate_limit", "500");`

Note: available since 6.0.13r14.

Maximum number of incoming VHA6 operations per second.
VHA6 operations include incoming VHA_BROADCAST and VHA_FETCH requests.
If unset, 2.5 * broadcast_rate_limit is used as a fallback.

See limits for more information.

`vha6_opts.set("origin_scheme", "");`

Scheme for peers to use to connect back to the VHA6 origin node
Defaults to the scheme defined in the nodes.conf
If no match is found and Varnish or Hitch is listening on port 443, https is used, otherwise http.

`vha6_opts.set("origin", "");`

Hostname or IP of the origin node
Defaults to the value defined in the nodes.conf, otherwise server.ip

`vha6_opts.set("origin_port", "");`

Port for peers to use to connect back to the VHA6 origin node
Defaults to the port defined in the nodes.conf
If no match is found, the listening port for Varnish or Hitch is used.

`vha6_opts.set("origin_ssl", "");`

Enables or disables SSL communication when talking to the VHA6 origin node
Defaults to the scheme defined in the nodes.conf
origin_scheme overrides this value.

`vha6_opts.set("origin_ssl_sni", "true");`

If using SSL, enable SNI when connecting to the origin node.

`vha6_opts.set("origin_ssl_verify_peer", "true");`

If using SSL, validate the origin node’s certificate chain.

`vha6_opts.set("origin_ssl_verify_host", "false");`

If using SSL, verify the origin node’s identity in the SSL certificate.

`vha6_opts.set("allow_locahost", "false");`

Allows a localhost address for the origin node

`vha6_opts.set("allow_stale", "false");`

Allows stale objects to be transferred

`vha6_opts.set("fetch_timeout", "");`

How long to wait for a VHA_FETCH to complete. This is a VCL duration.

`vha6_opts.set("force_fast304", "false");`

Attempts to use fast_304() on all VHA6 transactions
Only applies to objects with 304 origin responses

`vha6_opts.set("force_update", "false");`

Forces all VHA6 transactions to update an existing object in cache
Default keeps existing objects in cache when replicating a duplicate

`vha6_opts.set("keep_alive", "120s");`

Once a VHA6 request is validated, sets the session’s timeout_idle to this value
Default is twice the default value of backend_idle_timeout to maximize the re-use of connections from other nodes

`vha6_opts.set("min_ttl", "3s");`

If an object has a ttl value equal to or less than this value, the object won’t be replicated.

`vha6_opts.set("max_requests_sec", "200");`

Note: deprecated since 6.0.13r14.

The max_requests_sec setting used to serve three different purposes and was replaced by new settings:

broadcast_limit
broadcast_rate_limit
request_rate_limit

It is still present for existing VHA6 setups and serves as a fallback when the new settings are omitted from the configuration with a notable exception: the request_rate_limit setting falls back to 2.5 * max_requests_sec when all three new settings are unset.

See limits for more information on the new settings.

`vha6_opts.set("max_bytes", "25000000");`

If an object has a Content-Length larger than this value, it’s not replicated.
Chunked responses cannot be evaluated for this parameter.

`vha6_opts.set("peer_stream", "true");`

Stream objects through the peers

`vha6_opts.set("token_ttl", "2m");`

How long the token HMAC signature is valid on VHA6 transactions.
Represents the max clock skew allowed between servers

`vha6_opts.set("vcs", "true");`

Toggles VCS logging

`vha6_opts.set("vcs_key", "vcs-key");`

Changes the VCS key prefix

`vha6_opts.set("origin_backend_linger", "10s");`

Determines how long dynamic backends used for peer fetches should linger before being deleted

Requests

Request settings must be configured in vcl_backend_fetch or vcl_backend_response and are shown with the default values.

`vha6_request.set("skip", "false");`

If set to true, this request won’t be replicated.

`vha6_request.set("force_update", "false");`

Forces this VHA6 transaction to update an existing object in cache

Limits

In a cluster with VHA6, a cache insertion results in a broadcast to inform the other members of the cluster. The other members of the cluster may in return request the new contents to populate their own caches.

From a Varnish server’s perspective, it means that one outgoing broadcast can result in an amplification of up to N-1 incoming fetch requests from Varnish peers in a cluster of N servers as a consequence. In addition to the fetch requests, a Varnish server will receive broadcasts from its N-1 peers.

This traffic between Varnish nodes and the amplification factor could disturb traffic dedicated to actual clients despite an eventual higher hit ratio due to the continuous cache warm-up enabled by VHA6.

The content replication limits can be adjusted to keep VHA6 traffic under control:

broadcast_limit
broadcast_rate_limit
request_rate_limit

A Varnish server will broadcast in theory up to broadcast_rate_limit cache insertions per second. If all members of a cluster broadcast at full capacity one node should expect an amplification of:

(N-1) * broadcast_rate_limit incoming broadcast requests
(N-1) * broadcast_rate_limit incoming fetch requests

At full broadcast capacity, request_rate_limit should in theory be adjusted to 2 * (N-1) * broadcast_rate_limit to allow a node to process all incoming VHA6 requests without dropping anything.

In a cluster of 3 nodes, with a broadcast_rate_limit of 200, the value for request_rate_limit should in theory be 800 to process all VHA6 traffic. The fallback value however is computed as 2.5 * broadcast_rate_limit when the request_rate_limit setting is not set. This conservative fallback assumes a cluster of 3 nodes that will not process all the VHA6 traffic when all nodes broadcast at full capacity, but can still absorb cache insertion spikes happening sporadically on individual nodes.

The error_rate_limited counter is incremented when either rate limit is reached.

The other mechanism to reduce VHA6 replication pressure is broadcast_limit falling back to broadcast_rate_limit when the former is not set. This match between the two settings means that a broadcast request have a default budget of 1s on average at full replication capacity. One reason for broadcast requests to take time is the corresponding fetch requests coming back while a cache insertion is still being fetched by the Varnish origin that triggered the broadcast. A slow backend increases the chances of saturating the broadcast_limit before reaching the broadcast_rate_limit.

The error_max_broadcasts counter is incremented when this limit is reached.

See varnishstat on VHA6 monitoring for other noteworthy counters.

Hooks

Hooks allow for custom VCL code to execute during specific VHA6 states. To set up hooks, run the following commands:

mkdir -p /etc/varnish/vha6/hooks
cp /usr/share/varnish-plus/vcl/vha6/hooks/states.vcl /etc/varnish/vha6/hooks

This implements a default VHA6 hooks file at /etc/varnish/vha6/hooks/states.vcl. Edit this file to add custom VCL to the defined VHA6 hooks.

Bans and VHA

`vha.skip_ban()`

vha.skip_ban() is a function in vmod_vha that skips request-time ban evaluation for the current lookup. It can only be called from vcl_recv and vcl_hash.

When set, a banned object that has not yet been removed by the ban lurker is returned as a normal hit. The flag is per request, is not exposed as a VCL variable, and resets on every request.

The bundled VHA6 VCL calls vha.skip_ban() in two places:

On the peer, when receiving a VHA_BROADCAST, for the “do I already have this object?” lookup.
On the origin, when answering a VHA_FETCH, so the just-inserted object can be served to the peer even if a ban test would fire during lookup.

VHA lookups are internal bookkeeping, not client traffic. VHA deliberately leaves ban processing to real client requests and the ban lurker rather than having replication traffic evaluate and act on bans.

You normally do not call vha.skip_ban() yourself; the bundled VHA6 VCL already does. It only becomes relevant if you write your own replication VCL or VHA hooks and need internal lookups to bypass bans.

Bans and replication

Ban membership is positional: an object inserted after a ban never matches that ban. A freshly replicated copy on a peer is therefore immune to all pre-existing bans on that peer, while bans issued after the replication apply normally.

VHA does not propagate bans between peers. Cache invalidation must reach each node separately, for example via the Broadcaster. See cache invalidation.

PEER HIT and the ban lurker

When a ban is followed by a replication, a peer can answer PEER HIT and skip the replication, even though the object was just banned:

Object X is cached on nodes A and B.
A ban matching X is issued on both nodes.
A client request hits node A: the lookup evaluates the ban, kills A’s copy, fetches a fresh X from the backend and inserts it. VHA broadcasts the insert to B.
B receives the VHA_BROADCAST with skip_ban active. B’s old, banned copy of X is still physically resident, because the ban lurker ignores bans younger than ban_lurker_age (default 60s). The lookup hits the old copy and B answers 200 PEER HIT, skipping the replication.
B’s stale copy only dies when the ban lurker gets to it, at the earliest about 60 seconds after the ban, or when a real client request on B evaluates the ban. That replication generation is lost and B fetches from its own backend on the next client miss.

This is expected behavior, not data loss. The object is still invalidated for clients on B, because client lookups do evaluate bans. However, the fetch_peer_hit counter and the PEER HIT response right after a ban and a replication can look wrong without this explanation.

Note that the window is controlled by ban_lurker_age (default 60s, the ban lurker ignores bans until they are this old), not by ban_lurker_holdoff (default 0.010s, a lock-contention sleep). The related ban_lurker_sleep (default 0.010s) controls how often the lurker runs; setting it to 0 disables the ban lurker entirely.

To avoid losing the replication in this scenario:

Enable force_update, either globally or per request, or send the vha6-force: true header with the replication. This sets hash_always_miss on the peer, bypassing the PEER HIT and forcing the fresh copy in.
Lower ban_lurker_age to shrink the window in which a banned object can still answer a peer lookup.

This behavior can be observed through:

The fetch_peer_hit counter. See varnishstat.
The VHA_BROADCAST PEER: cache hit line in varnishlog.
The PEER HIT synth reason on the peer response.