Search
Varnish High Availability

varnish-discovery

In the case of cloud instances, or any other platform featuring clusters that can shrink and grow dynamically, the nodes.conf file needs to be updated and broadcaster needs to re-read it when a change occur. This is varnish-discovery’s role.

It will watch a given source, such as the VAC api or DNS information and will create or update a nodes.conf file tracking the relevant machines, in addition, it can send a SIGHUP to broadcaster (or any process) to warn it the configuration changed.

Its package is varnish-plus-discovery.

Preparing

First, broadcaster must expose a pid file using the -pid argument. By default, broadcaster and its service file use /run/varnish-broadcaster/broadcaster.pid by default

Getting started

varnish-discovery supports multiple backends to generate a nodes.conf, each with their own set of switches (most are optional), so to help you hit the ground running, here’s a selection of examples.

Note: if you just need to create the file then exit (eg. before you start broadcaster for the first time), you can just append -once to those commands.

AWS

Here, -group specifies an autoscaling group on AWS. This feature relies on the AWS SDK, so if awscli is properly configured and works, this subcommand will only require common arguments.

/usr/bin/varnish-discovery aws                                       \
                  -nodefile /etc/varnish/nodes.conf                 \
                  -warnpid /run/varnish-broadcaster/broadcaster.pid \
                  -group $DOMAIN_NAME

Azure

Here, -group specifies a virtual machine scale set on Azure. This feature relies on the Azure SDK, so if the az CLI is properly configured and works, this subcommand will only require common arguments.

/usr/bin/varnish-discovery azure                                     \
                  -nodefile /etc/varnish/nodes.conf                 \
                  -warnpid /run/varnish-broadcaster/broadcaster.pid \
                  -group $SCALESET_NAME

DNS

We need to specify a domain name that will be checked every now and then (see the -every option).

/usr/bin/varnish-discovery dns                                       \
                  -nodefile /etc/varnish/nodes.conf                 \
                  -warnpid /run/varnish-broadcaster/broadcaster.pid \
                  -group $DOMAIN_NAME

kubernetes

If running inside a pod, discovery will be able to find information about the namespace, cacert and token to access the API server , so there’s no need to specify them:

/usr/bin/varnish-discovery k8s                                       \
                  -nodefile /etc/varnish/nodes.conf                 \
                  -warnpid /run/varnish-broadcaster/broadcaster.pid \
                  -server "https://$K8S_API/"                       \
                  -group $ENDPOINT                                  \
                  -port $PORT

If you don’t specify the port, discovery will assume it’s 80 or 443 depending on the protocol you use (http if not specified). Any pod that doesn’t listen to TCP on the selected port will be omitted, but you can use -port 0 to use the first port listen for each pod.

If your node is only listed by more than one endpoint you must use -group otherwise the node will be present in multiple clusters and broadcaster won’t know what to do with it.

VAC

For the Varnish Administration Console, we only need to point to the API’s address, and varnish-discovery will be able to figure out its own cluster.

/usr/bin/varnish-discovery vac                                       \
                  -nodefile /etc/varnish/nodes.conf                 \
                  -warnpid /run/varnish-broadcaster/broadcaster.pid \
                  -server $LOGIN:$PASSWD@$VAC_IP

Service

All packages offer service files, how you edit them is going to depend on your platforms. For sysv, you can edit varnish-discovery.params located in either /etc/deafult/ or in /etc/sysconfig to change the ENABLE and DAEMON_OPTS= variables.

For systemd, run you can create the file /etc/systemd/system/varnish-discovery.service.d/exec.conf and redefine the ExecStart parameter, for example:

cat > /etc/systemd/system/varnish-discovery.service.d/exec.conf << EOF
[Service]
ExecStart=
ExecStart=/usr/bin/varnish-discovery dns -group localhost -ipv4 -nodefile /etc/varnish/nodes.conf -warnpid /run/varnish-broadcaster/broadcaster.pid
EOF
systemctl daemon-reload

Options

Common

  • -every DURATION (default: 2s)

    This specify how frequently varnish-discovery should contact the source. For DNS and VAC, this means the duration between two requests (from start to start). In the kubernetes case, which uses long-poll, it tells varnish-discovery how long it should wait before trying again in case of failure (again, this is he time since the start of the failed request).

  • -group NAME

    This option can be used multiple times and tell varnish-discovery the name to look for. The meaning is different depending on the source, but it always represent a handle behind which multiple IPs can hide::

    - AWS: corresponds to autoscaling group
    - Azure: corresponds to a virtual machine scale set
    - DNS: corresponds to hostname
    - k8s: corresponds to endpoint
    - VAC: corresponds to group
    
  • -ipv4 -ipv6

    Restrict what version of the IP protocol should be used (useful in the DNS case). If none is supplied, use both.

  • -nodefile [TEMPLATE:]NODEFILE (default: -)

    Output the cluster information in NODEFILE, with “-” meaning stdout. If TEMPLATE is specified, it points to a go template used to format the node file. More information below.

  • -once

    By default, varnish-discovery will monitor its source of information indefinitely, but with this option it only does it once before exiting. An non-zero return indicates an error during the run.

  • -proto PROTO (default: http)

    If the source of information doesn’t explicit the protocol used, we fallback to PROTO.

  • -port PORT

    If the source of information doesn’t explicit the port used, we fallback to PORT. If port is omitted, it is inferred from the protocol, and if “0” is specified, the first port is used (useful in the kubernetes case).

  • -postupdate COMMAND

    After a group update, run COMMAND. This is useful to reload a process or update a status file for example:

    varnish-discovery dns -group example.com -postupdate "touch /tmp/updated"
    

    COMMAND is parsed as a string following shell quoting logic, and -postupdate can be specified multiple times.

    Important notes: discovery will execute all commands in parallel but will wait for them to return before continuing. Also, if COMMAND doesn’t use an absolute filename for its binary, it must be in the PATH.

  • -warnpid PIDFILE

    To ensure that the broadcaster service reloads its configuration when changes occur in the node file, the -warnpid flag can be used to send a SIGHUP to the broadcaster service process, which will trigger a graceful reconfiguration. This option can be specified multiple times.

  • -version or -v

    Display the version number and exit

AWS

  • -region REGION

    If the AWS region hasn’t been configured yet (using aws configure), you may specify it here

Azure

For setting up Azure these environment variables needs to be configured.

AZURE_TENANT_ID="<app-tenant-id>"
AZURE_SUBSCRIPTION_ID="<subscription-id>"
AZURE_CLIENT_ID="<app-client-id>"
AZURE_CLIENT_SECRET="<app-secret>"
  • -resourcegroup NAME

    If the Azure base resource group name hasn’t been configured yet (using az configure or by setting the AZURE_BASE_GROUP_NAME environment variable), you may specify it here

  • -subscriptionid ID

    If the ID of your Azure subscription hasn’t been configured yet (using az configure or by setting the AZURE_SUBSCRIPTION_ID environment variable), you may specify it here

DNS

  • -no-hostname

    By default, varnish-discovery tries to find the IP corresponding to the local machine in the resolved list and replaces it with the local hostname.

    This switch disables this find-and-replace behavior.

kubernetes

  • -server URL (https://kubernetes/)

  • -token PATH (/var/run/secrets/kubernetes.io/serviceaccount/token)

  • -cacert PATH (/var/run/secrets/kubernetes.io/serviceaccount/ca.crt)

  • -namespace PATH (/var/run/secrets/kubernetes.io/serviceaccount/namespace)

    When operating inside a pod, varnish-discovery can find how to communicate to the kubernetes api using the conditional default, but they can all be overridden to work outside of a pod or to connect to a non-standard URL.

  • -group ENDPOINT

    Note that in k8s’ case, if no endpoint is specified, varnish-discovery will use the one owning the pod it’s running on. This allows for minimal configuration based on context.

VAC

  • -server URL

    Where to find the VAC API. URL must include the login and password to correctly authenticate to the API.

  • -no-hostname

    As for the dns command, varnish-discovery will try and replace the local ip with the local hostname.

Template

The default template to generate the nodes.conf file is:

# this was generated at: {{ .Time }}
# using command: "{{ .Command }}"

{{if .Err}}# node generation failed because an error occured: {{ FormatReplace (print .Err) "\n" "\n# " }}
{{ else }}{{ range $i, $grp := .List }}{{if $i}}
{{ end }}[{{ $grp.Name }}]
{{ if not $grp.Nodes}}# this cluster is empty
{{ else }}{{ range $j, $node := $grp.Nodes }}{{ $node.Name }} = {{ $node.Proto }}://{{ FormatIP $node.IP }}:{{ $node.Port }}
{{ end }}{{ end }}{{ end }}{{end}}

The FormatReplace function will ensure that new lines in a string get prefixed by a #, this is used to print error message in comments.

The template uses a Groups structure looking like this:

type Groups struct {
	Time    time.Time
	Err     error
	Command string
	List    []struct{
		Name  string
		Nodes []struct{
			Name  string
			Proto string
			Port  uint64
			IP    net.IP
		}
	}
}

Changelog

version 1.5.0 (2023-02-10)

  • Maintenance release, including:
    • Azure SDK minor version upgrade.
    • AWS SDK major version upgrade from v1 to v2.
    • Go 1.20.0

version 1.4.0 (2022-06-29)

  • Maintenance release, including AWS SDK version upgrade.

version 1.3.0 (2022-02-24)

  • Maintenance release, including dependency updates. No changes are required by the user.

version 1.2.0 (2021-01-27)

  • Sort nodes and groups before rendering
  • Add -postupdate argument

version 1.1.0 (2020-06-22)

  • Add Azure support
  • Fixed an AWS authentication issue where we set the AWS_SHARED_CREDENTIALS_FILE environment variable ourselves if systemd can’t retrieve it
  • Add --version and --v options to display the version number

version 1.0.4 (2019-06-07)

  • The node file argument can specify a template to format the output.

version 1.0.3 (2019-02-21)

  • Install the systemd service on Debian systems

version 1.0.2 (2018-12-24)

  • Add AWS support

version 1.0.1 (2018-10-15)

  • Fix an out-of-bounds exception
  • Failed HTTP request are now printing as comments to help debugging (vac and k8s modules)
  • When no group is found, a comment is printed on the first resolution, instead of looping silently

version 1.0.0 (2017-12-18)

  • initial release