Search
Varnish High Availability

varnish-discovery

Introduction

In the case of cloud instances or any other platform featuring clusters that can shrink and grow dynamically, the nodes.conf file needs to be updated and broadcaster needs to reread it when a change occurs. This is the role of varnish-discovery. It watches a given source (like the VAC API or DNS info) and creates or updates a nodes.conf file that tracks the relevant machines.

For processes that can’t actively watch their configuration files (like broadcaster does with -confwatch, discovery` can send a SIGHUP to any process as a signal that the configuration changed.

The package is varnish-plus-discovery.

Getting started

First, broadcaster must expose a pid file using the -pid argument. By default, broadcaster and its service file use /run/varnish-broadcaster/broadcaster.pid.

varnish-discovery supports multiple backends to generate a nodes.conf, each with its own set of switches (most are optional). To help you hit the ground running, here are a selection of examples.

Note: If you just need to create the file and then exit (for example, before you start broadcaster for the first time), just append -once to the commands.

AWS

-group specifies an autoscaling group on AWS. This feature relies on the AWS SDK, so if awscli is properly configured and works, this subcommand only requires common arguments:

/usr/bin/varnish-discovery aws                                       \
                  -nodefile /etc/varnish/nodes.conf                 \
                  -group $DOMAIN_NAME

Azure

-group specifies a virtual machine scale set on Azure. This feature relies on the Azure SDK, so if the az CLI is properly configured and works, this subcommand will only require common arguments:

/usr/bin/varnish-discovery azure                                     \
                  -nodefile /etc/varnish/nodes.conf                 \
                  -group $SCALESET_NAME

DNS

You’ll need to specify a domain name that will be checked every now and then (see -every under Common options:

/usr/bin/varnish-discovery dns                                       \
                  -nodefile /etc/varnish/nodes.conf                 \
                  -group $DOMAIN_NAME

Kubernetes

-group here is an endpoint, as listed by kubectl get endpoints.

If running inside a pod, discovery will be able to find info about the namespace, cacert, and token to access the API server, so there’s no need to specify them:

/usr/bin/varnish-discovery k8s                                       \
                  -nodefile /etc/varnish/nodes.conf                 \
                  -server "https://$K8S_API/"                       \
                  -group $ENDPOINT                                  \
                  -port 0

-port should almost always be 0, which tells varnish-discovery to use the first port listed for each port. If you don’t specify the port, discovery will assume it’s 80 or 443, depending on the protocol you use (http if not specified).

Important note: Any pod that doesn’t listen to TCP on the selected port will be omitted.

If your node is listed by more than one endpoint, you must use -group; otherwise, the node will be present in multiple clusters and broadcaster won’t know what to do with it.

VAC

For the Varnish Administration Console, we only need to point to the API’s address and varnish-discovery will be able to figure out its own cluster:

/usr/bin/varnish-discovery vac                                       \
                  -nodefile /etc/varnish/nodes.conf                 \
                  -server $LOGIN:$PASSWD@$VAC_IP

Service files

All the packages offer service files - how you edit them depends on your platform.

For sysv, edit varnish-discovery.params located in either /etc/deafult/ or /etc/sysconfig to change the ENABLE and DAEMON_OPTS= variables.

For systemd, create the file /etc/systemd/system/varnish-discovery.service.d/exec.conf and redefine the ExecStart parameter. For example:

cat > /etc/systemd/system/varnish-discovery.service.d/exec.conf << EOF
[Service]
ExecStart=
ExecStart=/usr/bin/varnish-discovery dns -group localhost -ipv4 -nodefile /etc/varnish/nodes.conf
EOF
systemctl daemon-reload

Options

Common options

  • -every DURATION (default: 2s)

    Specifies how frequently varnish-discovery should contact the source. For DNS and VAC, it’s the duration between two requests (from start to start). For Kubernetes, which uses long-poll, it tells varnish-discovery how long it should wait before trying again in case of failure (again, the time since the start of the failed request).

  • -group NAME

    Can be used multiple times and tells varnish-discovery the name to look for. The meaning is different depending on the source, but it always represents a handle behind which multiple IPs can hide::

    - AWS: corresponds to autoscaling group
    - Azure: corresponds to a virtual machine scale set
    - DNS: corresponds to hostname
    - k8s: corresponds to endpoint
    - VAC: corresponds to group
    
  • -ipv4 -ipv6

    Restricts what version of the IP protocol should be used (useful in the DNS case). If none are supplied, use both.

  • -nodefile [TEMPLATE:]NODEFILE (default: -)

    Outputs the cluster info in NODEFILE with “-” meaning stdout. If TEMPLATE is specified, it points to a go template used to format the node file. More info is below.

  • -once

    By default, varnish-discovery will monitor its source of information indefinitely, but with this option, it only does it once before exiting. A non-zero return indicates an error during the run.

  • -proto PROTO (default: http)

    If the source of information doesn’t explicate the protocol used, we fallback to PROTO.

  • -port PORT

    If the source of information doesn’t explicate the port used, we fallback to PORT. If the port is omitted, it’s inferred from the protocol. If “0” is specified, the first port is used (useful for Kubernetes).

  • -postupdate COMMAND

    After a group update, run COMMAND. This is useful to reload a process or update a status file. For example:

    varnish-discovery dns -group example.com -postupdate "touch /tmp/updated"
    

    COMMAND is parsed as a string following shell that quotes logic. -postupdate can be specified multiple times.

    Note: discovery will execute all commands in parallel, but will wait for them to return before continuing. Also, if COMMAND doesn’t use an absolute filename for its binary, it must be in the PATH.

  • ``-warnpid PIDFILE`

    As discovery usually generates a configuration file for another process, it needs a way to tell that process to reload said configuration. -warnpid does just this, sending a SIGHUP to the pid found in PIDFILE.

    In practice, when use with broadcaster, this option is deprecated thanks to the broadcasters’s -confwatch argument that will automatically pick up new versions on its configuration file. This option can be specified multiple times.

    Reminder: if using -warnipid to contact a process in another container, make sure you are sharing the PID namespace (shareProcessNamespace in kubernetes);

  • -version or -v

    This displays the version number and exit.

AWS

  • -region REGION

    If the AWS region hasn’t been configured yet (using aws configure), you can specify it here.

Azure

For setting up Azure, these environment variables need to be configured:

AZURE_TENANT_ID="<app-tenant-id>"
AZURE_SUBSCRIPTION_ID="<subscription-id>"
AZURE_CLIENT_ID="<app-client-id>"
AZURE_CLIENT_SECRET="<app-secret>"
  • -resourcegroup NAME

    If the Azure base resource group name hasn’t been configured yet (using az configure or by setting the AZURE_BASE_GROUP_NAME environment variable), you can specify it here.

  • -subscriptionid ID

    If the ID of your Azure subscription hasn’t been configured yet (using az configure or by setting the AZURE_SUBSCRIPTION_ID environment variable), you can specify it here.

DNS

  • -no-hostname

    By default, varnish-discovery tries to find the IP corresponding to the local machine in the resolved list and replace it with the local hostname. This switch disables this find-and-replace behavior.

Kubernetes

  • -server URL (https://kubernetes/)

  • -token PATH (/var/run/secrets/kubernetes.io/serviceaccount/token)

  • -cacert PATH (/var/run/secrets/kubernetes.io/serviceaccount/ca.crt)

  • -namespace PATH (/var/run/secrets/kubernetes.io/serviceaccount/namespace)

    When operating inside a pod, varnish-discovery can communicate with the Kubernetes API using the conditional default. However, they can all be overridden to work outside of a pod or to connect to a non-standard URL.

  • -group ENDPOINT

    In k8s’ case, if no endpoint is specified, varnish-discovery will use the one owning the pod it’s running on. This allows for minimal configuration based on context.

VAC

  • -server URL

    The location of the VAC API. The URL must include the login and password to correctly authenticate to the API.

  • -no-hostname

    As with the DNS command, varnish-discovery will try and replace the local IP with the local hostname.

Default template

The default template to generate the nodes.conf file is:

# this was generated at: {{ .Time }}
# using command: "{{ .Command }}"

{{if .Err}}# node generation failed because an error occured: {{ FormatReplace (print .Err) "\n" "\n# " }}
{{ else }}{{ range $i, $grp := .List }}{{if $i}}
{{ end }}[{{ $grp.Name }}]
{{ if not $grp.Nodes}}# this cluster is empty
{{ else }}{{ range $j, $node := $grp.Nodes }}{{ $node.Name }} = {{ $node.Proto }}://{{ FormatIP $node.IP }}:{{ $node.Port }}
{{ end }}{{ end }}{{ end }}{{end}}

The FormatReplace function ensures that new lines in a string are prefixed by a #, which is used to print error messages in comments.

The template uses a Groups structure that looks like this:

type Groups struct {
	Time    time.Time
	Err     error
	Command string
	List    []struct{
		Name  string
		Nodes []struct{
			Name  string
			Proto string
			Port  uint64
			IP    net.IP
		}
	}
}

Changelog

Version 1.5.0 (2023-02-10)

  • Maintenance release; includes:
    • Azure SDK minor version upgrade
    • AWS SDK major version upgrade from v1 to v2
    • Go 1.20.0

Version 1.4.0 (2022-06-29)

  • Maintenance release; includes AWS SDK version upgrade

Version 1.3.0 (2022-02-24)

  • Maintenance release; includes dependency updates and no changes are required by the user.

Version 1.2.0 (2021-01-27)

  • Sorts nodes and groups before rendering
  • Added -postupdate argument

Version 1.1.0 (2020-06-22)

  • Added Azure support
  • Fixed an AWS authentication issue; user can set the AWS_SHARED_CREDENTIALS_FILE environment variable if systemd can’t retrieve it
  • Added --version and --v options to display the version number

Version 1.0.4 (2019-06-07)

  • Node file argument can specify a template to format the output

Version 1.0.3 (2019-02-21)

  • Added systemd service install for Debian

Version 1.0.2 (2018-12-24)

  • Added AWS support

Version 1.0.1 (2018-10-15)

  • Fixed an out-of-bounds exception
  • Failed HTTP request now prints as comments to help with debugging (VAC and k8s modules)
  • When no group is found, a comment is printed on the first resolution instead of looping silently.

Version 1.0.0 (2017-12-18)

  • Initial release