SearchVarnish Enterprise
This VMOD provides functionality for using Varnish as a caching proxy in front of S3. This includes a dynamic backend director for S3 bucket endpoints with the following set of features:
Multiple IPs: Any number of IPs can be returned by DNS.Load-Balancing: Traffic is evenly balanced over each IP.Smart retries: Retried fetches always go to a different IP if possible.Dual-Stack: Both IPv4 and IPv6 are supported.Ready after load: DNS is resolved during vcl.load, no warmup needed.Async DNS resolution: IPs are kept up to date without blocking.Global resolution cache: DNS resolutions are cached across VCL reloads.To set up Varnish in front of a bucket at example.com, you can add the
following to your VCL
sub vcl_init {
new bucket = s3.director("bucket", "region", "example.com");
}
sub vcl_backend_fetch {
set bereq.backend = bucket.backend();
}
DNS is resolved using getaddrinfo() with the AF_UNSPEC flag set. This
means that backends will created based on IPv4 and/or IPv6 addresses. DNS is
resolved during vcl.load (blocking), which means that there is no period of
time after the VCL has been loaded where the director contains no backends. If
the initial DNS resolution fails or returns no results, we look for cached
results in the global endpoint cache. If no results are found, the VCL will fail
to load.
An asynchronous DNS resolver thread will continue to resolve the domain name at regular intervals. If these resolutions fail, the event will be logged, but otherwise nothing bad will happen. When the resolver thread makes a successful resolution, it will create a new set of backends, reusing currently active backends if possible. Each VCL gets one DNS resolver thread, and the resolver thread only runs while the VCL is warm.
Traffic is balanced by picking a random backend from the backend list. Each backend fetch task will remember which backend was initially picked, and will pick the next backend in the list on retries, Round-Robin style.
The buckets domain name is also used as the HTTP Host header for the backend
fetch. This is set when Varnish executes the directors resolve() function,
which happens after sub vcl_backend_fetch.
OBJECT director(STRING bucket, STRING region, STRING endpoint)
When created in sub vcl_init, this director resolves endpoint and
creates a backend list. The DNS resolver thread periodically re-resolves
endpoint and keeps the backend list up to date. endpoint is also used as
the Host header for backend fetches.
By default, backends are created with port number 80. endpoint may specify a
scheme or an explicit port number to use a different port. If both scheme and
port number are defined, the explicit port number takes precedence. For example,
setting the endpoint to “http://example.com:6081” will result in backends
with port number 6081.
Backends created with port number 443 are TLS-enabled, equivalent to
creating a static backend with .ssl = 1.
Parameters:
bucket: The name of the bucket.region: The name of the region.endpoint: The host name of your S3 endpoint.Arguments:
bucket accepts type STRING
region accepts type STRING
endpoint accepts type STRING
Type: Object
Returns: Object.
BACKEND .backend()
Returns a virtual backend object. Can be used to set either bereq.backend or
req.backend_hint.
Arguments: None
Type: Method
Returns: Backend
The s3 VMOD is available in Varnish Enterprise version 6.0.11r2 and later.