SearchVarnish Enterprise
AWS VCL is a VCL based library for accessing AWS services. AWS VCL supports the following:
AWS VCL is available in Varnish Enterprise 6.0.6r6 and later.
Manually setup signing parameters, sign a request and send it to an S3 backend:
import goto;
include "aws/init.vcl";
include "aws/sign.vcl";
sub vcl_init {
# Setup AWS request signing parameters
aws_config.set("aws_access_key_id", "AKAI...");
aws_config.set("aws_secret_access_key", "XXXX...");
aws_config.set("region", "us-west-2");
aws_config.set("host", "my-bucket.s3-us-west-2.amazonaws.com");
# S3 backend
new s3 = goto.dns_director(aws_config.get("host"), ssl = true);
}
sub vcl_recv {
set req.backend_hint = s3.backend();
set req.http.Host = aws_config.get("host");
}
sub vcl_backend_fetch {
call aws_sign_bereq;
}
Auto initialize signing parameters from the AWS CLI config and read a secret from the Secrets Manager. Use it for request authorization.
include "aws/auto_init.vcl";
include "aws/secrets_manager.vcl";
sub vcl_init
{
aws_secret.set("secret_id", "auth_token");
call aws_get_secret;
if(!aws_secret_vault.get("auth_token")) {
return (fail("Couldn't get auth_token"));
}
}
sub vcl_recv
{
if (req.http.Authorization != aws_secret_vault.get("auth_token")) {
return (synth(403));
}
}
Before AWS VCL can be used, it needs to be configured with the aws_access_key_id, aws_secret_access_key,
and region. There are 2 methods of configuration, manual initialization and auto initialization.
Manual initialization means that you supply the configuration keys via VCL.
Auto initialization means that AWS VCL automatically reads the configuration keys from your AWS CLI config.
Auto initialization mimics how other AWS SDK products function.
When doing manual initialization (VCL), please use the following include first:
include "aws/init.vcl";
When doing auto initialization (AWS CLI), use the following include first:
include "aws/auto_init.vcl";
After specifying your initialization include, include the AWS VCLs for the service(s) you require. Multiple AWS VCLs can be included.
VCL initialization requires setting the following three variables in vcl_init:
include "aws/init.vcl";
sub vcl_init {
# Setup AWS request signing parameters
aws_config.set("aws_access_key_id", "AKAI...");
aws_config.set("aws_secret_access_key", "XXXX...");
aws_config.set("region", "us-west-2");
}
Because initialization is VCL based, you are free to initialize these values using any means VCL provides. This includes reading keys from a file (vmod_kvstore) or even a remote service (vmod_http).
The following environment variables are supported for auto initializing your AWS signing keys:
AWS_ACCESS_KEY_IDAWS_SECRET_ACCESS_KEYAWS_DEFAULT_REGIONIf using systemd, these variables can be defined in your Varnish service file:
[Service]
Environment="AWS_ACCESS_KEY_ID="AKAI..."
Environment="AWS_SECRET_ACCESS_KEY=XXXX..."
Environment="AWS_DEFAULT_REGION=us-west-2"
Please see the systemd docs for more information.
If using a container management system like Docker or Kubernetes, please reference the container documentation on how to export environment variables to Varnish.
Once defined, be sure to use the auto initialization VCL:
include "aws/auto_init.vcl";
Note that VCL initialization can still be applied with this configuration. If auto initialization fails, you will get a VCL generated error when using the VCL API.
Auto initialization can also read AWS signing keys directly from the AWS CLI config.
This config is generally written out to ~/.aws/credentials and ~/.aws/config.
To configure your signing keys, run:
> aws configure
For more information, please see the AWS CLI docs.
By default, auto initialization will attempt to find the AWS CLI configs at these locations, in the defined order. Once each config is found, the search is stopped.
credentials
$AWS_SHARED_CREDENTIALS_FILE$AWS_HOME/.aws/credentialsconfig
$AWS_CONFIG_FILE$AWS_HOME/.aws/configYou can override any of these locations by setting the following environment variables:
AWS_SHARED_CREDENTIALS_FILEAWS_CONFIG_FILEAWS_HOMEPlease see the above section on how to set environment variables to be read by Varnish.
Note: When reading keys from a file, make sure the varnish user or group has read access to the necessary files and
read+execute permission on the parent directories.
Once defined, be sure to use the auto initialization VCL:
include "aws/auto_init.vcl";
Note that VCL initialization can still be applied with this configuration. If auto initialization fails, you will get a VCL generated error when using the VCL API.
AWS VCL supports multiple signing profiles. This lets you access multiple AWS services or AWS S3 buckets with a grainular set of AWS IAM roles.
If doing VCL initialization and you would like to setup profiles,
simply prepend the profile name and an underscore to each variable name you define.
So if you want to put your keys under profileABC, you would define your variables as
profileABC_aws_access_key_id, profileABC_aws_secret_access_key, and profileABC_region.
Repeat this process for all your different profiles.
If doing auto initialization, just use the --profile switch in the AWS CLI: aws configure --profile NAME
To setup a default profile used for all services, use the follow VCL initialization:
sub vcl_init {
aws_config.set("profile", "profileABC");
}
If no default profile is set, the default profile is no profile. If a profile is used that cannot be found, you will get a VCL generated error when using the VCL API.
To setup a default profile for auto initialization, use the $AWS_PROFILE environment variable
or simply configure a profile as the default using the AWS CLI (aws configure).
The best way to debug access problems to AWS services is to simply look at the AWS response. AWS will log what went wrong in the response body. You can use xbody.log_body() if you would like to log the body on the Varnish server.
If that does not work, another option would be to try and access your service via the AWS CLI.
First, setup your signing keys (if you are auto initializing, this step can be skipped):
> aws configure
Next, access the service. If debugging S3:
> aws s3 ls
Secrets Manager:
> aws secretsmanager get-secret-value --secret-id secret_token
Please reference the AWS CLI documentation for more advanced command usage.
Include the following VCL after your AWS VCL init:
include "aws/sign.vcl";
aws_sign.set("profile", "");Profile to use for the signing keys. Defaults to no profile or the global profile. Optional.
aws_sign.set("service", "s3");Service to use in the AWS v4 signature. Defaults to s3. Optional.
aws_sign.set("header_regex", "");Extra headers to add to the signature. To add multiple headers, use a regex pipe |.
Defaults to no extra headers. Optional.
call aws_sign_bereq;Sign the bereq. Must be called in vcl_backend_fetch.
call aws_sign_req_body;Get the req.body signature. This must be done if you have a PUT or POST payload. Must be called in vcl_recv.
Must call std.cache_req_body() before calling.
Optional.
Include the following VCL after your AWS VCL init:
include "aws/secrets_manager.vcl";
aws_secret.set("secret_id", "%SECRET_NAME%");The secret to lookup. %SECRET_NAME% is the name of the secret. Required.
aws_secret.set("version_stage", "");The secret version to lookup. Defaults to not specified (AWS default). Optional.
aws_secret.set("alias", "");The name to store the secret as in the aws_secret_vault. Defaults to %SECRET_NAME%.
Useful when looking up and storing multiple versions of a secret (ie: key rotation).
Optional.
Note: When looking up and storing multiple versions of a single key, the key lookup order should proceed from oldest to newest. Key check order should then proceed from newest to oldest. This ensures that there are no lapses in key coverage during rotation.
aws_secret.set("profile", "");Profile to use for the signing keys. Defaults to no profile or the global profile. Optional.
call aws_get_secret;Get the secret value from the Secrets Manager. Can be used in any VCL sub. Once called, the below variables are available.
aws_secret.get("secret");The secret value. If there is an error, this value is empty.
This value is only available to the current request.
See aws_secret_vault for permanent access.
aws_secret.get("response_status");The HTTP response status code from AWS. On a network error, this value is 0.
This value is only available to the current request.
aws_secret.get("response_body");The HTTP response body from AWS. On a network error, this value is empty. This value is only available to the current request.
aws_secret.get("response_code");Internal network error code (non zero). If the HTTP request was successful, this value is 0.
This value is only available to the current request.
aws_secret_vault.get("%SECRET_NAME%");The last successful secret value. %SECRET_NAME% is the name of the secret, unless an alias is used.
This value is available to all requests for the lifetime of the VCL, unless deleted.
The previous value is kept if there is an error, otherwise this value will be empty.
GCP supports the AWS Signature Version 4 request signing. This AWS VCL will also work for GCP with the following changes.
You must get an HMAC secret and access key from GCP. The secret is the
equivalent of the aws_secret_access_key. The access key is the equivalent
of the aws_access_key_id.
You must use the Google signing algorithm GOOG4-RSA-SHA256. The algorithm can
be specified with an additional parameter algorithm.
GCP signing parameters cannot be auto initialized from a CLI config Secrets Manager like with AWS.
import goto;
include "aws/init.vcl";
include "aws/sign.vcl";
sub vcl_init {
# Setup GCP request signing parameters
# Secret is: "so5Z4m..."
# Access key is: "GOOG1E..."
aws_config.set("aws_access_key_id", "GOOG1E...");
aws_config.set("aws_secret_access_key", "so5Z4m...");
aws_config.set("region", "us-west-2");
aws_config.set("algorithm", "GOOG4-RSA-SHA256");
aws_config.set("host", "my-bucket.storage.googleapis.com");
# S3 backend
new s3 = goto.dns_director(aws_config.get("host"), ssl = true);
}
sub vcl_recv {
set req.backend_hint = s3.backend();
set req.http.Host = aws_config.get("host");
}
sub vcl_backend_fetch {
call aws_sign_bereq;
}