Search
Varnish Cache Plus

AWS VCL

Description

AWS VCL is a VCL based library for accessing AWS services. AWS VCL supports the following:

  • AWS Signature Version 4 request signing
  • AWS S3
  • AWS Secrets Manager

AWS VCL is available in Varnish Cache Plus 6.0.6r6 and later.

Examples

AWS S3 Backend

Manually setup signing parameters, sign a request and send it to an S3 backend:

import goto;

include "aws/init.vcl";
include "aws/sign.vcl";

sub vcl_init {
    # Setup AWS request signing parameters
    aws_config.set("aws_access_key_id", "AKAI...");
    aws_config.set("aws_secret_access_key", "XXXX...");
    aws_config.set("region", "us-west-2");
    aws_config.set("host", "my-bucket.s3-us-west-2.amazonaws.com");

    # S3 backend
    new s3 = goto.dns_director(aws_config.get("host"), ssl = true);
}

sub vcl_recv {
    set req.backend_hint = s3.backend();
    set req.http.Host = aws_config.get("host");
}

sub vcl_backend_fetch {
    call aws_sign_bereq;
}

AWS Secrets Manager

Auto initialize signing parameters from the AWS CLI config and read a secret from the Secrets Manager. Use it for request authorization.

include "aws/auto_init.vcl";
include "aws/secrets_manager.vcl";

sub vcl_init
{
    aws_secret.set("secret_id", "auth_token");
    call aws_get_secret;

    if(!aws_secret_vault.get("auth_token")) {
        return (fail("Couldn't get auth_token"));
    }
}

sub vcl_recv
{
    if (req.http.Authorization != aws_secret_vault.get("auth_token")) {
        return (synth(403));
    }
}

Configuration

Before AWS VCL can be used, it needs to be configured with the aws_access_key_id, aws_secret_access_key, and region. There are 2 methods of configuration, manual initialization and auto initialization. Manual initialization means that you supply the configuration keys via VCL. Auto initialization means that AWS VCL automatically reads the configuration keys from your AWS CLI config. Auto initialization mimics how other AWS SDK products function.

When doing manual initialization (VCL), please use the following include first:

include "aws/init.vcl";

When doing auto initialization (AWS CLI), use the following include first:

include "aws/auto_init.vcl";

After specifying your initialization include, include the AWS VCLs for the service(s) you require. Multiple AWS VCLs can be included.

VCL Initialization

VCL initialization requires setting the following three variables in vcl_init:

include "aws/init.vcl";

sub vcl_init {
    # Setup AWS request signing parameters
    aws_config.set("aws_access_key_id", "AKAI...");
    aws_config.set("aws_secret_access_key", "XXXX...");
    aws_config.set("region", "us-west-2");
}

Because initialization is VCL based, you are free to initialize these values using any means VCL provides. This includes reading keys from a file (vmod_kvstore) or even a remote service (vmod_http).

Auto Initialization using Environment Variables

The following environment variables are supported for auto initializing your AWS signing keys:

  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY
  • AWS_DEFAULT_REGION

If using systemd, these variables can be defined in your Varnish service file:

[Service]
Environment="AWS_ACCESS_KEY_ID="AKAI..."
Environment="AWS_SECRET_ACCESS_KEY=XXXX..."
Environment="AWS_DEFAULT_REGION=us-west-2"

Please see the systemd docs for more information.

If using a container management system like Docker or Kubernetes, please reference the container documentation on how to export environment variables to Varnish.

Once defined, be sure to use the auto initialization VCL:

include "aws/auto_init.vcl";

Note that VCL initialization can still be applied with this configuration. If auto initialization fails, you will get a VCL generated error when using the VCL API.

Auto Initialization using the AWS CLI Config

Auto initialization can also read AWS signing keys directly from the AWS CLI config. This config is generally written out to ~/.aws/credentials and ~/.aws/config.

To configure your signing keys, run:

> aws configure

For more information, please see the AWS CLI docs.

By default, auto initialization will attempt to find the AWS CLI configs at these locations, in the defined order. Once each config is found, the search is stopped.

credentials

  • $AWS_SHARED_CREDENTIALS_FILE
  • $AWS_HOME/.aws/credentials
  • /.aws/credentials
  • /etc/varnish/.aws/credentials

config

  • $AWS_CONFIG_FILE
  • $AWS_HOME/.aws/config
  • /.aws/config
  • /etc/varnish/.aws/config

You can override any of these locations by setting the following environment variables:

  • AWS_SHARED_CREDENTIALS_FILE
  • AWS_CONFIG_FILE
  • AWS_HOME

Please see the above section on how to set environment variables to be read by Varnish.

Note: When reading keys from a file, make sure the varnish user or group has read access to the necessary files and read+execute permission on the parent directories.

Once defined, be sure to use the auto initialization VCL:

include "aws/auto_init.vcl";

Note that VCL initialization can still be applied with this configuration. If auto initialization fails, you will get a VCL generated error when using the VCL API.

Multiple Profiles

AWS VCL supports multiple signing profiles. This lets you access multiple AWS services or AWS S3 buckets with a grainular set of AWS IAM roles.

If doing VCL initialization and you would like to setup profiles, simply prepend the profile name and an underscore to each variable name you define. So if you want to put your keys under profileABC, you would define your variables as profileABC_aws_access_key_id, profileABC_aws_secret_access_key, and profileABC_region. Repeat this process for all your different profiles.

If doing auto initialization, just use the --profile switch in the AWS CLI: aws configure --profile NAME

To setup a default profile used for all services, use the follow VCL initialization:

sub vcl_init {
    aws_config.set("profile", "profileABC");
}

If no default profile is set, the default profile is no profile. If a profile is used that cannot be found, you will get a VCL generated error when using the VCL API.

To setup a default profile for auto initialization, use the $AWS_PROFILE environment variable or simply configure a profile as the default using the AWS CLI (aws configure).

Debugging

The best way to debug access problems to AWS services is to simply look at the AWS response. AWS will log what went wrong in the response body. You can use xbody.log_body() if you would like to log the body on the Varnish server.

If that does not work, another option would be to try and access your service via the AWS CLI.

First, setup your signing keys (if you are auto initializing, this step can be skipped):

> aws configure

Next, access the service. If debugging S3:

> aws s3 ls

Secrets Manager:

> aws secretsmanager get-secret-value --secret-id secret_token

Please reference the AWS CLI documentation for more advanced command usage.

API

Request Signing

Include the following VCL after your AWS VCL init:

include "aws/sign.vcl";

aws_sign.set("profile", "");

Profile to use for the signing keys. Defaults to no profile or the global profile. Optional.

aws_sign.set("service", "s3");

Service to use in the AWS v4 signature. Defaults to s3. Optional.

aws_sign.set("header_regex", "");

Extra headers to add to the signature. To add multiple headers, use a regex pipe |. Defaults to no extra headers. Optional.

call aws_sign_bereq;

Sign the bereq. Must be called in vcl_backend_fetch.

call aws_sign_req_body;

Get the req.body signature. This must be done if you have a PUT or POST payload. Must be called in vcl_recv. Must call std.cache_req_body() before calling. Optional.

Secrets Manager

Include the following VCL after your AWS VCL init:

include "aws/secrets_manager.vcl";

aws_secret.set("secret_id", "%SECRET_NAME%");

The secret to lookup. %SECRET_NAME% is the name of the secret. Required.

aws_secret.set("version_stage", "");

The secret version to lookup. Defaults to not specified (AWS default). Optional.

aws_secret.set("alias", "");

The name to store the secret as in the aws_secret_vault. Defaults to %SECRET_NAME%. Useful when looking up and storing multiple versions of a secret (ie: key rotation). Optional.

Note: When looking up and storing multiple versions of a single key, the key lookup order should proceed from oldest to newest. Key check order should then proceed from newest to oldest. This ensures that there are no lapses in key coverage during rotation.

aws_secret.set("profile", "");

Profile to use for the signing keys. Defaults to no profile or the global profile. Optional.

call aws_get_secret;

Get the secret value from the Secrets Manager. Can be used in any VCL sub. Once called, the below variables are available.

aws_secret.get("secret");

The secret value. If there is an error, this value is empty. This value is only available to the current request. See aws_secret_vault for permanent access.

aws_secret.get("response_status");

The HTTP response status code from AWS. On a network error, this value is 0. This value is only available to the current request.

aws_secret.get("response_body");

The HTTP response body from AWS. On a network error, this value is empty. This value is only available to the current request.

aws_secret.get("response_code");

Internal network error code (non zero). If the HTTP request was successful, this value is 0. This value is only available to the current request.

aws_secret_vault.get("%SECRET_NAME%");

The last successful secret value. %SECRET_NAME% is the name of the secret, unless an alias is used. This value is available to all requests for the lifetime of the VCL, unless deleted. The previous value is kept if there is an error, otherwise this value will be empty.