HTTP communication (http)

Description

The http vmod allows for HTTP communication to be made to other services from VCL. This includes communication back to Varnish in the form of a new request. vmod-http supports both synchronous and asynchronous calls and has automatic loop detection and prefetch URL generation capabilities.

vmod-http supports all string and integer CURLOPT options via http.req_set_sparam() and http.req_set_iparam().

Examples

Video Prefetch

Prefetch the next video segment:

vcl 4.0;

import http;

sub vcl_recv
{
  if (req.url ~ "^/vod/") {
    http.init(0);

    http.req_copy_headers(0);
    http.req_set_method(0, "HEAD");

    // Generate the next URL by incrementing the first
    // number sequence found and send it back to Varnish.
    // If no number sequence is found, the request is skipped.
    http.req_set_url(0, http.prefetch_next_url());

    http.req_send_and_finish(0);
  }
}

Search Prefetch

Prefetch the next two search pages using the page query parameter:

vcl 4.0;

import http;

sub vcl_recv
{
  http.init(0);

  // Allow this request to loop at most twice
  http.req_set_max_loops(0, 2);

  http.req_copy_headers(0);
  http.req_set_method(0, "HEAD");

  // Generate the next URL by incrementing the first number
  // sequence found after 'page=' and send it back to Varnish.
  // If 'page=' and the following number sequence is not found,
  // the request is skipped.
  http.req_set_url(0, http.prefetch_next_url("page="));

  http.req_send_and_finish(0);
}

Link Prefetch

Prefetch using a Link response header hint.

Example: Link: </images/big.jpeg>; rel=prefetch

vcl 4.0;

import http;

sub vcl_recv {
  // Store Varnish's local address for later use
  set req.http.X-prefetch = http.varnish_url("/");
}

sub vcl_backend_response {
  if (beresp.http.Link ~ "<.+>.*(prefetch|next)") {
    // Pull out the Link URL
    set bereq.http.X-link = regsub(beresp.http.Link, "^.*<([^>]*)>.*$", "\1");
    set bereq.http.X-prefetch = regsub(bereq.http.X-prefetch, "/$", bereq.http.X-link);

    // Prefetch the Link URL back through Varnish
    http.init(0);
    http.req_copy_headers(0);
    http.req_set_url(0, bereq.http.X-prefetch);
    http.req_send_and_finish(0);
  }
}

HTTP Request

Send a request to another service and include the response cookie.

vcl 4.0;

import cookieplus;
import http;

sub vcl_recv
{
  // We need to initialize the HTTP object here, as it is going to be used
  // in vcl_deliver with ``http.req_sent(0)``, which requires the object to be
  // initialized before usage.
  http.init(0);

  // Send the request if we do not have a session
  // We will read the response later
  if (!cookieplus.get("session")) {
    http.req_copy_headers(0);
    http.req_set_url(0, "https://example.com/api/authorize");

    http.req_send(0);
  }
}

sub vcl_deliver
{
  // Get the session response cookies
  if (http.req_sent(0)) {
    http.resp_wait(0);

    if (http.resp_get_status(0) == 200) {
      http.resp_copy_headers(0, "Set-Cookie");
    } else {
      return(synth(401, "UNAUTHORIZED"));
    }
  }
}

Request Mirroring

Send a read only copy of all requests to another service.

vcl 4.0;

import http;
import std;

sub vcl_recv
{
  std.cache_req_body(100KB);

  http.init(0);

  http.req_copy_headers(0);
  http.req_set_header(0, "X-Varnish-fork", http.version());
  http.req_set_max_loops(0, -1); // Remove loop counter header
  http.req_set_method(0, req.method);
  http.req_copy_body(0);
  http.req_set_url(0, "http://service_host" + req.url);

  http.req_send_and_finish(0);
}

POST Request

Send a POST request with a custom body using CURLOPT_POSTFIELDS. The CURLOPT_ prefix is optional on all curl options.

vcl 4.0;

import http;

sub vcl_deliver
{
  http.init(0);

  http.req_set_method(0, "POST");
  http.req_set_url(0, "https://some.host/data/service");
  http.req_set_sparam(0, "POSTFIELDS", "postdata is here");

  http.req_send_and_finish(0);
}

JSON API Call With Caching

vcl 4.0;

import goto;
import http;
import json;

sub vcl_init {
  new api = goto.dns_director("api.example.com");
}

sub vcl_recv {
  if (req.http.Host == "www.example.com") {
    // Pull JSON through Varnish
    http.init(0);
    http.req_copy_headers(0);
    http.req_set_url(0, http.varnish_url("/auth/"));
    http.req_set_header(0, "Host", "api.example.com");
    http.req_send(0);
    http.resp_wait(0);

    // Invalid response
    if (http.resp_get_status(0) != 200) {
      return (synth(401));
    }

    // Parse the response
    json.parse(http.resp_get_body(0));

    // Invalid authorization
    if (!json.is_object() || !json.get("auth") || json.get("error")) {
      return (synth(401));
    }

    // Pull data out of the JSON response
    set req.http.X-auth = json.get("auth");
    unset req.http.Authorization;
  } else if (req.http.Host == "api.example.com") {
    set req.backend_hint = api.backend();
    return (hash);
  }
}

API

init

VOID init(INT name, ENUM {NONE, LOW, MEDIUM, HIGH, DEBUG} log_level = HIGH, BOOL validate_charset = 0)

Initialize an http call, by name. The name must be used on all calls which reference this instance, it must also be non-negative and less than HTTP_MAX_OBJECTS. Referencing an uninitialized name in a http call will result in a Varnish assertion error.

When validate_charset is true, the URL, method and headers passed in will be checked for valid characters. Invalid characters (like a control character in a header field) will cause VCL failure.

Arguments:

• name accepts type INT

• validate_charset accepts type BOOL with a default value of 0 optional

• log_level is an ENUM that accepts values of NONE, LOW, MEDIUM, HIGH, and DEBUG with a default value of HIGH optional

Type: Function

Returns: None

prefetch_next_url

STRING prefetch_next_url(STRING prefix = "", STRING url_prefix = "", STRING url = "", INT count = 1, ENUM {DECIMAL, HEX, HEX_UPPER} base = DECIMAL)

Generate the next prefetch URL. This works by finding a numeric sequence in the current URL and incrementing it by count. If no numeric sequence is found, an empty URL is returned which will cause the http request to be skipped. Cannot be used in a backend context (sub vcl_backend_*).

If prefix is used, the numeric sequence used is searched for after prefix. prefix is a regular expression. If prefix is not found, an empty URL is returned. This value is static and cannot change between calls.

url_prefix is the URL prefix. It is appended before req.url to compose the full URL needed for the http request. It must start with http:// or https://, followed by a hostname or IP, and an optional port. It must not contain a trailing slash character, /. If not defined, server.ip and std.port(server.ip) are used with http://. If the port detected is 443, https:// is used.

count is the amount to add to the numeric sequence found when generating the next prefetch URL. Defaults to 1.

base is the format that the url is in. Either DECIMAL, HEX, or HEX_UPPER. DECIMAL indicates that the url string will be in decimal, base 10. HEX indicates that the URL string will be in hexadecimal, base 16. It’s alpha characters of the string are lower case. HEX_UPPER is the same as HEX but the alpha characters are upper case. If not defined, base will be set to DECIMAL.

Arguments:

• prefix accepts type STRING with a default value of empty. optional

• url_prefix accepts type STRING with a default value of empty. optional

• url accepts type STRING with a default value of empty. optional

• count accepts type INT with a default value of 1 optional

• base is an ENUM that accepts values of DECIMAL, HEX, and HEX_UPPER with a default value of DECIMAL optional

Type: Function

Returns: String

Restricted to: client

varnish_url

STRING varnish_url(STRING url)

Generate a Varnish callback URL. This is done by appending the URL to server.ip and std.port(server.ip) with http:// as the protocol. If the port detected is 443, https:// is used. Cannot be used in a backend context (sub vcl_backend_*).

Arguments:

• url accepts type STRING

Type: Function

Returns: String

Restricted to: client

backend_url

STRING backend_url(BACKEND be, STRING url)

Generate a URL from a backend or director. This is done by appending the URL to the resolved backend’s IP address and port. If the SSL flag is set, https:// is used as the protocol. Otherwise, http:// is used as the protocol. This can only be used in sub vcl_backend_fetch.

Arguments:

• be accepts type BACKEND

• url accepts type STRING

Type: Function

Returns: String

Restricted to: vcl_backend_fetch

req_set_url

VOID req_set_url(INT name, STRING url)

Set the URL for a request. If url is empty, the request is not performed. url must start with http:// or https:// and contain the host or IP destination and the trailing URL.

When the URL has an IPv6 address, it must be enclosed between square brackets:

  http.set_url(0, "http://[::1]:8080/auth");

The other functions http.varnish_url(), http.backend_url() and http.prefetch_next_url() already enclose IPv6 addresses by default.

Arguments:

• name accepts type INT

• url accepts type STRING

Type: Function

Returns: None

req_set_method

VOID req_set_method(INT name, STRING method)

Set the request method. E.g.: GET, HEAD, POST.

Arguments:

• name accepts type INT

• method accepts type STRING

Type: Function

Returns: None

req_set_header

VOID req_set_header(INT name, STRING header, STRING value)

Set a request header.

Arguments:

• name accepts type INT

• header accepts type STRING

• value accepts type STRING

Type: Function

Returns: None

req_get_header

STRING req_get_header(INT name, STRING header_name, STRING default = 0)

Copy all Varnish request headers into this request.

Arguments:

• name accepts type INT

• header_name accepts type STRING

• default accepts type STRING with a default value of 0 optional

Type: Function

Returns: String

req_copy_headers

VOID req_copy_headers(INT name, STRING list = "")

Copy all Varnish request headers into this request. If list is blank then all request headers are copied over. Defaults to copying all headers.

Arguments:

• name accepts type INT

• list accepts type STRING with a default value of empty. optional

Type: Function

Returns: None

Restricted to: client, backend

req_copy_body

VOID req_copy_body(INT name)

Copy the request body to this request. std.cache_req_body() must be called before using this function.

Arguments:

• name accepts type INT

Type: Function

Returns: None

Restricted to: client, backend

req_unset_header

VOID req_unset_header(INT name, STRING header)

Unset a request header.

Arguments:

• name accepts type INT

• header accepts type STRING

Type: Function

Returns: None

req_set_sparam

VOID req_set_sparam(INT name, STRING param, STRING value, INT debug_param = 0)

Set a string parameter for this request. Please see CURLOPT options for supported parameters. The CURLOPT_ prefix is optional.

Arguments:

• name accepts type INT

• param accepts type STRING

• value accepts type STRING

• debug_param accepts type INT with a default value of 0 optional

Type: Function

Returns: None

req_set_iparam

VOID req_set_iparam(INT name, STRING param, INT value, INT debug_param = 0)

Set an integer parameter for this request. Please see CURLOPT options for supported parameters. The CURLOPT_ prefix is optional.

Arguments:

• name accepts type INT

• param accepts type STRING

• value accepts type INT

• debug_param accepts type INT with a default value of 0 optional

Type: Function

Returns: None

req_set_max_loops

VOID req_set_max_loops(INT name, INT loops)

Set the maximum number of times this request can loop when doing a request back to Varnish. This is done via a special request header, VMOD-http-loops, which keeps track of the number of times the request passes through this VMOD. This has a default value of 1, meaning this VMOD will only make a single call to Varnish. Increasing this value in conjunction with prefetch_next_url() allows multiple URLs to be prefetched from a single client call.

Arguments:

• name accepts type INT

• loops accepts type INT

Type: Function

Returns: None

req_get_loops

INT req_get_loops(INT name)

Get the current loop iteration for this request. The first request starts at 0 and the first loop is 1.

Arguments:

• name accepts type INT

Type: Function

Returns: Int

req_is_loop

BOOL req_is_loop(INT name)

Return true if the request has looped at least once.

Arguments:

• name accepts type INT

Type: Function

Returns: Bool

req_send

BOOL req_send(INT name, BOOL resp_skip_body = 0)

Send the request.

If resp_skip_body is true then the body of the response will be ignored. Once a response is ready, the calls accessing the body or body length will behave as if the response had no body.

Arguments:

• name accepts type INT

• resp_skip_body accepts type BOOL with a default value of 0 optional

Type: Function

Returns: Bool

req_send_and_finish

BOOL req_send_and_finish(INT name, BOOL flush = 0)

Send the request and finish the name. Any overhead associated with response parsing will be skipped. The request will be safely processed in the background and finished when completed.

name cannot be used after calling this function.

Arguments:

• name accepts type INT

• flush accepts type BOOL with a default value of 0 optional

Type: Function

Returns: Bool

finish

VOID finish(INT name)

Cleanup and finish an http call. All resources associated with the http call will be released, including the name. Referencing the name again will result in a Varnish assertion error. name can be safely reused in another init() call.

Calling finish() on an http request is optional. All http calls will be safely finished when both the parent request and the actual http call are completed.

Note: A negative value removes the loop counting header and potentially allows unlimited loops.

Arguments:

• name accepts type INT

Type: Function

Returns: None

req_sent

BOOL req_sent(INT name)

Has the request been sent? If true, it is safe to call resp_wait(). Useful if a request has been conditionally sent.

Arguments:

• name accepts type INT

Type: Function

Returns: Bool

resp_is_ready

BOOL resp_is_ready(INT name)

Is the response ready? Errors are considered ready responses. Can be called at anytime, before or after the request has been sent.

Arguments:

• name accepts type INT

Type: Function

Returns: Bool

resp_wait

VOID resp_wait(INT name)

Wait for a response. When this function returns, the response is ready. Errors are considered ready responses. Can only be called after a request has been sent.

Arguments:

• name accepts type INT

Type: Function

Returns: None

resp_get_status

INT resp_get_status(INT name)

Get the status code for a response. If no response status was received, due to error, this function will return 0. Can only be called after a response is ready.

Arguments:

• name accepts type INT

Type: Function

Returns: Int

resp_get_errorcode

INT resp_get_errorcode(INT name)

Get the internal response error code. If no error has happened, this function will return 0. Otherwise a nonzero error code will be returned. Can only be called after a response is ready.

Arguments:

• name accepts type INT

Type: Function

Returns: Int

resp_get_reason

STRING resp_get_reason(INT name)

Get the response reason. Can only be called after a response is ready. This is the string after the response status, ex: 200 OK, the reason is OK.

Arguments:

• name accepts type INT

Type: Function

Returns: String

resp_get_header

STRING resp_get_header(INT name, STRING header_name, STRING default = 0)

Get a response header. If not found, default is returned. Can only be called after a response is ready.

Arguments:

• name accepts type INT

• header_name accepts type STRING

• default accepts type STRING with a default value of 0 optional

Type: Function

Returns: String

resp_copy_headers

VOID resp_copy_headers(INT name, STRING header_name = "")

Copy all matching response headers into the Varnish response. Can only be called after a response is ready. Cannot be called from vcl_recv, as the Varnish response has not been initialized yet.

Arguments:

• name accepts type INT

• header_name accepts type STRING with a default value of empty. optional

Type: Function

Returns: None

Restricted to: vcl_deliver, vcl_synth, vcl_backend_response

resp_get_body_blob

BLOB resp_get_body_blob(INT name)

Get the response body as a blob. Can only be called after a response is ready.

Arguments:

• name accepts type INT

Type: Function

Returns: Blob

resp_get_body

STRING resp_get_body(INT name)

Get the response body. Can only be called after a response is ready.

Arguments:

• name accepts type INT

Type: Function

Returns: String

resp_get_body_len

INT resp_get_body_len(INT name)

Get the response body length. Can only be called after a response is ready.

Arguments:

• name accepts type INT

Type: Function

Returns: Int

version

STRING version()

Get the vmod-http version.

Arguments: None

Type: Function

Returns: String

set_log_level

VOID set_log_level(INT name, ENUM {NONE, LOW, MEDIUM, HIGH, DEBUG} log_level)

Change the logging level of a request.

Arguments:

• name accepts type INT

• log_level is an ENUM that accepts values of NONE, LOW, MEDIUM, HIGH, and DEBUG

Type: Function

Returns: None

debug_print

VOID debug_print(INT name)

Print the internal vmod-http stats.

Arguments:

• name accepts type INT

Type: Function

Returns: None

get_request_count

INT get_request_count(BOOL async_only = 0)

Returns the number of active HTTP requests, or the length of the abandon queue.

GLOBAL PARAMETERS

This vmod also exposes some parameters that are set at the global level, either using the -p argument in the varnishd command line, or with varnishadm param.set while varnishd is running (in which case the value won’t persist after a restart).

vmod_http_max_conn

• Default: 25 connections

Limits the number of connections kept open for reuse in subsequent tasks.

Each vmod_http thread has its own connection cache and will prioritize reusing connections from the cache if possible. A new connection is opened if there are no available connections in the cache.

This parameter can be increased to reduce the number of new connection establishments in vmod_http at the cost of a higher number of long-lived connections. For more details, see documentation of libcurl’s s CURLMOPT_MAXCONNECTS option.

vmod_http_max_tasks_total

• Default: 1000 tasks

Limits the number of concurrent tasks across all vmod_http threads.

A task is either a synchronous or a asynchronous request initiated by http.req_send() or http.req_send_and_finish(). The task is completed when the response has been fully received or it has failed.

This parameter can be adjusted to limit concurrent vmod_http requests. It applies per VCL, so each loaded VCL will have its own limit. If the limit is reached, the new task is aborted and the handle_limited varnishstat counter is incremented.

vmod_http_min_tasks_thread

• Default: 100 tasks

Threshold for delegating tasks to multiple threads.

Threads in vmod_http can manage many tasks at once by active polling. To minimize the resource usage of vmod_http when there are few concurrent tasks to perform, tasks are first delegated to only one thread. When that thread has reached its threshold, new tasks are delegated to another thread. If all threads have reached their threshold, new tasks are delegated to the least busy thread.

This parameter can be adjusted to influence task delegation, but it is generally recommended to leave it on default.

vmod_http_pool_timeout

• Default: 118 seconds

Timeout for idle connections in the connection cache.

Cached connections stay open until they are either reused or reach the timeout. If they are reused, the timeout countdown resets. The timeout also applies to connections in the CLOSE-WAIT state.

This parameter can be increased to make connection reuse more likely, or reduced to close idle connections earlier.

vmod_http_prefer_http_11

• Default: on

Prefer HTTP/1.1 when negotiating new connections.

This parameter can be turned off to use libcurl defaults.

vmod_http_require_http

• Default: on

Only enable the HTTP and HTTPS protocols.

This parameter can be turned off to use libcurl defaults.

vmod_http_threads

• Default: 10 threads

Number of vmod_http threads.

The threads spawned by vmod_http are responsible for carrying out tasks initiated from VCL. Each loaded VCL has its own set of threads.

This parameter can be increased to handle more tasks in total with a higher potential resource consumption.

Arguments:

• async_only accepts type BOOL with a default value of 0 optional

Type: Function

Returns: Int

Availability

The http VMOD is available in Varnish Enterprise version 6.0.0r0 and later.