URL Plus (urlplus)

Description

The urlplus vmod is a URL/query string normalization, parsing and manipulation VMOD. This VMOD allows you to get, add, delete, and keep URL segments and query parameters. You can also sort query parameters. vmod-urlplus keeps an internal representation of the URL up until it is written out.

Examples

Query Filtering

Keep only the page_id.

vcl 4.0;

import urlplus;

sub vcl_recv
{
  //Keep page_id
  urlplus.query_keep("page_id");

  //Sort query string and write URL out to req.url
  //Any query that is not kept is not written to req.url
  urlplus.write();
}

Remove Google Analytics

vcl 4.0;

import urlplus;

sub vcl_recv
{
  //Remove all Google Analytics
  urlplus.query_delete_regex("utm_");

  //Sort query string and write URL out to req.url
  urlplus.write();
}

Remove random query parameter

vcl 4.0;

import urlplus;

sub vcl_recv
{
  urlplus.query_delete("random");

  // Write URL out to req.url without sorting
  urlplus.write(sort_query = false);
}

Normalize, sort and write query string to req.url

vcl 4.0;

import urlplus;

sub vcl_recv
{
  // Write URL out to req.url
  urlplus.write();
}

File Extension Specific Logic

Specify TTL by file type.

vcl 4.0;

import urlplus;

sub vcl_backend_response
{
  //Give files with these extensions a TTL of 1 day
  if (urlplus.get_extension() ~ "gif|jpg|jpeg|bmp|png|tiff|tif|img") {
      set beresp.ttl = 1d;
  }
}

API

parse

VOID parse(STRING url)

Parse URL and query string. Any internal state is reset.

Arguments:

• url accepts type STRING

Type: Function

Returns: None

reset

VOID reset()

Reset the internal state.

Arguments: None

Type: Function

Returns: None

write

VOID write(BOOL sort_query = 1, ENUM {NONE, KEY, KEY_VALUE} query_unique = "NONE", ENUM {FROM_INPUT, TRUE, FALSE} leading_slash = "FROM_INPUT", ENUM {FROM_INPUT, TRUE, FALSE} trailing_slash = "FROM_INPUT", BOOL query_keep_equal_sign = 0)

Write URL back out to be/req.url. sort_query indicates the query string should be sorted. sort_query defaults to TRUE. query_unique indicates if the query string should be unique by either key or key and value. This is done before sorting. The first occurrence is kept and each proceeding occurrence is removed. Defaults to NONE. leading_slash has three states. When set to TRUE, a leading slash will be added to the URL. When set to FALSE, no leading slash will be added to the URL. When set to FROM_INPUT a leading slash will be added or not depending on the way the original string was parsed. trailing_slash follows the same rules except the slash will be at the end of the URL. query_keep_equal_sign indicates that a trailing equal sign should be kept when the query string value is empty.

Arguments:

• sort_query accepts type BOOL with a default value of 1 optional

• query_keep_equal_sign accepts type BOOL with a default value of 0 optional

• query_unique is an ENUM that accepts values of NONE, KEY, and KEY_VALUE with a default value of NONE optional

• leading_slash is an ENUM that accepts values of FROM_INPUT, TRUE, and FALSE with a default value of FROM_INPUT optional

• trailing_slash is an ENUM that accepts values of FROM_INPUT, TRUE, and FALSE with a default value of FROM_INPUT optional

Type: Function

Returns: None

as_string

STRING as_string(BOOL sort_query = 1, ENUM {NONE, KEY, KEY_VALUE} query_unique = "NONE", ENUM {FROM_INPUT, TRUE, FALSE} leading_slash = "FROM_INPUT", ENUM {FROM_INPUT, TRUE, FALSE} trailing_slash = "FROM_INPUT", BOOL query_keep_equal_sign = 0)

Return the combined URL and query as a string. sort_query indicates the query string should be sorted. sort_query defaults to TRUE. query_unique indicates if the query string should be unique by either key or key and value. This is done before sorting. The first occurrence is kept and each proceeding occurrence is removed. Defaults to NONE. leading_slash has three states. When set to TRUE, a leading slash will be added to the URL. When set to FALSE, no leading slash will be added to the URL. When set to FROM_INPUT a leading slash will be added or not depending on the way the original string was parsed. trailing_slash follows the same rules except the slash will be at the end of the URL. query_keep_equal_sign indicates that a trailing equal sign should be kept when the query string value is empty.

Arguments:

• sort_query accepts type BOOL with a default value of 1 optional

• query_keep_equal_sign accepts type BOOL with a default value of 0 optional

• query_unique is an ENUM that accepts values of NONE, KEY, and KEY_VALUE with a default value of NONE optional

• leading_slash is an ENUM that accepts values of FROM_INPUT, TRUE, and FALSE with a default value of FROM_INPUT optional

• trailing_slash is an ENUM that accepts values of FROM_INPUT, TRUE, and FALSE with a default value of FROM_INPUT optional

Type: Function

Returns: String

get_basename

STRING get_basename()

Given /foo/bar.baz return the location’s basename (bar.baz). If no extension is present, return the last URL segment.

Arguments: None

Type: Function

Returns: String

get_filename

STRING get_filename()

Given /foo/bar.baz return the location’s filename (bar). If no extension is present, return NULL.

Arguments: None

Type: Function

Returns: String

get_extension

STRING get_extension()

Given /foo/bar.baz return the location’s extension (baz). If no extension is present, return NULL.

Arguments: None

Type: Function

Returns: String

get_dirname

STRING get_dirname()

Return a string with everything but the last segment. If only one segment, return /. Given /foo/bar/bar.baz return the location’s directory (/foo/bar).

Arguments: None

Type: Function

Returns: String

tolower

VOID tolower(ENUM {ALL, URL, QUERY} convert = ALL)

Convert the URL to lowercase. part_to_lower indicates which part of the URL will be converted. If convert is set to URL only the URL will be converted. If set to QUERY only the query string will be converted. If convert is set to ALL both the URL and query string will be converted. convert defaults to ALL.

Arguments:

• convert is an ENUM that accepts values of ALL, URL, and QUERY with a default value of ALL optional

Type: Function

Returns: None

toupper

VOID toupper(ENUM {ALL, URL, QUERY} convert = ALL)

Convert the URL to uppercase. convert indicates which part of the URL will be converted. If convert is set to URL only the URL will be converted. If set to QUERY only the query string will be converted. If part_to_lower is set to ALL both the URL and query string will be converted. convert defaults to ALL.

Arguments:

• convert is an ENUM that accepts values of ALL, URL, and QUERY with a default value of ALL optional

Type: Function

Returns: None

query_get

STRING query_get(STRING name = "", STRING def = 0, INT position = -1)

Return the value of query name. If not found, def is returned(NULL by default). Position when set, ignores name, and returns the value of the query at position (starting from zero). Otherwise returns def. Returns def when name is unset, and position is out of bounds. If both name and position are set, position will be returned.

Arguments:

• name accepts type STRING with a default value of empty. optional

• def accepts type STRING with a default value of 0 optional

• position accepts type INT with a default value of empty. optional

Type: Function

Returns: String

query_get_regex

STRING query_get_regex(STRING regex, STRING def = 0)

Return the value of query that matches regex. If not found, def is returned(NULL by default).

Arguments:

• regex accepts type STRING

• def accepts type STRING with a default value of 0 optional

Type: Function

Returns: String

query_add

VOID query_add(STRING name, STRING value = "", BOOL keep = 1, INT position = -1)

Add a query of query, value with parameter keep. Value is NULL by default. keep is 1 by default. position is a base 0 integer. By default, position = -1 which will add the URL segment to the end. Adding to position = 0 will add to the front of the list. Adding to position = 1 will add to the second spot and so on.

Arguments:

• name accepts type STRING

• value accepts type STRING with a default value of empty. optional

• keep accepts type BOOL with a default value of 1 optional

• position accepts type INT with a default value of empty. optional

Type: Function

Returns: None

query_delete

VOID query_delete(STRING name, BOOL delete_keep = 0)

Delete all queries of name. If delete_keep is set to TRUE, kept queries will also be removed. delete_keep is FALSE by default.

Arguments:

• name accepts type STRING

• delete_keep accepts type BOOL with a default value of 0 optional

Type: Function

Returns: None

query_delete_regex

VOID query_delete_regex(STRING regex, BOOL delete_keep = 0)

Delete all queries matching regex. If delete_keep is set values of name with keep set, will also be removed.

Arguments:

• regex accepts type STRING

• delete_keep accepts type BOOL with a default value of 0 optional

Type: Function

Returns: None

query_keep

VOID query_keep(STRING name)

Set keep to TRUE for all queries of name. This initiates keep_mode.

Arguments:

• name accepts type STRING

Type: Function

Returns: None

query_keep_regex

VOID query_keep_regex(STRING regex)

Set keep to TRUE for all queries with a name matching regex. This initiates keep_mode.

Arguments:

• regex accepts type STRING

Type: Function

Returns: None

query_count

INT query_count()

Return the number of query pairs stored. If keep_mode is enabled, only kept query pairs will be counted.

Arguments: None

Type: Function

Returns: Int

query_as_string

STRING query_as_string(BOOL sort_query = 1, ENUM {NONE, KEY, KEY_VALUE} query_unique = "NONE", BOOL query_keep_equal_sign = 0)

Return a string of queries in the format name=value&name2. sort_query indicates the query string should be sorted. sort_query defaults to TRUE. query_unique indicates if the query string should be unique by either key or key and value. This is done before sorting. The first occurrence is kept and each proceeding occurrence is removed. Defaults to NONE. If keep_mode is enabled, only kept query pairs will be added to string. query_keep_equal_sign indicates that a trailing equal sign should be kept when the query string value is empty.

Arguments:

• sort_query accepts type BOOL with a default value of 1 optional

• query_keep_equal_sign accepts type BOOL with a default value of 0 optional

• query_unique is an ENUM that accepts values of NONE, KEY, and KEY_VALUE with a default value of NONE optional

Type: Function

Returns: String

query_set

VOID query_set(STRING name, STRING value = "", BOOL keep = 1, BOOL all = 0)

Updates the value of the query ’name’ with the value ‘value’. If the query does not exist a new one with the provided name and value is added. Requesting all will result in the value of all matching queries being updated.

Arguments:

• name accepts type STRING

• value accepts type STRING with a default value of empty. optional

• keep accepts type BOOL with a default value of 1 optional

• all accepts type BOOL with a default value of 0 optional

Type: Function

Returns: None

url_get

STRING url_get(INT start_range = 0, INT end_range = -1, ENUM {FROM_INPUT, TRUE, FALSE} leading_slash = "FROM_INPUT", ENUM {FROM_INPUT, TRUE, FALSE} trailing_slash = FROM_INPUT)

Return the URL as a string. This can be done by range. By default the start_range and end_range are set to 0 and -1. This is considered the entire URL. The range is 0 based. leading_slash has three states. When set to TRUE, a leading slash will be added to the URL. When set to FALSE, no leading slash will be added to the URL. When set to FROM_INPUT a leading slash will be added or not depending on the way the original string was parsed. trailing_slash follows the same rules except the slash will be at the end of the URL.

Arguments:

• start_range accepts type INT with a default value of 0 optional

• end_range accepts type INT with a default value of empty. optional

• leading_slash is an ENUM that accepts values of FROM_INPUT, TRUE, and FALSE with a default value of FROM_INPUT optional

• trailing_slash is an ENUM that accepts values of FROM_INPUT, TRUE, and FALSE with a default value of FROM_INPUT optional

Type: Function

Returns: String

url_add

VOID url_add(STRING name, BOOL keep = 1, INT position = -1)

Add a URL segment with parameter keep. position is a base 0 integer. By default, position = -1 which will add the URL segment to the end. Adding to position = 0 will add to the front of the list. Adding to position = 1 will add to the second spot and so on.

Arguments:

• name accepts type STRING

• keep accepts type BOOL with a default value of 1 optional

• position accepts type INT with a default value of empty. optional

Type: Function

Returns: None

url_delete

VOID url_delete(STRING name, BOOL delete_keep = 0)

Delete all URL segments of name. If delete_keep is set kept URL segments will also be removed. delete_keep is 0 by default.

Arguments:

• name accepts type STRING

• delete_keep accepts type BOOL with a default value of 0 optional

Type: Function

Returns: None

url_delete_range

VOID url_delete_range(INT start_range, INT end_range, BOOL delete_keep = 0)

Delete all URL segments between the given start_range and end_range indices. start_range and end_range are base 0 and inclusive. If delete_keep is set values of within range with keep set, will also be removed. delete_keep is 0 by default. If end = -1 it will be set to the number of URL segments.

Arguments:

• start_range accepts type INT

• end_range accepts type INT

• delete_keep accepts type BOOL with a default value of 0 optional

Type: Function

Returns: None

url_delete_regex

VOID url_delete_regex(STRING regex, BOOL delete_keep = 0)

Delete all URL segments matching regex. If delete_keep kept URL segments will also be removed. delete_keep is 0 by default.

Arguments:

• regex accepts type STRING

• delete_keep accepts type BOOL with a default value of 0 optional

Type: Function

Returns: None

url_keep

VOID url_keep(STRING name)

Set keep to TRUE for all URL segments of name. This initiates keep_mode.

Arguments:

• name accepts type STRING

Type: Function

Returns: None

url_keep_regex

VOID url_keep_regex(STRING regex)

Set keep to TRUE for all URL segments with name matching regex. This initiates keep_mode.

Arguments:

• regex accepts type STRING

Type: Function

Returns: None

url_count

INT url_count()

Return the number of URL segments stored. If keep_mode is enabled, only kept URL segments will be counted.

Arguments: None

Type: Function

Returns: Int

url_as_string

STRING url_as_string(ENUM {FROM_INPUT, TRUE, FALSE} leading_slash = "FROM_INPUT", ENUM {FROM_INPUT, TRUE, FALSE} trailing_slash = FROM_INPUT)

Return the URL formatted as, some/test/url. If keep_mode is enabled, only kept URL segments will be added to string. leading_slash has three states. When set to TRUE, a leading slash will be added to the URL. When set to FALSE, no leading slash will be added to the URL. When set to FROM_INPUT a leading slash will be added or not depending on the way the original string was parsed.trailing_slash follows the same rules except the slash will be at the end of the URL.

Arguments:

• leading_slash is an ENUM that accepts values of FROM_INPUT, TRUE, and FALSE with a default value of FROM_INPUT optional

• trailing_slash is an ENUM that accepts values of FROM_INPUT, TRUE, and FALSE with a default value of FROM_INPUT optional

Type: Function

Returns: String

Availability

The urlplus VMOD is available in Varnish Enterprise version 6.0.0r1 and later.