Varnish Administration Console

Advanced configuration

Logging of exceptions or errors in VAC

sudo journalctl -u vac

Client browser requirements

All recent versions of Firefox and Chrome work without problems.

Firewall recommendations

Always keep in mind that through some of non-authenticated endpoints, the VAC exposes sensitive information. As an example /api/rest/status/all as the following output.

License: OK

Database:
Mongo db 127.0.0.1:27017 - Status: Ok

Groups:
Group: dev
 - Cache: 127.0.0.1:6002_/var/lib/varnish/b - Status: Ok
 - Cache: 127.0.0.1:6003_/var/lib/varnish/c - Status: Ok
 - Cache: 127.0.0.1:6004_/var/lib/varnish/d - Status: Ok
Group: oi
 - Cache: 127.0.0.1:6001_/var/lib/varnish/a - Status: Ok

Notice how this endpoint exposes IP’s and ports for Varnish instance(s) and the VAC database.

We strongly recommended that you disable unnecessary ports to prevent potentially malicious usage of this data.

Ports required

The communication between the VAC, the Varnish Agent and Varnish Cache server(s) requires a few ports to be open. The port that need to be open on the Varnish Cache server(s) are:

  • Port 80 (6081 by default), open for the world for HTTP access to Varnish Cache
  • Port 6085, for Varnish-agent by default, open from the server where the VAC is installed
  • Port 80 ( by default) or the port defined by vacListeningPort on the VAC server from the management network, to access the VAC

    note: vacListeningPort is defined in /opt/vac/etc/defaults in the VAC server

Varnish-agent utilizes the Varnish API internally for fetching and issuing commands to and from Varnish Cache.

All ports are configurable.

Configuring Varnish Administration Console

VAC does not usually need to be configured. However, the main configuration files are

  • /opt/vac/etc/defaults
  • /opt/vac/etc/log4j.xml
  • /opt/vac/etc/logback.xml
  • /opt/vac/etc/rrd_conf.xml

You can find a detailed list of configuration properties here

Logback

We use logback as a abstraction layer for log4j. The possible levels are, in order of precedence: TRACE, DEBUG, INFO, WARN and ERROR.

Log4j

The log levels in VAC can be tweaked in log4j.xml configuration file. The usual level of logging are TRACE, DEBUG, INFO, WARN, ERROR or FATAL.

The namespace used by VAC is prefixed with com.varnish. The log4j.xml file ship with a console appender and a rolling file appender(only sysv). By default the rolling file appender is used for sysv and console appender for systemd.

RRD

The rrd_conf.xml is used to configure the archive and datasource used in VAC. A single cache in VAC has an associated RRD. This RRD is stored in memory based on the specified datasource and archives. Similar to rrdtools, once the RRD is created for a single cache, its definition cannot be dynamically changed. Therefore RRD changes are only applied to Varnish Cache servers added to VAC after said change.

Adding a new RRD archive

After you edit the rrd_config.xml file with the appropriate archives, you need to make VAC aware of the updated file in order to make it choose the best existing archive to query. In order to do that you need to follow the following steps:

  • Restart the vac - in order to pick up the new changes in the rrd_config.xml file
  • Restart the agents and make the node registration - in order for the nodes to re-register themselves with the new RRD configuration known by the VAC.

Memory/Heap

A memory requirement of 2GB dedicated to VAC is recommended. This is a very optimistic allocation as the real consumer of memory in VAC is the real-time RRD component. A rule of thumb would be to make available 200MB ~ 300MB per cache for the RRD.

Out of the box, VAC starts with the following:

max_heap_size=2048m         # jvm max heap size
initial_heap_size=512m      # jvm initial heap size

It is therefore recommended to monitor memory usage and ensure that enough memory is made available for VAC as the number of cache being managed by it increases.

See the configuration file, /opt/vac/etc/defaults, for more detail.

Supported counters in Varnish Administration Console


Graph name Counters
Hit Rate % cache_hit, cache_miss, cache_hitpass
Request per second cache_req
Hitrate cache_hit, cache_miss, cache_hitpass
Hit vs Miss vs Hit pass cache_hit, cache_miss, cache_hitpass, client_req
Request vs Connection client_req, client_conn
Bandwitdh s_hdrbytes, s_bodybytes
Bad things backend_fail, n_wrk_drop, client_drop, n_wrk_queued, n_lru_nuked

Graphs

With the help of Varnish-agent, a snapshot of Varnishstat is pushed to VAC and aggregated in memory. The end result is a time series for each key counter for each Varnish instance. The Real-time Statistic API supports queries based on time range in seconds past epoch, and number of samples based on current time. The API also supports the usual consolidation function, average, min, max, last, first, and total. VAC is a natural datasource for providing statistics to any existing monitoring tool.

The graphs provided in VAC UI are in two forms of realtime and historical. The realtime graph indicates time series data across a group as based on timestamp and usual consolidation function. The historical graph represents time-series data from and to specific ending time point across a cache group. The counters of interest, and the sampling resolution and window are completely configurable within VAC.

Upgrading from previous versions

Upgrading from previous versions should adhere to the following steps:

  1. Stop and upgrade Varnish agent individually
  2. Stop and upgrade VAC. Note the listening IP and port
  3. Ensuring that VAC and Varnish agent are not running, configure Varnish agent as specified above
  4. Fix verbose and excessive logging in systemd (if any): If you have made any changes to the log4j.xml, defaults file and service unit file update them:
    • Add a argument in default settings file vacLoggingPropFileLogback with value /opt/vac/etc/logback.xml
    • Add a javaArgs - Dlogback.configurationFile=file://${vacLoggingPropFileLogback} before $CONSISTENCY_OPTION
    • Replace all occurances of <appender-ref ref="productionLog"/> with <appender-ref ref="consoleLog"/> in the log4j.xml.
    • Add SyslogIdentifier=vac to the vac.service unit file
  5. Add mongoClient connection string (optional):
    • Since 3.8.1 we have introduced mongoClient connection string. If you want to make use of this you must add a argument mongoURI and add javaArgs -Dcom.varnish.vac.mongo.uri=$mongoURI along with other mongoDB arguments in the defaults settings file
  6. Restart VAC
  7. Restart Varnish agent individually

After step 4 and 5 your new javaArgs should look like this: javaArgs="-server -Xms$initial_heap_size -Xmx$max_heap_size -Djava.library.path=$vacNativeLibDir/$VAC_ARCH -Dcom.varnish.rrd.default.sample_offset=$vacRrdSampleOffset -Dcom.varnish.rrd.default.xml=$vacRrdCounters -Dcom.varnish.vac.log.dir=$vacLogDir -Dcom.varnish.vac.data.dir=$vacDataDir -Dcom.varnish.vac.user=$serviceUser -Dcom.varnish.vac.group=$serviceGroup -Dcom.varnish.vac.host=$vacListeningHost -Dcom.varnish.vac.port=$vacListeningPort -Dcom.varnish.vac.mongo.uri=$mongoURI -Dcom.varnish.vac.mongo.port=$mongoPort -Dcom.varnish.vac.mongo.host=$mongoHost -Dcom.varnish.vac.mongo.db=$mongoDB -Dlog4j.configuration=file://${vacLoggingPropFile} -Dorg.terracotta.quartz.skipUpdateCheck=true -Dlogback.configurationFile=file://${vacLoggingPropFileLogback} -Dcom.varnish.license.file=$licenseFile -Dcom.varnish.cc.dir=$vacDir $CONSISTENCY_OPTION $MSG_TO_KEEP_OPTION $DAEMON_OPTIONS $VCP_OPTIONS $JDI_OPTIONS $API_LOG_OPTION $AGENT_LOG_OPTION -classpath $vacClassPath com.varnish.VacService"

Note that VCL names in 3.0.0 must now adhere to VCL naming syntax as imposed by Varnish Cache. Therefore previous VCL name is prepended with “vac_” during the upgrade process.

In versions lower than 3.1.0, user roles where not supported by VAC. When upgrading to - Add a javaArgs - Dlogback.configurationFile=file://${vacLoggingPropFileLogback} before $CONSISTENCY_OPTION - Replace all occurances of <appender-ref ref="productionLog"/> with <appender-ref ref="consoleLog"/> in the log4j.xml. - Add SyslogIdentifier=vac to the vac.service unit file 5. Add mongoClient connection string (optional): - Since 3.8.1 we have introduced mongoClient connection string. If you want to make use of this you must add a argument mongoURI and add javaArgs -Dcom.varnish.vac.mongo.uri=$mongoURI along with other mongoDB arguments in the defaults settings file 6. Restart VAC 7. Restart Varnish agent individually

After step 4 and 5 your new javaArgs should look like this: javaArgs="-server -Xms$initial_heap_size -Xmx$max_heap_size -Djava.library.path=$vacNativeLibDir/$VAC_ARCH -Dcom.varnish.rrd.default.sample_offset=$vacRrdSampleOffset -Dcom.varnish.rrd.default.xml=$vacRrdCounters -Dcom.varnish.vac.log.dir=$vacLogDir -Dcom.varnish.vac.data.dir=$vacDataDir -Dcom.varnish.vac.user=$serviceUser -Dcom.varnish.vac.group=$serviceGroup -Dcom.varnish.vac.host=$vacListeningHost -Dcom.varnish.vac.port=$vacListeningPort -Dcom.varnish.vac.mongo.uri=$mongoURI -Dcom.varnish.vac.mongo.port=$mongoPort -Dcom.varnish.vac.mongo.host=$mongoHost -Dcom.varnish.vac.mongo.db=$mongoDB -Dlog4j.configuration=file://${vacLoggingPropFile} -Dorg.terracotta.quartz.skipUpdateCheck=true -Dlogback.configurationFile=file://${vacLoggingPropFileLogback} -Dcom.varnish.license.file=$licenseFile -Dcom.varnish.cc.dir=$vacDir $CONSISTENCY_OPTION $MSG_TO_KEEP_OPTION $DAEMON_OPTIONS $VCP_OPTIONS $JDI_OPTIONS $API_LOG_OPTION $AGENT_LOG_OPTION -classpath $vacClassPath com.varnish.VacService"

Note that VCL names in 3.0.0 must now adhere to VCL naming syntax as imposed by Varnish Cache. Therefore previous VCL name is prepended with “vac_” during the upgrade process.

In versions lower than 3.1.0, user roles where not supported by VAC. When upgrading to 3.1.0, existing users inherit, by default, the Admin role. This role can be updated later. As in 3.1.0 version there are two roles defined for VAC, “Admin” and “ReadOnly”.

Maintaining the VAC

Backup and migration

VAC stores state in MongoDB, /var/opt/vac/ and /opt/vac/.

You will need to save the database:

mongodump -d vcc -o <backup_dir>

And then import mongo backup on new server:

mongorestore -d vcc <backup_dir> --drop

You also need to save all the configuration files which you can find under:

/opt/vac/etc/defaults
/opt/vac/etc/log4j.xml
/opt/vac/etc/logback.xml (if exists)
/opt/vac/etc/roles.js
/opt/vac/etc/rrd_conf.xml

After you have everything in place, you can change varnish agents to target new VAC and restart the agent to pick the new configuration.

Upgrade

The preferred method of upgrading is using the Debian / Red Hat packages.

It is not necessary to do any manual database migration steps, or change anything on the Varnish Cache server(s).

Advanced properties

In the defaults file, located in /var/opt/vac, these are the properties that we support.

Property Default Description
vacDir /opt/vac This is where every static asset lives
vacVarDir /var/opt/vac And this is where we place variable data according to FHS
vacDataDir $vacVarDir/data This is where rrd and parsed ephemeral data will live
vacTmpDir $vacVarDir/tmp This is where tmp folder for the vac will live. This is not used by default unless specified below in DAEMON_OPTIONS
vacListeningPort 80 listening port
vacListeningHost 0.0.0.0 listening host
mongoHost 127.0.0.1 mongo endpoint, host
mongoPort 27017 mongo endpoint, port
mongoDB vcc db name
mongoURI "" mongo connection
max_files 131072 ulimit
max_heap_size 2048m jvm max heap size
initial_heap_size 512m jvm initial heap size
jdi_listening_host 0.0.0.0 The listening host for the Super Fast Purger.
jdi_listening_port 8088 The listening port for the Super Fast Purger.
jdi_client_timeout 5000 Client socket timeout. Client being the API user.
jdi_cache_timeout 3000 Cache socket timeout. Cache refers to Varnish Cache. This can also be interpreted as worst-case response time from Varnish Cache.
jdi_corepool_size 1000 The number of threads to keep in the pool, even if they are idle.
jdi_maxpool_size 3000 The maximum number of threads to allow in the pool.
jdi_thread_keepalive 5 When the number of thread is greater than the core, this is the maximum time that excess idle threads will wait for new tasks before terminating. The value here is in minutes.
jdi_retries 3 The number of retries the VAC PURGER will attempt.
jdi_db_cache_ttl 600000 he time-to-live for objects in the db cache used only by the Super Fast Purger. Value here is in ms. (10min ttl for each obj in our db cache)
vcp_max_active_obj 2 maximum active objs in pool
vcp_max_idle_obj 2 maximum idle objs in pool
vcp_min_idle_obj 1 mininum idle objs in pool
vcp_exhausted_action 0 actions to take when exhausted. 0 = fail, 1 = block, 2 = grow
vcp_test_on_borrow true perform connection test on borrowing obj from pool
vcp_test_on_return false perform connection test on returning obj to pool
vcp_max_wait_on_borrow 200 max wait time on borrow object in ms
vcp_prepare_pool_immediately true prepare pools on startup
vcp_http_keep_connection_alive true keep connection to agent alive
vcp_http_between_bytes_timeout 5000 between bytes timeout on receiving response from varnish-agent
vcp_http_connect_timeout 5000 connection timeout to the varnish-agent
vacRrdSampleOffset 1 sampling offset for end time when fetching based on resolution
scriptFile install the absolute, dereferenced path of this script file
serviceName vac service name
serviceUser vac OS user name for the service
serviceGroup vac OS group name for the service
maxShutdownTime 20 maximum number of seconds to wait for the daemon to terminate normally
maxStartupTime 60 maximum number of seconds to wait for the daemon to start properly
serviceUserHome $applDir location of vac dir
vacJarFile $vacShareDir/vac.jar location of vac.jar file