Advanced configuration

Logging of exceptions or errors in VAC

sudo journalctl -u vac

Client browser requirements

All recent versions of Firefox and Chrome work without problems.

Firewall recommendations

Always keep in mind that through some of non-authenticated endpoints, the VAC exposes sensitive information. As an example /api/rest/status/all as the following output.

License: OK

Database:
Mongo db 127.0.0.1:27017 - Status: Ok

Groups:
Group: dev
 - Cache: 127.0.0.1:6002_/var/lib/varnish/b - Status: Ok
 - Cache: 127.0.0.1:6003_/var/lib/varnish/c - Status: Ok
 - Cache: 127.0.0.1:6004_/var/lib/varnish/d - Status: Ok
Group: oi
 - Cache: 127.0.0.1:6001_/var/lib/varnish/a - Status: Ok

Notice how this endpoint exposes IP’s and ports for Varnish instance(s) and the VAC database.

We strongly recommended that you disable unnecessary ports to prevent potentially malicious usage of this data.

Ports required

The communication between the VAC, the Varnish Agent and Varnish Cache server(s) requires a few ports to be open. The port that need to be open on the Varnish Cache server(s) are:

Port 80 (6081 by default), open for the world for HTTP access to Varnish Cache
Port 6085, for Varnish-agent by default, open from the server where the VAC is installed
Port 80 ( by default) or the port defined by vacListeningPort on the VAC server from the management network, to access the VAC

note: vacListeningPort is defined in /opt/vac/etc/defaults in the VAC server

Varnish-agent utilizes the Varnish API internally for fetching and issuing commands to and from Varnish Cache.

All ports are configurable.

Configuring Varnish Administration Console

VAC does not usually need to be configured. However, the main configuration files are

/opt/vac/etc/defaults
/opt/vac/etc/log4j.xml
/opt/vac/etc/logback.xml
/opt/vac/etc/rrd_conf.xml

You can find a detailed list of configuration properties here

Logback

We use logback as a abstraction layer for log4j. The possible levels are, in order of precedence: TRACE, DEBUG, INFO, WARN and ERROR.

Log4j

The log levels in VAC can be tweaked in log4j.xml configuration file. The usual level of logging are TRACE, DEBUG, INFO, WARN, ERROR or FATAL.

The namespace used by VAC is prefixed with com.varnish. The log4j.xml file ship with a console appender and a rolling file appender(only sysv). By default the rolling file appender is used for sysv and console appender for systemd.

RRD

The rrd_conf.xml is used to configure the archive and datasource used in VAC. A single cache in VAC has an associated RRD. This RRD is stored in memory based on the specified datasource and archives. Similar to rrdtools, once the RRD is created for a single cache, its definition cannot be dynamically changed. Therefore RRD changes are only applied to Varnish Cache servers added to VAC after said change.

Adding a new RRD archive

After you edit the rrd_config.xml file with the appropriate archives, you need to make VAC aware of the updated file in order to make it choose the best existing archive to query. In order to do that you need to follow the following steps:

Restart the vac - in order to pick up the new changes in the rrd_config.xml file
Restart the agents and make the node registration - in order for the nodes to re-register themselves with the new RRD configuration known by the VAC.

Memory/Heap

A memory requirement of 2GB dedicated to VAC is recommended. This is a very optimistic allocation as the real consumer of memory in VAC is the real-time RRD component. A rule of thumb would be to make available 200MB ~ 300MB per cache for the RRD.

Out of the box, VAC starts with the following:

max_heap_size=2048m         # jvm max heap size
initial_heap_size=512m      # jvm initial heap size

It is therefore recommended to monitor memory usage and ensure that enough memory is made available for VAC as the number of cache being managed by it increases.

See the configuration file, /opt/vac/etc/defaults, for more detail.

Supported counters in Varnish Administration Console

Graph name	Counters
`Hit Rate %`	cache_hit, cache_miss, cache_hitpass
`Request per second`	cache_req
`Hitrate`	cache_hit, cache_miss, cache_hitpass
`Hit vs Miss vs Hit pass`	cache_hit, cache_miss, cache_hitpass, client_req
`Request vs Connection`	client_req, client_conn
`Bandwitdh`	s_hdrbytes, s_bodybytes
`Bad things`	backend_fail, n_wrk_drop, client_drop, n_wrk_queued, n_lru_nuked

Graphs

With the help of Varnish-agent, a snapshot of Varnishstat is pushed to VAC and aggregated in memory. The end result is a time series for each key counter for each Varnish instance. The Real-time Statistic API supports queries based on time range in seconds past epoch, and number of samples based on current time. The API also supports the usual consolidation function, average, min, max, last, first, and total. VAC is a natural datasource for providing statistics to any existing monitoring tool.

The graphs provided in VAC UI are in two forms of realtime and historical. The realtime graph indicates time series data across a group as based on timestamp and usual consolidation function. The historical graph represents time-series data from and to specific ending time point across a cache group. The counters of interest, and the sampling resolution and window are completely configurable within VAC.

Upgrading from previous versions

Upgrading from previous versions should adhere to the following steps:

Stop and upgrade Varnish agent individually
Stop and upgrade VAC. Note the listening IP and port
Ensuring that VAC and Varnish agent are not running, configure Varnish agent as specified above
Fix verbose and excessive logging in systemd (if any): If you have made any changes to the log4j.xml, defaults file and service unit file update them:
- Add a argument in default settings file vacLoggingPropFileLogback with value /opt/vac/etc/logback.xml
- Add a javaArgs - Dlogback.configurationFile=file://${vacLoggingPropFileLogback} before $CONSISTENCY_OPTION
- Replace all occurances of <appender-ref ref="productionLog"/> with <appender-ref ref="consoleLog"/> in the log4j.xml.
- Add SyslogIdentifier=vac to the vac.service unit file
Add mongoClient connection string (optional):
- Since 3.8.1 we have introduced mongoClient connection string. If you want to make use of this you must add a argument mongoURI and add javaArgs -Dcom.varnish.vac.mongo.uri=$mongoURI along with other mongoDB arguments in the defaults settings file
Restart VAC
Restart Varnish agent individually

After step 4 and 5 your new javaArgs should look like this: javaArgs="-server -Xms$initial_heap_size -Xmx$max_heap_size -Djava.library.path=$vacNativeLibDir/$VAC_ARCH -Dcom.varnish.rrd.default.sample_offset=$vacRrdSampleOffset -Dcom.varnish.rrd.default.xml=$vacRrdCounters -Dcom.varnish.vac.log.dir=$vacLogDir -Dcom.varnish.vac.data.dir=$vacDataDir -Dcom.varnish.vac.user=$serviceUser -Dcom.varnish.vac.group=$serviceGroup -Dcom.varnish.vac.host=$vacListeningHost -Dcom.varnish.vac.port=$vacListeningPort -Dcom.varnish.vac.mongo.uri=$mongoURI -Dcom.varnish.vac.mongo.port=$mongoPort -Dcom.varnish.vac.mongo.host=$mongoHost -Dcom.varnish.vac.mongo.db=$mongoDB -Dlog4j.configuration=file://${vacLoggingPropFile} -Dorg.terracotta.quartz.skipUpdateCheck=true -Dlogback.configurationFile=file://${vacLoggingPropFileLogback} -Dcom.varnish.license.file=$licenseFile -Dcom.varnish.cc.dir=$vacDir $CONSISTENCY_OPTION $MSG_TO_KEEP_OPTION $DAEMON_OPTIONS $VCP_OPTIONS $JDI_OPTIONS $API_LOG_OPTION $AGENT_LOG_OPTION -classpath $vacClassPath com.varnish.VacService"

Note that VCL names in 3.0.0 must now adhere to VCL naming syntax as imposed by Varnish Cache. Therefore previous VCL name is prepended with “vac_” during the upgrade process.

In versions lower than 3.1.0, user roles where not supported by VAC. When upgrading to - Add a javaArgs - Dlogback.configurationFile=file://${vacLoggingPropFileLogback} before $CONSISTENCY_OPTION - Replace all occurances of <appender-ref ref="productionLog"/> with <appender-ref ref="consoleLog"/> in the log4j.xml. - Add SyslogIdentifier=vac to the vac.service unit file 5. Add mongoClient connection string (optional): - Since 3.8.1 we have introduced mongoClient connection string. If you want to make use of this you must add a argument mongoURI and add javaArgs -Dcom.varnish.vac.mongo.uri=$mongoURI along with other mongoDB arguments in the defaults settings file 6. Restart VAC 7. Restart Varnish agent individually

Note that VCL names in 3.0.0 must now adhere to VCL naming syntax as imposed by Varnish Cache. Therefore previous VCL name is prepended with “vac_” during the upgrade process.

In versions lower than 3.1.0, user roles where not supported by VAC. When upgrading to 3.1.0, existing users inherit, by default, the Admin role. This role can be updated later. As in 3.1.0 version there are two roles defined for VAC, “Admin” and “ReadOnly”.

Maintaining the VAC

Backup and migration

VAC stores state in MongoDB, /var/opt/vac/ and /opt/vac/.

You will need to save the database:

mongodump -d vcc -o <backup_dir>

And then import mongo backup on new server:

mongorestore -d vcc <backup_dir> --drop

You also need to save all the configuration files which you can find under:

/opt/vac/etc/defaults
/opt/vac/etc/log4j.xml
/opt/vac/etc/logback.xml (if exists)
/opt/vac/etc/roles.js
/opt/vac/etc/rrd_conf.xml

After you have everything in place, you can change varnish agents to target new VAC and restart the agent to pick the new configuration.

Upgrade

The preferred method of upgrading is using the Debian / Red Hat packages.

It is not necessary to do any manual database migration steps, or change anything on the Varnish Cache server(s).

Advanced properties

In the defaults file, located in /var/opt/vac, these are the properties that we support.

Property	Default	Description
vacDir	`/opt/vac`	This is where every static asset lives
vacVarDir	`/var/opt/vac`	And this is where we place variable data according to FHS
vacDataDir	`$vacVarDir/data`	This is where rrd and parsed ephemeral data will live
vacTmpDir	`$vacVarDir/tmp`	This is where tmp folder for the vac will live. This is not used by default unless specified below in DAEMON_OPTIONS
vacListeningPort	`80`	listening port
vacListeningHost	`0.0.0.0`	listening host
mongoHost	`127.0.0.1`	mongo endpoint, host
mongoPort	`27017`	mongo endpoint, port
mongoDB	`vcc`	db name
mongoURI	`""`	mongo connection
max_files	`131072`	ulimit
max_heap_size	`2048m`	jvm max heap size
initial_heap_size	`512m`	jvm initial heap size
jdi_listening_host	`0.0.0.0`	The listening host for the Super Fast Purger.
jdi_listening_port	`8088`	The listening port for the Super Fast Purger.
jdi_client_timeout	`5000`	Client socket timeout. Client being the API user.
jdi_cache_timeout	`3000`	Cache socket timeout. Cache refers to Varnish Cache. This can also be interpreted as worst-case response time from Varnish Cache.
jdi_corepool_size	`1000`	The number of threads to keep in the pool, even if they are idle.
jdi_maxpool_size	`3000`	The maximum number of threads to allow in the pool.
jdi_thread_keepalive	`5`	When the number of thread is greater than the core, this is the maximum time that excess idle threads will wait for new tasks before terminating. The value here is in minutes.
jdi_retries	`3`	The number of retries the VAC PURGER will attempt.
jdi_db_cache_ttl	`600000`	he time-to-live for objects in the db cache used only by the Super Fast Purger. Value here is in ms. (10min ttl for each obj in our db cache)
vcp_max_active_obj	`2`	maximum active objs in pool
vcp_max_idle_obj	`2`	maximum idle objs in pool
vcp_min_idle_obj	`1`	mininum idle objs in pool
vcp_exhausted_action	`0`	actions to take when exhausted. 0 = fail, 1 = block, 2 = grow
vcp_test_on_borrow	`true`	perform connection test on borrowing obj from pool
vcp_test_on_return	`false`	perform connection test on returning obj to pool
vcp_max_wait_on_borrow	`200`	max wait time on borrow object in ms
vcp_prepare_pool_immediately	`true`	prepare pools on startup
vcp_http_keep_connection_alive	`true`	keep connection to agent alive
vcp_http_between_bytes_timeout	`5000`	between bytes timeout on receiving response from varnish-agent
vcp_http_connect_timeout	`5000`	connection timeout to the varnish-agent
vacRrdSampleOffset	`1`	sampling offset for end time when fetching based on resolution
scriptFile	`install`	the absolute, dereferenced path of this script file
serviceName	`vac`	service name
serviceUser	`vac`	OS user name for the service
serviceGroup	`vac`	OS group name for the service
maxShutdownTime	`20`	maximum number of seconds to wait for the daemon to terminate normally
maxStartupTime	`60`	maximum number of seconds to wait for the daemon to start properly
serviceUserHome	`$applDir`	location of vac dir
vacJarFile	`$vacShareDir/vac.jar`	location of vac.jar file