Troubleshooting

Support

Varnish Enterprise

Varnish Enterprise users are entitled to support according to the support SLA stated in their contract.

Email: support@varnish-software.com

Portal: https://support.varnish-software.com

Phone: See support contract.

Varnish Cache

For open source Varnish Cache users, see https://varnish-cache.org/support/index.html for more information.

Panics

Varnish is defensively programmed, sprinkled with asserts that aim to catch any undefined scenario. Under normal use, these asserts will not be visible. If you do encounter a scenario that puts Varnish in an undefined state, then the child process will “panic” and be automatically restarted to a known/safe state.

A panic restart will cause all connections to be closed immediately, and can therefore impact ongoing transactions. The restart is usually almost instant, so a panic should not impact traffic for a long period of time unless the panics happen frequently.

When a panic is encountered, the reason for the panic including a stack trace is written to syslog. The most recent panic can also be retrieved from Varnish by running varnishadm panic.show.

If you have encountered a panic, please contact Varnish support.

Errors

Most errors are logged to the Varnish Shared memory Log (VSL), and can be observed with tools like varnishlog and varnishncsa.

A quick overview of the different types of error messages:

• VCL_Error: Error messages logged during VCL execution.
• Error: General error messages.
• FetchError: Error messages logged during a backend fetch.

FetchError

http format error

Varnish has received and rejected a badly formatted response.

Backend log:

BogoHeader     Header without ':' bad-header
HttpGarbage    HTTP/1.1
FetchError     http format error

Result: VCL execution moves to sub vcl_backend_error.

timeout

Varnish has not received any header bytes from an ongoing backend fetch within the configured between_bytes_timeout (default 60s).

Backend log:

FetchError     timeout

Result: VCL execution moves to sub vcl_backend_error.

Known causes:

• The backend is slow to send response headers. Can be fixed by increasing between_bytes_timeout, or reducing load on the backend.
• The backend is unresponsive.

overflow

Varnish received a response from the backend which was too large for the backend workspace.

Backend log:

FetchError     overflow

Result: VCL execution moves to sub vcl_backend_error.

Known causes:

• The backends response headers consume more space than workspace_backend (default 64k). Can be fixed by increasing workspace_backend.

out of workspace

Varnish was unable to complete a backend fetch due to running out of workspace.

There are several variations of the “out of workspace” error, and they may be generated by Varnish core and VMODs. The exact error message will typically indicate which component ran out of workspace.

Backend log:

FetchError     out of workspace

Result: VCL execution moves to sub vcl_backend_error.

Known causes:

• A backend task runs out of workspace and performs a retry. Can be fixed by increasing workspace_backend (default 64k).

HTC eof (-1)

Varnish tried to use a connection that has been closed by the backend.

Backend log:

FetchError     HTC eof (-1)

Result: VCL execution moves to sub vcl_backend_error.

Known causes:

• The backends connection keepalive is set lower than backend_idle_timeout (default 60s). Can be fixed by increasing the backends connection keepalive or lowering backend_idle_timeout.
• Degraded connectivity between Varnish and the Backend.
• Middleware between Varnish and the backend is closing the connection prematurely.

HTC idle (3)

Varnish sent a request to the backend and did not get any response bytes within the configured first_byte_timeout (default 60s).

Backend log:

FetchError     HTC idle (3)

Result: VCL execution moves to sub vcl_backend_error.

Known causes:

• The backend is slow to send response headers. Can be fixed by increasing first_byte_timeout, or reducing load on the backend.
• The backend is unresponsive.

Timed out reusing backend connection

Varnish sent a request on a reused connection to the backend and did not get any response bytes within the configured first_byte_timeout (default 60s).

Backend log:

FetchError     Timed out reusing backend connection

Result: VCL execution moves to sub vcl_backend_error.

Known causes:

• The backends connection keepalive is set lower than backend_idle_timeout (default 60s). Can be fixed by increasing the backends connection keepalive or lowering backend_idle_timeout.
• The backend is slow to send response headers. Can be fixed by increasing first_byte_timeout, or reducing load on the backend.
• The backend is unresponsive.

straight insufficient bytes

Varnish hit a timeout when receiving either the request or the response. This causes the request or response to be aborted. Which timeout has been hit can be determined by looking at the additional log output.

Result: VCL execution moves to sub vcl_backend_error.

timeout_idle

Varnish has not received any bytes from an ongoing client request within the configured idle_timeout (default 5s).

Client log:

FetchError     Resource temporarily unavailable
FetchError     straight insufficient bytes

Backend log:

FetchError     req.body read error: 11 (Resource temporarily unavailable)
FetchError     backend write error: 11 (Resource temporarily unavailable)

timeout_req / timeout_reqbody

Varnish has not received the entire request within the configured timeout_req (default 5s), or the entire body within the configured timeout_reqbody (default not enabled).

Client log:

FetchError     Success
FetchError     straight insufficient bytes

Backend log:

FetchError     req.body read error: 0 (Success)
FetchError     backend write error: 0 (Success)

between_bytes_timeout

Varnish has not received any bytes from an ongoing backend fetch within the configured between_bytes_timeout (default 60s).

Backend log:

FetchError     Resource temporarily unavailable
FetchError     straight insufficient bytes

last_byte_timeout

Varnish has not received the entire response within the configured last_byte_timeout (default not enabled).

Backend log:

FetchError     Success
FetchError     straight insufficient bytes

backend unhealthy

The backend selected in VCL was unhealthy.

Backend log:

FetchError     backend vcl1.default: unhealthy

Result: VCL execution moves to sub vcl_backend_error.

Known causes:

• A backend health probe has marked the backend as unhealthy. This can either be caused by poor backend connectivity, a struggling backend, or probe settings that are too strict.
• The health of the backend has been set to sick through varnishadm.

backend busy

The backend selected in VCL has reached it’s max_connections limit (default not enabled). This means that there are max_connections concurrent ongoing backend fetches.

Backend log:

FetchError     backend vcl1.default: busy

Result: VCL execution moves to sub vcl_backend_error.

Known causes:

• The backend is struggling.
• The max_connections limit is too low. Consider increasing max_connections or enabling connection queue to handle spikes with backend_wait_limit and backend_wait_timeout.

backend fail

Varnish failed to establish a connection to the backend. The errno number gives more details about why the connection failed. If you see a number not documented here, consult the connect() system call documentation for more information or contact Varnish support.

Backend log format:

FetchError     backend <vcl>.<backend>: fail errno <number> (<reason>)

errno 71 (Protocol error)

Varnish failed to establish a connection to the backend due to a protocol error.

Backend log:

FetchError     backend vcl1.default: fail errno 71 (Protocol error)

Result: VCL execution moves to sub vcl_backend_error.

Known causes:

• If using TLS towards the backend and the backend is not able to perform TLS, the backend may refuse the request and close the connection.

errno 101 (Network is unreachable)

Varnish failed to establish a connection to the backend.

Backend log:

FetchError     backend vcl1.default: fail errno 101 (Network is unreachable)

Result: VCL execution moves to sub vcl_backend_error.

Known causes:

• The IP or port number of the backend is incorrect.
• Firewall or security group rules prevent Varnish from opening a TCP connection to the backend.
• The backend is unavailable.

errno 104 (Connection reset by peer)

The backend closed the connection to Varnish unexpectedly.

Backend log:

FetchError     backend vcl1.default: fail errno 104 (Connection reset by peer)

Result: VCL execution moves to sub vcl_backend_error.

Known causes:

• If using TLS towards backend and Varnish sends the wrong SNI, the backend may refuse the request and close the connection.

errno 110 (Connection timed out)

Varnish failed to establish a new connection to the backend within the configured connect_timeout (default 3.5s).

Backend log:

FetchError     backend vcl1.default: fail errno 110 (Connection timed out)

Result: VCL execution moves to sub vcl_backend_error.

Known causes:

• The IP or port number of the backend is incorrect.
• Firewall or security group rules prevent Varnish from opening a TCP connection to the backend.
• The backend is unavailable.

errno 111 (Connection refused)

Varnish failed to establish a connection to the backend.

Backend log:

FetchError     backend vcl1.default: fail errno 111 (Connection refused)

Result: VCL execution moves to sub vcl_backend_error.

Known causes:

• The IP or port number of the backend is incorrect.
• Firewall or security group rules prevent Varnish from opening a TCP connection to the backend.
• The backend is unavailable.

Director returned no backend

The director selected in VCL had no usable backends to fetch from.

Backend log:

FetchError     Director dir returned no backend
FetchError     No backend

Result: VCL execution moves to sub vcl_backend_error.

Known causes:

• No backends have been added to the director.
• If using a dynamic director, no backends have been generated.
• All backends in the director are unhealthy.
• If using a director that keeps track of which backends have been used for a given transaction, all backends have been attempted for the current transaction.

No backend

No backend was selected to fetch from.

Backend log:

FetchError     No backend

Result: VCL execution moves to sub vcl_backend_error.

Known causes:

• No backend was selected in VCL, and no backend named default was defined.
• The selected director did not return any backends.

No thread available for bgfetch

Varnish attempted a background fetch, but no worker threads were available to handle the fetch.

Backend log:

FetchError     No thread available for bgfetch
BereqAcct      0 0 0 0 0 0
End

Result: The fetch is abandoned.

Known causes:

• All thread pools have reached thread_pool_max (default 5000) and all worker threads are busy. Consider investigating why the thread pools are exhausted before increasing thread_pool_max.

Request body too big to cache

A client sent a request body larger than the configured std.cache_req_body() limit.

Backend log:

FetchError     Request body too big to cache

Result: The request is abandoned.

Known causes:

• The request body limit is too low, consider increasing it.

Client chunked read err

The client used chunked transfer encoding and closed the connection before transmitting the final zero chunk

Client log:

FetchError     chunked read err

Backend log:

FetchError     req.body read error: 0 (Success)
FetchError     backend write error: 0 (Success)

Result: The request is abandoned.

Known causes:

• The client has an unstable connection.
• The client aborted the upload.