Operations

This page describes how MSE4 behaves in operation: how device failures are handled, what the CLI commands do, what happens when a store fills up, and how startup handles missing or damaged devices. For counters to alert on, see the Monitoring and alerting page.

Book and store states

Every book and store is in one of three states:

• ONLINE: the device is healthy and in service.
• FAILING: the device has failed and is being taken out of service. Objects on the device are being removed while in-flight deliveries are allowed to finish.
• OFFLINE: the device is out of service.

The current states are shown by the CLI command mse4.status:

varnishadm mse4.status
#  Type   State    ID      Dependency
1  book   ONLINE   book1   -
2  store  ONLINE   store1  book1
3  store  ONLINE   store2  book1

The same information is available as the counter gauges MSE4_BOOK.<id>.online and MSE4_STORE.<id>.online, which are non-zero only in the ONLINE state.

Store failure handling

When an I/O error (read or write) occurs against a store, MSE4 fails the whole store rather than retrying against a suspect device:

1. The object whose I/O failed is removed from the cache. If the failure aborts an in-progress delivery, that delivery fails.
2. The store transitions from ONLINE to FAILING and stops accepting new work. varnishd does not panic or restart; the book and the remaining stores keep serving.
3. The FAILING state lasts at least 10 seconds, and extends for as long as in-flight deliveries still hold references to objects on the store.
4. The store then transitions to OFFLINE, its file is closed, and the state change is recorded in the statelog and printed to standard error.

Objects lost with the failed store are transparently re-fetched from the backend on the next request and stored on the remaining stores.

An I/O error against a book (slot journal, slot table, or the book’s ban journal) fails the entire book and every store attached to it, through the same FAILING to OFFLINE sequence. An I/O error against the environment-level ban file is not survivable and restarts the cache process.

Failover behavior

The following behavior was recorded with two 512 MB stores under load with mse4.fail:

• The store’s online gauge dropped to 0 within milliseconds, and objects on the failed store were removed almost immediately.
• 18,816 requests were served during the failure window with zero non-200 responses, zero client errors, and no hung requests.
• Six in-flight 50 MB streaming downloads all completed with full byte counts, with no truncation or resets.
• With no references held, FAILING lasted 10.2 seconds; with a slow streaming client holding references, it extended past 38 seconds until the delivery drained.

Note that mse4.fail is a graceful, orderly failure. A real disk throwing I/O errors mid-request may be uglier. In particular, a delivery whose read actually fails is aborted. This has not been verified on failing hardware.

Failing and resetting devices from the CLI

mse4.fail <id> puts a book or store through the failure sequence described above. This is useful for rehearsing failures and for taking a device out of service before maintenance. Failing an already OFFLINE device reports an error.

mse4.reset <id> brings an OFFLINE book or store back into service: the device file is deleted, recreated, and initialized with a new unique ID, and the device returns to ONLINE with zero objects.

Persisted objects on healthy stores are unaffected by restarting varnishd: they are revived from the book at startup.

The statelog

MSE4 records book and store state changes in a plain-text statelog file, so that device failures are on record for the management process. The file is human-readable: entries consist of a # TIME: line, a # TEXT: line with the reason, and a <name>/<unique-id> ONLINE|OFFLINE line. The last entry for a given device wins. mkfs.mse4 appends ONLINE entries when it creates devices, and mse4.reset appends a “CLI reset” entry.

The statelog file is created lazily, on the first state change. On a fresh installation, startup therefore logs:

MSE4: [statelog] Failed to read statelog (Could not open state log)

This warning is benign and expected until a state change has occurred.

When a store fills up

Writes to persisted storage are write-behind: objects are cached in memory first and flushed to the store asynchronously, so clients do not wait for store allocation on the fetch path.

When a store reaches capacity, MSE4 prunes segments: objects are evicted from disk, chosen by the location they occupy so that contiguous free areas are created, to maintain a band of free space for new writes. On a store running at capacity this is normal, continuous operation, visible in the counters MSE4_STORE.<id>.c_segment_prune and c_segment_pruned_objects (and in aggregate as MSE4.c_removed). Note that MAIN.n_lru_nuked does not count MSE4 evictions.

On slower disks, the place where back-pressure appears is the allocation queue: fetch threads needing store space queue up (MSE4_STORE.<id>.g_allocation_queue, c_allocation_queued) until pruning produces free space. Watch those gauges, and the c_allocation_failure* counters, on spinning-disk or heavily loaded deployments.

Startup with missing or damaged devices

At startup, MSE4 validates every configured book and store. The general rule is that a bad device is skipped and service continues without it, unless the configuration marks it as required:

• A book or store file that is missing or cannot be opened is skipped with a warning. If the device has required set, or the environment has required_all set, startup fails instead.
• A device with a corrupt header is skipped, as is a device left in maintenance mode by an interrupted maintenance operation.
• A device whose size or layout does not match the configuration is used anyway, with a warning.
• A device whose on-disk format version does not match the varnishd binary always aborts startup. Upgrades that change the on-disk format require recreating the devices.
• A store whose header does not match the book it is configured under (for example after mixing files from different environments) is set OFFLINE.
• If processing a book’s ban journal fails, the book and its stores are set OFFLINE. If an error occurs while reviving objects and objects were already revived, the cache process restarts once to start from a clean slate; otherwise the book is set OFFLINE and service continues.

At runtime, a chunk read from disk that fails checksum verification (counted in MSE4_STORE.<id>.c_checksum_verification_failed) causes the affected object to be removed from the cache and that delivery to be aborted; it does not fail the store.