Log

Changelog (last updated v2.2)

v2.2: earlier epochs rejected at startup

A node whose log is at an earlier epoch than its indexed storage now refuses to start, pointing at Out of Sync Log & Intact Storage. See Epochs.

Previously any epoch difference — in either direction — was treated as the start of a new epoch, logged at INFO, and offset validation was skipped. That is only safe going forwards. Because a message id packs the epoch above the offset, every id on an earlier-epoch log sorts below the storage watermark, so the log replayed as entirely stale and transactions submitted afterwards were acknowledged and then discarded without ever committing or aborting.

To upgrade: nothing to do, unless a node is already running against an earlier epoch than its storage — in which case it will now fail to start, and the recovery is to restore the storage backup that matches the log, or to move forwards to a higher epoch.

v2.2: source/replica log split

Each database now uses two logs under the hood — a source log (client writes) and a replica log (the indexing leader’s output) — to support single-writer indexing. See the Kafka log documentation for configuration details. The in-memory and local-disk log implementations are single-node only and unaffected.

v2.1: multi-database support

As part of the multi-database support, the Kafka log-clusters were extracted - see the Kafka log documentation for more details.

One of the key components of an XTDB node is the log - this is a totally ordered log of all operations that have been applied to the database, generally persistent & shared between nodes.

Implementations

We offer a number of separate implementations of the log, currently:

Single-node log implementations, within xtdb-core:
- In memory: transient in-memory log.
- Local disk: log using the local filesystem.
Remote: multi-node log implementations using a remote service.

In memory

By default, the log is a transient, in-memory log:

## default, no need to explicitly specify

## log: !InMemory

If configured as an in-process node, you can also specify an InstantSource implementation - this is used to override the local machine’s clock when providing a system-time timestamp for each message.

Local disk

A single-node persistent log implementation that writes to a local directory.

log: !Local
  # -- required

  # The path to the local directory to store the log in.
  # (Can be set as an !Env value)
  path: /var/lib/xtdb/log

  # -- optional

  # The number of entries of the buffer to use when writing to the log.
  # bufferSize: 4096

Remote

A multi-node persistent log implementation that uses a remote service to store the log.

We currently offer the following remote log implementations, available in their own modules:

Kafka: a log implementation that uses a Apache Kafka topic to store the log.

Epochs

An epoch is a manually assigned, monotonically increasing integer used to identify the generation of the log in XTDB:

Epochs allow a cluster to safely reset its log state following partial log loss, corruption, or intentional recovery operations, without requiring full reindexing of storage data.
If not explicitly configured, nodes assume epoch = 0.

Configuration

To configure an epoch, specify the epoch field inside the node’s log configuration:

log: !<LogType>
  epoch: <new-epoch>

Where:

<LogType> is the chosen log implementation (e.g., !Kafka, !Local).
<new-epoch> is a positive integer greater than the previous epoch.

All nodes within the same cluster must use an identical epoch value at startup.

Epochs only move forwards. A node started with an epoch below the one its storage has already indexed refuses to start, and points at Out of Sync Log & Intact Storage — there is no way to reconcile the two, because everything on the earlier log ranks below what storage has already processed.

Log