Transaction Log

One of the key components of an XTDB node is the transaction log - this is a totally ordered log of all transaction operations that have been applied to the database, generally persistent & shared between nodes.

Implementations

We offer a number of separate implementations of the transaction log, currently:

Single-node log implementations, within xtdb-core:
- In memory: transient in-memory log.
- Local disk: log using the local filesystem.
Remote: multi-node log implementations using a remote service.

In memory

By default, the transaction log is a transient, in-memory log:

# default, no need to explicitly specify
# txLog: !InMemory

If configured as an in-process node, you can also specify an InstantSource implementation - this is used to override the local machine’s clock when providing a system-time timestamp for each transaction.

Local disk

A single-node persistent transaction log implementation that writes to a local directory.

txLog: !Local
  # -- required

  # The path to the local directory to store the transaction log in.
  # (Can be set as an !Env value)
  path: /var/lib/xtdb/log

  # -- optional

  # The number of entries of the buffer to use when writing to the transaction log.
  # bufferSize: 4096

  # The duration to sleep for when polling for new transactions written to the transaction log.
  # pollSleepDuration: PT0.1S

Remote

A multi-node persistent transaction log implementation that uses a remote service to store both the transaction log itself, and other shared information between the nodes such as file change notifications from the storage module.

We currently offer the following remote transaction log implementations, available in their own modules:

Kafka: a log implementation that uses Apache Kafka topics to store the transaction log and file change notifications.