Skip to content
XTDB

Storage

One of the key components of an XTDB node is the storage module - used to store the data and indexes that make up the database.

We offer the following implementations of the storage module:

  • In memory: transient in-memory storage.

  • Local disk: storage persisted to the local filesystem.

  • Remote: storage persisted remotely.

In memory

By default, the storage module is configured to use transient, in-memory storage.

# default, no need to explicitly specify
# storage: !InMemory

Local disk

A persistent storage implementation that writes to a local directory, also maintaining an in-memory cache of the working set.

storage: !Local
  # -- required

  # The path to the local directory to persist the data to.
  # (Can be set as an !Env value)
  path: /var/lib/xtdb/storage

  # -- optional

  # The maximum number of entries to store in the in-memory cache.
  # maxCacheEntries: 1024

  # The maximum number of bytes to store in the in-memory cache.
  # maxCacheBytes: 536870912

Remote

A persistent storage implementation that:

  • Persists data remotely to a provided object store.

  • Maintains an local-disk cache and in-memory cache of the working set.

Note
 When using a remote storage implementation as part of a distributed cluster of XTDB nodes, we must ensure that all nodes are able to efficiently communicate the stream of file changes they make to the remote storage. We achieve this inter-node messaging using a remote transaction log implementation. 
storage: !Remote
  # -- required

  # Configuration of the Object Store to use for remote storage
  # Each of these is configured separately - see below for more information.
  objectStore: <ObjectStoreImplementation>

  # Local directory to store the working-set cache in.
  # (Can be set as an !Env value)
  localDiskCache: /var/lib/xtdb/remote-cache

  ## -- optional

  # The maximum number of entries to store in the in-memory cache.
  # maxCacheEntries: 1024

  # The maximum number of bytes to store in the in-memory cache.
  # maxCacheBytes: 536870912

  # The max percentage of space to use on the filesystem for the localDiskCache directory (overriden by maxDiskCacheBytes, if set).
  # maxDiskCachePercentage: 75

  # The upper limit of bytes that can be stored within the localDiskCache directory (unset by default).
  # maxDiskCacheBytes: 107374182400

Each Object Store implementation is configured separately - see the individual documentation for more information: