Clojure Configuration Cookbook

Changelog (last updated v2.1)

v2.1: multi-database support

Prior to 2.1, :log and :storage were top-level keys, and :databases was not used.

For example:

{:log [:in-memory {...}]
 :storage [:in-memory {...}]}}

The :disk-cache and :memory-cache keys were nested under the local/remote storage:

{:storage [:local
           {;; -- required

            :path "/var/lib/xtdb/storage"

            ;; -- optional

            ;; :max-cache-bytes 1024
            ;; :max-cache-entries 536870912
           }]}

{:storage [:remote {;; --required
                    :object-store [:object-store-implementation {}]

                    :local-disk-cache "/tmp/local-disk-cache"

                    ;; -- optional
                    ;; :max-cache-entries 1024
                    ;; :max-cache-bytes 536870912
                    ;; :max-disk-cache-percentage 75
                    ;; :max-disk-cache-bytes 107374182400
                    }]}

This document provides examples for the EDN configuration of XTDB components, to be supplied to xtdb.node/start-node.

See the main configuration documentation for more details.

Log

Main article: Log

In-Memory

Main article: in-memory log

This is the default, and can be omitted.

{:databases
 {:xtdb
  {:log [:in-memory
         {;; -- optional

          ;; :instant-src (java.time.InstantSource/system)
          }]}}}

Local disk

Main article: local-disk log

{:databases
 {:xtdb
  {:log [:local
         {;; -- required
          ;; accepts `String`, `File` or `Path`
          :path "/tmp/log"

          ;; -- optional

          ;; accepts `java.time.InstantSource`
          ;; :instant-src (InstantSource/system)

          ;; :buffer-size 4096
          ;; :poll-sleep-duration "PT1S"
          }]}}}

Kafka

Main article: Kafka

{:databases
 {:xtdb
  {:log [:kafka
         {;; -- required
          :bootstrap-servers "localhost:9092"
          :topic-name "xtdb-log"

          ;; -- optional

          ;; :create-topic? true
          ;; :poll-duration #xt/duration "PT1S"
          ;; :properties-file "kafka.properties"
          ;; :properties-map {}
          ;; :replication-factor 1
          ;; :topic-config {}
          }]}}}

Storage

Main article: Storage

In-Memory

Main article: in-memory storage

This is the default, and should be omitted.

Local disk

Main article: local-disk storage

{:databases
 {:xtdb
  {:storage [:local
             {;; -- required

              ;; accepts `String`, `File` or `Path`
              :path "/var/lib/xtdb/storage"

              ;; -- optional

              ;; :max-cache-bytes 536870912
             }]}}}

Remote

Main article: remote storage

{:databases
 {:xtdb
  {:storage [:remote {;; -- required

                      ;; Each object store implementation has its own configuration -
                      ;; see below for some examples.
                      :object-store [:object-store-implementation {}]}]}}

 ;; -- required for remote storage
 ;; Local directory to store the working-set cache in.
 :disk-cache {;; -- required

              ;; accepts `String`, `File` or `Path`
              :path "/tmp/local-disk-cache"

              ;; -- optional
              ;; The maximum proportion of space to use on the filesystem for the diskCache directory
              ;; (overridden by maxSizeBytes, if set).
              :max-size-ratio 0.75

              ;; The upper limit of bytes that can be stored within the diskCache directory (unset by default).
              :max-size-bytes 107374182400}

 ;; -- optional - in-memory cache created with default config if not supplied
 ;; configuration for XTDB's in-memory cache
 ;; if not provided, an in-memory cache will still be created, with the default size
 :memory-cache {;; -- optional

                ;; The maximum proportion of the JVM's direct-memory space to use for the in-memory cache
                ;; (overridden by `:max-size-bytes`, if set).
                :max-size-ratio 0.5

                ;; unset by default
                :max-size-bytes 536870912}

}

S3

Main article: S3

{:databases
 {:xtdb
  {:storage [:remote
             {:object-store [:s3
                             {;; -- required
                              :bucket "my-bucket"

                              ;; -- optional

                              ;; :prefix "my-xtdb-node"
                              ;; :configurator (reify S3Configurator
                              ;;                 ...)
                             }]}]}}}

Azure Blob Storage

Main article: Azure Blob Storage

{:databases
 {:xtdb
  {:storage [:remote
             {:object-store [:azure
                             {;; -- required
                              ;; --- At least one of storage-account or storage-account-endpoint is required
                              :storage-account "storage-account"
                              ;; :storage-account-endpoint "https://storage-account.privatelink.blob.core.windows.net"
                              :container "xtdb-container"

                              ;; -- optional

                              ;; :prefix "my-xtdb-node"
                              ;; :user-managed-identity-client-id "user-managed-identity-client-id"
                             }]}]}}}

Google Cloud Storage

Main article: Google Cloud Storage

{:databases
 {:xtdb
  {:storage [:remote
             {:object-store [:google-cloud
                             {;; -- required
                              :project-id "xtdb-project"
                              :bucket "xtdb-bucket"

                              ;; -- optional

                              ;; :prefix "my-xtdb-node"
                             }]}]}}}