Kafka

Apache Kafka can be used as an XTDB transaction log.

Setup

1. Add a dependency to the com.xtdb/xtdb-kafka module in your dependency manager.

2. On your Kafka cluster, XTDB requires a Kafka topic to use for its transaction log. This can be created manually, or automatically by XTDB (see below).

   • If setting up the topic manually, ensure that the topic has:

     • A single partition - this ensures that the transaction log is strictly ordered.

     • message.timestamp.type set to LogAppendTime - this ensures that the timestamp of the message is the time it was appended to the log, rather than the time it was sent by the producer.

   • If allowing XTDB to create the topic automatically, ensure that the connection properties supplied to the XTDB node have the appropriate permissions to create topics.

Configuration

To use the Kafka module, include the following in your node configuration:

txLog: !Kafka
  # -- required

  # A comma-separated list of host:port pairs to use for establishing the
  # initial connection to the Kafka cluster.
  # (Can be set as an !Env value)
  bootstrapServers: "localhost:9092"

  # Name of the Kafka topic to use for the transaction log.
  # (Can be set as an !Env value)
  topicName: "xtdb-log"

  # -- optional

  # Whether or not to automatically create the topic, if it does not already exist.
  # autoCreateTopic: true

  # The maximum amount of time to block waiting for records to be returned by the Kafka consumer.
  # pollDuration: "PT1S"

  # Path to a Java properties file containing Kafka connection properties,
  # supplied directly to the Kafka client.
  # (Can be set as an !Env value)
  # propertiesFile: "kafka.properties"

  # A map of Kafka connection properties, supplied directly to the Kafka client.
  # propertiesMap:

  # The replication factor of the transaction log topic (if it is automatically created by XTDB).
  # see https://kafka.apache.org/documentation/#replication.factor
  # replicationFactor: 1

  # A map of topic configuration options to use when creating the transaction log topic
  # (if it is automatically created by XTDB).
  # see https://kafka.apache.org/documentation/#topicconfigs
  # topicConfig:

For examples on how to enable/configure the Kafka module as part of your node, for each client library, see the individual driver documentation: