Skip to content
XTDB

Google Cloud

XTDB provides modular support for Google Cloud environments, including a prebuilt Docker image, integrations with Google Cloud Storage, and configuration options for deploying onto Google Cloud infrastructure.

Required Infrastructure

In order to run a Google Cloud based XTDB cluster, the following infrastructure is required:

  • A Google Cloud Storage bucket for remote storage.

  • A Kafka cluster for transaction logs.

    • For more information on setting up Kafka for usage with XTDB, see the Kafka configuration docs.

  • A service account with the necessary permissions to access the storage bucket and Kafka cluster.

  • XTDB nodes configured to communicate with the Kafka cluster and Google Cloud Storage.

Note
 We would recommend running XTDB in a Google Kubernetes Engine (GKE) cluster, which provides a managed Kubernetes environment in Google Cloud.

xtdb-google-cloud Docker Image

The xtdb-google-cloud image is optimized for running XTDB in Google Cloud environments and is deployed on every release to XTDB.

xtdb/xtdb-google-cloud:2.0.0-beta4

By default, it will use Google Cloud Storage for storage and Kafka for transaction logs, including dependencies for both.

Configuration

The following environment variables are used to configure the xtdb-google-cloud image:

Variable Description

KAFKA_BOOTSTRAP_SERVERS

Kafka bootstrap server containing the XTDB topics.

XTDB_TX_TOPIC

Kafka topic to be used as the transaction log.

XTDB_FILES_TOPIC

Kafka topic to be used for node file notifications.

XTDB_GCP_PROJECT_ID

GCP project ID containing the bucket.

XTDB_GCP_BUCKET

Name of the Google Cloud Storage bucket used for remote storage.

XTDB_GCP_LOCAL_DISK_CACHE_PATH

Path to the local disk cache.

XTDB_LOGGING_LEVEL

Sets the root logging level for the node - defaults to INFO. See "Increasing the Log Level" for more information.

XTDB_NODE_ID

Persistent node id for labelling Prometheus metrics.

Using a Custom Node Configuration

For advanced usage, XTDB allows the above YAML configuration to be overridden to customize the running node’s system/modules.

In order to override the default configuration:

  1. Mount a custom YAML configuration file to the container.

  2. Override the COMMAND of the docker container to use the custom configuration file, ie:

    CMD ["-f", "/path/to/custom-config.yaml"]

Google Cloud Storage

Google Cloud Storage can be used as a shared object-store for XTDB’s remote storage module.

Infrastructure Requirements

To use Google Cloud Storage as the object store, the following infrastructure is required:

  1. A Google Cloud Storage bucket

  2. A custom role with the necessary permissions for XTDB to use the bucket:

    type: gcp-types/iam-v1:projects.roles
name: custom-role-name
properties:
  parent: projects/project-name
  roleId: custom-role-name
  role:
    title: XTDB Custom Role
    stage: GA
    description: Custom role for XTDB - allows usage of containers.
    includedPermissions:
      - storage.objects.create
      - storage.objects.delete
      - storage.objects.get
      - storage.objects.list
      - storage.objects.update
      - storage.buckets.get

Authentication

XTDB uses Google’s "Application Default Credentials" for authentication. See the Google Cloud documentation for setup instructions.

Configuration

To use the Google Cloud module, include the following in your node configuration:

storage: !Remote
  objectStore: !GoogleCloud
    ## -- required

    # The name of the GCP project containing the bucket
    # (Can be set as an !Env value)
    projectId: xtdb-project

    # The Cloud Storage bucket to store documents
    # (Can be set as an !Env value)
    bucket: xtdb-bucket

    ## -- optional
    # A file path to prefix all files with
    # - for example, if "foo" is provided, all XTDB files will be under a "foo" sub-directory
    # (Can be set as an !Env value)
    # prefix: my-xtdb-node

  localDiskCache: /var/cache/xtdb/object-store