Skip to content
XTDB

Google Cloud Storage

Google Cloud Storage can be used as a shared object-store for XTDB’s remote storage module.

Setup

First, ensure the com.xtdb/xtdb-google-cloud Maven dependency is added to your XTDB node.

Infrastructure

Prior to deploying any infrastructure, you will need to ensure that the following APIs are active within your Google Cloud Project:

  • Cloud Storage API

  • IAM API

  • Cloud Deployment Manager API (if using our template)

Note
Cloud Deployment Manager template

We provide a parameterized 'Cloud Deployment Manager' template to help set up everything that you need.

To use Google Cloud Storage as the object store, the following infrastructure is required:

  • A Cloud Storage bucket

  • A custom role for all of the necessary permissions for XTDB to use the bucket (get, create, delete, list, and update storage objects - see below for example role from the template)

    type: gcp-types/iam-v1:projects.roles
name: custom-role-name
properties:
  parent: projects/project-name
  roleId: custom-role-name
  role:
    title: XTDB Custom role
    stage: GA
    description: Custom role for XTDB - allows usage of containers.
    includedPermissions:
      - storage.objects.create
      - storage.objects.delete
      - storage.objects.get
      - storage.objects.list
      - storage.objects.update
      - storage.buckets.get

Authentication

Authentication is done via Google’s "Application Default Credentials" - see the Google Cloud documentation for setup instructions.

Configuration

To use the Google Cloud module, include the following in your node configuration:

storage: !Remote
  objectStore: !GoogleCloud
    # -- required

    # The name of the GCP project that the bucket is contained within
    # (Can be set as an !Env value)
    projectId: xtdb-project

    # The Cloud Storage bucket which the documents will be stored within
    # (Can be set as an !Env value)
    bucket: xtdb-bucket

    # -- optional
    # A file path to prefix all of your files with
    # - for example, if "foo" is provided, all XTDB files will be located under a "foo" sub-directory
    # (Can be set as an !Env value)
    # prefix: my-xtdb-node

  localDiskCache: /var/cache/xtdb/object-store