Skip to content
XTDB

Azure Blob Storage

Azure Blob Storage can be used as a shared object-store for XTDB’s remote storage module.

Note

The Azure Blob Storage module uses an Event Grid topic and Service Bus Topic to maintain a local copy of the file listings on Azure, saving on expensive/lengthy operations to list objects on Azure Blob Storage.

If not using the Resource Manager template, ensure that you have a similar setup to track file changes on whichever storage account/container you are using, and that XTDB has relevant permissions to use all of the services.

Setup

First, ensure the com.xtdb/xtdb-azure Maven dependency is added to your XTDB node.

Infrastructure

Note
Resource Manager template

We provide a parameterized 'Azure Resource Manager' stack to help set up everything that you need.

To use Azure Blob Storage as the object store, the following infrastructure is required:

  1. An Azure Storage Account.

    1. A Storage Account Container within the storage account.

  2. An EventGrid System Topic, subscribed to blob creation/deletion events on the container.

  3. A standard tier Service Bus Namespace

    1. A Service Bus Topic within the namespace.

  4. A Custom Role Definition, granting XTDB permission to:

    1. Use the created storage container (Get, Put, Delete and List)

    2. Read messages sent to EventGrid

    3. Create Subscriptions on the Service Bus Topic

{
  "permissions": [
    {
      "actions": [
        "Microsoft.ServiceBus/*",
        "Microsoft.Storage/storageAccounts/blobServices/containers/write",
        "Microsoft.Storage/storageAccounts/blobServices/containers/delete",
        "Microsoft.Storage/storageAccounts/blobServices/containers/read",
        "Microsoft.EventGrid/systemTopics/read",
        "Microsoft.EventGrid/systemTopics/eventSubscriptions/read",
        "Microsoft.EventGrid/systemTopics/eventSubscriptions/write",
        "Microsoft.EventGrid/systemTopics/eventSubscriptions/delete",
        "Microsoft.EventGrid/systemTopics/eventSubscriptions/getFullUrl/action",
        "Microsoft.EventGrid/systemTopics/eventSubscriptions/getDeliveryAttributes/action"
      ],
      "notActions": [],
      "dataActions": [
        "Microsoft.ServiceBus/*",
        "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read",
        "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/write",
        "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/delete",
        "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/add/action",
        "Microsoft.Storage/storageAccounts/blobServices/containers/blobs/move/action"
      ],
      "notDataActions": []
    }
  ]
}

Authentication

Authentication is done via the Azure SDK, which uses the DefaultAzureCredential. See the Azure Documentation for setup instructions.

Configuration

To use the Azure module, include the following in your node configuration:

storage: !Remote
  objectStore: !Azure
    # -- required

    # The storage account which has the storage container
    # (Can be set as an !Env value)
    storageAccount: storage-account

    # The name of the blob storage container to be used as the object store
    # (Can be set as an !Env value)
    container: xtdb-container

    # The service bus namespace which contains the serviceBusTopicName
    # (Can be set as an !Env value)
    serviceBusNamespace: xtdb-service-bus

    # The name of the service bus topic which is collecting notifications from the container
    # (Can be set as an !Env value)
    serviceBusTopicName: xtdb-service-bus-topic

    # -- optional
    # A file path to prefix all of your files with
    # - for example, if "foo" is provided, all XTDB files will be located under a "foo" sub-directory
    # (Can be set as an !Env value)
    # prefix: my-xtdb-node

  localDiskCache: /var/cache/xtdb/object-store

Examples

Clojure Kotlin

For examples on how to enable/configure the Azure module as part of your node, for each client library, see the individual driver documentation: