Setting up a cluster on Azure

This guide will walk you through the process of configuring and running an XTDB Cluster on Azure. This setup includes:

• Using Azure Blob Storage as the remote storage implementation.

• Utilizing Apache Kafka as the shared transaction log implementation.

• Exposing the cluster to the internet via HTTP and the Postgres Wire Protocol, while collecting & exposing metrics via a Prometheus exporter.

The application is deployed on the Azure Managed Kubernetes Service (AKS) using the provided Kubernetes configuration files. The Azure infrastructure — including the AKS cluster and all related resources, services, and roles used by XTDB — is deployed via the provided Terraform templates. These can all be found on the XTDB repository.

Although we provide numerous parameters to configure the templates, you are encouraged to edit them, use them as a foundation for more advanced use cases, and reuse existing infrastructure when suitable. These templates serve as a simple starting point for running XTDB on Azure and Kubernetes, and should be adapted to meet your specific needs, especially in production environments.

This guide assumes that you are using the default templates.

Requirements

Before starting, ensure you have the following installed:

• The Azure CLI - See the Installation Instructions.

• Terraform - See the Installation Instructions.

• kubectl - The Kubernetes CLI, used to interact with the AKS cluster. See the Installation Instructions.

• Download the contents of the folder containing the Terraform templates & Kubernetes config, from the XTDB repository.

Authenticating the Azure CLI

Within Azure, ensure that you have an existing Subscription, and that you are authenticated with the Azure CLI.

Ensure that your existing Subscription has the necessary resource providers - see this article for more information. This guide requires the following providers:

• Microsoft.ContainerService - for the AKS resources.

• Microsoft.ManagedIdentity - for the user assigned managed identity resources.

• Microsoft.Storage - for the storage account resources.

To login to Azure using the command line, run the following:

az login --scope https://management.azure.com//.default

To explicitly check that CLI commands run against the correct subscription, run:

az account set --subscription "Subscription Name"

This allows you to perform necessary operations on Azure via Terraform using the User Principal on the Azure CLI.

Note	There are other ways to authenticate Terraform with Azure besides using the User Principal available via the Azure CLI. For other authentication scenarios, see the azurerm backend authentication docs.

Getting started with Terraform

The following assumes that you are authenticated on the Azure CLI, have Terraform installed on your machine, and are located within the root terraform directory provided above.

From the root of the directory, run:

cd terraform

Before applying or creating any resources, initialize the Terraform working directory. This will download the necessary provider plugins for Azure, set up the sample modules, and configure the backend for storing the state.

First, create a new resource group to store the Terraform state. For this guide, we’ll use "xtdbterraform" as the resource group name and "East US" as the location. These can be changed as appropriate.

az group create --name xtdbterraform --location "East US"

Create the storage account to store the Terraform state - the name of the $TfStateStorageAccount must be globally unique.

az storage account create --name $TfStateStorageAccount --resource-group xtdbterraform --location "East US" --sku Standard_LRS

Within this storage account, create the storage container to store the Terraform state.

az storage container create --name terraformstate --account-name $TfStateStorageAccount

Run the following command, substituting the names of the above as appropriate:

terraform init \
  -backend-config="resource_group_name=xtdbterraform" \
  -backend-config="container_name=terraformstate" \
  -backend-config="storage_account_name=$TfStateStorageAccount"

The directory is now initialized, and ready to be used to create the resources.

What is being deployed on Azure?

The sample Terraform directory contains different modules - each responsible for different parts of the infrastructure. If using the default configuration, the following will be created:

• At the top level: an XTDB resource group and user assigned managed identity.

• Remote Storage infrastructure - contains the Azure infrastructure XTDB needs for using Azure Storage Blobs as Remote Storage. This sets up a storage account, containing the blob storage container XTDB will use.

• AKS infrastructure - sets up the Azure infrastructure necessary for setting up an XTDB cluster & simple kafka on Managed Kubernetes:

  • An AKS cluster with a default system node pool - size configured by the aks_node_pool_vm_size

    • This will default to Standard_D2_v2 (2 vCPUs, 7 GiB memory).

  • A separate application node pool - size configured by the aks_app_node_pool_vm_size - the XTDB nodes and sample Kafka will run on this pool.

    • This will default to Standard_D8_v3 (8 vCPUs, 32 GiB memory).

  • A federated identity for the AKS cluster, which will use the user assigned managed identity to access the storage account.

Note	The above infrastructure is designed for creating a simple, production ready starting point for running XTDB on Azure & Kubernetes. The VM sizes and resource tiers can & should be adjusted to suit your specific requirements and cost constraints.

Deploying the Azure Infrastructure

Before creating the Terraform resources, review and update the terraform.tfvars file to ensure the parameters are correctly set for your environment:

• You are required to set a unique and valid storage_account_name for your environment.

• You may also wish to change resource tiers, the location of the resource group, or the VM sizes used by the AKS cluster.

  • The VM sizes used within the examples may not always be available in your subscription - if this is the case, see alternative/equivalent VM sizes that you can use within the Azure VM Sizes document.

  • Ensure that the quota for the VM size and region is set appropriately in Subscription > Settings > Usage + Quotas.

To get a full list of the resources that will be deployed by the templates, run:

terraform plan

Finally, to create the resources, run:

terraform apply

This will create all the resources within the Azure subscription and save the state of the resources within the storage account created earlier.

Fetching the Terraform Outputs

The Terraform templates will generate several outputs required for setting up the XTDB nodes on the AKS cluster.

To retrieve these outputs, execute the following command:

terraform output

This will return the following outputs:

• kubernetes_namespace

• kubernetes_service_account_name

• storage_account_container

• storage_account_name

• user_managed_identity_client_id

Deploying on Kubernetes

With the infrastructure created on Azure, you can now deploy the XTDB nodes and a simple Kafka instance on the AKS cluster.

Prior to deploying the Kubernetes resources, ensure that the kubectl CLI is installed and configured to deploy and connect to the AKS cluster. Run the following command:

az aks get-credentials --resource-group xtdb-resources --name xtdb-aks-cluster

This will configure kubectl to use the credentials for the Managed Kubernetes cluster.

You will also need to create both a namespace and a service account for the deployments. The values for these are provided in the Terraform configuration (terraform.tfvars) to create the necessary federated identity used by the AKS cluster - and can also be found in the Terraform outputs.

The names used by Terraform default to xtdb-deployment for the namespace and xtdb-service-account for the service account. To create them on the AKS cluster, run:

kubectl create namespace xtdb-deployment
kubectl create serviceaccount xtdb-service-account --namespace xtdb-deployment

The AKS cluster is now ready for deployment,

Deploying the Kafka Instance

Note

The Kafka module deployed within the AKS cluster is simple and unauthenticated, and is not intended for production use. We allow XTDB itself to manage the Kafka topic creation and configuration in this example - in practice, we recommend using a production ready Kafka deployment, creating the topic in advance, and configuring XTDB to use it. See the XTDB Kafka Setup Docs for more information on Kafka configuration recommendations.

To deploy the Kafka instance, run the following command from the base of the kubernetes directory:

kubectl apply -f kafka.yaml

This will create:

• A 100GiB persistent volume claim for the Kafka data, which is backed by Azure Disks.

• A simple Kafka Deployment on the AKS cluster, which XTDB will use as the Transaction Log.

• A Kubernetes service to expose the Kafka instance to the XTDB cluster.

To check the status of the Kafka deployment, run:

kubectl get pods --namespace xtdb-deployment

To view the logs of the Kafka deployment, run:

kubectl logs -f deployments/kafka-app --namespace xtdb-deployment

Deploying the XTDB cluster

Prior to deploying the XTDB cluster, there are a few configuration settings that need to be updated. These can be found within the xtdb.yaml file, specifically in the ConfigMap at the top of the file. The following values should be updated based on the Terraform outputs:

• XTDB_AZURE_USER_MANAGED_IDENTITY_CLIENT_ID - set this to the user_managed_identity_client_id output.

• XTDB_AZURE_STORAGE_ACCOUNT - set this to the storage_account_name output.

• XTDB_AZURE_STORAGE_CONTAINER - set this to the storage_account_container output.

To deploy the XTDB cluster, run the following command from the base of the kubernetes directory:

kubectl apply -f xtdb.yaml

This will create:

• A ConfigMap containing shared configuration for the XTDB nodes.

• A StatefulSet containing the XTDB nodes.

• A 50 GiB persistent volume claim for each member of the stateful set.

• A LoadBalancer Kubernetes service to expose the XTDB cluster to the internet.

• A Headless Kubernetes service to expose Prometheus exporter endpoints to the rest of the AKS cluster.

To check the status of the XTDB deployment, run:

kubectl get pods --namespace xtdb-deployment

To view the logs of each individual StatefulSet member, run:

kubectl logs -f xtdb-statefulset-n --namespace xtdb-deployment

Note

We recommend using the templates provided as a starting point, modifying the configuration to suit the specific requirements of your environment. Of particular note is the options passed in via the ConfigMap, and the settings within the XTDB stateful set. It is also important to note that the load balanced service is exposed to the internet within the base configuration, so this should also be adjusted to suit security requirements.

Accessing the XTDB Cluster

Once the XTDB cluster is up and running, you can access it via the LoadBalancer service that was created.

To get the external IP of the LoadBalancer service, run:

kubectl get svc xtdb-service --namespace xtdb-deployment

This will return the external IP of the LoadBalancer service. You can use this IP to access the XTDB cluster via the Postgres Wire Protocol (on port 5432), or over HTTP (on port 3000).

curl http://$ExternalIP:3000/status

If the above command succeeds, you now have a load-balanced XTDB cluster accessible over the internet.

Extension: Monitoring with Grafana

Below we give an example for setting up a basic version of Grafana & Prometheus - and then setup our XTDB monitoring dashboards.

For this you will require:

• The Kubernetes config for these - found under the azure-setup-guide/kubernetes directory.

• The Grafana dashboards - found under our shared grafana/dashboards directory on the XTDB Repo.

Deploying Prometheus & Grafana to AKS

The XTDB nodes each run a Prometheus exporter, which exposes metrics about the Java process, node resource usage and performance. These can be scraped by a Prometheus instance running on the AKS cluster, and visualized in Grafana.

We provide a simple deployment for both Prometheus & Grafana in the kubernetes directory, which can be deployed by running:

kubectl apply -f metrics.yaml

This will create:

• A Prometheus deployment that scrapes the Prometheus exporter endpoints on the XTDB nodes.

• A Kubernetes service to expose the Prometheus instance to the rest of the AKS cluster.

• A 5GiB persistent volume claim for the Grafana data.

• A Deployment for the Grafana instance.

• A Kubernetes service to expose the Grafana instance to the internet.

To check the status of the Prometheus and Grafana deployments, run:

kubectl get pods --namespace xtdb-deployment

To access the Grafana instance, you can use the external IP of the LoadBalancer service created for Grafana:

kubectl get svc grafana --namespace xtdb-deployment

The Grafana dashboard can be accessed via the external IP of the LoadBalancer service, on port 3000. The default credentials are admin/admin.

Adding Prometheus Data Source to Grafana

XTDB can be added as a Prometheus data source to Grafana, allowing for the creation of dashboards and alerts based on the metrics exposed by the XTDB nodes.

To add the Prometheus data source:

• Navigate to the Grafana instance in the browser

• Go to Connections → Add new connection.

• Search for Prometheus in the data source list.

• Click Add new data source.

• Set the Name to prometheus.

• Set the Prometheus server URL to http://prometheus:9090, then Save & test.

Adding the XTDB Dashboard to Grafana

With the Prometheus data source added, the XTDB metrics are now being scraped by Grafana. We provide pre-configured Grafana dashboards for:

• XTDB Cluster Monitoring

• XTDB Node Debugging

These are located under the shared grafana/dashboards directory on the XTDB Repo.

To add these dashboards:

• Navigate to the Dashboards → New → New dashboard.

• Click Import dashboard.

• Upload dashboard JSON file → Upload the dashboard file from the example directory.

  • Select the Prometheus data source.

  • Click Import.

These will create useful dashboards for monitoring the XTDB cluster and the nodes.