S3

S3 can be used as a shared object-store for XTDB’s remote storage module.

Setup

First, ensure the com.xtdb/xtdb-aws Maven dependency is added to your XTDB node.

Infrastructure

Note	CloudFormation stack We provide a parameterized CloudFormation stack to help set up everything that you need. The stack accepts the (globally unique) name of an S3 bucket as an input - this will be created, and referenced in associated resources - to be used in your XTDB configuration.

To use S3 as the object store, the following infrastructure is required:

1. An S3 bucket.

2. IAM policies which grant XTDB permission to the S3 bucket:

   Statement:
   - Effect: Allow
     Action:
       - 's3:GetObject'
       - 's3:PutObject'
       - 's3:DeleteObject'
       - 's3:ListBucket'
       - 's3:AbortMultipartUpload'
       - 's3:ListBucketMultipartUploads'
     Resource:
       - !Ref S3BucketArn
       - !Join [ '', [ !Ref S3BucketArn, '/*'] ]

Authentication

Authentication is done via the AWS SDK, using the default AWS credential provider chain. See the AWS documentation for setup instructions.

Configuration

To use the S3 module, include the following in your node configuration:

storage: !Remote
  objectStore: !S3
    ## -- required

    ## The name of the S3 bucket to use for the object store
    ## (Can be set as an !Env value)
    bucket: "my-s3-bucket"

    ## -- optional

    ## A file path to prefix all of your files with
    ## - for example, if "foo" is provided, all XTDB files will be located under a "foo" sub-directory
    ## (Can be set as an !Env value)
    # prefix: my-xtdb-node

    ## Basic credentials for AWS.
    ## If not provided, will default to AWS's standard credential resolution.
    ## see: https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials-chain.html
    # credentials:
    #   accessKey: "..."
    #   secretKey: "..."

    ## Endpoint URI
    ## If not provided, will default to the standard S3 endpoint for the resolved region.
    # endpoint: "https://..."

  localDiskCache: /var/cache/xtdb/object-store

If configured as an in-process node, you can also specify an S3Configurator instance - this is used to modify the requests sent to S3.