ADBC reference

Changelog (last updated v2-NEXT)

v2-NEXT: FlightSQL clients authenticate per the node’s authentication rules

The FlightSQL listener applies the same authentication rules as pgwire, and a session’s writes run as the user it authenticated as — see Authentication.

Previously the listener accepted every caller and its writes ran as no user, so rules restricting pgwire didn’t restrict FlightSQL, and a role granted to a user didn’t apply to their FlightSQL session.

Where the rules ask for a password, give FlightSQL clients the driver’s username and password options. Under the default rules nothing changes for a client sending no credentials; one sending a username runs as that user.

v2-NEXT: a FlightSQL transaction or prepared statement belongs to the session that created it

Another session presenting its handle is told it doesn’t exist — see Session options.

Previously any client presenting a handle could use it, and commit a transaction as whoever began it.

No upgrade steps for a client that keeps its handles within its own session; one passing a handle to another authenticated session has to stop. Clients sending no credentials share one identity, so they still reach one another’s handles.

XTDB exposes ADBC, the Arrow Database Connectivity standard, through two surfaces:

• In-process, via node.connect() on the JVM, which returns an org.apache.arrow.adbc.core.AdbcConnection. Zero copies, direct access to the in-memory Arrow buffers.
• Over the wire, via the Apache Arrow Flight SQL server bundled into XTDB. Any ADBC FlightSQL driver (Python, Rust, Go, C, R, Java) connects to it.

The FlightSQL producer is a thin shim over the same in-process AdbcConnection / AdbcStatement implementation, so a behaviour documented here holds on either surface, barring the per-client caveats noted below. Signatures and examples are given in the in-process Kotlin API; over the wire the same operations are whatever your driver’s ADBC binding calls them.

Connecting

In-process (Kotlin / JVM)

An org.apache.arrow.adbc.core.AdbcConnection is obtained from a running node via node.connect():

import xtdb.api.Xtdb

Xtdb.openNode().use { node ->
    node.connect().use { conn ->
        conn.createStatement().use { stmt ->
            stmt.setSqlQuery("SELECT 1")
            stmt.executeQuery().use { result ->
                // result.reader: org.apache.arrow.vector.ipc.ArrowReader
            }
        }
    }
}

Over the wire (FlightSQL)

Enable the FlightSQL listener in your node config; the Docker standalone image does this by default on port 3000:

flightSql:
  host: '*'
  port: 3000

Then connect with any ADBC FlightSQL driver pointed at grpc://localhost:3000. Connection setup is driver-specific: see the connection snippet on the Python, Java, Kotlin, or Go driver page, or the Apache ADBC driver matrix for other languages.

The FlightSQL listener doesn’t serve TLS, so credentials travel in the clear: front it with a TLS-terminating proxy in production.

Authentication

The FlightSQL listener applies the node’s authentication rules, as pgwire does, matching them against the user a client names and the address it connects from.

A client authenticates with the driver’s username and password options, which it sends as a Basic authorization header. A client authenticating with OAuth client credentials sends them as authorization: Client <base64(client-id:client-secret)> instead: with the Go driver, set adbc.flight.sql.authorization_header; with the Java driver, set adbc.flight.sql.rpc.call_header.authorization, along with adbc.flight.sql.rpc.with_cookie_middleware=true so the session carries across calls. Such a client names no user, so its credentials are verified with the identity provider first, and the rules are then matched against the user that provider reports for them - for Keycloak, the client’s service-account user id - and let it in only where they ask for CLIENT_CREDENTIALS. A rule admitting it over FlightSQL therefore names that user id, where a pgwire rule names the username its client connects with. A client sending no credentials is let in only where the rules ask for TRUST. Where they ask for TRUST, a client sending a username is taken at its word as that user, whatever password it sends, as over pgwire. The rules are checked when a session starts, as pgwire checks them when a connection opens.

The response to the first authenticated call carries authorization: Bearer <session>, which the drivers send back on every later call in place of the credentials. A session’s user doesn’t change: a later call presenting different credentials from those the session was authenticated with fails, as does a bearer token naming a closed session. Authentication failures are UNAUTHENTICATED; a call an authenticated user isn’t permitted to make is UNAUTHORIZED.

A rule asking for DEVICE_AUTH rejects a FlightSQL client with UNIMPLEMENTED: no driver can show the approval URL mid-call.

Statements

createStatement()

opens an AdbcStatement, the entry point for every query and DML.

setSqlQuery(sql) / prepare() / bind(batch) / executeQuery()

the standard ADBC statement lifecycle. prepare() is optional but required before bind(); bind() takes one Arrow record batch of parameters. executeQuery() returns an ArrowReader that yields one Arrow batch at a time, so large result sets need not fit in memory.

val stmt = conn.createStatement()
stmt.setSqlQuery("SELECT ?, ?")
stmt.prepare()
stmt.bind(argBatch)
val result = stmt.executeQuery()

executeUpdate()

runs DML. Returns -1 (XTDB does not pre-count affected rows). Effects are visible to the next query on the same connection (see Transactions for multi-statement semantics).

executeSchema()

returns the Arrow schema of a query’s result set without running it, for tooling that needs the result columns up front. Works on prepared statements (read from getResultSetSchema) and ad-hoc queries (the wire path opens a transient PreparedQuery). Because it runs before any bind, a result schema that depends on parameter types is resolved against null-typed placeholders. For SELECT cols FROM t WHERE _id = ? (where the projection doesn’t depend on the parameter) the schema is accurate; a query that projects a parameter directly (SELECT ?, SELECT ? + 1) comes back null-typed for those fields. Project through a column expression that fixes the type if you need it resolved.

Bulk ingest

bulkIngest(table, mode)

lands an Arrow table (or any Arrow-shaped data: record batches, streams, anything exposing the Arrow C Data Interface) in table in a single round trip. Over the wire it maps to the FlightSQL CommandStatementIngest command; batches stream in one at a time, so the whole table needn’t fit in memory at either end. Each call commits atomically on its own, regardless of the connection’s commit mode.

conn.bulkIngest("people", BulkIngestMode.CREATE_APPEND).use { stmt ->
    stmt.bind(peopleBatch)        // _id, name, age …
    stmt.executeUpdate()
}

Because XTDB creates tables on demand, the ingest modes either coincide or don’t apply:

Mode	Behaviour
create	Accepted. Table is auto-created if missing.
append	Accepted. Upserts on _id.
create_append	Accepted (the common case).
replace	Rejected (INVALID_ARGUMENT). Would require an explicit ERASE step.
create with fail-if-exists	Rejected. XTDB auto-creates; “fail if exists” can’t be honoured without an existence check.
append with fail-if-not-exists	Rejected. XTDB auto-creates on insert; silently accepting would violate the ADBC contract.

Further constraints:

• Every row must have an _id column. Rows without one are rejected; materialise an _id client-side before ingesting if your source Arrow lacks one.
• A temporal _valid_from / _valid_to column must be a timestamp, not a date.
• A per-call catalog override is rejected (the connection-scoped catalog is the only source of truth); a per-call schema override is honoured, defaulting to public.

Mode rejections return a descriptive gRPC INVALID_ARGUMENT. Other ingest-time failures are not yet uniformly mapped: a missing _id surfaces as gRPC INTERNAL, so match on the message, not the status code, until that mapping is tightened.

Transactions

autoCommit = false / commit() / rollback()

multi-statement transactions, on both surfaces. Switch off autocommit, run statements, then commit or roll back on the connection.

conn.autoCommit = false
conn.createStatement().use { stmt ->
    stmt.setSqlQuery("INSERT INTO users (_id, name) VALUES ('alice', 'Alice')")
    stmt.executeUpdate()
}
conn.commit()

Because XTDB advertises FlightSQL transaction support (FLIGHT_SQL_SERVER_TRANSACTION, SqlInfo id 8), a PEP 249 wire client (e.g. the Python dbapi) defaults to manual-commit: DML stays in an uncommitted transaction until you commit. bulkIngest is the exception (each call commits on its own).

Transaction options. The connection’s transaction API is the only way to open and end a transaction: BEGIN, START TRANSACTION, COMMIT and ROLLBACK are rejected as SQL. To make a transaction read-only, pin its snapshot, or back-fill at a given system-time, run SET TRANSACTION as its first statement - e.g. SET TRANSACTION READ ONLY, SNAPSHOT_TOKEN = 'ChYKBHh0ZGISDgoMCKHqs8cGEPCP2poB'.

Visibility inside an open transaction. XTDB buffers a transaction’s DML and applies it atomically on COMMIT, so reads on the same connection do not see that transaction’s own pending writes before it commits. A SELECT issued between an uncommitted INSERT and the COMMIT returns the pre-transaction state.

Committed-write visibility. Once a write is committed (in autocommit mode, or after commit()), subsequent reads on the same connection see it with no manual await: the connection tracks its own write tokens and threads them through the planner. Cross-connection visibility follows the usual transactional semantics; another connection sees the write once it commits and its read snapshot advances.

See Prepared statements and transactions for the full lifecycle, including the execute vs executemany DML routing on the Python dbapi.

Metadata

getObjects(depth, …)

returns the catalog → schema → table → column tree, to the requested depth. At depth = ALL each table’s Arrow schema is carried as bytes in the standard table_schema VarBinaryVector, which the client deserialises into a Schema.

getTableSchema(catalog, schema, table)

returns the Arrow schema of a single table.

val objs   = conn.getObjects(GetObjectsDepth.ALL, null, "public", "users", null, null)
val schema = conn.getTableSchema(null, "public", "users")

Both reflect live data, not just flushed-to-block state: freshly-inserted rows show up immediately, the same way they do via SQL SELECT.

getTableTypes()

returns TABLE (XTDB has one table type).

getInfo(…)

returns VENDOR_NAME = "XTDB" and DRIVER_NAME = "XTDB ADBC Driver", with VENDOR_VERSION and DRIVER_VERSION both XTDB’s version, e.g. "2.2.0" — the same value as current_setting('xtdb.version').

Session options

Every call runs in a FlightSQL session. A call naming no session — neither the session cookie nor the bearer token — starts a new one, authenticated from its credentials, and its response carries the cookie; the client keeps session state - the catalog, a SET - across calls only by sending that cookie, or the bearer token, back. A session is closed once it goes flightSql.sessionIdleTimeout (default 30 minutes) without a call presenting its cookie, or flightSql.sessionUnclaimedTimeout (default 1 minute) if its cookie never comes back. A call presenting the cookie of a closed session fails with NOT_FOUND.

getCurrentCatalog() / getCurrentDbSchema()

return the session catalog (default xtdb) and schema (public). Read via getSessionOptions.

setSessionOptions("catalog", db)

selects the database for the session. The value is validated against the node’s known databases: an unknown name returns INVALID_VALUE, an empty string clears it.

closeSession()

ends the session, closes its connections, and invalidates the cookie. An open transaction is left open; an anonymous client can still commit or roll it back by its handle, but an authenticated session’s is reachable only from that session, so it rolls back once it has gone flightSql.transactionIdleTimeout (default 30 minutes) without a call naming it.

A transaction or prepared statement is reached by its handle, from the session that created it: another session presenting the handle is told it doesn’t exist. Clients connecting without credentials, where the rules allow it, share one identity, so any of them can reach the others’ handles, as before.

The schema is not settable: public is the only accepted value (a confirming no-op), anything else is rejected.

Not supported

• Setting the schema session option to anything but public.
• Bulk-ingest replace and existence-check modes (rejected with an explanatory INVALID_ARGUMENT).
• Per-call catalog override on bulk ingest.
• The Substrait variant of executeSchema (getSchemaSubstraitPlan): no current driver consumer.
• getCurrentCatalog / getCurrentDbSchema on the Java FlightSQL client: an upstream Apache ADBC gap (its Java client doesn’t query getSessionOptions), not an XTDB limitation. The in-process JVM connection is unaffected.
• Parameter type inference for executeSchema (placeholder types only).
• TLS on the FlightSQL listener.
• DEVICE_AUTH rules over FlightSQL.