Lotse guides
Lotse is a message broker written in Rust. It offers reliable queues and persistent streams on one replicated log, with exactly-once delivery.
These guides are for four readers:
- Setup: install Lotse, start a cluster and send a first message.
- Administration: run a cluster: certificates, users and roles, nodes, retention, monitoring.
- Development: write a client against the native protocol, use the Rust, Java and Kotlin clients, or change Lotse itself.
- Use: work with streams, queues, exchanges, transactions and the console.
The guides say how to do things. The reasons behind each decision are in the architecture decision records; the storage and protocol design is in the design document (German).
Install
Lotse is one binary, lotse. It runs one node of a cluster, makes the cluster’s certificates and checks the cluster file. There are three ways to get it.
What you need
- Linux on x86_64 or aarch64. The release binaries are static.
- Three machines for a cluster that survives the loss of one. One machine is enough to try Lotse (a one-machine trial).
- Two TCP ports per node. The examples use 7000 for the other nodes (
peer) and 9000 for clients (client). Both are set per node in the cluster file. - A disk for the data directory. Retention keeps 7 days by default, so the size follows your traffic.
Each release has static binaries, Debian packages and a container image (ADR 0049).
Debian or Ubuntu
The package installs the binary, the unit lotse@.service, the system user lotse and an example cluster file at /usr/share/doc/lotse/cluster.toml.example.
sudo apt install ./lotse_<version>_amd64.deb
lotse --version
Replace <version> with the release you downloaded. The next chapter, A first cluster, continues from here.
Container image
The image ghcr.io/pierrejochem/lotse is a static build on a distroless base. A node takes its id and cluster file from the environment:
| Variable | Same as | Meaning |
|---|---|---|
LOTSE_NODE | --node | the node’s id |
LOTSE_CONFIG | --config | the cluster file |
LOTSE_JOIN | --join | 1 or true: a new node joining a running cluster |
docker run -e LOTSE_NODE=1 \
-v /etc/lotse:/etc/lotse:ro -v lotse-1:/var/lib/lotse \
-p 7000:7000 -p 9000:9000 ghcr.io/pierrejochem/lotse
Mount the cluster file and the certificates read-only and the data in a volume. In the image the data directory is /var/lib/lotse, so data_dir in the cluster file should be that path or one below it. The image sets LOTSE_CONFIG=/etc/lotse/cluster.toml, so mount the cluster file there. It exposes ports 7000 and 9000.
Build from source
You need a current stable Rust toolchain.
git clone https://github.com/pierrejochem/lotse
cd lotse
cargo build --release -p lotse-node # target/release/lotse
cargo deb -p lotse-node builds the Debian package from the repository (install the tool with cargo install cargo-deb), and docker build -f packaging/Dockerfile -t lotse . builds the image.
Check the install
lotse --version
prints the version, for example lotse 0.0.1. lotse with no command prints the usage of run, certs, check and node.
A first cluster
This chapter sets up three nodes with TLS 1.3. Every node reads the same cluster file. Streams, queues and exchanges are not in the file; clients create them.
The cluster file
Copy the example to /etc/lotse/cluster.toml and change the addresses. The Debian package ships it as /usr/share/doc/lotse/cluster.toml.example.
# A Lotse cluster file (ADR 0047): the same on every node. Copy it to
# /etc/lotse/cluster.toml, name your nodes, make the certificates with
#
# lotse certs --config /etc/lotse/cluster.toml
#
# and start each node with `systemctl enable --now lotse@<id>`.
[cluster]
data_dir = "/var/lib/lotse"
[[node]]
id = 1
peer = "10.0.0.1:7000"
client = "10.0.0.1:9000"
metrics = "10.0.0.1:9464" # /metrics, /healthz, /readyz, /safe-to-stop (ADR 0117); leave out to serve none
[[node]]
id = 2
peer = "10.0.0.2:7000"
client = "10.0.0.2:9000"
metrics = "10.0.0.2:9464"
[[node]]
id = 3
peer = "10.0.0.3:7000"
client = "10.0.0.3:9000"
metrics = "10.0.0.3:9464"
[tls]
ca = "/etc/lotse/ca.pem"
cert = "/etc/lotse/node-{id}.pem"
key = "/etc/lotse/node-{id}.key"
crl = "/etc/lotse/crl.pem" # `lotse certs --revoke <cert>` adds to it; same file on every node
[retention]
ms = 604800000 # 7 days; -1 keeps everything
[auth]
admins = ["admin"] # signs in with `lotse certs --client admin`
What the keys mean:
| Key | Meaning |
|---|---|
[cluster] data_dir | where a node keeps its data. {id} becomes the node’s id, so /var/lib/lotse/node-{id} gives each node its own directory on a shared machine. |
[[node]] id | a number from 1 up, unique. |
[[node]] peer | host:port where the other nodes reach this node. |
[[node]] client | host:port where clients reach this node. |
[tls] ca, cert, key | the CA certificate and the node’s certificate and key. {id} is replaced as in data_dir. |
[tls] crl | the revocation list. Optional. |
[retention] ms | how long streams keep records, in milliseconds. -1 keeps everything. The default is 7 days. |
[auth] admins | the principals that may create tenants and grant roles. Required while auth.required is on, which is the default. |
The file is checked strictly: an unknown key is an error. Administration will list every key; this guide uses only those above.
Check the file
lotse check --config /etc/lotse/cluster.toml
For the example it prints:
3 nodes: 1, 2, 3
TLS: certificates from [tls]
retention: 604800000 ms by age, no limit per partition
clients sign in with: certificate, SCRAM-SHA-512; admins: admin
Once certificates exist, lotse check also names any that expire within 30 days.
Make the certificates
lotse certs --config /etc/lotse/cluster.toml
lotse certs --config /etc/lotse/cluster.toml --client admin --out ./admin
The first command writes a CA (ca.pem, ca.key), a certificate and key per node, and an empty revocation list where [tls] names them. Node certificates last one year, the CA ten; --days sets another length. lotse certs keeps files that exist, so you can run it again after you add a node to the file.
The second command writes a client certificate ./admin/admin.pem and key ./admin/admin.key for the principal admin, the name auth.admins lists. Use this certificate to sign in as the first administrator.
ca.key signs every certificate of the cluster. Move it off the nodes after you have made the certificates. Copy ca.pem, crl.pem and each node’s own node-<id>.pem and node-<id>.key to its machine, readable by the user that runs lotse.
Start the nodes
On each machine, with its own id:
lotse run --config /etc/lotse/cluster.toml --node 1
A node stops cleanly on Ctrl-C or SIGTERM. LOTSE_LOG sets the log filter, for example LOTSE_LOG=debug; the default is info.
Run under systemd
The Debian package installs lotse@.service. The text after @ is the node’s id. The unit reads /etc/lotse/cluster.toml, runs as the user lotse and restarts the node on failure.
sudo install -d /etc/lotse
sudo cp /usr/share/doc/lotse/cluster.toml.example /etc/lotse/cluster.toml # then edit it
sudo lotse certs --config /etc/lotse/cluster.toml
sudo chown -R lotse: /etc/lotse
sudo systemctl enable --now lotse@1
Run the last command on each machine with its own id (lotse@2, lotse@3). Keep data_dir at /var/lib/lotse or below: the unit lets the node write nowhere else. Follow a node with journalctl -u lotse@1 -f.
This guide ran lotse check and lotse certs on the example file’s shape and started three nodes on one machine (the trial). It did not run the systemd unit or a cluster across three machines.
Next
A first message sends and reads a record, and shows the cluster in the console. Certificates, renewal, revocation, authentication, roles, and adding and draining nodes belong to the administration guide.
A one-machine trial
For a first look you do not need three machines, certificates or logins. Two settings switch them off, and only on loopback.
The cluster file
Three nodes on 127.0.0.1, each with its own ports and data directory:
[cluster]
data_dir = "/tmp/lotse-trial/node-{id}"
[[node]]
id = 1
peer = "127.0.0.1:7001"
client = "127.0.0.1:9001"
[[node]]
id = 2
peer = "127.0.0.1:7002"
client = "127.0.0.1:9002"
[[node]]
id = 3
peer = "127.0.0.1:7003"
client = "127.0.0.1:9003"
[tls]
plaintext_on_loopback = true
[auth]
required = false
Save it as trial.toml. plaintext_on_loopback = true runs the node links and the client port as plain TCP. auth.required = false lets every client do everything. Neither is a setting for production, and the node enforces the limits:
[tls]may not name certificates together withplaintext_on_loopback.- Every address in the file must be on loopback.
auth.requiredwithplaintext_on_loopbackis refused, because nodes prove who they are by their certificates.
lotse check --config trial.toml
3 nodes: 1, 2, 3
TLS: none, loopback only
retention: 604800000 ms by age, no limit per partition
clients: no authentication, everyone may do everything
Run the nodes
Start each node in its own terminal:
lotse run --config trial.toml --node 1
lotse run --config trial.toml --node 2
lotse run --config trial.toml --node 3
Each node logs a warning that it listens with plain TCP. That is expected here. Clients connect to 127.0.0.1:9001, 127.0.0.1:9002 and 127.0.0.1:9003. Stop a node with Ctrl-C and start it again to see it rejoin from its data directory.
Or: make dev
From a checkout of the repository, make dev starts three real nodes and the console with sample traffic:
make dev # web app, three nodes, console on http://127.0.0.1:8080
make dev ADDR=127.0.0.1:8081
It builds the web app first if console/web/dist is missing, which needs Node.js and npm. The nodes keep their data in a fresh temporary directory that is removed when you stop with Ctrl-C. A client writes to a stream orders (3 partitions) and a queue jobs every 250 ms, so the console has something to show. This cluster also runs without TLS and authentication, and you sign in to the console as the user local.
When you are done
Delete /tmp/lotse-trial. The next chapter, A first message, works against either cluster.
A first message
This chapter writes ten records to a stream and reads them back, then looks at them in the console. It assumes the one-machine trial is running. Against a TLS cluster you set the CA and a client certificate instead; see below.
With the Java client
The Java 17 client is in sdk/java of the repository. The example StreamExample creates the stream example-orders with 2 partitions and 3 replicas, writes ten records and reads them back. Its source is compiled by ./gradlew check, so it keeps up with the API:
package dev.lotse.examples;
import dev.lotse.client.ClientException;
import dev.lotse.client.Fetched;
import dev.lotse.client.LotseClient;
import dev.lotse.client.OutMessage;
import dev.lotse.client.ProduceResult;
import dev.lotse.record.BatchRecord;
import java.time.Duration;
/** Creates a stream, writes ten messages to it and reads them back. */
public final class StreamExample {
private StreamExample() {}
public static void main(String[] args) {
try (LotseClient client = new LotseClient(Env.config().build())) {
try {
client.createStream("example-orders", 2, 3);
} catch (ClientException.ResourceExists e) {
// made by an earlier run
}
for (int i = 0; i < 10; i++) {
// The partition is the caller's to choose; here, by key.
String customer = "customer-" + (i % 3);
int partition = Math.floorMod(customer.hashCode(), 2);
ProduceResult r = client.produce(
"example-orders", partition, OutMessage.value("order-" + i).withKey(customer));
System.out.printf("order-%d at partition %d, offset %s%n", i, partition,
Long.toUnsignedString(r.baseOffset()));
}
for (int partition = 0; partition < 2; partition++) {
long offset = 0;
while (true) {
Fetched f = client.fetch("example-orders", partition, offset, Duration.ofMillis(200));
for (BatchRecord record : f.committedMessages()) {
System.out.printf("read %s from partition %d%n", record.value().toUtf8(), partition);
}
long next = f.nextOffset(offset);
if (next == offset) {
break; // nothing more for now
}
offset = next;
}
}
}
}
}
The examples read their connection settings from the environment (sdk/java/src/examples/java/dev/lotse/examples/Env.java):
| Variable | Meaning |
|---|---|
LOTSE_BOOTSTRAP | brokers as host:port, separated by commas. The default is 127.0.0.1:9092, so set it. |
LOTSE_CA | the CA certificate, to use TLS. Without it the example uses plain TCP, which the client allows on loopback only. |
LOTSE_CERT, LOTSE_KEY | a client certificate and its key, with LOTSE_CA. |
LOTSE_USER, LOTSE_PASSWORD | a SCRAM-SHA-512 login, if the cluster asks for one. |
For the trial:
export LOTSE_BOOTSTRAP=127.0.0.1:9001,127.0.0.1:9002,127.0.0.1:9003
For the TLS cluster of A first cluster:
export LOTSE_BOOTSTRAP=10.0.0.1:9000,10.0.0.2:9000,10.0.0.3:9000
export LOTSE_CA=/etc/lotse/ca.pem LOTSE_CERT=admin/admin.pem LOTSE_KEY=admin/admin.key
The expected output has ten lines of the form order-3 at partition 1, offset 0, then ten lines of the form read order-3 from partition 1. The offsets start at 0 on a new stream and the order across partitions is not defined; within a partition it is the order written.
This guide has not run the example end to end. The repository’s Gradle build has no task that starts an example; it only compiles them. Put the compiled classes of the examples source set and the main classes on the class path and run dev.lotse.examples.StreamExample. The developer guide covers the clients in full.
With the Rust client
lotse-client is the Rust client over the native protocol. Client::new(rt, ClientConfig::new(bootstrap)) connects, create_stream(name, partitions, replication) creates a stream, produce(stream, partition, batch) writes a record batch and returns its offset, and fetch reads from an offset. The client is generic over the runtime in lotse-rt; on a real machine that is TokioRuntime from lotse-rt-tokio with .plaintext_on_loopback() for the trial or .with_tls(...) for a TLS cluster. The working code to start from is crates/lotse-console/examples/local_cluster.rs, which produces to a stream and a queue in a loop.
Check it in the console
The console shows what the cluster holds. For the trial cluster, write a console file lotse-console.toml:
listen = "127.0.0.1:8080"
[auth]
mode = "dev"
user = "trial"
[[clusters]]
id = "trial"
kind = "native"
bootstrap = ["127.0.0.1:9001", "127.0.0.1:9002", "127.0.0.1:9003"]
Build the web app once, then start the console:
cd console/web && npm ci && npm run build && cd ../..
cargo run -p lotse-console -- --config lotse-console.toml
Open http://127.0.0.1:8080. The stream example-orders is listed with its partitions, replicas and offsets, and the charts show produce and fetch rates. This guide has not run this exact console file; the keys follow console/lotse-console.example.toml and the comments in crates/lotse-console/src/config.rs, where a native cluster’s ca_file is optional and plain TCP is allowed on loopback only.
Against a TLS cluster, add ca_file = "/etc/lotse/ca.pem" to the cluster entry. The console passes the signed-in user’s identity on to the cluster, so each user sees only what their roles allow (ADR 0040). In the dev mode used here the console acts as the one configured user; use it only on a machine only you can reach.
With make dev you can skip all this: the console is already running at http://127.0.0.1:8080 with the cluster local, and your Java client can connect to the client addresses in the log line three nodes running, because that cluster picks free ports at random.
The cluster file
One TOML file describes the whole cluster. It is the same file on every node; a node learns which entry is its own from --node (or LOTSE_NODE). Streams, queues and exchanges are not in the file: clients create them at runtime (ADR 0047).
Unknown keys are refused. So is a file that could not run: no [[node]], a duplicate id or address, missing certificates, plain TCP off loopback, authentication without TLS, or authentication required with no admin. lotse check --config cluster.toml reads the file and says what it describes.
Example
[cluster]
data_dir = "/var/lib/lotse"
[[node]]
id = 1
peer = "10.0.0.1:7000"
client = "10.0.0.1:9000"
[[node]]
id = 2
peer = "10.0.0.2:7000"
client = "10.0.0.2:9000"
[[node]]
id = 3
peer = "10.0.0.3:7000"
client = "10.0.0.3:9000"
[tls]
ca = "/etc/lotse/ca.pem"
cert = "/etc/lotse/node-{id}.pem"
key = "/etc/lotse/node-{id}.key"
crl = "/etc/lotse/crl.pem"
[retention]
ms = 604800000
[auth]
admins = ["admin"]
The Debian package installs the same example as /usr/share/doc/lotse/cluster.toml.example.
{id} in paths
In cluster.data_dir, tls.cert and tls.key, the text {id} becomes the node’s id. Several nodes can then share one machine and one file. tls.ca and tls.crl are taken as written.
[cluster]
| Key | Type, unit | Default | Meaning |
|---|---|---|---|
data_dir | string, required | none | Where a node keeps its data. |
tick_ms | integer, ms | 10 | The Raft tick. Must be at least 1. |
segment_bytes | integer, bytes | 134217728 (128 MiB) | Log segment size. Retention deletes whole segments. Must be at least 4096. |
group_session_timeout_ms | integer, ms | 45000 | A consumer group member without a heartbeat for this long is removed. |
group_heartbeat_ms | integer, ms | 5000 | How often group members send heartbeats. |
follower_max_lag_ms | integer, ms | 10000 | How long a follower may go without having applied what its leader said was committed and still serve reads to clients in its zone; after that it sends them to the leader, like Kafka’s replica.lag.time.max.ms (ADR 0116). |
[[node]]
One entry per node, at least one.
| Key | Type | Default | Meaning |
|---|---|---|---|
id | integer, required | none | The node’s id, 1 or more, unique. |
peer | host:port, required | none | The address other nodes dial. |
client | host:port, required | none | The address clients dial. |
peer_bind | host:port | the peer address | Where the node listens for nodes, if not on peer, such as 0.0.0.0:7000. |
client_bind | host:port | the client address | Where the node listens for clients, if not on client. |
zone | string | none | The node’s zone (a data centre or availability zone). Clients that name the same zone fetch from its replicas (ADR 0104). A node added later takes it with lotse node add --zone. |
metrics | host:port | none | Where the node serves /metrics, /healthz, /readyz and /safe-to-stop over HTTP (ADR 0117). Plain HTTP on a loopback address, TLS with the node’s certificate anywhere else. Left out, nothing listens. See Monitoring. |
Every peer, client and metrics address must be unique in the file.
[tls]
Either name ca, cert and key, or set plaintext_on_loopback. Naming both is refused. See Certificates.
| Key | Type | Default | Meaning |
|---|---|---|---|
ca | path | none | The CA certificate, PEM. |
cert | path with {id} | none | The node’s certificate chain, PEM. |
key | path with {id} | none | The node’s private key, PEM. |
crl | path | none | The certificate revocation list, PEM. Optional. A certificate it names is refused in every handshake (ADR 0060). |
renewal_ca | path | none | The renewal CA’s certificate, PEM, made by lotse certs --renewal-ca. Optional. With it, a client certificate is renewed over a connection made with it (ADR 0123). Not with plaintext_on_loopback. |
renewal_ca_key | path | renewal_ca ending in .key | The renewal CA’s private key, PEM. Only with renewal_ca. |
plaintext_on_loopback | boolean | false | No TLS. Allowed only when every address, including peer_bind and client_bind, is on loopback. Needs auth.required = false. For trying a cluster on one machine. |
Plain TCP is refused off loopback (ADR 0037).
[retention]
What streams keep unless they say otherwise. A stream created at runtime can override them; see Retention and compaction.
| Key | Type, unit | Default | Meaning |
|---|---|---|---|
ms | integer, ms | 604800000 (7 days) | Records older than this go. -1 for no limit. |
bytes | integer, bytes | no limit | Each partition keeps at least this much. -1 for no limit. |
Any other value must be positive.
[compaction]
How compacted streams are compacted; see Retention and compaction.
| Key | Type, unit | Default | Meaning |
|---|---|---|---|
map_bytes | integer, bytes | 67108864 (64 MiB) | Memory the key map of the node’s one compaction sweep may take, 48 bytes per key. |
[disk]
How much free space each node keeps on its data volume (ADR 0122).
| Key | Type, unit | Default | Meaning |
|---|---|---|---|
free_limit | bytes or a share | the larger of 1 GiB and 5 % | Below this much free space a node takes no new data in the groups it votes in. Bytes as a number (2147483648) or text with a unit ("2GiB", "500MB"; KB, MB, GB, TB, KiB, MiB, GiB, TiB), a share of the volume ("5%", "2.5%"), or 0 for no limit. |
check_interval_ms | integer | 1000 | How often a node reads its volume’s free space, 10 to 60000. A node that crosses the limit reports it at its next read (ADR 0124). |
A node reads its volume every check_interval_ms. Below the limit it is low: the leaders of every group with a replica on it refuse produces, publishes and dead letters with STORAGE_FULL, which clients retry; reads, acks and metadata changes go on. It is no longer low once its free space is a tenth above the limit. The node tells the metadata group at once, so other nodes learn of it within a few seconds; the node itself refuses at once for the groups it leads. See Nodes.
[producers]
How each partition keeps idempotent producers (ADR 0110). Use the same values on every node.
| Key | Type, unit | Default | Meaning |
|---|---|---|---|
max | integer | 10000 | Producers a partition admits. While it knows this many, a new producer is refused with PRODUCER_LIMIT and its client retries. A known producer is never refused or forgotten to make room. |
expiration_ms | integer, ms | 86400000 (a day) | How long a producer may write nothing to a partition before its leader forgets it. A retry after that is taken as a new batch. |
Both must be at least 1.
[auth]
| Key | Type, unit | Default | Meaning |
|---|---|---|---|
required | boolean | true | Clients must authenticate. With false, everyone may do everything. |
mtls | boolean | true | A client certificate names its principal. |
scram | boolean | true | SASL SCRAM-SHA-512, with users set at runtime. |
admins | list of strings | empty | Principals that are cluster admins from the start. Must not be empty while required is true. |
tenants | list of strings | empty | Tenants besides default that exist from the start. |
max_session_s | integer, s | 3600 | A session ends after this at the latest. |
With required true, at least one of mtls, scram and oidc must be on, and TLS is required: nodes prove who they are by their certificates.
[auth.oidc]
Present only when tokens from an identity provider are accepted (SASL OAUTHBEARER).
| Key | Type | Default | Meaning |
|---|---|---|---|
issuer | string, required | none | The issuer, as in the tokens’ iss. |
audience | string, required | none | The audience tokens must be for. |
principal_claim | string | sub | The claim that names the principal. |
groups_claim | string | groups | The claim that lists the user’s groups. |
jwks_url | URL | the issuer’s discovery document | The key set’s URL. |
Environment variables
| Variable | Used by | Meaning |
|---|---|---|
LOTSE_CONFIG | every command that reads the file | The cluster file, when --config is not given. |
LOTSE_NODE | lotse run | The node’s id, when --node is not given. |
LOTSE_JOIN | lotse run | 1 or true makes the node join a running cluster, like --join. |
LOTSE_LOG | lotse run | The log filter, in the syntax of the tracing env filter. Default info. |
LOTSE_PASSWORD | lotse node | The password for --user. |
A flag wins over the variable. The systemd unit and the container image use these variables (ADR 0049).
Commands
lotse run --config <cluster.toml> --node <id> [--join]
lotse certs --config <cluster.toml> [--days <n>]
lotse certs --config <cluster.toml> --client <principal> --out <dir> [--days <n>]
lotse certs --config <cluster.toml> --revoke <cert.pem>
lotse check --config <cluster.toml>
lotse node add --config <cluster.toml> --node <id> [--peer <host:port> --client-addr <host:port>] <admin>
lotse node remove --config <cluster.toml> --node <id> <admin>
lotse node list --config <cluster.toml> <admin>
lotse --version
<admin> is --cert <file> --key <file> (a client certificate) or --user <name> with the password in LOTSE_PASSWORD.
lotse run stops cleanly on Ctrl-C or SIGTERM, after handing the groups its node leads to other voters (at most 30 s). lotse check prints the node count, the TLS mode, the certificates on this machine that expire within 30 days, the retention defaults and the sign-in methods.
Certificates
Every connection speaks TLS 1.3 with certificates from the cluster’s CA. Node links need a node certificate on both ends; clients may show a certificate that names their principal (ADR 0037).
Issue
lotse certs --config cluster.toml
lotse certs --config cluster.toml --client alice --out ./alice
The first command makes, at the paths [tls] names:
- a CA:
ca.pem, with its key beside it asca.key, valid for 3650 days; - for each node, a certificate and key valid for 365 days. The certificate carries
lotse://node/<id>and the hosts of the node’s addresses; - an empty revocation list, if
[tls] crlnames a file that does not exist.
Files that exist are kept, so the command can run again after a node is added to the file. Keys are readable by their owner only. --days <n> sets another length for new node and client certificates.
The second command makes a client certificate whose common name is the principal: alice/alice.pem and alice/alice.key. It never overwrites existing files. This is how the first admin signs in; the name must be listed in auth.admins.
Keep ca.key off the nodes. It signs every certificate of the cluster. A larger setup can bring certificates from its own CA to the same paths instead.
Check expiry
lotse check --config cluster.toml
warns about the CA and node certificates on this machine that expire within 30 days or already have. Files that are not on this machine are skipped.
Renew
Remove the node’s certificate and key, then run lotse certs again:
rm /etc/lotse/node-2.pem /etc/lotse/node-2.key
lotse certs --config cluster.toml
A running node reads its CA, certificate, key and revocation list every 10 s. If any of them changed and the files make a working configuration, every handshake from then on uses them. No restart is needed (ADR 0060).
If the new files do not load (half written, a key that does not match the certificate, a list that does not parse), the node logs “TLS files do not load; keeping the material in use” once and goes on with what it has.
New files must reach the node’s machine yourself; lotse certs writes where [tls] says.
Expiry ends connections
A connection ends when the certificate its other end showed expires. Both ends do this. The client or node then reconnects; with renewed files in place the next handshake succeeds, without them it fails (ADR 0068). Clients that sign in with SCRAM or a token show no certificate and are not affected.
Revoke
lotse certs --config cluster.toml --revoke alice/alice.pem
adds the certificate to the revocation list [tls] crl names. The command keeps the entries already there, raises the list’s number, and refuses a certificate the cluster’s CA did not issue. --revoke goes alone, without --client or --days.
Then copy the list to every node. Within 10 s each node:
- refuses the revoked certificate in every handshake, in both directions and on both link kinds;
- ends every open connection whose other end showed that certificate.
Other connections stay open. Revocation is a file; there is no OCSP. Distributing the file to the nodes is up to you.
Revoke from the console or a client
A cluster admin can also revoke one client certificate without the CA key and without copying a file: the console’s Users and access page (tab Certificates) or lotse-client’s revoke_certificate send RevokeCertificate with the issuer (the SHA-256 of the issuer’s DER name and authority key identifier, so that two CAs with one name are told apart), the serial and the time the certificate expires (ADR 0090). Every node applies it at once, ends the connections that showed the certificate and refuses it until it expires. The CRL file stays for revoking offline and for node certificates, which this request does not cover.
See which certificates are in use
Every node reports the client certificates it accepts, and the cluster keeps them for 30 days after they expire. The console’s Certificates tab lists them, by expiry, with the principal each names, when a node last accepted it (at most an hour late) and its state: valid, expiring within 30 days, expired or revoked. A cluster admin can revoke a valid or expiring one from the list. lotse-client’s describe_certificates returns the same list (ADR 0091). Certificates revoked only in the CRL file and node certificates are not listed.
Ask for a client certificate
In the console’s Certificates tab, Request certificate asks the CA holder for a client certificate for a principal. Your browser makes an ECDSA P-256 key and a signing request; the private key is saved as <principal>.key once and goes nowhere else, and the console and the cluster hold only the request. The request waits for 7 days.
A cluster admin asks for any principal. Any other signed-in user asks only in their own name and sees only their own requests; asking again replaces their request that is still waiting, so each has at most one. Users’ own requests take at most 192 of the 256 places, which leaves room for admins. A certificate signs in without the identity provider’s groups, so it grants only the roles bound to the user’s name itself; the console says so when none is (ADR 0119).
Every node reports how many requests wait as lotse_certificate_requests_pending on its metrics endpoint, and the Certificates tab shows the number to cluster admins. An alert on it tells the CA holder when to sign (see Monitoring).
The CA holder, on the machine with the CA key, signs what is waiting:
lotse certs --config cluster.toml --sign --user admin # password in LOTSE_PASSWORD, or --cert and --key
The command shows each request’s principal, who asked and the fingerprint of the key, and signs those you confirm. --request <id> (the id or its first digits) with --yes signs that one without asking, never all of them; --out <dir> also writes each certificate as <principal>.pem; --days sets the validity (default 365). The certificate is made from the key and the principal only: whatever else a request asks for is ignored. The signed certificate is recorded in the cluster, which takes it only if it has the request’s key and principal and is signed by a CA certificate in the nodes’ [tls] ca file, and the console’s list offers it for download; it works with the key file saved earlier (ADR 0092).
Renew a client certificate over its connection
A client certificate renews itself, without the CA holder, once the cluster has a renewal CA (ADR 0123):
lotse certs --config cluster.toml --renewal-ca # on the machine with ca.key
makes, under the cluster’s CA, an intermediate CA that may sign client certificates only: renewal-ca.pem and its key renewal-ca.key at the path [tls] renewal_ca names. Add renewal_ca to the cluster file and copy both files to every node. Each node takes them up within 10 s. Keep ca.key offline as before.
A certificate may then be renewed from the earlier of its last 30 days and two thirds of its validity, until it expires; a year-long certificate opens on day 243. The client sends a signing request for a new key over a connection made with the certificate. The metadata leader signs a certificate for the same principal, valid for 90 days, and records it in the cluster. The old certificate keeps working until it expires.
lotse certs --config cluster.toml --renew --cert alice/alice.pem --key alice/alice.key
lotse certs --config cluster.toml --renew --if-due --cert alice/alice.pem --key alice/alice.key # from a daily timer
renews the certificate in --cert and writes the new key, then the new certificate (with the renewal CA’s after it), over the files. With --if-due it does nothing before the window opens. The clients for Java, Kotlin, C# and Node.js can renew by themselves, and every client has a call for it (see the client guides).
A certificate from the renewal CA signs in only while the cluster holds its record for that principal. A copy of the renewal CA’s key therefore mints nothing that signs in. Renewing the same certificate a second time replaces the first renewal, whose connections end. The Certificates tab lists renewed certificates like any other.
A revoked certificate does not end a user’s other ways in. To cut off a user’s sessions, see Revoking sessions.
Authentication and roles
Ways in
A client signs in in one of three ways (ADR 0038). Each can be switched off in [auth].
| Way | Key | How the principal is named |
|---|---|---|
| Client certificate | mtls | The certificate’s common name. |
| SASL SCRAM-SHA-512 | scram | The user name. Users and passwords are set at runtime. |
| SASL OAUTHBEARER | [auth.oidc] | The token’s principal_claim (default sub). |
With auth.required = true, the default, a connection may send nothing but ApiVersions until it has authenticated. With required = false, clients act as anonymous and may do everything; this needs plaintext_on_loopback and is for trying Lotse on one machine.
Certificates
Make a client certificate with lotse certs --client <principal> --out <dir>. See Certificates.
SCRAM users
Credentials live in the metadata group and are checked by every node from its applied state. The broker stores a salt, an iteration count and two derived keys, never the password; the client derives them, so the password does not leave it either. A cluster admin sets or removes a user through a client (set_user_password and delete_user in lotse-client, AlterUser on the wire) and lists users with DescribeUsers.
lotse node commands can sign in as a SCRAM user: pass --user <name> and put the password in LOTSE_PASSWORD.
OIDC tokens
Set [auth.oidc] with the issuer and the audience. A node fetches the issuer’s discovery document and key set itself and caches the keys. A token must:
- be signed with an asymmetric algorithm by a key of the set;
- name the configured issuer and audience;
- not be expired (60 s of slack for clocks);
- carry the principal in
principal_claim.
The groups in groups_claim (default groups) go with the principal for authorization. A token with an unknown key id makes a node fetch the key set again, at most every 30 s.
Sessions
A session lasts max_session_s (default 3600 s), and a token’s session no longer than the token. A client authenticates again on the same connection after four fifths of that time. Authenticating again cannot change the principal. A removed user or an expired token is thus cut off within the session time.
Tenants
Every resource belongs to a tenant. The tenant default always exists. auth.tenants names more that exist from the start; a cluster admin creates and deletes others at runtime (CreateTenant, DeleteTenant). A tenant that holds resources cannot be deleted. Resource names are unique across the cluster (ADR 0039).
Transactional ids and consumer group ids live in a tenant too (ADR 0062). A tenant deleted and created again starts afresh (ADR 0069).
Roles
Six roles. Four are per tenant; two are cluster-wide.
| Role | May |
|---|---|
cluster-admin | Everything, everywhere: users, tenants, role bindings, nodes, pinned leaders, session revocation. |
cluster-viewer | See every resource and tenant and the cluster’s own metrics, in every tenant. Reads no messages, writes nothing, sees neither users nor other principals’ bindings (ADR 0043). For monitoring. |
tenant-admin | Create, change, delete and bind resources in the tenant, and what the three roles below may. |
producer | Write: produce, publish, transactions. |
consumer | Read: fetch, subscribe, acknowledge, consumer groups and their offsets. |
viewer | See: metadata, settings, metrics. |
A node principal, shown by a node certificate, may look up metadata and write transaction markers, and nothing else.
What a principal may not see is unknown to it: metadata leaves it out, and DescribeTenants and DescribeMetrics show only what its roles allow. A request that needs a role the principal lacks is refused with UNAUTHORIZED and a reason.
Bindings
A binding gives a role to a user or to a group at the identity provider, in a tenant, for the resources whose names start with a prefix. An empty prefix means all resources of the tenant. The bindings cluster-admin and cluster-viewer are stored without tenant and prefix.
Bindings are stored in the metadata group. A cluster admin changes them with AlterRoleBindings (alter_role_bindings in lotse-client) and lists them with DescribeRoleBindings. A client asks which bindings apply to it with WhoAmI. A removed binding takes effect on each node once that node applies the metadata group.
The principals in auth.admins are cluster admins from the start. They create the first tenants and bindings. These are configured bindings and cannot be removed at runtime.
Revoking sessions
A cluster admin revokes the sessions of a user or of an IdP group with RevokeSessions (revoke_sessions in lotse-client) (ADR 0070). The request names the user or group and a time; a time of 0 means now, by the metadata leader’s clock.
A session counts as older than the revocation when its credential dates from before it:
- a token’s
iat(a token without one counts as time 0); - the moment of a SCRAM login;
- the moment a certificate connection was made.
Every node then closes the connections of revoked sessions and refuses logins and tokens that date from before the revocation. The client connects again:
- a SCRAM user signs in with the password and gets a new session;
- a token user needs a token issued after the revocation;
- a certificate user gets in again. To keep a certificate out, revoke it in the CRL (Certificates).
Revocation is per user or group, not per token.
See open sessions
Every node lists the client sessions it holds with DescribeSessions (node_sessions in lotse-client, nodeSessions in the SDKs, which ask every node): who, signed in how (certificate, SCRAM-SHA-512 or OAUTHBEARER), from which address, since when, when the credential dates from and when the session ends unless renewed (ADR 0121). Cluster admins and cluster viewers see every session; anyone else sees only the sessions made with their own name. The console shows the list on the Sessions tab of Users and access, with a revoke per user for those who see everyone.
A client connected to several nodes appears once per node.
Where to do it
lotse node add, remove and list are the only admin requests the lotse command line sends. Users, tenants, role bindings, settings, pins and session revocation go through a client: lotse-client, the Java and Kotlin SDKs, or the console, which manages users, role bindings, open sessions and their revocation on its Users and access page as the signed-in user: the browser derives a password’s SCRAM credential, so the console never sees the password (ADR 0089). The console signs users in with OpenID Connect and acts as the signed-in user (ADR 0040).
Nodes
The list of nodes lives in the metadata group. The nodes in the cluster file are the first ones. Later changes are made at runtime and need no restart of the other nodes (ADR 0048).
Every node hosts the metadata group (Raft group 1) and the coordinator group (group 2). Both have as many voters as there are active nodes, at most five, and an odd number where possible. Every other node follows the metadata group as a learner, so each has the metadata state.
Adding and removing nodes needs a cluster admin. Listing them needs a cluster viewer. <admin> below is --cert <file> --key <file> or --user <name> with LOTSE_PASSWORD set.
Add a node
-
Give the node a
[[node]]entry in the cluster file and copy the file to every node. Node ids are never used twice. -
Make its certificate:
lotse certs --config cluster.toml. Existing files are kept. -
Announce it to the cluster:
lotse node add --config cluster.toml --node 4 <admin>The addresses come from the node’s entry. If the node is not in the file, give
--peer <host:port>and--client-addr <host:port>. -
Start it on its machine with
--join:lotse run --config cluster.toml --node 4 --joinIn a container or under systemd, set
LOTSE_JOIN=1instead. With systemd, a drop-in made withsystemctl edit lotse@4can set it.
A joining node starts outside the configured groups. Their leaders make it a member. The voters the configured groups started with are kept on the node’s disk (first-voters) from its first start, so a later change of the file never alters them.
Rebalancing
A new node fills up by itself (ADR 0050). While no node drains, the metadata leader moves one replica at a time from the node with the most replicas to the one with the fewest. It stops when the two differ by at most one replica or a tenth of the fullest node’s count. At most four groups move at a time across the cluster. When a move is due, the group that evens out the two nodes’ bytes most goes first (ADR 0052).
Leaders spread too. Every node computes the same preferred leader for each group, one of its voters. A leader hands over to it with a Raft leader transfer, once that voter has held every committed entry for a minute: at most one handover per node every 100 ticks and per group every 1000. After a failover, the group goes back to a returning node about a minute after it has caught up. Each recent election beyond the first doubles the wait, up to 30 minutes, so a node short of CPU that keeps losing its groups stops getting them back every few seconds.
Each move is a full copy of the group through Raft. Rebalancing counts replicas; it uses sizes only to choose which group moves, so nodes with equal counts and very different sizes stay as they are. Traffic is not weighed.
Drain and remove a node
lotse node remove --config cluster.toml --node 2 <admin>
lotse node list --config cluster.toml <admin>
remove marks the node draining. Its replicas move to other nodes: for each group, the least loaded active node takes its place. lotse node list shows, per node, its addresses, whether it is active or draining, its replica count (replicas) and the groups moving (moving).
When no group counts the node as a voter and no move off it is under way, the metadata leader forgets it. The node itself stops, closing its listeners and groups, and lotse run exits with status 0, so a supervisor such as systemd (Restart=on-failure) does not start it again (ADR 0051).
A draining node that was cut off while the cluster forgot it asks the metadata leader for the node list every 100 ticks and stops when the list leaves it out. A node that cannot reach any node it lists keeps running until it can.
A drained node cannot come back under its old id. To return the machine to the cluster, join it under a new id.
Change replication
Every partition of a stream or queue created at runtime keeps the replica count it was created with until you change it (ADR 0115).
- For all partitions: set
replication.factorwithAlterConfigs, on its own in the request, from 1 to the number of active nodes. The planner adds the least loaded nodes or removes the busiest ones, four groups at a time; each new replica is a full copy through Raft. Resetting the key leaves the partitions as they are.DescribeConfigsshows the factor, or the first partition’s count. - For one partition, on nodes you choose: a cluster admin sends
ReassignPartition(reassign_partitioninlotse-client,reassignPartitionin the SDKs) with the partition’s voters, as with Kafka’s partition reassignment. Rebalancing and the factor then leave the partition where you put it; only a drain moves a replica off it. An empty list hands it back to the planner. A placement is refused while the partition is still moving: ask again once it has settled. - Setting
replication.factorreleases the placed partitions of that resource. - In the console: a stream’s and a queue’s page have Change replication, and each partition a Place button with the cluster’s nodes to choose from; a placed partition can be released there too.
Metadatasays which partitions are placed and which leaders are pinned, and the console marks them (ADR 0121).
Pin a leader
A cluster admin can pin a partition’s leader to one of its voters with MoveLeader (move_leader in lotse-client) (ADR 0065). It works for partitions of streams and queues created at runtime. A node id of 0 unpins.
- The pin is kept in the metadata state, so it holds across failovers and restarts. When the pinned node comes back, leadership returns to it.
- The handover waits until the target has every entry and is paced like the automatic one, but skips the minute’s wait: you asked for it.
- A pin to a node that is no longer a voter counts for nothing, so a drain or rebalance is never blocked by it. A pin is dropped when its resource is deleted or its node leaves the cluster.
The console can move leaders as the signed-in user (ADR 0066) and marks a pinned partition with its node; Metadata carries the pin (ADR 0121).
A node low on disk
Each node keeps free space on its data volume: [disk] free_limit in the cluster file, by default the larger of 1 GiB and 5 % (ADR 0122). A node below it is low.
- What stops: every group with a replica on the node takes no new data. Produces, publishes and dead letters are refused with
STORAGE_FULL; clients retry with the same bytes and give up with that refusal when their time runs out. On a cluster where every group has a replica on every node, that is every group. - What goes on: reads, acks and nacks, so consumers drain queues; transaction markers; metadata changes, such as creating or deleting a resource or changing
replication.factor; moving replicas. - How it shows:
lotse_disk_lowis 1 on the node’s/metrics,DescribeMetricssays so, and the console marks the node “Low on disk” with an alert. - How it ends: free space on the volume, delete data (a stream’s retention, or a resource you no longer need), or move replicas off the node by draining it or placing partitions elsewhere. New data is taken again once free space is a tenth above the limit, so a volume that hovers at the limit does not switch on and off.
A node reports that it is low, or back, to the metadata group at its next read of its volume (every second by default, [disk] check_interval_ms), so other nodes hear of it within a few seconds; the node refuses at once for the groups it leads. A low node that cannot reach the metadata group stays as it was recorded until it can. The default margin of 5 % leaves room for that and for snapshots.
Limits
- Four groups move at a time.
- A node that restarts with an old view of the metadata group asks for the node list and takes only the metadata group’s frames from the nodes it is missing until its own state lists them (ADR 0050).
Retention and compaction
Retention
A log is trimmed only at a snapshot point. When applying reaches the base offset of a segment, a replica records a snapshot of its state there, in a file of the group’s directory. Retention picks a point and the replica cuts at the newest such boundary at or below it, deleting whole segments (ADR 0046). Each node runs retention for each of its groups once a second, by its own clock.
Because only whole segments go, a partition keeps up to a segment more than its limits. cluster.segment_bytes (default 128 MiB) sets the size of a segment. The active segment is never deleted. Nothing at or above the last stable offset is deleted, so an open transaction’s records stay until it ends.
What each kind of log keeps
| Log | Rule |
|---|---|
| Stream | By age: segments whose newest record is older than retention.ms, by the record’s timestamp. By size: the oldest segments, as long as what stays still holds retention.bytes. The limit is a floor. |
| Queue | Everything below the queue’s ack floor goes. There is no setting. |
| Metadata and coordinator groups | Everything their state holds, once a whole segment can go. |
Age goes by the timestamps producers write.
Node defaults
In the cluster file:
[retention]
ms = 604800000 # 7 days, the default
bytes = -1 # no limit, the default
-1 means no limit. See The cluster file.
Per stream
A stream created at runtime can set its own values:
| Key | Unit | Meaning |
|---|---|---|
retention.ms | ms | Age limit. -1 for none. |
retention.bytes | bytes | Size floor per partition. -1 for none. |
They are set when the stream is created (CreateStream), changed or reset with AlterConfigs, and read with DescribeConfigs. A client does this: alter_configs(name, set, reset) and describe_configs(name) in lotse-client. A key that is not set comes back with the node’s value and the source default. A change names only the keys it sets or resets, so two changes of different keys both hold.
Streams from a node’s configuration keep the node’s defaults and cannot be changed at runtime. The lotse binary has no such streams: clients create everything.
What readers see
Fetch reports the log start offset. A read below it is answered with OFFSET_OUT_OF_RANGE. The Rust client reports that as ClientError::OffsetOutOfRange with the start. A consumer group member whose position was deleted goes on from the oldest record left.
Compaction
A stream with cleanup.policy set to compact or compact,delete is compacted (ADR 0067). Only streams created at runtime can be compacted.
| Key | Values | Default |
|---|---|---|
cleanup.policy | delete, compact, compact,delete | delete |
delete.retention.ms | ms; 0 removes tombstones at once, -1 keeps them for ever | 86400000 (1 day) |
compact deletes nothing by age or size. compact,delete does both. Set both keys in CreateStream or change them with AlterConfigs.
How it works
- Each replica compacts its own log, at its own pace. Compaction is not replicated, so replicas’ bytes may differ. Their entries never do.
- Sealed segments below the last stable offset and the applied index are rewritten whole. Batches keep their offsets, term and producer fields; records go.
- A record with a key goes once a newer record of the same key is in a sealed segment. A record of an aborted transaction goes. A record without a key stays.
- A tombstone, once it is the newest record of its key, goes when all three hold: it is older than
delete.retention.msby its timestamp; the replica leads and every member of the group, learners included, is known to hold it; and this replica’s older values of the key are gone. - A sweep first maps keys to their newest offset, from where the last sweep’s map ended, until the map is full or the sealed data ends; then it rewrites the segments below that point. A node runs one sweep at a time and its map may take
[compaction] map_bytes(default 64 MiB, about 1.4 million keys); partitions take turns. A partition with more keys than that takes several sweeps. - A sweep starts when new sealed data is there or the last one did not reach the end, otherwise a minute after the last one began. A step looks at up to eight segments and rewrites at most one.
- A segment is written as
<base>.log.cleaned, synced and renamed over the old one. A crash leaves the old segment or the new one whole, and opening the log removes a leftover.cleanedfile.
Limits
- Keys are told apart by 16 bytes of their SHA-256: two keys with one hash (odds of 2^-128 per pair) would count as one. There is no dirty ratio or minimum lag. After a restart the sweeps start from the log start again.
- Markers, Raft entries and the broker’s own batches are never compacted.
- A tombstone dropped while a replica that left the group still holds an older value is not covered by a test.
Monitoring
A node tells what it sees in two ways, and both need no separate exporter:
- Its operator’s endpoint,
/metrics,/healthz,/readyzand/safe-to-stopover HTTP, for Prometheus, Kubernetes and restart scripts (ADR 0117). DescribeMetricsover the native protocol, for the console and clients (ADR 0036).
The operator’s endpoint
Give a node metrics = "host:port" in its [[node]] entry. Without it, nothing listens.
[[node]]
id = 1
peer = "10.0.0.1:7000"
client = "10.0.0.1:9000"
metrics = "10.0.0.1:9464"
On a loopback address (127.0.0.1:9464) the endpoint speaks plain HTTP, and anyone on the machine may read all of it, for a Prometheus agent on each machine. Anywhere else it speaks only TLS 1.3, with the node’s certificate; lotse certs puts the metrics host into new node certificates (not a wildcard such as 0.0.0.0).
| Path | Answers | Who |
|---|---|---|
/metrics | OpenMetrics text: everything the node reports, with names. | Over TLS, a client certificate from the cluster CA whose principal holds cluster-viewer or cluster-admin, and which no cluster admin revoked. Anyone else gets 403. |
/healthz | 200 ok while the node’s main loop answers within 5 s, else 503. | Anyone. |
/readyz | 200 ready once every group on the node is ready, else 503 and each group with its reason. | Anyone. |
/safe-to-stop | 200 unless stopping the node now would leave a group without a quorum, else 503 and the groups. | Anyone. |
A group is ready on a node that leads it with a quorum that answered within an election timeout, or that follows a leader it hears and keeps up with, the rule followers serve reads by (ADR 0116). /safe-to-stop asks every node: for each group the node votes in, the newest leader’s list of members that answered lately, without this node, must still be a quorum. A group no node reports leading counts as not safe; one whose only voter is this node does not count, since waiting would not help. The health paths name group and node ids only, so a Kubernetes HTTPS probe needs no certificate.
Prometheus scrapes each node with a client certificate. Make one and bind it:
lotse certs --config cluster.toml --client prometheus --out prometheus/
Then bind the user prometheus to cluster-viewer, in the console or with AlterRoleBindings.
scrape_configs:
- job_name: lotse
scheme: https
tls_config:
ca_file: /etc/prometheus/lotse/ca.pem
cert_file: /etc/prometheus/lotse/prometheus.pem
key_file: /etc/prometheus/lotse/prometheus.key
static_configs:
- targets: ["10.0.0.1:9464", "10.0.0.2:9464", "10.0.0.3:9464"]
In Kubernetes:
readinessProbe:
httpGet: { path: /readyz, port: 9464, scheme: HTTPS }
livenessProbe:
httpGet: { path: /healthz, port: 9464, scheme: HTTPS }
failureThreshold: 6
A rolling restart waits for every node’s /readyz and for the next node’s /safe-to-stop before it stops that node. /readyz is 503 on every node while a group has no leader anywhere.
Metrics
Every name starts with lotse_; every sample has the label node. Partition metrics carry group, kind (stream, queue, metadata or coordinator), tenant, resource and partition. Counters run since the node started, and a group’s counters grow only while this node leads it, so summing over nodes counts each produce once.
| Metric | Type | Meaning |
|---|---|---|
lotse_node_info{version} | info | The node and its version. |
lotse_node_uptime_seconds | gauge | Time since start. |
lotse_node_ready | gauge | 1 while /readyz would answer 200. |
lotse_node_connections | gauge | Open client connections. |
lotse_certificate_requests_pending | gauge | Client certificate requests waiting for the CA holder, as this node has applied them (ADR 0119). |
lotse_client_received_bytes_total, lotse_client_sent_bytes_total | counter | Traffic with clients. |
lotse_peer_received_bytes_total, lotse_peer_sent_bytes_total | counter | Traffic with other nodes. |
lotse_commit_latency_seconds | histogram | Produces by time to acknowledgement, at the leader; buckets at powers of two in ms. |
lotse_host_cpu_ratio, lotse_host_memory_bytes, lotse_host_memory_limit_bytes, lotse_host_disk_free_bytes, lotse_host_disk_total_bytes | gauge | The host, read every 5 s; absent where the node cannot tell. |
lotse_disk_free_limit_bytes, lotse_disk_low | gauge | The node’s free space limit, 0 for none, and 1 while the node is below it: then the groups it votes in take no new data (ADR 0122). |
lotse_partition_leader | gauge | 1 while this node leads the group with a quorum. |
lotse_partition_term | gauge | The Raft term here. |
lotse_partition_high_watermark, lotse_partition_last_stable_offset, lotse_partition_log_start_offset, lotse_partition_log_end_offset | gauge | Offsets, as in DescribeMetrics. |
lotse_partition_size_bytes | gauge | Bytes of the log on this node. |
lotse_partition_oldest_record_timestamp_seconds | gauge | When the oldest record stored here was written, seconds since the epoch: the producer’s timestamp, on a queue the leader’s append time. Absent for an empty log and the cluster’s own groups. |
lotse_partition_open_transactions | gauge | Transactions open on the partition. |
lotse_partition_produced_bytes_total, lotse_partition_produced_records_total, lotse_partition_fetched_bytes_total | counter | While this node led. |
lotse_partition_voters | gauge | Voters of the group. |
lotse_partition_responsive_voters | gauge | On the leader: voters that answered within an election timeout. |
lotse_partition_under_replicated | gauge | On the leader: 1 while a voter has not. |
lotse_replica_log_end_offset{replica} | gauge | On the leader: what each replica holds. |
lotse_queue_ready_messages, lotse_queue_in_flight_messages, lotse_queue_in_open_transactions_messages, lotse_queue_consumers | gauge | On the queue’s leader; labels tenant, queue. |
lotse_queue_in_flight_bytes | gauge | Bytes of the messages delivered and not yet acknowledged, each its share of its batch. |
lotse_queue_delivered_messages_total, lotse_queue_acked_messages_total | counter | While this node led the queue. |
lotse_queue_consumers_in_state{state} | gauge | Subscriptions by state: shared, or a single active consumer’s active and standby ones (ADR 0120). |
lotse_queue_ready_by_level_messages{level}, lotse_queue_ready_level_unread_messages | gauge | On a queue with priority levels: ready messages per level, and those whose level a new leader has not read yet; together they make lotse_queue_ready_messages. |
lotse_consumer_group_members, lotse_consumer_group_epoch, lotse_consumer_group_committed_offset{resource,partition} | gauge | On the coordinator’s leader; labels tenant, consumer_group. |
DescribeMetrics
To see the cluster through the native protocol, ask every node and combine the answers. The console does this; lotse-client has node_metrics(), which returns one answer or error per node. The answer shows what the principal may see; the cluster’s own groups are shown to cluster admins and cluster viewers. A monitoring principal is meant to hold cluster-viewer.
What a node reports
Counters run since the node started and only grow. A rate is the difference of two answers.
| Field | Unit | Meaning |
|---|---|---|
node_id, version, uptime_ms | none, none, ms | Which node, its version, time since start. |
connections | count | Open client connections. |
client_in_bytes, client_out_bytes | bytes | Traffic with clients. |
peer_in_bytes, peer_out_bytes | bytes | Traffic with other nodes. |
commit_latency | 17 buckets | Acknowledged produces by latency. Bucket i counts those under 2^i ms; the last bucket counts all slower ones. |
host.cpu_permille | per mille | CPU use over the last 5 s. |
host.memory_bytes, host.memory_limit_bytes | bytes | Memory the node uses, and what is available to it. |
host.disk_free_bytes, host.disk_total_bytes | bytes | Free and total space of the data volume. |
disk.free_limit_bytes, disk.low | bytes, flag | The node’s free space limit and whether it is below it (tag 6). |
Host numbers are read every 5 s, not per request. A value is absent where the node cannot tell.
For every Raft group the node hosts (partitions):
| Field | Meaning |
|---|---|
group, resource_id, partition | The group, the stream or queue it belongs to (0 for the metadata and coordinator groups), the partition. |
leader, term | Whether this node leads, and the Raft term. |
high_watermark, last_stable, log_start, log_end | Offsets: committed end, below the first open transaction, first stored, after the last in the log. |
size_bytes | Bytes of the log on this node. |
produce_bytes, produce_records, fetch_bytes | Counters, growing only while this node leads the group. Summing over all nodes counts each produce once. |
replicas | On the leader: the last offset each replica holds, plus one. |
oldest_timestamp_ms, open_transactions | When the oldest data record stored here was written (ms since the epoch), and the transactions open on the partition (tag 4). Each replica looks at its oldest record once a second. |
For each queue the node leads (queues): ready, in_flight, in_open_txns, the counters delivered and acked, and per subscription its in_flight, credit, state and in_flight_bytes; with priority levels, the ready messages per level. On the coordinator’s leader, groups lists consumer groups with their members, the partitions they own and the committed offsets.
The console
The console (kind = "native") asks every node and turns counters into rates. It keeps the samples of the last few minutes and takes each rate over the newest sample at least 5 s old. Until there is one, and for a node that restarted in between, there is no rate. It takes p99 commit latency from the difference of two histogram samples, to the bucket. Chart history gets coarser with age (ADR 0045).
The console shows each user what their roles allow, and its metrics sampler signs in with client credentials. Charts sum only what the sampler may see; give it the cluster-viewer role so that one binding covers all tenants, also those created later (ADR 0043, ADR 0041).
What a real cluster does not report shows as a dash: quotas, a consumer’s credit in bytes and acks per second, and the age of a priority level’s oldest message. Zones come from the nodes’ zone settings; a stream’s oldest record is the oldest its partitions’ leaders report, and its open transactions are summed over partitions, so a transaction that writes to two partitions counts in each.
What to alert on
The repository ships no alert rules. These follow from the metrics above; the thresholds are yours to choose.
- A node whose scrape fails (
up == 0), orlotse_node_ready == 0for longer than a restart takes. lotse_disk_low == 1: the groups the node votes in take no new data. Before that,lotse_host_disk_free_bytesapproachinglotse_disk_free_limit_bytes.lotse_partition_under_replicated == 1for longer than a minute.- A group with no
lotse_partition_leader == 1on any node, or withlotse_partition_termrising fast. - A replica whose
lotse_replica_log_end_offsetlags the leader’slotse_partition_log_end_offsetfor long. lotse_commit_latency_secondsmoving to higher buckets:histogram_quantile(0.99, sum by (le) (rate(lotse_commit_latency_seconds_bucket[5m]))).- A queue whose
lotse_queue_ready_messagesgrows whilelotse_queue_consumersis 0, or whose in-flight count does not fall. - A consumer group whose committed offsets fall behind the partition’s high watermark.
- Certificates within 30 days of expiry: run
lotse checkon each node. - Certificate requests waiting for the CA holder:
max(lotse_certificate_requests_pending) > 0for an hour or so, routed to whoever holds the CA key. A request lapses after 7 days.
Logs: set LOTSE_LOG for the filter (default info). A node logs “starting”, “stopping: handing the groups this node leads to other voters”, “stopped”, “drained”, and “TLS files changed: reloaded” with the number of revoked connections closed. It logs “TLS files do not load; keeping the material in use” once when new certificate files are broken.
Operations
This chapter says what the code supports: where the data is, what a restart does, and how to read common failures. The repository documents no backup or upgrade procedure. Those are listed as open at the end, not invented.
What is in the data directory
cluster.data_dir (with {id} replaced by the node’s id) holds everything a node keeps. The node creates it if it is missing. Under the systemd unit and in the image it is /var/lib/lotse, the only place the unit lets the node write.
| Path | Content |
|---|---|
first-voters | The voters the configured groups started with, written at the node’s first start. |
groups/<group id>/ | One directory per Raft group the node hosts. Group 1 is the metadata group, group 2 the coordinator group; every stream partition and queue has a group of its own. |
groups/<id>/log/<base offset>.log | The segments of the group’s log. The name is the first offset, 20 digits. The Raft log and the partition log are the same log. |
groups/<id>/raft.state | The group’s term and vote. |
groups/<id>/snapshot | The group’s latest snapshot. |
groups/<id>/snapshot.<index>.boundary | A snapshot taken where a segment begins, for retention to cut at later. |
groups/<id>/snapshot.<index>.sent | A replaced snapshot a follower is still being sent. |
groups/<id>/snapshot.incoming | The part of a snapshot received so far. |
groups/<id>/moved | A record of an earlier move of the replica off this node. |
The node writes these files itself. Of the snapshot files, only snapshot outlives a restart: the node removes the others when it opens the group (ADR 0099). The code does not offer a supported way to edit or copy them while the node runs. Resources, users, tenants, role bindings and the node list are in the metadata group, so they are in these files too; the cluster file does not hold them.
Restart and crash
- A node stops cleanly on Ctrl-C or SIGTERM. First it hands every group it leads to another voter with a Raft leader transfer, as Kafka’s controlled shutdown does, and stops once it leads none, at most 30 s later. Clients then meet a planned move, not a failover (ADR 0114). The systemd unit sends SIGTERM and allows 60 s (
TimeoutStopSec), and restarts the node after a failure (Restart=on-failure,RestartSec=5). A supervisor of your own must allow at least 35 s. - After a crash, opening a log scans the active segment, cuts a torn or corrupt tail and syncs the result. Appended data survives a crash only once the log has been synced.
- Raft state is written in a fixed order: snapshot, truncation, entries (synced), hard state, then apply and send (ADR 0023).
- Each sealed segment has an
.indexfile beside it. On open a node reads the active segment and, of each sealed segment, its index file and last batch; a missing or mismatched index file is rebuilt by scanning that segment (ADR 0094). Damage inside a sealed segment is reported when the batch is read (Corrupt), not at start-up; delete the segment’s.indexfile to have the next start check every batch of it. The key map of a compacted stream is still rebuilt by scanning, so a restart of a node with large compacted streams takes longer for it. - After a restart a node comes back with an old view of the metadata group if it was down for a while. It asks for the node list and catches up (ADR 0050).
- Consumer group members time out after
group_session_timeout_ms(default 45 s) without a heartbeat. Queue messages in flight to a disconnected consumer are redelivered.
A cluster of three nodes needs two to commit. The code does not state how many restarts at once a given cluster survives beyond what Raft’s majority rule says.
A drained node
lotse run exits with status 0 when its node is drained, so a supervisor does not start it again. A drained node’s data directory is no longer used. The id cannot be used again (Nodes).
Troubleshooting
Messages below are those the code prints.
| Symptom | Cause and action |
|---|---|
lotse run or check fails with --config (or LOTSE_CONFIG) is missing or --node (or LOTSE_NODE) is missing | Give the flag or set the variable. |
there is no node N in the file; it names ... | The node id is not in [[node]]. For a new node, add its entry first. |
[tls] needs ca, cert and key ... or plaintext_on_loopback = true | The file names no certificates. Run lotse certs, or set plaintext_on_loopback on a machine you use alone. |
auth.admins is empty: nobody could create tenants or grant roles | Name at least one principal in auth.admins, then make its certificate with lotse certs --client. |
auth.required needs TLS ... | Authentication needs certificates. Use TLS, or set auth.required = false with plaintext_on_loopback to try a cluster without either. |
... is not on loopback; without TLS every address must be | plaintext_on_loopback allows only loopback addresses, peer_bind and client_bind included. |
TLS: ... at start | A certificate, key or list could not be read or does not match. Check the paths and that the lotse user can read them. |
Log: TLS files do not load; keeping the material in use | New certificate files are broken. The node keeps the old ones. Fix the files; the node tries again within 10 s. |
| Connections drop at a fixed time and reconnect fails | A certificate expired (ADR 0068). Renew it and let the node pick it up. lotse check warns 30 days before. |
A client gets UNAUTHORIZED | The principal lacks a role for that request, or has not authenticated. Use WhoAmI to see its bindings (Authentication and roles). |
AUTHENTICATION_FAILED for SCRAM | The answer says no more, so that names cannot be probed. Check user and password. |
OFFSET_OUT_OF_RANGE on a fetch | The position is below the log start; retention deleted it. Read the log start from the answer and go on from there. |
lotse node ... with --user fails with --user needs LOTSE_PASSWORD | Put the password in LOTSE_PASSWORD. |
lotse node add says node N is not in the file: give --peer and --client-addr | Add the [[node]] entry or pass both addresses. |
A node sits in draining in lotse node list | Its groups are still moving; look at moving. At most four groups move at a time. |
| A consumer does not get queue messages | DescribeMetrics shows the queue’s ready, in_flight and each subscription’s credit. A queue with single.active.consumer serves one subscription at a time. |
For more detail raise the log level, for instance LOTSE_LOG=debug. The filter uses the tracing env filter syntax.
Open: backup
The repository documents no backup procedure, and the code offers no snapshot or export command. These questions are open:
- Whether copying a stopped node’s data directory gives a usable backup, and how it relates to the other replicas.
- Whether a backup of one node is enough, since each partition is replicated.
- How to restore a whole cluster, and whether a restored node may rejoin with its old id.
- How to back up the CA key, the certificates and the cluster file. These are ordinary files and you hold them yourself;
ca.keymust stay off the nodes.
Open: upgrade
The repository documents no upgrade procedure. These questions are open:
- Whether nodes of different versions can run together, and so whether a rolling restart is supported.
- Which versions can read each other’s data directory, snapshots and node-to-node format.
- Whether the native protocol versions across releases are compatible;
ApiVersionslets a client ask a node which versions it speaks. - The order in which to restart nodes, and how to know a restarted node has caught up before the next goes down.
DescribeMetricsreports every replica’slog_endon the leader, which is a starting point but is not documented as a procedure.
Until these are answered, test an upgrade on a copy of your cluster first.
Open: other gaps
- Releases and the image are built only by GitHub Actions. The package targets systemd systems.
- A drained node must join under a new id to come back.
- Certificate revocation is a file you copy to every node; there is no OCSP.
Clients and the native protocol
Lotse speaks one binary protocol, the native protocol, over TLS 1.3. Every client is a program that speaks it. This chapter says which clients exist, how they relate and how they are kept correct. The rest of the developer guide covers each client and the recipes for exactly-once processing.
The clients
| Client | Where | Notes |
|---|---|---|
| Rust | crates/lotse-client | The reference. Written against the lotse-rt traits, so it runs on Tokio and in the deterministic simulation. |
| Java | sdk/java | Java 17, Gradle. No runtime dependencies; zstd and lz4 batches need two optional ones. |
| Kotlin | sdk/kotlin | A thin layer on the Java client: suspend functions on Dispatchers.IO, flows and transaction { }. |
| C# | sdk/csharp | .NET 10. No dependencies; zstd and LZ4 batches need the optional Lotse.Client.Codecs package. |
| Node.js | sdk/node | TypeScript, Node.js 20 and later. No runtime dependencies; zstd needs Node.js 22.15 or later. |
The Rust code is the authority. lotse-proto and lotse-record define every byte on the wire; lotse-client defines how a client behaves: routing to leaders, retries, idempotent sequences, transactions, groups. The Java, C# and Node.js clients follow it rule for rule. The reasons are in ADR 0056; the Kotlin layer is in ADR 0080; the C# and Node.js clients are in ADR 0083, their codecs in ADR 0084 and ADR 0085.
Pages in this guide:
- The native protocol: byte layouts and client behaviour, for anyone writing a client in another language.
- Rust client
- Java and Kotlin
- C#
- Node.js
- Exactly-once recipes
- Contributing
What a client does
A client does four things on top of the frame format:
- Finds leaders. It asks any broker for
Metadata, then sendsProduceandFetchto the partition’s leader and followsNOT_LEADERhints. - Retries safely. Idempotent produce stamps every batch with a producer id and a sequence per partition, so a retry after a lost answer cannot write a batch twice.
- Keeps sessions. It authenticates each connection (certificate, SCRAM-SHA-512 or OAUTHBEARER) and renews the session before it ends.
- Owns the protocols above the log. Queue credit and acks, transactions, consumer group heartbeats and cooperative rebalancing are all client behaviour driven by broker requests.
A client chooses the partition itself; it does not hash keys for you.
Golden vectors
Two implementations of a byte format agree only if something checks it. The Rust codec writes the check files, and every SDK reads them.
sdk/conformance/holds the vectors: encoded requests, responses and pushes, record batches (plain, compressed, compacted), varints, CRC32C values, error codes, exchange routing cases and SCRAM exchanges.- They are written by
crates/lotse-proto/examples/vectors.rs. Never edit them by hand. - A decoder passes when it reads every vector and writes it again to the same bytes, refuses each vector cut at every length and with one byte added, and agrees with the field files. The protocol page lists the files.
Regenerate the vectors after you change a message or the batch format:
make sdk-vectors
make sdk-test first checks that sdk/conformance still matches what the Rust codec writes, then runs the Java SDK and Kotlin layer tests and compiles their examples (./gradlew check in sdk/java and sdk/kotlin), then the C# tests (dotnet test in sdk/csharp) and the Node.js tests (npm test in sdk/node). make sdk-csharp-test and make sdk-node-test run one each:
make sdk-test
You need a JDK 17 or later, the .NET 10 SDK and Node.js 20 or later. Set LOTSE_BIN to a lotse executable to add the TLS and real-node tests. CI runs the same target with LOTSE_BIN set to the binary it builds, and runs the Node.js tests once more on Node.js 20.
A real-node test that relies on what a leader keeps in memory (messages in flight, delivery counts, a single active consumer, produce counters) pins that leader first: pinLeader (Java, Node.js) or RealWaits.PinLeaderAsync (C#) pins it where it is with MoveLeader and waits until it leads there in one term for five reads in a row. A new group’s leader otherwise moves to the group’s preferred node at a time no test can know. Tests that stop or restart nodes do not pin: a pinned leader would move back to a node that returns. A pinned leader can still move when a starved node loses an election on a slow runner, so such a scenario runs inside retryIfLeaderMoved (Java, Node.js) or RealWaits.RetryIfLeaderMovedAsync (C#): it watches the leaders it relies on and runs the scenario again, on new queues or streams, when it failed after one of them moved or with NOT_LEADER. A failure with every leader where it was is thrown at once.
Writing a client in another language
- Read the protocol page and the vectors.
- Pass the vectors first: field codec, frames, record batch.
- Add connections, authentication and metadata routing.
- Add idempotent produce and read_committed fetch.
- Add queues, transactions and groups, in that order of need.
- Follow
lotse-clientfor behaviour. Where the protocol page and the Rust code differ, the Rust code wins and the page is a bug: fix it in the same change.
The native protocol
This page is sdk/PROTOCOL.md from the repository, included as it stands, so it cannot drift from the file. It is written for people who build a client in another language.
The Rust code is the authority. docs/design.md (German) gives the design, and the golden vectors in sdk/conformance/ pin the layouts below to the broker’s bytes.
All of it is version 0 of every API key.
Integers and fields
Fixed-width integers are little-endian. The one exception is the frame length prefix.
| Type | Wire | Notes |
|---|---|---|
u8, u16, u32, u64, i32, i64 | 1, 2, 4, 8, 4, 8 bytes | booleans are a u8: write 0 or 1, read any non-zero as true |
varint | ZigZag, then LEB128, at most 10 bytes | array counts, tag numbers, record fields |
string | u16 length, UTF-8 | length 0xFFFF is null; at most 65,534 bytes |
bytes | u32 length, data | no null form |
array<T> | varint count, elements | the count is ZigZag: 2 elements is the byte 04 |
| optional | u8 0 or 1, then the value if 1 | a different form from a null string |
| tagged fields | varint count, then {tag varint, bytes} | see below |
Unsigned types in Java. u64 travels in a long as its bit pattern; compare, divide and print it with the Long.*Unsigned methods. u32 fits a long, u16 an int. NO_OFFSET (a group’s “no committed offset”) is u64::MAX, which is -1 as a long. Offsets are u64 in the protocol but the first offset of a log is 0: base_offset is the Raft index of the first record minus one (the broker stamps index - 1), although the doc comment in lotse-record says “index of the first record”.
Counts are bounded before allocating. A reader must refuse an array whose count times the smallest encoding of one element exceeds the bytes left in the frame. The minimum element sizes are part of the broker’s decoder (the array(min_size, ..) arguments in messages.rs); a decoder that guesses a smaller minimum is merely more permissive, one that guesses a larger minimum refuses valid frames of nothing but minimal elements.
Varints. The reader accepts overlong encodings such as 80 00 for 0 and does not tell a truncated varint from an overlong one in a frame (both are “truncated”); inside a record they are different errors. Writers never produce overlong forms. The tenth byte of a 10-byte varint may only be 00 or 01.
Frames
On a connection every frame is preceded by its length, a big-endian u32, at most 16 MiB (MAX_FRAME). A larger length ends the connection.
Request: api_key u16 | api_version u16 | correlation_id u32 | flags u16 | tagged_fields | body
Response: correlation_id u32 | flags u16 | tagged_fields | body
Push: correlation_id u32 (0) | flags u16 (PUSH) | tagged_fields | api_key u16 | api_version u16 | body
- Flags:
PUSH= 1,ERROR= 2. - Correlation ids of requests start at 1 and are never 0; 0 marks pushes. Requests are pipelined and responses may arrive in another order: match them by id.
- A response has no API key. The decoder must be told which request it answers.
- Check
ERRORbefore decoding a typed body. A response with that flag has the bodyerror_code u16 | tagged_fields, whatever the request was (unsupported version, connection not authenticated). If tagged field 0 is present, its bytes are the reason, in UTF-8. Decoding such a frame as aProduceresponse misreads it. - Push detection: the first four bytes are zero and bit 0 of the flags is set. A response with correlation id 0 cannot exist.
- A push’s body begins with the subscription id;
docs/design.mdused to say the id sat in the header. Creditis never answered. No other request is unanswered.- After the header’s tagged fields,
ApiVersionsstill has a body: one more, empty block of tagged fields (00).
Tagged fields: not at the same place everywhere
Most structures end with a block of tagged fields, written empty (00) and skipped by readers (unknown tags come from newer versions and are dropped, not preserved). That is not uniform, and a generic “every struct ends in tags” decoder is wrong. The leader hint is tag 0 of the block it sits in: leader u32 | term u64, present only after NOT_LEADER.
The tenant of a transactional or group id is tag 1 of a request’s trailing block: one string (ADR 0062). It is written only for a tenant other than default; a reader that finds none takes default. The requests that carry it are Produce (it applies to its transactional id), InitProducer, AddPartitionsToTxn, EndTxn, GroupHeartbeat, CommitOffsets, OffsetFetch and AddOffsetsToTxn. A client sends the tenant of its configuration. DescribeMetrics gives each group’s tenant as a plain string after its id.
| Where the block is | Messages |
|---|---|
| Hint is the trailing block | InitProducer, Subscribe, QueueAction (answer to PurgeQueue, Requeue), Outcome (answer to Ack, Nack, Unsubscribe, Bind, Unbind), Create (CreateStream, CreateQueue, DeclareExchange), Admin (DeleteResource, AlterConfigs, AlterUser, CreateTenant, DeleteTenant, AlterRoleBindings, AddNode, RemoveNode, MoveLeader, ReassignPartition, RevokeSessions, RevokeCertificate, RecordCertificates, RequestCertificate, RecordCertificate), RenewCertificate |
| Hint in the middle, no trailing block | DescribeConfigs, DescribeUsers, DescribeTenants, DescribeRoleBindings, DescribeNodes (after the error code, before the array) |
| Hint is the element’s block, one per partition | Produce and Fetch partitions (the resource around them still ends in an empty block) |
| Hint is the last field of a push | Deliver |
| No block at all | SaslHandshake and SaslAuthenticate responses, WhoAmI response |
| Empty block after the body fields | everything else |
Elements of arrays carry their own empty block in most requests (TxnResource, produce and fetch partitions, offset commits) and none in a few (ApiVersions entries, queue ranges, config pairs, role bindings, metrics sub-structures). Do not infer: the layouts are in Request.java and Response.java of the Java SDK, and each is covered by a vector.
Responses worth a second look
Metadata: a resource carries its ownerror. An unknown name is a resource withUNKNOWN_RESOURCE, not a frame error. Leader0means unknown. A partition’s tag block may hold tag 0,pinned_leader(u32, the node a cluster admin pinned its leader to), and tag 1,placed(empty: a cluster admin placed it on named nodes); each is present only when set (ADR 0121).OffsetFetch: an offset ofu64::MAXmeans none.DescribeMetrics:connectionsis sent as au64but must fit au32. Between nodes the answer may carry tag 1,removed_through(oneu64, ADR 0100), and tag 2,responsive(an array of: groupu64, termu64, an array of node idsu32; ADR 0117), which a principal that sees the whole cluster gets too; a client skips both like any unknown tag. Tag 3,queue_details(ADR 0120), goes to every principal: an array of entries, each a queue idu64, its ready messages per priority level as an array ofu64(level 0 first, empty without levels), the ready messages whose level the leader has not read yetu64, and an array of subscription idu32and stateu8(0 shared, 1 the single active consumer, 2 a standby; read an unknown code as shared). It lists only queues with levels or a subscription that is not shared, and a client reads it into the queue entries of the same id; a queue or subscription it does not list has no levels and is shared. ADR 0122 adds three tags, each read into the entries of the same id, and each left out when it has nothing to say: tag 4, an array of groupu64, the oldest data record’s timestampi64(ms since the epoch, -1 for none) and the open transactionsu32, read into the partitions; tag 5, an array of a queue idu64and an array of subscription idu32and the bytes it holdsu64, read into the subscriptions; tag 6, the node’s free disk limit in bytesu64(0 for none) and whether it is below itu8. A partition or subscription a tag does not list has no oldest record, no open transactions and holds 0 bytes; without tag 6 the node did not say.DescribeNodes: between nodes the request may carry tag 0,applied_at_least(oneu64, ADR 0105), and the answer is thenNOT_LEADERfrom a leader that has applied less. Clients do not send it.ProduceandFetch: every partition has its own error; a request can partly succeed.
Error codes
Codes 0 to 34, with names and retry classes, are in ErrorCode.java and in conformance/error-codes.txt. Retriable: NOT_LEADER, FENCED_LEADER_TERM, NOT_ENOUGH_REPLICAS, REQUEST_TIMED_OUT, COORDINATOR_NOT_AVAILABLE, CONCURRENT_TRANSACTIONS, UNSTABLE_OFFSET_COMMIT, CONCURRENT_GROUP_CHANGE, PRODUCER_LIMIT, STORAGE_FULL. STORAGE_FULL (11, ADR 0122) refuses a new batch while a voter of the partition is below its free disk limit: nothing was written, so send the same batch again, with the same sequence, after a backoff, and when the time for retries runs out, fail with STORAGE_FULL itself rather than a timeout, the sequence still free; after an attempt whose answer was lost the outcome stays unknown. PRODUCER_LIMIT (ADR 0110) refuses a produce from a producer the partition does not know while it holds as many as it admits: nothing was written, so send the same batch again, with the same sequence, after a backoff. QUEUE_FULL (34, ADR 0118) refuses a write to a queue at its length limit with overflow = reject-publish: nothing was written, it is not retried, and the caller decides; after an attempt whose answer was lost it says nothing about that attempt, so report the outcome as unknown and start a new producer, as after UNAUTHORIZED then. REQUEST_TIMED_OUT means the outcome is unknown: retry only what is idempotent. A code a client does not know is kept as it came and named UNKNOWN.
The record batch
A batch is a 65-byte header and a block of records. All fixed-width header fields are little-endian.
offset size field
0 8 base_offset stamped by the leader: Raft index of the first record, minus one
8 4 batch_len bytes after this field, so the batch is batch_len + 12 bytes
12 8 term stamped by the leader
20 1 magic 1
21 4 crc32c CRC32C (Castagnoli) over bytes 25 up to the end of this batch
25 2 attributes compression(3) | transactional(1) | control(1) | kind(3) | compacted(1), bit 0 first
27 4 last_offset_delta record_count - 1
31 8 base_ts milliseconds since the epoch
39 8 max_ts
47 8 producer_id 0 = not idempotent
55 2 producer_epoch
57 4 base_sequence i32, -1 when there is none
61 4 record_count
65 . records
base_offset and term sit before the checksummed range: the leader stamps them without invalidating the CRC. A client sends both as zero.
Kinds: 0 data, 1 ack, 2 queue state, 3 timer, 4 dead letter, 5 retire, 6 checkout (ADR 0109: which consumer a leader handed which messages to; clients never read or write one). Compression: 0 none, 1 zstd, 2 lz4. Bits 8 to 15 must be zero.
Compression (ADR 0058) covers the record block only; the header stays as it is, and the CRC is over the compressed bytes. With zstd the block is one zstd frame; with lz4 it is one LZ4 frame (the frame format with its magic 04 22 4d 18, not the bare block format). Once decompressed the block reads exactly like an uncompressed one, and it is at most 16 MiB: a reader stops decompressing past that and treats the batch as damaged. The batch itself, compressed, stays within 1 MiB. A node checks a compressed batch’s records when it is produced and refuses one that does not decode (CORRUPT_BATCH); it stores, replicates and serves the batch compressed, so a consumer must read every codec its producers use. Control batches are never compressed.
A record, every varint ZigZag:
length varint | attributes i8 (0) | ts_delta varint | offset_delta varint
| key_len varint (-1 = null) | key | value_len varint (-1 = null) | value
| header_count varint | [key_len varint, key, value_len varint, value]*
ts_delta is relative to base_ts and wraps like a signed 64-bit subtraction; offset_delta is the record’s index and must run 0, 1, 2, and so on, except in a compacted batch, where records that compaction removed leave gaps (vectors in compacted.hex). A header key is never null.
Checks a parser makes, in this order: the buffer holds the 12-byte prefix; the length is at most 1 MiB (the length is a u32, so test it without overflow); it is at least 65; the whole batch is present; magic is 1; the CRC matches; the attributes are valid; record_count is not 0; last_offset_delta == record_count - 1; max_ts >= base_ts; a control batch holds exactly one record. A compacted batch (bit 8, ADR 0067) replaces the two record-count checks: it holds at most last_offset_delta + 1 records, possibly none, and is never a control batch. Decoding the records adds: the block decompresses, to at most 16 MiB; consecutive offset deltas (in a compacted batch: strictly rising and at most last_offset_delta; an emptied one has no block at all, whatever its codec), no bytes left inside a record, no bytes left after the last record, lengths and counts not negative, and every length within the record.
Editing in place. A client sets producer fields (set_producer), the transactional bit (set_transactional) and, for the leader’s TTL stamping, the base timestamp, on the encoded bytes and recomputes the CRC over bytes 25 to the end of that batch. BatchBuilder takes the first record’s timestamp as base_ts and the largest as max_ts; it does not read a clock.
Several batches in a row (the records of a Fetch partition, a log segment): read batch_len, add 12, cut, repeat. A fetch response holds whole batches.
Control batches. One record. Key: version u16 (0) | type u16. Types 0 abort and 1 commit are transaction markers whose value is the coordinator epoch as a u32. Types 2 and 3 are Raft’s own entries, 4 to 8 the coordinator and metadata groups’ records. Consumers skip all of them. A read_committed consumer uses the fetch response’s aborted-transaction list and the markers to hide aborted data.
Connections
One TCP connection per broker address, kept open. Every connection is TLS 1.3 with the cluster’s CA (ADR 0037), or plain TCP on loopback when a node was set up for development; a client never falls back from one to the other. The client checks the node’s certificate against the host it dialled, including IP entries: lotse certs writes the node’s addresses from the cluster file as IP entries, so dial the same address the file lists. A client certificate, if the node asks for one, names the principal by its common name. There is no hello, no ALPN and no version byte: the first bytes after the handshake are the first frame.
- Set
TCP_NODELAY. Frames are written whole; writes from several threads must not interleave. - One reader per connection decodes frames and completes requests by correlation id. A response nobody waits for (its request timed out) is dropped without ending the connection.
- A frame whose length exceeds 16 MiB, one that does not decode, or end of stream ends the connection: fail every pending request and every subscription. A refused request (
ERRORflag) does not end it. - A request that timed out has an unknown outcome. Retry only what is idempotent.
ApiVersionsis answered before authentication. The Rust client never sends it; the broker does not require it.- A request on a connection that is not authenticated, or whose session ended, gets an
ERRORresponse withUNAUTHORIZEDand no reason. The connection stays open. - Pushes of a subscription can arrive before the response to
Subscribehas been handled, so the client keeps pushes for subscriptions nobody has registered yet. The Rust client keeps up to 64 such subscriptions and drops the lowest id for another.
Authentication
Two SASL mechanisms, chosen by name in SaslHandshake. A node offers the ones its configuration enables; asking for another gives UNSUPPORTED_SASL_MECHANISM (28) with the node’s list in the same response.
SaslHandshake(mechanism) -> error u16 | mechanisms array<string> (no tagged fields)
SaslAuthenticate(bytes) -> error u16 | error_msg optional string | auth_bytes bytes | session_lifetime_ms u64
An unsuccessful SaslAuthenticate carries AUTHENTICATION_FAILED (27) and a message, and the node closes the connection after sending it. It does not say whether the user or the password was wrong.
OAUTHBEARER is one round: n,,\x01auth=Bearer <token>\x01\x01. Ask for the token on every authentication, including renewals.
SCRAM-SHA-512 (RFC 5802, RFC 7677) is two rounds, with these particulars that differ from what a library might assume:
| Step | Detail |
|---|---|
| client first | n,,n=<user>,r=<nonce>; in the user name = becomes =3D and , becomes =2C. The nonce is 24 random bytes in hex |
| server first | r=<nonce>,s=<base64 salt>,i=<iterations>. The nonce must strictly extend the client’s. Refuse iterations below 4096 or above 1,048,576 |
| password | the raw UTF-8 bytes, no SASLprep, as PBKDF2-HMAC-SHA512 input with a 64-byte output |
| client final | c=biws,r=<nonce>,p=<base64 proof>; biws is base64 of n,, (no channel binding) |
| AuthMessage | <client first without n,,>,<server first>,<client final without ,p=...> |
| proof | ClientKey xor HMAC(StoredKey, AuthMessage), where ClientKey = HMAC(SaltedPassword, "Client Key") and StoredKey = SHA-512(ClientKey) |
| server final | v=<base64 HMAC(ServerKey, AuthMessage)>; the client must verify it, it is the server’s proof that it knows the credential |
An empty password is valid. HMAC pads a short key with zeros, so an implementation whose library refuses an empty key can use 128 zero bytes.
conformance/scram.txt holds whole exchanges with fixed nonces, written by the broker’s own code, and the credential computed for each.
Setting a password (AlterUser, admin) sends a credential, never the password: a 16-byte random salt, 4096 iterations, StoredKey and ServerKey = HMAC(SaltedPassword, "Server Key") (each 64 bytes). A removal sends no credential.
Sessions. session_lifetime_ms of the last response is how long the session lasts, counted by the node from when it completes the login (its last answer); 0 means as long as the connection (a certificate login has no SASL session at all). The node caps it at its max_session_s and, for OAUTHBEARER, at the token’s expiry. Authenticate again on the same connection (SaslHandshake, then SaslAuthenticate) before it ends: the Rust client does so at four fifths of the lifetime, counted from before its first request (so it never believes in a session longer than the node does), and the node insists on the same principal. Requests in flight meanwhile are served. After the end the node answers UNAUTHORIZED to everything but SASL and ApiVersions; the Rust client does not try to renew then but opens a new connection and logs in again, and so does the Java one.
Client behaviour: streams
The bytes above are enough to talk to a node. These rules are what makes a client behave like lotse-client when nodes fail; they come from crates/lotse-client/src/lib.rs and the Java client follows them rule for rule.
Errors of a call
Every request ends in one of four ways, and the retry rules depend on which:
| Outcome | Meaning | Retried by |
|---|---|---|
| not sent | the connection was not there, or the node refused the whole frame without a reason | everything |
| lost | the request may have reached the node (timeout, broken connection after sending, an answer that does not decode) | idempotent produce, and requests that can be repeated |
| denied | the node requires authentication, or login failed | nothing: Authentication at once |
| refused | the node answered with ERROR and a reason (a role is missing) | nothing |
A connection that failed in any of these ways is closed and not trusted again. A failed login is remembered until a new connection logs in, so that a retry loop whose last attempt only saw a metadata lookup fail still reports the login. If an earlier attempt of a produce was lost and a later one is denied, the answer is Unknown, not Authentication: the first attempt may have been written.
The retry loop tries again until retry_for (default 30 s) has passed, pausing backoff (50 ms) between attempts. When time runs out it reports GaveUp if every attempt was certainly not carried out, and Unknown if one may have been.
Routes
The client keeps (stream, partition) -> {resource_id, leader, term}, learnt from Metadata, and uses a route until a request fails. Routes do not expire by age; metadata_max_age applies to exchange bindings only.
Metadatafor a name a node does not know returns a resource withUNKNOWN_RESOURCE. The client then asks every broker it knows (sorted by address, the bootstrap list included) and keeps, per name, the resource with the highestresource_id(compared asu64): a name deleted and created again has a higher id, and a node that lags may still report the old one. A resource with an error names nothing and is never merged.leader == 0means unknown here, not that there is none. A partition created at runtime lives on some nodes only; the client asks the partition’s voters, one after the other, forMetadataof that name and takes the first non-zero leader. If none knows, the attempt is retried.NOT_LEADERandFENCED_LEADER_TERMcarry a leader hint (leader,term) when the node knows one. The client switches the route to the hint only ifhint.term >= route.term(unsigned) andhint.leader != route.leader. Otherwise (no hint, an older term, the same leader) it drops the route, and the next attempt asks for metadata again. An older term comes from a node that lags; following it would send the client back to a leader that has been replaced.- A leader whose node id has no known address drops the route and retries.
UNKNOWN_RESOURCEin aProduceorFetchdrops the route too: the name may come back as a new resource.
Idempotent produce
- On the first produce the client asks any broker
FindCoordinator(key_type = PRODUCER, key = "")and then the coordinatorInitProducer(txn_id = null). The answer isproducer_idandepoch; both are kept for the client’s session. - Per
(resource_id, partition)the client counts the next sequence, starting at 0 (ADR 0107): a stream deleted and created again under its name has a new resource id and sequences of its own. Note the resource id the partition routes to before stamping; if a retry routes to another resource id, end the produce with an unknown-stream error rather than send the batch there, and if an earlier attempt’s outcome was unknown, drop the session as in step 6. It stamps the batch (set_producer(id, epoch, sequence), which recomputes the CRC) and sends it. - Produces to one partition go out one at a time. A second batch is not sent until the first has an answer, so the leader sees sequences in order. Different partitions run in parallel.
- Retrying a batch whose outcome is unknown (lost connection,
REQUEST_TIMED_OUT) sends the same bytes: same id, epoch and sequence. The leader recognises a batch it has and answers with its offset.PRODUCER_LIMIT(ADR 0110) is retried the same way after a backoff, though nothing was written: the partition takes the producer once it has forgotten an idle one. - On success the sequence advances by the batch’s record count (a wrapping
i32). - When the call ends with
Unknown,OUT_OF_ORDER_SEQUENCEorPRODUCER_FENCED, the whole session is dropped; the next produce gets a new producer id and starts every partition at 0. A gap in the sequence would otherwise block everything behind it. A call abandoned before its outcome is known (a caller’s timeout, a cancelled task) counts asUnknowntoo: reusing its sequence would let the leader take the next batch for the abandoned one, answer with that one’s offset and write nothing. Any other failure means the batch was not written, and its sequence is free. REQUEST_TIMED_OUTmeans the quorum did not confirm in time: the batch may or may not be written.- Without idempotence nothing is stamped and an unknown outcome fails at once, because a retry could write the batch twice. Errors that prove the batch was not written (
NOT_LEADER, a connection that was never opened) are still retried.
The timestamp of a record is the producer’s. A stream’s retention judges a batch’s age by it, so a record stamped 0 is old at once and may be deleted at the next retention pass. The Rust helper OutMessage defaults to 0; the Java client uses the time of the call.
Fetch
Fetch asks for min_bytes = 1, max_bytes 4 MiB, 1 MiB per partition, isolation = 1 (committed) and leader_term = the route’s term; the broker holds the request up to max_wait_ms when there is nothing to return. Keep max_wait well below the request timeout.
Fetching in a zone (ADR 0104). Metadata names each broker’s zone as rack (empty: none). A client configured with a zone sends a partition’s Fetch to the first voter whose rack is that zone, unless the leader is in it, and sets the request’s tag 0, allow_follower (one u8, 1). A follower then serves the committed data it has: it may lag the leader, it ignores leader_term, and past its log it answers with no records and no error. On any error answer from that replica, or a failed connection, the client reads the partition from the leader until it next learns metadata. Without the tag a follower answers NOT_LEADER as before. A follower also answers NOT_LEADER when it lags (ADR 0116): it has not heard its leader within an election timeout, or not applied what the leader said was committed within follower_max_lag (10 s by default). The records are whole batches; a response may contain only a control batch, so a reader moves on by the offset after the last batch, not by the number of messages it got.
A read_committed reader drops what it must not show:
- Sort the
aborted_txnsby first offset (asu64). - For each batch in order, every aborted entry whose first offset is at or below the batch’s base offset puts its producer into the skipping set.
- Control batches are never shown. A control batch that is an abort marker (type 0) takes its producer out of the skipping set; commit markers and the broker’s own control entries do nothing.
- A transactional data batch of a producer in the skipping set is dropped. A non-transactional batch of the same producer is kept.
The first record of a fresh partition is not at offset 0: the log begins with entries of the broker’s own. Do not assume the offset of the first message.
OFFSET_OUT_OF_RANGE is not retried. The partition response’s log_start_offset says where the log starts now; a reader that was overtaken by retention continues from there.
ListOffsets
ListOffsets (key 5, ADR 0059) goes to the partition’s leader, like Fetch, and answers per partition with an offset and a timestamp. With request tag 0 allow_follower (one u8, 1) a follower answers too, from its own log start, commit index and last stable offset, which may lag the leader’s; a client with a zone sends it where it sends Fetch, and leaves a replica that answers with an error for the leader until its next metadata (ADR 0116). timestamp = -2 asks for the first offset still stored, -1 for the offset after the last one the reader may see (the high watermark, or the last stable offset with isolation = 1), and a time in milliseconds since the epoch for the first offset of the first data batch whose newest record is at or after it; that batch’s newest timestamp comes back too. If no batch qualifies, the answer is the latest offset with timestamp -1. Other negative timestamps are INVALID_REQUEST. A time lookup goes by batches: the batch it finds may begin with records older than the time asked for. A consumer group member uses it for a partition the group has committed no offset for, when it is told to start at the end or at a time.
Client behaviour: queues
These rules come from crates/lotse-client/src/consumer.rs and the push handling in lib.rs. A queue is a resource of kind 1 with one partition, 0, so routes, leader hints and Produce work as for a stream; messages are produced with a normal Produce to partition 0.
Subscribing
- Look up the route of
(queue, 0)and open (or reuse) the connection to its leader. - Send
Subscribe(queue_id, consumer_tag = "", credit_msgs, credit_bytes = 0xFFFFFFFF, exclusive = false).exclusiveis refused withINVALID_REQUEST; do not offer it. On a queue with a single active consumer (ADR 0076) the answer may carry tag 1standby: the subscription is accepted but gets nothing until it is its turn. A client that sends tag 1notify_active = 1gets a push when that happens: errorNONE, an empty batch and no deliveries; one that does not send it never gets such a push. The Rust and Java clients send it and reportisStandbyuntil the activation push or the first message. The answer carries tag 2consumer_id(oneu64, ADR 0109), which the leader gives a new consumer; keep the latest nonzero one, and send it as tag 2resume(oneu64) whenever the consumer subscribes again, so its messages stay its own at the new subscription. Send nothing in tag 2 before a consumer has an id. Withresume, send tag 3held(ranges[from, to), built like an ack’s, ADR 0112): the messages the consumer holds, which are those handed out and not yet acknowledged, nacked or acknowledged in a transaction, those received and not handed out, and those whose requeue waits to be sent. The leader takes back only these, and hands the consumer’s other messages out again at once: their pushes never reached it, because the leader went or the connection broke after it wrote their checkout. An empty list is sent as an empty array, not left out: without the tag the leader takes back everything handed to the consumer, as an older client expects. - On
NONE, register thesubscription_idon that connection. Subscription ids are per node, andCredit,Unsubscribeand a requeueingNackare honoured only on the connection that owns the subscription (UNKNOWN_SUBSCRIPTIONotherwise). Pushes can arrive before the answer toSubscribehas been handled: the connection keeps them (see Connections) and hands them over on registration. - A retriable error (
NOT_LEADER) follows the leader hint and retries.UNKNOWN_RESOURCEdrops the route and fails the call. A lost answer drops the connection (a subscription the broker may have made ends with it), drops the route and retries.
Receiving
A push holds one batch exactly as stored and deliveries, the offsets in it that are meant for this subscription, each with its delivery count. Only records whose offset is in deliveries are the consumer’s; the rest of the batch belongs to others. Read the first batch of the push only.
The consumer hands out what it holds first and reads the next push (or the end of the subscription) only when its buffer is empty. A push with a non-zero error ends the subscription:
| Error | Then |
|---|---|
NOT_LEADER | follow the leader hint (as for a request), subscribe again |
UNKNOWN_RESOURCE | drop the route, subscribe again |
ACK_TIMED_OUT, any other | subscribe again; the broker took the unacknowledged messages back |
A connection that breaks ends the subscription in the same way. The next read subscribes again, on the next call and not before, resuming with the consumer id. After NOT_LEADER or a broken connection a consumer with an id keeps what it received, both the messages parsed into the buffer and the pushes that arrived and were not read yet: the leader writes each delivery to the log before it pushes it (ADR 0109), and a new leader keeps the messages for their consumer until it resumes, for the queue’s resume window (30 s by default). After ACK_TIMED_OUT or any other error, or without an id, drop the buffer and the unread pushes: the broker gave them to others. Messages handed out before can still arrive twice, and a delivery count that rose by one for each time says so; counts are in the log, so they go on across failovers.
Credit
The consumer keeps two counters: messages handed out and bytes received (the length of each pushed batch, added when it arrives, saturating at u32::MAX). After handing out a message, if messages used is at least ceil(credit / 2), it sends Credit(subscription_id, add_msgs = used, add_bytes = bytes) on the subscription’s connection and sets both counters to 0. Credit is never answered; a lost one shows as a broken connection on the next read.
Settling
- Ack goes to the queue’s leader, on any connection, with ranges
[from, to)built from the offsets: sorted asu64, duplicates dropped, neighbours joined.txn_idis null outside a transaction (see below for one inside). An ack names an offset, not a delivery, so it settles a message that went to someone else meanwhile, and settling twice is harmless: a lost answer is retried, and so isNOT_LEADERafter following the hint. - Nack without requeue discards the message (or dead-letters it, if the queue’s limits say so) and is sent like an ack.
- Nack with requeue is sent on the subscription’s own connection and names the subscription its messages came on in tag 0,
subscription_id(oneu32, ADR 0106): the broker takes back only what that subscription holds. Without the tag it takes back what any subscription of the connection holds, so a lateNackfrom a consumer that ended could take a message a newer subscription on the connection received again since; always send it.delay_msis the ms the message waits before it is delivered again (ADR 0077; refused withoutrequeue); the Java client’snacksends 0 andnackAfterthe delay asked for. A producer delays a message with the headerx-lotse-delay-ms(decimal ms up to 4294967295), counted from the leader’s append time; messages of a queue carry that time as their batch’s base timestamp. A consumer with an id sends it on its current subscription, also for a message an earlier subscription received: the current one took it over. If there is no current subscription, or theNackfails with a retriable error or a broken connection, keep the offset and delay and send them once the consumer has subscribed again. Without an id the message has gone back already when the subscription ended, and the call does nothing; a message of an earlier subscription is then not asked for on the new connection, which does not hold it.
Closing
Send Unsubscribe(subscription_id) and wait for its answer (up to the request timeout). Then, for the messages received but not handed out (the ones parsed into the buffer and those in pushes that arrived until the answer, which the node holds in flight just the same), send one Nack(requeue = true) with their offsets as ranges and the ending subscription in tag 0, on the subscription’s connection and without waiting for its answer. Pushes and answers keep their order on a connection, and a push waits for its checkout to commit (ADR 0109), so a push can still arrive after the Unsubscribe was sent; waiting for the answer catches it. Messages handed out stay with the consumer until it acknowledges them or the connection closes.
If the answer is not NONE (the node no longer knows the subscription, a lost answer, a broken connection), or the subscription had ended already after NOT_LEADER or a broken connection and the consumer did not subscribe again, a new leader may keep the consumer’s messages for it until its resume window is over (ADR 0112). A consumer with an id then subscribes once more (retried like any subscribe, within retry_for) with resume, credit_msgs = 0 and held naming only the messages handed out and not settled, and unsubscribes that subscription as above: the leader hands the rest out again at once. The Rust client does this in close, not when a consumer is dropped.
Publishing through an exchange
Publish (key 9, ADR 0079) goes to any node, with entries of exchange id, routing key, mandatory and a batch of exactly one non-transactional data record; txn_id must be null. The node routes by the bindings it has applied and writes each queue in turn. An entry’s error is UNKNOWN_RESOURCE (exchange unknown: look the name up again, it may have a new id), NOT_LEADER (the node has not heard of the exchange yet, ADR 0101: nothing was written, so send the publish again to another node or after a pause; this holds for every retriable entry error), INVALID_REQUEST, UNROUTABLE (mandatory and no queue) or UNAUTHORIZED (no producer role on a routed queue; nothing written); otherwise routed lists each queue by id and name (the publisher may never have seen a queue bound and deleted between two of its lookups; the name is empty only when the node no longer knows the queue), with NONE and the offset, REQUEST_TIMED_OUT when its outcome is unknown, or the code that refused it. A publish that is not mandatory and routes nowhere has NONE and no queues. When the answer is lost, do not send the publish again: the node may have written it, and a second one could write it twice. Report the outcome as unknown.
Dead letters
A queue with limits (max.deliveries, message.ttl.ms) and a dead.letter.queue moves a message there as a one-record batch with the original key, value, headers and timestamp, and these headers (UTF-8 text, decimal numbers; they describe the latest move and replace those a message arrived with): x-lotse-dead-letter-reason (max-deliveries, expired, requeued or max-length, ADR 0118), -queue (source queue name), -queue-id, -offset, -deliveries and -floor (the source’s ack floor when it was sent). A client reads them like any header; the Java client offers them typed as Delivery.deadLetter().
Client behaviour: transactions
A client configured with a transactional id runs one transaction at a time (ADR 0029):
- Begin. The first transaction asks the coordinator
InitProducer(txn_id)for a producer id and epoch. That fences an earlier client with the same id and aborts what it left open. Later transactions reuse the id with the epoch the lastEndTxnanswer gave. After a transaction that was left open, or anEndTxnthat failed, the next one asksInitProduceragain. - Add. Before the first batch to a partition (a queue is partition 0), send
AddPartitionsToTxnto the coordinator, once per partition per transaction. A retriable error means: forget the coordinator and try again. - Produce. Stamp the batch with the producer id, epoch and a sequence per partition starting at 0 in each transaction, set the transactional bit, and send it with
txn_idset. Retry an unknown outcome with the same bytes, as for an idempotent produce. On success the sequence advances by the record count. - Must abort. After any error except “no broker in time” (
GaveUp) and “no such stream” the transaction may hold a write nobody knows, so only an abort is left. That includes a call that ends before its outcome is known. A client refuses further writes and the commit, without asking the coordinator. - End.
EndTxn(commit)to the coordinator. A lost request or a retriable error may have been decided already: retry, and reportUnknownif time runs out. A transaction that added no partition and sent no offsets ends withoutEndTxn. One that sent offsets (AddOffsetsToTxn, even if the answer was lost or refused) is ended all the same, with the sameEndTxn: the coordinator holds the offsets as pending until it hears, and every reader of the group’s offsets waits for them until the transaction times out.
Publishing in a transaction is routed by the client (ADR 0032): read the exchange’s type and bindings from Metadata (kept for metadata_max_age), pick the queues as routing.txt describes, and produce the message to each of them in the transaction. If a write fails after some queues have it, only an abort is left.
Acking a queue message in a transaction (ADR 0075): add the queue to the transaction, then send Ack with txn_id, the producer id and the epoch on the subscription’s own connection, since only the connection that holds a message can ack it so. A lost answer is sent again on the same connection, up to twice. A consumer with an id (ADR 0109) that then meets a retriable error, a broken connection or a third lost answer subscribes again, resuming with the id, and sends the ack on the new subscription’s connection, within retry_for; the new subscription holds the message. Without an id a broken connection means the message went back: MESSAGE_NOT_HELD, which also comes when another transaction holds it, or when the consumer did not come back within the resume window. Any other error leaves only an abort.
Offsets in a transaction: AddOffsetsToTxn to the coordinator, with the group, member id, member epoch and positions; it answers like CommitOffsets. A refusal (a fenced member, say) or an unknown outcome leaves only an abort.
Client behaviour: consumer groups
A member heartbeats with GroupHeartbeat to the coordinator (ADR 0033). The Rust and Java clients heartbeat only from their poll: a caller that takes longer than the session timeout between polls loses its partitions.
- Join with member id
"", epoch 0, and the subscribed stream ids, or the pattern instead (ADR 0073), and the instance id if any (ADR 0072). Later heartbeats carry the member id and epoch the answer gave, the owned partitions, and no subscription. - When. The next heartbeat is due after
min(heartbeat_interval_ms, session_timeout / 3), counted from when the last one was sent; after a failure, after one backoff. - Assignment. An answer with an assignment replaces the target. A partition no longer in it is not read from that moment on. With auto-commit its position is committed, and the next heartbeat goes out at once, reporting it released. Only then may another member get it. New partitions start without a position.
- Positions. A partition without one asks
OffsetFetchwithrequire_stable = 1, so an offset an open transaction holds waits.NO_OFFSETstarts at 0, or where the member was told to start (ListOffsetswith-1or a time). AnOFFSET_OUT_OF_RANGEwhose log start is past the position goes on from the log start. - Lost. When no heartbeat has succeeded for the session timeout, counted from when the last successful one was sent, every partition is dropped at once, without a commit, and the member joins again.
UNKNOWN_MEMBER_IDandFENCED_MEMBER_EPOCHmean the same.FENCED_INSTANCE_IDmeans a newer instance took the place: stop, and do not join again. - Commit sends
CommitOffsetswith the member id and epoch; a retriable error is retried as maybe done. - Leave with epoch
-1after committing, if auto-commit is on.
Client behaviour: admin
Admin requests (creates, deletes, bindings, settings, users, tenants, roles, nodes, pins, placements, partitions, revoking sessions) go to the metadata group’s leader (ADR 0034). Until a client knows it, it asks the brokers in turn. A retriable error carries a leader hint, which becomes the address for the next attempt. A node named in a hint whose address is unknown means: ask for metadata first.
-
Every admin request may be retried: a lost one may have been carried out. A create retried after a lost answer may find what its first attempt made and answer
RESOURCE_EXISTSwith the resource’s id; report it so, with the id, rather than as an error of the create. A delete answeredUNKNOWN_RESOURCEis done. -
Names are turned into ids with
Metadataasked of every known broker, keeping the newest resource per name, so a name made a moment ago is found even on a broker that lags. -
A bind or unbind changes routing: forget the exchange’s cached bindings.
-
PurgeQueueandRequeuego to the queue’s leader and work in rounds: sendup_to0, then theup_tothe answer gave, untilmoreis false, adding upcount. -
RevokeCertificate(key 52) carries the issuer (32 bytes: the SHA-256 of the issuer’s distinguished name in DER form followed by the certificate’s authority key identifier, if it has one; ADR 0096), the serial, the certificate’snot_after_msand a reason. A cluster admin sends it; every node then ends connections that showed that certificate and refuses it until it expires (ADR 0090). -
DescribeCertificates(key 53, cluster admin) lists the client certificates nodes accepted and the revoked ones: per entry issuer, serial, principal, validity, first and last sight, a state (0 valid, 1 expiring within 30 days, 2 expired, 3 revoked), the reason and time of a revocation; the answer carries the leader’s clock the states are computed against.RecordCertificates(key 54) is internal: only broker nodes send it, and a client never does (ADR 0091). So isReportDisk(key 61), by which a node tells the metadata leader that it went low on disk or back (ADR 0124); it is not in the vectors. -
RequestCertificate(key 55) carries a principal and a PKCS#10 signing request in DER form; its id is the SHA-256 of the request, so the same request twice is one. A cluster admin asks in any name; any other user only in their own (another name isUNAUTHORIZED), and their new request replaces their pending one; users’ own requests take at most 192 of the 256 places (ADR 0119).DescribeCertificateRequests(key 56) lists them, newest first, all of them to a cluster admin and to anyone else what they asked for: id, principal, request, who asked, when, when it lapses (7 days), a state (0 pending, 1 signed, 2 expired) and, once signed, the certificate (DER) and the time.RecordCertificate(key 57, cluster admin) is what the CA holder’slotse certs --signsends after signing: the request id, the certificate, the issuer fingerprint, serial and validity beside it (ADR 0092). The broker refuses it withINVALID_REQUESTunless the certificate has the request’s key and principal as its only common name, is no CA or node certificate, is signed by one of the cluster’s CA certificates, and says of itself what the fields beside it say (ADR 0096). -
DescribeSessions(key 59) is answered by the node it is sent to, for itself: an erroru16, the node’s idu32, its clocku64(ms since the Unix epoch) and an array of sessions, each the principal’s kindu8(0 anonymous, 1 user, 2 node), its name (a node’s id in decimal), the IdP groups (array of strings), the mechanismu8(0 none, 1 certificate, 2 SCRAM-SHA-512, 3 OAUTHBEARER), the peer address, when the connection was made, when its credential dates from and when it ends unless renewed (eachu64ms; 0: with the connection), then an empty tag block. Cluster admins and viewers get every session that has signed in, anyone else only those made with their own name (ADR 0121). A client asks every broker, as forDescribeMetrics. -
RenewCertificate(key 60, ADR 0123) renews the client certificate the connection was made with. The request is one field, a PKCS#10 signing request in DER form (at most 4,096 bytes) signed with the new key, then an empty tag block; its subject and extensions are ignored. The answer is the erroru16, the reason (optional string), the chain (array of bytes: the new certificate, then the renewal CA’s, DER), the new certificate’s end (u64, ms since the Unix epoch) and the metadata leader’s hint as the trailing block. It goes to the metadata leader like an admin change. It is refused withINVALID_REQUEST:- on a connection that showed no client certificate;
- outside the certificate’s window, which opens at the earlier of its last 30 days and two thirds of its validity, as the leader’s clock sees it;
- for a request that does not verify;
- on a cluster without a renewal CA;
- when 4,096 renewed certificates are held and this one replaces none.
A connection that signed in again with SASL as someone else gets
UNAUTHORIZED. The new certificate names the same principal and is valid for 90 days. A later renewal of the same certificate replaces it, so a client that retries after a lost answer keeps only the last chain it was given. A client makes a new P-256 key for each call and writes the key before the chain. -
ReassignPartition(key 58, cluster admin) carries a resource id, a partition and the voters (an array ofu64node ids, each once, active nodes); an empty array releases the partition to the planner (ADR 0115). It is refused withINVALID_REQUESTwhile the partition is moving: ask again later. The replica count of every partition is the settingreplication.factor, changed withAlterConfigson its own (no other key in the same request). -
AlterUsercarries a SCRAM-SHA-512 credential: a random 16-byte salt, at least 4096 iterations, the stored and server keys. The password never leaves the client.
Conformance vectors
sdk/conformance/* is written by cargo run -p lotse-proto --features samples --example vectors (make sdk-vectors). One vector per line, # for comments.
| File | Line |
|---|---|
requests.hex | label hex (frame without the length prefix; correlation id is index + 1) |
responses.hex | label api_key hex (correlation id is the index in the sample list) |
pushes.hex | label hex |
batches.hex | label hex |
batch-fields.txt, batch-records.txt | the header fields and records of each batch, as the Rust parser reads them |
batch-ops.txt | op arguments input output: an in-place edit and its result |
fetch-records.hex | several batches in a row |
varints.txt, crc32c.txt, error-codes.txt | fixed cases, including CRC32C of 123456789 = E3069283 |
routing.txt | label type bindings routing_key headers queues: exchange routing as lotse-route does it; the header of the file explains the fields |
scram.txt | label user_hex password salt_hex iterations client_nonce server_nonce and the five messages, stored_key server_key; the password is = and its hex |
A decoder passes when it reads every vector and writes it again to the same bytes, refuses each vector cut at every length and with one byte added, and agrees with the field files.
Rust client
lotse-client is the reference client. It is generic over the runtime traits of lotse-rt (Net, Clock, Spawn, Rng), so the same code runs on Tokio in production and in the deterministic simulation in tests. It is a workspace crate; depend on it by path or Git until a release is published.
[dependencies]
lotse-client = { git = "https://github.com/pierrejochem/lotse" }
lotse-rt-tokio = { git = "https://github.com/pierrejochem/lotse" }
lotse-record = { git = "https://github.com/pierrejochem/lotse" }
bytes = "1"
Connecting
ClientConfig::new(bootstrap) takes brokers as host:port strings and fills in defaults. The fields you will change most:
| Field | Default | Meaning |
|---|---|---|
request_timeout | 10 s | How long one request may take before the connection is given up. |
retry_for | 30 s | How long retriable failures are retried before ClientError::GaveUp. |
produce_timeout | 5 s | How long the leader waits for the quorum on a produce. |
idempotent | true | Idempotent produce. Needs the coordinator group. |
transactional_id | None | Needed for begin_transaction. One client instance per id. |
producer_name | None | A stable name for an idempotent producer (ADR 0071). |
txn_timeout | 60 s | How long a transaction may stay open before the coordinator aborts it. |
credentials | None | Credentials::Scram { user, password } or Credentials::OAuthBearer(..). |
tenant | default | Where created resources, transactional ids and group ids live. |
zone | none | The zone the client runs in. Fetches and offset lookups (list_offset) go to a replica of the partition in that zone, which serves the committed data it has and may lag the leader a little; to the leader if none is there, that replica fails, or it lags too far (ADR 0116). Everything else goes to the leader. Nodes name their zone in the cluster file (ADR 0104). |
The runtime decides the transport. A TokioRuntime uses the network only after with_tls (a TLS 1.3 configuration) or plaintext_on_loopback; plain TCP is refused off loopback.
use lotse_client::{Client, ClientConfig};
use lotse_rt_tokio::TokioRuntime;
let rt = TokioRuntime::new(std::env::temp_dir()).plaintext_on_loopback();
let client = Client::new(rt, ClientConfig::new(vec!["127.0.0.1:9092".into()]));
Streams
Create a stream, produce a batch and read it back. produce takes a record batch built with lotse_record::BatchBuilder; it returns the offset of the batch’s first record once the batch is committed.
use std::time::Duration;
use lotse_record::{BatchBuilder, BatchOptions};
client.create_stream("orders", 2, 3).await?; // 2 partitions, 3 replicas
let mut batch = BatchBuilder::new(BatchOptions::default());
batch.push(now_ms, Some(b"customer-7"), Some(b"order-1"), &[]);
let offset = client.produce("orders", 0, batch.finish()?).await?;
let fetched = client.fetch("orders", 0, offset, Duration::from_millis(500)).await?;
for b in fetched.committed_batches()? {
for record in b.records()? {
// record.offset, record.key, record.value
}
}
let next = fetched.next_offset(offset);
- You choose the partition. The client does not hash keys.
fetchis read_committed.fetch_uncommittedalso returns open and aborted transactions and their markers; use it to inspect a log, not to process it.Fetched::committed_batchesdrops what a read_committed reader must not see.Fetched::next_offsetmoves on even when no message came, because control entries advance the offset.list_offset(stream, partition, at)takesOffsetAt::Earliest,OffsetAt::LatestorOffsetAt::Time(ms)(ADR 0059).ClientError::OffsetOutOfRangemeans retention deleted the offset. Read on from the log start it reports.
Idempotent produce and unknown outcomes
With idempotent on, the client gets a producer id from the coordinator, stamps each batch and retries a batch whose outcome is unknown with the same sequence. The leader recognises a batch it already holds and answers with its offset (ADR 0028).
Only when retrying runs out of retry_for with the outcome still unknown does the caller get ClientError::Unknown. The client then starts over with a new producer id so later batches do not wait for a gap. Read the partition, or produce again and accept a possible duplicate. Without idempotent, an unknown outcome is never retried.
Queues
create_queue(name, replication, ack_timeout) creates a queue; create_queue_with and create_queue_in take more options and a tenant. subscribe(queue, credit) returns a Consumer. The queue’s leader pushes messages within the credit; the consumer settles each one.
client.create_queue("jobs", 3, None).await?;
let mut jobs = client.subscribe("jobs", 16).await?;
while let Some(message) = jobs.next(Duration::from_secs(5)).await? {
// message.record, message.delivery_count
jobs.ack(&message).await?; // or jobs.nack(&message, true) / jobs.nack_after(&message, delay)
}
jobs.close().await;
acksettles a message for good.nack(&message, requeue)delivers it again withrequeue, and discards it like an acknowledged one without;nack_after(&message, delay)returns it after a delay (ADR 0077).is_standby()is true while a subscription waits behind another on a queue withsingle.active.consumer.- After a leader change or a broken connection the consumer subscribes again by itself, naming the consumer id the leader gave it, and keeps the messages it received;
ack_insends a transactional ack again on the new subscription (ADR 0109). If it does not come back within the queue’s resume window, its messages are delivered again, possibly to another consumer. Message::delivery_countis how often the message has been delivered, this time included.
Publishing through an exchange
declare_exchange(name, kind), bind(exchange, queue, key), bind_headers and unbind set up routing. publish goes to any node, which routes by the bindings it has applied (ADR 0079).
use lotse_client::OutMessage;
let published = client.publish("orders-x", "eu.created", OutMessage::value("order-1"), false).await?;
// published: each routed queue and the message's offset there
With mandatory set, a message no queue takes is refused; otherwise it is dropped. If a node’s answer is lost the result is ClientError::Unknown, not a second publish. OutMessage has a timestamp, an optional key and value, and headers; a headers exchange routes by them.
Transactions
Set transactional_id, then begin_transaction. Starting one fences any older instance with the same id. See Exactly-once recipes for the patterns.
| Call | What it does |
|---|---|
Transaction::produce(stream, partition, batch) | Writes a batch that becomes visible on commit. |
Transaction::publish(exchange, key, message, mandatory) | Routes in the client and writes to each queue atomically (ADR 0032). |
Transaction::send_offsets(&group) | Commits a GroupConsumer’s positions with the transaction. |
Consumer::ack_in(&mut txn, &message) | Acknowledges a queue message within the transaction (ADR 0075). |
Transaction::commit(), abort() | End it. |
A transaction dropped without commit or abort is left open; the client’s next transaction starts with a fresh producer id, which makes the coordinator abort the old one. After a partial failure a transaction reports ClientError::MustAbort; only abort is left.
Consumer groups
join_group(group, &["orders"], GroupConfig::default()) joins; partitions arrive with the first poll calls. join_group_matching(group, pattern, cfg) subscribes by a regular expression (ADR 0073).
use lotse_client::GroupConfig;
let mut group = client.join_group("billing", &["orders"], GroupConfig::default()).await?;
loop {
for r in group.poll(Duration::from_millis(500)).await? { // heartbeats too: call it often
// r: stream, partition, record
}
group.commit().await?; // with auto_commit on, positions are also committed on revoke and close
}
GroupConfig holds session_timeout (45 s), rebalance_timeout (60 s), auto_commit (on), start_from (an OffsetAt, Earliest) and instance_id for static membership (ADR 0072). rewind() forgets the read positions so every owned partition starts again from the committed offset; use it after an aborted transaction. positions() and assignment() show what the member holds.
Admin
The admin calls are methods of Client, in crates/lotse-client/src/admin.rs:
- Resources:
create_stream,create_stream_with,create_queue,create_queue_in,declare_exchange,bind,unbind,delete,delete_if_empty,create_partitions. - Queues:
purge,requeue,alter_configs,describe_configs. - Access:
set_user_password,delete_user,describe_users,create_tenant,delete_tenant,describe_tenants,alter_role_bindings,describe_role_bindings,who_am_i,revoke_sessions. - Cluster:
add_node,remove_node,describe_nodes,move_leader,reassign_partition,node_metrics.
Each is checked against the signed-in principal’s roles on the broker (ADR 0039).
Renewing the client certificate
On a cluster with a renewal CA, renew_certificate renews the certificate the client’s connections are made with, once it is in its window: its last 30 days, or past two thirds of its validity (ADR 0123). It makes a new P-256 key and returns RenewedCertificate with the key and chain as PEM, for you to write where the runtime’s TLS reads them; a TokioRuntime takes new files up with reload_tls. renew_certificate_if_due takes the certificate (DER) and does nothing before the window opens. A retry after a lost answer renews again, and the earlier renewal then stops working, so keep only the one returned.
Testing against the simulation
Because the client takes its runtime as a parameter, you can run it on lotse-sim with crashing disks and a faulty network. The client simulation in crates/lotse-server/tests/clients.rs does this. See Contributing.
Java and Kotlin
The Java client is in sdk/java (Java 17, Gradle). It follows lotse-client rule for rule; the Kotlin layer in sdk/kotlin wraps it for coroutines. The design is in ADR 0056 and ADR 0080. The artifact group is dev.lotse, version 0.0.1; nothing is published to a repository yet, so build it from the source tree.
Packages
| Package | What |
|---|---|
dev.lotse.proto | Field codec, frames, every client-facing request and response, error codes. |
dev.lotse.record | Record batch, BatchBuilder, in-place edits, control batches. |
dev.lotse.net | Connections, TLS 1.3 (TlsConfig), pipelining, pushes, session renewal. |
dev.lotse.auth | Credentials: SCRAM-SHA-512 and OAUTHBEARER. |
dev.lotse.client | LotseClient, QueueConsumer, GroupConsumer, Transaction and the admin calls. |
The client has no runtime dependencies. Reading and writing zstd and lz4 batches needs zstd-jni and lz4-java, which are optional; their absence is named in the error (ADR 0061).
Connecting
ClientConfig.builder(bootstrap...) builds a configuration. There is no transport default: call tls(TlsConfig.fromFiles(ca, cert, key)) or plaintextOnLoopback().
var config = ClientConfig.builder("127.0.0.1:9092", "127.0.0.1:9093")
.plaintextOnLoopback()
.build();
try (var client = new LotseClient(config)) {
// ...
}
Other builder methods: requestTimeout, retryFor, backoff, produceTimeout, idempotent, txnTimeout, producerName, transactionalId, metadataMaxAge, tenant, zone, credentials. They mean what the Rust fields mean (see Rust client).
Streams
LotseClient offers createStream, produce, fetch, fetchUncommitted and listOffset. Every call has an ...Async form that returns a CompletableFuture. Two async produces to one partition run one after the other in no promised order: await the first if order matters.
- You choose the partition; the client does not hash keys.
u64values such as offsets travel in alongas their bit pattern. Print and compare them withLong.toUnsignedStringand the otherLong.*Unsignedmethods.- A message without a timestamp is stamped with the time of the call. Retention judges data by producers’ timestamps, so a message stamped 0 would be deleted at the next pass.
ClientException.Unknownis the one error that needs a decision: afterretryForthe client cannot tell whether the batch was written. Read the partition, or produce again and accept a possible duplicate.ClientException.OffsetOutOfRangecarrieslogStart().
The example below creates a stream, writes ten messages and reads them back. It is compiled by gradlew check.
package dev.lotse.examples;
import dev.lotse.client.ClientException;
import dev.lotse.client.Fetched;
import dev.lotse.client.LotseClient;
import dev.lotse.client.OutMessage;
import dev.lotse.client.ProduceResult;
import dev.lotse.record.BatchRecord;
import java.time.Duration;
/** Creates a stream, writes ten messages to it and reads them back. */
public final class StreamExample {
private StreamExample() {}
public static void main(String[] args) {
try (LotseClient client = new LotseClient(Env.config().build())) {
try {
client.createStream("example-orders", 2, 3);
} catch (ClientException.ResourceExists e) {
// made by an earlier run
}
for (int i = 0; i < 10; i++) {
// The partition is the caller's to choose; here, by key.
String customer = "customer-" + (i % 3);
int partition = Math.floorMod(customer.hashCode(), 2);
ProduceResult r = client.produce(
"example-orders", partition, OutMessage.value("order-" + i).withKey(customer));
System.out.printf("order-%d at partition %d, offset %s%n", i, partition,
Long.toUnsignedString(r.baseOffset()));
}
for (int partition = 0; partition < 2; partition++) {
long offset = 0;
while (true) {
Fetched f = client.fetch("example-orders", partition, offset, Duration.ofMillis(200));
for (BatchRecord record : f.committedMessages()) {
System.out.printf("read %s from partition %d%n", record.value().toUtf8(), partition);
}
long next = f.nextOffset(offset);
if (next == offset) {
break; // nothing more for now
}
offset = next;
}
}
}
}
}
Queues
createQueue(name, QueueOptions) and subscribe(queue, SubscribeOptions) return a QueueConsumer. poll(Duration) returns a Delivery or null; settle with ack, nack(delivery, requeue) or nackAfter(delivery, delay). isStandby() is true while a subscription waits behind another on a single active consumer queue. The consumer resubscribes by itself after a leader change, as the same consumer, and keeps the messages it received, also those acked in a transaction (ADR 0109).
QueueOptions.builder() sets maxDeliveries, deadLetterQueue, ackTimeout and the other queue settings. A retired message arrives in the dead-letter queue with Delivery.deadLetter() describing where it came from and why.
package dev.lotse.examples;
import dev.lotse.client.ClientException;
import dev.lotse.client.DeadLetter;
import dev.lotse.client.Delivery;
import dev.lotse.client.LotseClient;
import dev.lotse.client.OutMessage;
import dev.lotse.client.QueueConsumer;
import dev.lotse.client.QueueOptions;
import dev.lotse.client.SubscribeOptions;
import java.time.Duration;
/**
* A work queue with a dead-letter queue: jobs that fail are tried again after a delay, and a job that fails three
* times moves to the dead-letter queue.
*/
public final class QueueExample {
private QueueExample() {}
public static void main(String[] args) {
try (LotseClient client = new LotseClient(Env.config().build())) {
createIfMissing(client, "example-jobs-dead", QueueOptions.defaults());
createIfMissing(client, "example-jobs", QueueOptions.builder()
.maxDeliveries(3)
.deadLetterQueue("example-jobs-dead")
.ackTimeout(Duration.ofSeconds(30))
.build());
for (int i = 0; i < 5; i++) {
client.produce("example-jobs", 0, OutMessage.value("job-" + i));
}
try (QueueConsumer jobs = client.subscribe("example-jobs", SubscribeOptions.withCredit(16))) {
while (true) {
Delivery job = jobs.poll(Duration.ofSeconds(5));
if (job == null) {
break; // nothing for 5 s: done
}
String name = job.value().toUtf8();
if (name.equals("job-3")) {
// Fails every time: back in one second, until max.deliveries retires it.
System.out.printf("%s failed (delivery %d)%n", name, job.deliveryCount());
jobs.nackAfter(job, Duration.ofSeconds(1));
continue;
}
System.out.println("done " + name);
jobs.ack(job);
}
}
// The retired job, with where it came from and why.
try (QueueConsumer dead = client.subscribe("example-jobs-dead")) {
Delivery job = dead.poll(Duration.ofSeconds(5));
if (job != null) {
System.out.printf("dead letter %s: %s after %d deliveries%n", job.value().toUtf8(),
job.deadLetter().map(DeadLetter::reason).orElse("?"),
job.deadLetter().map(DeadLetter::deliveries).orElse(0));
dead.ack(job);
}
}
}
}
private static void createIfMissing(LotseClient client, String name, QueueOptions options) {
try {
client.createQueue(name, options);
} catch (ClientException.ResourceExists e) {
// made by an earlier run
}
}
}
Publishing through an exchange
declareExchange, bind, bindHeaders and unbind set up routing; LotseClient.publish(exchange, routingKey, message, mandatory) goes to a node, which routes it (ADR 0079). Inside a transaction Transaction.publish routes in the client, checked against sdk/conformance/routing.txt.
Transactions and groups
beginTransaction() returns a Transaction; set transactionalId first. It has produce, produceBatch, publish, sendOffsets(GroupConsumer), commit and abort; QueueConsumer.ackIn(txn, delivery) acknowledges within it. close() without a decision ends the handle; the coordinator aborts what is left when the next transaction starts with a fresh producer id.
joinGroup(group, streams, GroupConfig) and joinGroupMatching(group, pattern, config) return a GroupConsumer. poll(Duration) heartbeats too, so call it often. commit(), rewind(), assignment(), positions() and close() match the Rust client. Exactly-once recipes shows the combination.
package dev.lotse.examples;
import dev.lotse.client.ClientException;
import dev.lotse.client.GroupConfig;
import dev.lotse.client.GroupConsumer;
import dev.lotse.client.GroupRecord;
import dev.lotse.client.LotseClient;
import dev.lotse.client.OutMessage;
import dev.lotse.client.Transaction;
import java.time.Duration;
import java.util.List;
/**
* Consume, transform, produce, exactly once: a group member reads orders, and each batch of invoices it writes
* commits in one transaction with the group's offsets. A crash between the two leaves neither.
*/
public final class ExactlyOnceExample {
private ExactlyOnceExample() {}
public static void main(String[] args) {
// One running client per transactional id: a second one fences the first.
try (LotseClient client = new LotseClient(Env.config().transactionalId("example-billing-1").build())) {
try {
client.createStream("example-invoices", 1, 3);
} catch (ClientException.ResourceExists e) {
// made by an earlier run
}
GroupConfig config = GroupConfig.builder().autoCommit(false).build(); // offsets go with the transaction
try (GroupConsumer group = client.joinGroup("example-billing", List.of("example-orders"), config)) {
long deadline = System.nanoTime() + Duration.ofSeconds(20).toNanos();
while (System.nanoTime() < deadline) {
List<GroupRecord> orders = group.poll(Duration.ofMillis(500)); // heartbeats too: call it often
if (orders.isEmpty()) {
continue;
}
try (Transaction txn = client.beginTransaction()) {
for (GroupRecord order : orders) {
txn.produce("example-invoices", 0,
OutMessage.value("invoice for " + order.record().value().toUtf8()));
}
txn.sendOffsets(group);
txn.commit();
System.out.printf("invoiced %d orders%n", orders.size());
} catch (ClientException e) {
// Not committed: read the same orders again from the group's committed offsets.
System.out.println("transaction failed, reading again: " + e.getMessage());
group.rewind();
}
}
}
}
}
}
Admin
The admin calls are methods of LotseClient: delete, deleteIfEmpty, purge, requeue, alterConfigs, describeConfigs, setUserPassword, deleteUser, describeUsers, createTenant, deleteTenant, describeTenants, alterRoleBindings, describeRoleBindings, whoAmI, revokeSessions, addNode, removeNode, describeNodes, moveLeader, reassignPartition, createPartitions and nodeMetrics.
Renewing the client certificate
On a cluster with a renewal CA, a client that signs in with a certificate renews it in its window: its last 30 days, or past two thirds of its validity (ADR 0123).
TlsConfig tls = TlsConfig.fromFiles(ca, cert, key); // reads the files again when they change
LotseClient client = new LotseClient(ClientConfig.builder("10.0.0.1:9000")
.tls(tls)
.renewCertificate(cert, key) // renews by itself in the window; looks every hour
.build());
renewCertificate(cert, key) on the client renews once and writes the new key, then the new chain, over the files; renewCertificate() only returns them. CertificateRenewal.due(cert, now) says whether the window is open. renewalFailure() tells why the last automatic renewal failed. Connections made after the files changed show the new certificate; the old one keeps working until it expires.
Kotlin
dev.lotse.kotlin wraps the Java client: every call is a suspend function that runs the blocking Java call on Dispatchers.IO.
lotse(vararg bootstrap) { ... }builds aLotse(the block configuresClientConfig.Builder). BuildersgroupConfig { }andqueueOptions { }do the same for their types, andmessage(value, key, headers)makes anOutMessage.Lotse.records(stream, partition, from)andLotseQueue.deliveries()andLotseGroup.records()areFlows.Lotse.transaction { }begins a transaction, runs the block with aLotseTransactionreceiver, commits when the block returns, and aborts on an exception or cancellation.LotseTransaction,LotseQueueandLotseGroupexpose the Java object asjava.
package dev.lotse.examples
import dev.lotse.client.ClientException
import dev.lotse.client.GroupRecord
import dev.lotse.kotlin.groupConfig
import dev.lotse.kotlin.lotse
import dev.lotse.kotlin.message
import java.time.Duration
import kotlinx.coroutines.flow.take
import kotlinx.coroutines.runBlocking
import kotlinx.coroutines.withTimeoutOrNull
/**
* Writes orders, then invoices them exactly once from a consumer group, each poll's orders in one transaction with
* the group's offsets. Brokers from `LOTSE_BOOTSTRAP` (default `127.0.0.1:9092`), plain TCP on loopback.
*/
fun main(): Unit = runBlocking {
val bootstrap = (System.getenv("LOTSE_BOOTSTRAP") ?: "127.0.0.1:9092").split(",").toTypedArray()
lotse(*bootstrap) {
plaintextOnLoopback()
transactionalId("example-billing-kt")
}.use { lotse ->
for (stream in listOf("example-orders-kt", "example-invoices-kt")) {
try {
lotse.createStream(stream, partitions = 1, replication = 3)
} catch (e: ClientException.ResourceExists) {
// made by an earlier run
}
}
repeat(10) { lotse.produce("example-orders-kt", 0, message("order-$it", key = "customer-${it % 3}")) }
lotse.joinGroup("example-billing-kt", listOf("example-orders-kt"), groupConfig { autoCommit(false) }).use { group ->
withTimeoutOrNull(Duration.ofSeconds(20).toMillis()) {
var invoiced = 0
while (invoiced < 10) {
val orders: List<GroupRecord> = group.poll(Duration.ofMillis(500))
if (orders.isEmpty()) continue
try {
lotse.transaction {
for (order in orders) {
produce("example-invoices-kt", 0, message("invoice for ${order.record().value().toUtf8()}"))
}
sendOffsets(group)
}
invoiced += orders.size
println("invoiced $invoiced orders")
} catch (e: ClientException) {
group.rewind() // not committed: read them again
}
}
}
}
lotse.records("example-invoices-kt", 0, from = 0).take(10).collect { println(it.value().toUtf8()) }
}
}
Running the examples
The examples read LOTSE_BOOTSTRAP (default 127.0.0.1:9092). The Java examples use plain TCP on loopback unless LOTSE_CA is set, with optional LOTSE_CERT, LOTSE_KEY, LOTSE_USER and LOTSE_PASSWORD. make dev starts a local three-node cluster.
C#
The C# client is in sdk/csharp (.NET 10). It follows lotse-client and the Java client rule for rule. The design is in ADR 0083; the optional codecs are in ADR 0084. Nothing is published to NuGet yet, so build it from the source tree. sdk/csharp/README.md lists every rule in full.
Build and test
global.json pins the SDK to 10.0.403 and lets it roll forward to a later feature band. From the repository root:
make sdk-csharp-test # cd sdk/csharp && dotnet test Lotse.sln
LOTSE_BIN=/path/to/lotse make sdk-csharp-test # adds the TLS and real-node tests
Without LOTSE_BIN the tests that need a lotse executable are skipped with a message. make sdk-test runs this after the Java and Kotlin tests.
Projects and namespaces
| Project | What |
|---|---|
src/Lotse.Client | Everything below. No dependencies beyond the base class library. |
src/Lotse.Client.Codecs | zstd and LZ4 for compressed batches, optional. |
| Namespace | What |
|---|---|
Lotse.Proto | Field reader and writer, frames, every client-facing request and response, error codes. |
Lotse.Record | Record batch parser and builder, in-place edits, control batches. |
Lotse.Net | Connection, ConnectionPool, ConnectionOptions, TlsConfig, Address, the exceptions. |
Lotse.Auth | Credentials (SCRAM-SHA-512 and OAUTHBEARER), ScramCredential. |
Lotse.Client | LotseClient, QueueConsumer, GroupConsumer, Transaction and the admin calls. |
Unsigned wire types keep their width: offsets are ulong, so no unsigned-compare helpers are needed as in Java.
Connecting
ClientConfig.CreateBuilder(bootstrap...) builds a configuration. There is no transport default: call Tls(TlsConfig.FromFiles(ca, cert, key)) or PlaintextOnLoopback(), or the client refuses with InsecureTransportException before it sends a byte.
ClientConfig config = ClientConfig.CreateBuilder("127.0.0.1:9092", "127.0.0.1:9093")
.PlaintextOnLoopback()
.Build();
using var client = new LotseClient(config);
Other builder methods include WithCredentials(Credentials.Scram(user, password)), WithTransactionalId, WithProducerName, WithIdempotence, WithTenant, WithZone, WithRetryFor and WithMetadataMaxAge. Defaults are the Rust client’s: 10 s per request, 30 s of retrying, 50 ms between attempts, 5 s for the leader’s quorum (see Rust client).
Streams
await client.CreateStreamAsync("orders", partitions: 2, replication: 3);
ProduceResult sent = await client.ProduceAsync("orders", 0,
OutMessage.FromValue("order-1").WithKey("k1"));
Fetched fetched = await client.FetchAsync("orders", 0, sent.BaseOffset, TimeSpan.FromMilliseconds(500));
foreach (BatchRecord record in fetched.CommittedMessages())
{
Console.WriteLine($"{record.Offset} {record.Value!.ToUtf8()}");
}
ulong next = fetched.NextOffset(sent.BaseOffset);
- You choose the partition; the client does not hash keys.
FetchAsyncis read_committed. Continue fromNextOffset, not from the number of records: a response may hold only control batches.- A message without a timestamp gets the time of the call. A message stamped 0 would be deleted at the next retention pass.
OutcomeUnknownExceptionis the one error that needs a decision: afterRetryForthe client cannot tell whether the batch was written. Read the partition, or produce again and accept a possible duplicate.OffsetOutOfRangeExceptionsays where the partition starts now.- Every call takes a
CancellationTokenand throwsOperationCanceledException. A produce cancelled while its outcome is unknown drops the producer id, as an unknown outcome does.
Queues
CreateQueueAsync(name, QueueOptions) and SubscribeAsync(queue, SubscribeOptions) return a QueueConsumer. NextAsync waits for a message, PollAsync(wait) returns null when none came, and DeliveriesAsync() is an IAsyncEnumerable<Delivery>. Settle with AckAsync, NackAsync(delivery, requeue) or NackAfterAsync(delivery, delay).
await client.CreateQueueAsync("jobs", QueueOptions.CreateBuilder()
.WithAckTimeout(TimeSpan.FromSeconds(30))
.WithMaxDeliveries(5)
.WithDeadLetterQueue("jobs-dead")
.Build());
await using QueueConsumer consumer = await client.SubscribeAsync("jobs", SubscribeOptions.WithCredit(16));
await foreach (Delivery delivery in consumer.DeliveriesAsync(cancellationToken))
{
try
{
Handle(delivery.Value!.ToUtf8());
await consumer.AckAsync(delivery);
}
catch (TransientException)
{
await consumer.NackAfterAsync(delivery, TimeSpan.FromSeconds(10));
}
}
- Credit defaults to 64 messages. The consumer resubscribes by itself after a leader change, as the same consumer, and keeps the messages it received (ADR 0109); after an ack timeout they may arrive twice, with a higher
DeliveryCount. IsStandbyis true while a subscription waits behind another on a single active consumer queue.Delivery.DeadLettersays where a retired message came from and why.DisposeAsyncsendsUnsubscribeand then hands back what the consumer holds and has not handed out. The Java consumer leaves those messages with the broker until the ack timeout.
Publishing through an exchange
DeclareExchangeAsync, BindAsync, BindHeadersAsync and UnbindAsync set up routing. PublishAsync(exchange, routingKey, message, mandatory) goes to a node, which routes it (ADR 0079); inside a transaction Transaction.PublishAsync routes in the client, checked against sdk/conformance/routing.txt.
await client.DeclareExchangeAsync("events", Messages.ExchangeType.Topic);
await client.BindAsync("events", "audit", "order.#");
Published published = await client.PublishAsync("events", "order.paid", OutMessage.FromValue("o-1"), mandatory: true);
If the node’s answer is lost the call throws OutcomeUnknownException and is not sent again, since the node may have written the message.
Transactions and groups
Set WithTransactionalId first. BeginTransactionAsync() returns a Transaction with ProduceAsync, PublishAsync, SendOffsetsAsync(GroupConsumer), CommitAsync and AbortAsync; QueueConsumer.AckInAsync(txn, delivery) acknowledges within it. Transaction is IAsyncDisposable only: await using aborts a transaction that was neither committed nor aborted, also when the block ends with an exception.
JoinGroupAsync(group, streams, GroupConfig) and JoinGroupMatchingAsync(group, pattern, config) return a GroupConsumer. PollAsync heartbeats too, so call it more often than the session timeout; RecordsAsync() repeats it. CommitAsync(), Rewind() and DisposeAsync() match the Rust client.
GroupConfig groupConfig = GroupConfig.CreateBuilder().WithAutoCommit(false).Build();
await using GroupConsumer group = await client.JoinGroupAsync("pipeline", new[] { "orders" }, groupConfig);
while (!stop.IsCancellationRequested)
{
IReadOnlyList<GroupRecord> records = await group.PollAsync(TimeSpan.FromMilliseconds(500), stop);
if (records.Count == 0)
{
continue;
}
await using Transaction txn = await client.BeginTransactionAsync(stop);
foreach (GroupRecord r in records)
{
await txn.ProduceAsync("shipments", 0, Transform(r.Record), stop);
}
await txn.SendOffsetsAsync(group, stop);
await txn.CommitAsync(stop);
}
After an exception that aborted the transaction, call group.Rewind() to read the records again. Exactly-once recipes shows the rules. Runnable programs are in sdk/csharp/examples.
Admin
The admin calls are methods of LotseClient: DeleteAsync, DeleteIfEmptyAsync, PurgeAsync, RequeueAsync, AlterConfigsAsync, DescribeConfigsAsync, SetUserPasswordAsync, DeleteUserAsync, DescribeUsersAsync, CreateTenantAsync, DeleteTenantAsync, DescribeTenantsAsync, AlterRoleBindingsAsync, DescribeRoleBindingsAsync, WhoAmIAsync, RevokeSessionsAsync, AddNodeAsync, RemoveNodeAsync, DescribeNodesAsync, MoveLeaderAsync, ReassignPartitionAsync, CreatePartitionsAsync and NodeMetricsAsync. Each needs the role the broker asks for. A create whose answer was lost is retried; if the first attempt did create the resource, the call throws ResourceExistsException with the id.
Renewing the client certificate
On a cluster with a renewal CA, a client that signs in with a certificate renews it in its window: its last 30 days, or past two thirds of its validity (ADR 0123).
var client = new LotseClient(ClientConfig.CreateBuilder("10.0.0.1:9000")
.Tls(TlsConfig.FromFiles("ca.pem", "alice.pem", "alice.key")) // read again when they change
.RenewCertificate("alice.pem", "alice.key") // renews by itself; looks every hour
.Build());
RenewCertificateAsync(cert, key) renews once and writes the new key, then the new chain; without the files it only returns them. CertificateRenewal.Due(file, nowMs) says whether the window is open, and RenewalFailure why the last automatic renewal failed.
Compressed batches
Add the Lotse.Client.Codecs package to read and write zstd and LZ4 batches. It brings ZstdSharp.Port and K4os.Compression.LZ4.Streams, and is found by name the first time a compressed batch is read or built. A program that is trimmed or compiled ahead of time calls LotseCodecs.Register() once.
Without the package a compressed batch still parses, but reading its records or building one throws RecordFormatException of kind UnsupportedCompression, naming the package to add. Nothing compresses by default. A decompressed block may hold at most 16 MiB.
Differences from Java
async/awaitandCancellationTokenreplaceCompletableFuture; there is no blocking form.ulongreplaces thelongbit pattern for offsets.Transaction,QueueConsumerandGroupConsumerareIAsyncDisposable. Closing a queue consumer also returns the messages it received and did not read, which the Java consumer does not.- The client ends its own side of a connection when the node’s certificate expires (ADR 0068); the Java SDK leaves that to the node.
- The codecs are a separate package, found by name, where Java uses capabilities of the Gradle module.
Node.js
The Node.js client is in sdk/node (TypeScript, Node.js 20 and later). It follows lotse-client and the Java client rule for rule. The design is in ADR 0083; the codecs are in ADR 0085. The package is named @lotse/client and is private: nothing is published to npm yet, so build it from the source tree. It has no runtime dependencies. sdk/node/README.md lists every rule in full.
Build and test
make sdk-node-test # cd sdk/node && npm ci && npm test
LOTSE_BIN=/path/to/lotse make sdk-node-test # adds the TLS and real-node tests
npm test compiles to dist/ and runs node:test. Without LOTSE_BIN the tests that need a lotse executable are skipped with a message. make sdk-test runs this after the Java and Kotlin tests.
Modules
| Module | What |
|---|---|
src/proto | Field codec, frames, every client-facing request and response, error codes. |
src/record | Record batch: parse, build, in-place edits, control batches, CRC32C, varints, the codecs. |
src/net | Connection, ConnectionPool, TlsConfig, Address, the errors. |
src/auth | scramCredentials, oauthBearerCredentials, ScramCredential. |
src/client | LotseClient, QueueConsumer, GroupConsumer, Transaction and the admin calls. |
Values of 64 bits are bigint. A u64 is held as its signed 64-bit bit pattern, as in Java, so “no offset” is -1n. Compare, add and print such values with compareU64, addU64Exact and u64ToString, not with < or String.
Connecting
new LotseClient(options) takes bootstrap and a transport. There is no transport default: give tls or plaintextOnLoopback: true, or the client refuses with InsecureTransportError, which is never retried.
import { LotseClient, TlsConfig, scramCredentials } from "@lotse/client";
const client = new LotseClient({
bootstrap: ["10.0.0.1:9092", "10.0.0.2:9092"],
tls: await TlsConfig.fromFiles("ca.pem", "alice.pem", "alice.key"), // the certificate is optional
credentials: scramCredentials("bob", "secret"), // optional
});
// ... client.close() when done
Durations are milliseconds: requestTimeoutMs 10 000, retryForMs 30 000, backoffMs 50, produceTimeoutMs 5 000, txnTimeoutMs 60 000, metadataMaxAgeMs 30 000. Other options: idempotent, producerName, transactionalId, tenant, zone. They mean what the Rust fields mean (see Rust client).
Streams
await client.createStream("orders", 2, 3, { "retention.ms": "86400000" }); // 2 partitions, 3 replicas each
const { baseOffset } = await client.produce("orders", 0, { key: "k1", value: "hello" });
for await (const record of client.readRecords("orders", 0, { from: baseOffset, untilEnd: true })) {
console.log(record.offset, Buffer.from(record.value ?? []).toString());
}
const end = await client.listOffset("orders", 0, OffsetAt.LATEST);
- You choose the partition; the client does not hash keys.
fetchis read_committed. Continue fromnextOffset(from), not from the number of records.readRecordsdoes this for you and yields committed records.- Keys, values and header values are
string(UTF-8) orUint8Array;nulland empty differ. - A message without a timestamp gets the time of the call.
UnknownOutcomeErroris the one error that needs a decision: afterretryForMsthe client cannot tell whether the batch was written. Read the partition, or produce again and accept a possible duplicate.- Calls take
{ signal }(anAbortSignal) last. A produce given up while its answer may still come rejects withUnknownOutcomeErrorand drops the producer id. - All errors extend
ClientError;OffsetOutOfRangeErrorsays where the partition starts now.
Queues
createQueue(name, options) and subscribe(queue, options) return a QueueConsumer. next() waits for a message, poll(waitMs) resolves null when none came, and the consumer is an async iterable. Settle with ack, nack(delivery, requeue) or nackAfter(delivery, delayMs).
await client.createQueue("jobs", { ackTimeoutMs: 30_000, maxDeliveries: 5, deadLetterQueue: "jobs-dead" });
const consumer = await client.subscribe("jobs", { credit: 16 });
for await (const d of consumer) {
try {
await handle(d.text());
await consumer.ack(d);
} catch {
await consumer.nackAfter(d, 5_000); // back in the queue after 5 s
}
}
consumer.close();
- Credit defaults to 64 messages. The consumer subscribes again by itself after a leader change, as the same consumer, and keeps the messages it received (ADR 0109); after an ack timeout they may arrive twice, with a higher
deliveryCount. - Leaving a
for awaitloop does not close the consumer. consumer.isStandbyis true while a subscription waits behind another on a single active consumer queue.delivery.deadLetter()says where a retired message came from and why.close()returns the messages received and not handed out, with one requeueingNack; it does not wait for answers.
Publishing through an exchange
declareExchange(name, ExchangeType.TOPIC), bind, bindHeaders and unbind set up routing. publish(exchange, routingKey, message, { mandatory }) goes to a node, which routes it (ADR 0079); inside a transaction txn.publish routes in the client, checked against sdk/conformance/routing.txt.
await client.declareExchange("orders", ExchangeType.TOPIC);
await client.bind("orders", "orders-eu", "*.eu");
const { routed } = await client.publish("orders", "orders.eu", { value: "m" }, { mandatory: true });
If the node’s answer is lost the call rejects with UnknownOutcomeError and is not sent again, since the node may have written the message.
Transactions and groups
Set transactionalId first. beginTransaction() returns a Transaction with produce, produceBatch, publish, sendOffsets(group), commit and abort; consumer.ackIn(txn, delivery) acknowledges within it. With await using, leaving the scope aborts a transaction that was not ended; if the abort does not get through, the transaction is abandoned and the error that left the scope is not hidden.
{
await using txn = await client.beginTransaction();
await txn.produce("ledger", 0, { key: "acct-7", value: "debit 40" });
await txn.publish("events", "ledger.debit", { value: "debit 40" });
await txn.commit();
}
await using needs TypeScript 5.2 or later (the SDK builds with 5.9). Without it, call abort() yourself in a finally block.
joinGroup(group, streams, config) and joinGroupMatching(group, pattern, config) return a GroupConsumer. poll(waitMs, { signal }) heartbeats when one is due, so call it more often than the session timeout; for await (const { stream, partition, record } of group) polls without end. commit(), rewind() and close() match the Rust client.
const group = await client.joinGroup("billing", ["orders"], { autoCommit: true, startFrom: OffsetAt.EARLIEST });
for await (const { record } of group) {
await handle(record);
}
await group.close(); // commits the positions and leaves
Exactly-once recipes shows the rules. Runnable programs are in sdk/node/examples.
Admin
The admin calls are methods of LotseClient: delete, deleteIfEmpty, purge, requeue, alterConfigs, describeConfigs, setUserPassword, deleteUser, describeUsers, createTenant, deleteTenant, describeTenants, alterRoleBindings, describeRoleBindings, whoAmI, revokeSessions, addNode, removeNode, describeNodes, moveLeader, reassignPartition, createPartitions and nodeMetrics. Each needs the role the broker asks for. A create whose answer was lost is retried; if the first attempt did create the resource, the call rejects with ResourceExistsError and the id.
Renewing the client certificate
On a cluster with a renewal CA, a client that signs in with a certificate renews it in its window: its last 30 days, or past two thirds of its validity (ADR 0123).
const files = { certPemFile: "alice.pem", keyPemFile: "alice.key" };
const client = new LotseClient({
bootstrap: ["10.0.0.1:9000"],
tls: await TlsConfig.fromFiles("ca.pem", files.certPemFile, files.keyPemFile), // read again when they change
renewCertificate: files, // renews by itself in the window; looks every hour (renewCheckMs)
});
client.renewCertificate(files) renews once and writes the new key, then the new chain; without files it only returns them. renewalDue(file) says whether the window is open, and client.renewalFailure why the last automatic renewal failed.
Compressed batches
zstd and lz4 are read and written without a package to install. Set attributes.compression on a BatchBuilder; nothing compresses by default.
- lz4 is implemented in the SDK and works on every supported Node.js.
- zstd is
node:zlib’s, which has it from Node.js 22.15 and 23.8, and in 24. On Node.js 20 and earlier 22 versions, reading or building a zstd batch rejects withRecordFormatErrorof kindUNSUPPORTED_COMPRESSION. Give the SDK a codec of your own withregisterCodec(Compression.ZSTD, { compress, decompress })to read zstd there. - A decompressed block may hold at most 16 MiB.
Differences from Java
- Promises and async iterators replace
CompletableFutureand blocking calls; there is no blocking form. AbortSignalreplaces thread interruption. A call given up after it was sent rejects withUnknownOutcomeErrorwhere the outcome may be unknown.- Values of 64 bits are
bigint. - The SDK has no runtime dependencies at all: lz4 is its own, zstd is Node’s. Java needs two optional libraries.
Exactly-once recipes
Lotse gives exactly-once effects from three pieces that build on each other. Use the smallest one that covers your case.
| Need | Use |
|---|---|
| A retry must not write a message twice | An idempotent producer |
| Several writes must appear together or not at all | A transaction |
| Read, transform and write must happen once as a unit | A transaction that also commits the consumer’s position |
1. Idempotent producer
Idempotent produce is on by default in both clients. The client gets a producer id from the coordinator and stamps each batch with a sequence per partition. If a connection breaks after sending, or a request times out, it sends the batch again with the same sequence. The leader checks the sequence against the whole log and answers with the offset of the batch it already has (ADR 0028).
What you must handle:
- Unknown outcome. When retrying runs out of
retry_for(Rust) orretryFor(Java) with the result unknown, the call fails withClientError::UnknownorClientException.Unknown. The client drops its producer id. Read the partition to see whether the batch is there, or produce again and accept a possible duplicate. - Restarts. Without a name, each client instance gets a fresh producer id, so a retry from before a restart is a different producer. Set a name (
producer_namein Rust,producerNamein Java) and the coordinator hands out the same id with a raised epoch on every start. The earlier instance is fenced: its retries are refused instead of written after the new instance’s batches (ADR 0071). Names share a namespace with transactional ids, and one client instance may use a name at a time. - Idle producers. A partition forgets a producer that has written nothing to it for a day (
[producers] expiration_ms); a retry after that is a new batch. It never forgets a live producer to make room. While a partition knows[producers] maxproducers (10,000), it refuses new ones withPRODUCER_LIMIT, and the client retries until one is forgotten orretry_forruns out (ADR 0110). Many short-lived producers without a name fill that table; give long-running ones a name. - Publishing through an exchange. A node writes each routed queue with an idempotent producer of its own. If its answer is lost, you get
Unknown, not a second publish (ADR 0079).
2. Transactions
Set a transactional id. One client instance per id: starting a transaction fences any older instance with the same id.
A transaction can write to several partitions and queues atomically. A read_committed reader (the default for fetch) sees all of a committed transaction and nothing of an aborted one (ADR 0029).
let mut txn = client.begin_transaction().await?;
txn.produce("payments", 0, payment_batch).await?;
txn.produce("audit", 1, audit_batch).await?;
txn.commit().await?; // or txn.abort().await?
try (Transaction txn = client.beginTransaction()) {
txn.produce("payments", 0, OutMessage.value("paid"));
txn.produce("audit", 1, OutMessage.value("payment recorded"));
txn.commit();
}
lotse.transaction {
produce("payments", 0, message("paid"))
produce("audit", 1, message("payment recorded"))
} // commits when the block returns; aborts on an exception or cancellation
Rules to keep in mind:
- A transaction that fails part way reports
MustAbort(Rust) and can only be aborted. Rust’sTransaction::publishdoes this when some queues hold a message and others do not. - A transaction left open times out after
txn_timeout(60 s by default) and the coordinator aborts it. The next transaction of the same client starts with a fresh producer id and aborts it sooner. - An open transaction holds back the last stable offset of its partitions, so read_committed readers wait behind it. Keep transactions short.
- A commit that fails with
Unknownmay have committed. The outcome stays unknown to the caller; check the data before you repeat the work.
3. Consume, transform, produce
From a stream
A consumer group member reads records; the transaction writes the results and commits the member’s positions together. A crash between the two leaves neither.
- Join the group with
auto_commitoff. Offsets then go with each transaction, not on a timer. pollfor records. It also heartbeats, so call it often.- Begin a transaction, produce the results, call
send_offsets(Rust) orsendOffsets(Java) with the group member, commit. - If the transaction fails, call
rewind(). The member reads the same records again from the group’s committed offsets.
let cfg = GroupConfig { auto_commit: false, ..GroupConfig::default() };
let mut group = client.join_group("billing", &["orders"], cfg).await?;
loop {
let orders = group.poll(Duration::from_millis(500)).await?;
if orders.is_empty() { continue; }
let mut txn = client.begin_transaction().await?;
for order in &orders {
txn.produce("invoices", 0, invoice_batch(order)).await?;
}
txn.send_offsets(&group).await?;
match txn.commit().await {
Ok(()) => {}
Err(_) => group.rewind(),
}
}
A member that no longer owns a partition is fenced: send_offsets fails and the transaction can only abort. That is the point: the new owner reads from the last committed offset, and nothing is written twice.
This is the complete Java version, compiled by gradlew check:
package dev.lotse.examples;
import dev.lotse.client.ClientException;
import dev.lotse.client.GroupConfig;
import dev.lotse.client.GroupConsumer;
import dev.lotse.client.GroupRecord;
import dev.lotse.client.LotseClient;
import dev.lotse.client.OutMessage;
import dev.lotse.client.Transaction;
import java.time.Duration;
import java.util.List;
/**
* Consume, transform, produce, exactly once: a group member reads orders, and each batch of invoices it writes
* commits in one transaction with the group's offsets. A crash between the two leaves neither.
*/
public final class ExactlyOnceExample {
private ExactlyOnceExample() {}
public static void main(String[] args) {
// One running client per transactional id: a second one fences the first.
try (LotseClient client = new LotseClient(Env.config().transactionalId("example-billing-1").build())) {
try {
client.createStream("example-invoices", 1, 3);
} catch (ClientException.ResourceExists e) {
// made by an earlier run
}
GroupConfig config = GroupConfig.builder().autoCommit(false).build(); // offsets go with the transaction
try (GroupConsumer group = client.joinGroup("example-billing", List.of("example-orders"), config)) {
long deadline = System.nanoTime() + Duration.ofSeconds(20).toNanos();
while (System.nanoTime() < deadline) {
List<GroupRecord> orders = group.poll(Duration.ofMillis(500)); // heartbeats too: call it often
if (orders.isEmpty()) {
continue;
}
try (Transaction txn = client.beginTransaction()) {
for (GroupRecord order : orders) {
txn.produce("example-invoices", 0,
OutMessage.value("invoice for " + order.record().value().toUtf8()));
}
txn.sendOffsets(group);
txn.commit();
System.out.printf("invoiced %d orders%n", orders.size());
} catch (ClientException e) {
// Not committed: read the same orders again from the group's committed offsets.
System.out.println("transaction failed, reading again: " + e.getMessage());
group.rewind();
}
}
}
}
}
}
And the same in Kotlin with transaction { }:
package dev.lotse.examples
import dev.lotse.client.ClientException
import dev.lotse.client.GroupRecord
import dev.lotse.kotlin.groupConfig
import dev.lotse.kotlin.lotse
import dev.lotse.kotlin.message
import java.time.Duration
import kotlinx.coroutines.flow.take
import kotlinx.coroutines.runBlocking
import kotlinx.coroutines.withTimeoutOrNull
/**
* Writes orders, then invoices them exactly once from a consumer group, each poll's orders in one transaction with
* the group's offsets. Brokers from `LOTSE_BOOTSTRAP` (default `127.0.0.1:9092`), plain TCP on loopback.
*/
fun main(): Unit = runBlocking {
val bootstrap = (System.getenv("LOTSE_BOOTSTRAP") ?: "127.0.0.1:9092").split(",").toTypedArray()
lotse(*bootstrap) {
plaintextOnLoopback()
transactionalId("example-billing-kt")
}.use { lotse ->
for (stream in listOf("example-orders-kt", "example-invoices-kt")) {
try {
lotse.createStream(stream, partitions = 1, replication = 3)
} catch (e: ClientException.ResourceExists) {
// made by an earlier run
}
}
repeat(10) { lotse.produce("example-orders-kt", 0, message("order-$it", key = "customer-${it % 3}")) }
lotse.joinGroup("example-billing-kt", listOf("example-orders-kt"), groupConfig { autoCommit(false) }).use { group ->
withTimeoutOrNull(Duration.ofSeconds(20).toMillis()) {
var invoiced = 0
while (invoiced < 10) {
val orders: List<GroupRecord> = group.poll(Duration.ofMillis(500))
if (orders.isEmpty()) continue
try {
lotse.transaction {
for (order in orders) {
produce("example-invoices-kt", 0, message("invoice for ${order.record().value().toUtf8()}"))
}
sendOffsets(group)
}
invoiced += orders.size
println("invoiced $invoiced orders")
} catch (e: ClientException) {
group.rewind() // not committed: read them again
}
}
}
}
lotse.records("example-invoices-kt", 0, from = 0).take(10).collect { println(it.value().toUtf8()) }
}
}
From a queue
A consumer acknowledges messages inside the transaction (ADR 0075). If it commits, the messages are settled; if it aborts, they are delivered again.
let mut jobs = client.subscribe("jobs", 16).await?;
while let Some(job) = jobs.next(Duration::from_secs(5)).await? {
let mut txn = client.begin_transaction().await?;
txn.produce("results", 0, result_batch(&job)).await?;
jobs.ack_in(&mut txn, &job).await?; // adds the queue to the transaction first
txn.commit().await?;
}
In Java, queue.ackIn(txn, delivery); in Kotlin, ack(queue, delivery) inside transaction { }.
- Acknowledge on the subscription that holds the message. The ack goes on the connection the message came on.
- Of two transactions that try to hold one message, the first in the log holds it and the other gets
MESSAGE_NOT_HELD. Any error fromack_inleaves the transaction to abort. - Acks alone do not hold the last stable offset back.
- Messages that stay unsettled past the queue’s ack timeout are delivered again; keep the transaction shorter than that.
What exactly-once does not cover
The guarantee holds for effects inside Lotse: the log, the queue ack index and the group offsets. A side effect outside it, such as a payment or an e-mail, can run twice if your process crashes after the effect and before the commit. Make such effects idempotent on their own side, with a key derived from the stream, partition and offset or from the queue message.
Contributing
The rules for everyone who changes the repository, people and AI agents alike, are in AGENTS.md. They are binding. This page is a map for newcomers and does not replace it: when the two differ, AGENTS.md wins.
For the design, read docs/design.md (German) and the decision records in docs/adr. The design document describes the current system; the ADRs keep the reasons.
Crate map
| Crate | What |
|---|---|
lotse-rt | The only seam to the outside world: Clock, Disk, SegmentFile, Spawn, Rng, Net, Host, Http traits |
lotse-rt-tokio | Production runtime: Tokio, file system, TLS 1.3 or plain TCP on loopback |
lotse-record | Batch format: header, ZigZag-varint records, CRC32C, zstd and lz4 blocks, control batches |
lotse-log | Segmented partition log, which is also the Raft log |
lotse-raft | Sans-I/O Raft core and Multi-Raft |
lotse-queue | Queue semantics free of I/O: ack index and dispatcher |
lotse-auth | SCRAM, OAUTHBEARER, roles; free of I/O |
lotse-route | Exchanges and bindings: direct, topic, fanout, headers |
lotse-proto | Native protocol: frames, error codes, requests and responses |
lotse-server | Broker node: Replica, Node, client serving, metadata, membership |
lotse-node | The lotse binary |
lotse-client | Rust client |
lotse-sim | Deterministic simulation |
lotse-console | Management console backend; the frontend is in console/web |
The full map, with the files inside lotse-server, is in AGENTS.md.
The runtime seam
Nothing above lotse-rt touches the real world directly. Code takes a Clock, Disk, Spawn, Rng or Net instead of calling Instant::now, SystemTime::now, std::thread, OS randomness or ambient Tokio APIs. Use BTreeMap and BTreeSet, never HashMap or HashSet, because iteration order would make simulations unrepeatable. clippy.toml enforces most of this. unsafe is forbidden. The reasons are in ADR 0006.
The Raft core is sans-I/O: it never does I/O. A driver persists a Ready in a fixed order: snapshot, truncate, entries (synced), hard state, apply, send (ADR 0023).
Deterministic simulation
lotse-sim runs the real code on a seeded executor with virtual time, disks that crash and lose or tear unsynced writes, and a network that drops, delays and partitions. A run is a function of its seed, so a failure is reproduced exactly.
cargo test --workspace # unit tests and short simulations
LOTSE_SIM_SEEDS=5000 cargo test -p lotse-sim -p lotse-server --release # CI depth
LOTSE_SIM_SEED=1234 cargo test -p lotse-sim # exactly one seed
- Simulations take seeds from
lotse_sim::check_seeds, which panics with the seed and the command that reproduces it. Scenario tests whose value does not grow with seeds usecheck_few_seeds. - A simulation’s oracle is written in the test, never borrowed from the code it checks.
- Simulations live in
crates/lotse-sim/tests/and in a crate’stests/; for examplecrates/lotse-server/tests/clients.rsruns the Rust client against a faulty cluster. - A failing seed is a bug until proven otherwise. Find the cause, then decide whether the code or the check is wrong, and say which in the commit. Never weaken a check to make a test pass.
- After you add faults or operations, check with a quick statistic that they happen (crashes per run, snapshots installed, reads served) before you trust a green run.
Pull requests do not run the simulations at depth. The release workflow runs all of them at 5,000 seeds before anything is built for a release, and a failure there blocks the release (ADR 0088). So run the simulations your change touches yourself, at CI depth where it matters, or run the release workflow by hand on your branch: run by hand, it runs the simulations only (ADR 0010).
Mutation checking
When you add or change a rule that safety depends on, break it on purpose and confirm that a test fails. If none does, add one: a simulation, or a deterministic unit test for rules that depend on timing, because random simulation rarely reaches those. Restore the rule afterwards. State which mutants you tried in your report.
Before you push
cargo fmt --all
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace
Also run the affected simulation at CI depth. After you change a message or the batch format, run make sdk-vectors and make sdk-test; a change to a message needs the codecs of the Java, C# and Node.js SDKs updated. After you change an API type of the console, run npm run api in console/web.
Decisions: the ADR process
A changed decision gets an ADR in the same change.
- Copy
docs/adr/template.mdto the next number, for example0082-short-title.md. - Write context, decision and consequences. State what you rejected and why.
- Add it to the index in
docs/adr/README.md. - Update the design document to match. It stays German.
A superseded decision keeps its ADR, marked as superseded, so the reasons stay readable.
Conventions
- Code, comments, docs and commit messages are in English; the design document is German.
- Comments explain why, not what. Public items have doc comments.
- Errors use
thiserrorand say what happened and, where useful, what to do. - Dependencies go through
[workspace.dependencies]. - Docs and UI copy follow the brand voice: calm, exact, short, units named, no hype, no exclamation marks, no emoji.
- Commits have an imperative subject line and a body that explains why.
Working on this guide
The guides live in docs/guide/ and are built with mdBook (ADR 0081). Examples that could drift are included with {{#include}}, not pasted. Where the repository documents no procedure, the guide says so.
Concepts
Lotse keeps messages in replicated logs. A log can be read like a stream, or worked off like a queue. This chapter names the parts you meet as an application developer.
Stream and partition
A stream is a named, ordered log of messages. A stream has one or more partitions; each partition is its own log with its own offsets. Messages in one partition keep the order they were written in. Across partitions there is no order.
Every message in a partition has an offset, a number that starts at 0 and only grows. Reading does not remove anything: a reader keeps its own position and may read the same offsets again. What leaves a stream is decided by retention (see Streams).
Each partition is a Raft group on several nodes, the stream’s replication factor. A write is acknowledged once a majority of those nodes holds it.
Queue
A queue hands each message to one consumer, then waits for that consumer to say what became of it: ack (done), nack (give it back, now or after a delay). A message nobody acknowledged is delivered again after a failure, a disconnect or the ack timeout. A queue is one partition, so it has one order, and offsets exist but are rarely needed.
A queue can limit deliveries, expire messages and send the ones it gives up on to a dead-letter queue. See Queues.
Exchange and binding
An exchange takes a message and a routing key and passes it on to the queues bound to it. A binding says which queues an exchange serves and under which key or headers. There are four kinds: direct, topic, fanout and headers. See Exchanges and routing.
You can also write to a queue directly, as to a stream’s partition 0, and skip the exchange.
Tenant
Every stream, queue and exchange belongs to a tenant. Names are unique inside a tenant, not across the cluster. A client works in one tenant, set when it is configured; without one it is default. What a user may do is decided by roles bound per tenant and name prefix: cluster-admin, cluster-viewer, tenant-admin, producer, consumer and viewer. A resource you may not see is unknown to you, so a missing permission can look like a missing queue. See ADR 0039.
Producer and delivery guarantees
The Java and Rust clients are idempotent producers by default: a retried write after a lost answer does not create a duplicate. A producer with a name, or with a transactional id, gets its identity from the cluster, so a restarted instance fences the earlier one. See ADR 0028 and ADR 0071.
Transactions write to several partitions and queues, and commit offsets or acks, all or nothing. See Transactions and exactly-once.
Consumer group
A consumer group shares the partitions of one or more streams among its members and stores their committed offsets in the cluster. Each partition has one owner at a time. When members join or leave, partitions move without any partition having two owners. A member can keep its place across a restart with a fixed instance id, and a group can subscribe to streams by a regular expression. See ADR 0033, ADR 0072 and ADR 0073.
Which one to use
| You want | Use |
|---|---|
| Keep history, read it again, several independent readers | A stream |
| Share work among workers, retry failures, dead letters | A queue |
| Send one message to several queues by a rule | An exchange with bindings |
| Spread a stream’s partitions over a team of workers | A stream and a consumer group |
| Read, change and write elsewhere with no loss or repeat | A transaction |
About the examples
The Java examples in this guide are the files under sdk/java/src/examples; the build compiles them, so they match the SDK. They read the cluster address and credentials from Env.java. The Kotlin layer in sdk/kotlin offers the same calls as suspend functions and flows. Short snippets that are not from those files are marked as such.
Streams
A stream is a log you append to and read by offset. This chapter creates one, writes to it, reads it back, reads it as a consumer group and describes what keeps it from growing for ever.
Create, write and read
The example creates a stream with two partitions on three nodes, writes ten messages keyed by customer, and reads each partition from offset 0.
package dev.lotse.examples;
import dev.lotse.client.ClientException;
import dev.lotse.client.Fetched;
import dev.lotse.client.LotseClient;
import dev.lotse.client.OutMessage;
import dev.lotse.client.ProduceResult;
import dev.lotse.record.BatchRecord;
import java.time.Duration;
/** Creates a stream, writes ten messages to it and reads them back. */
public final class StreamExample {
private StreamExample() {}
public static void main(String[] args) {
try (LotseClient client = new LotseClient(Env.config().build())) {
try {
client.createStream("example-orders", 2, 3);
} catch (ClientException.ResourceExists e) {
// made by an earlier run
}
for (int i = 0; i < 10; i++) {
// The partition is the caller's to choose; here, by key.
String customer = "customer-" + (i % 3);
int partition = Math.floorMod(customer.hashCode(), 2);
ProduceResult r = client.produce(
"example-orders", partition, OutMessage.value("order-" + i).withKey(customer));
System.out.printf("order-%d at partition %d, offset %s%n", i, partition,
Long.toUnsignedString(r.baseOffset()));
}
for (int partition = 0; partition < 2; partition++) {
long offset = 0;
while (true) {
Fetched f = client.fetch("example-orders", partition, offset, Duration.ofMillis(200));
for (BatchRecord record : f.committedMessages()) {
System.out.printf("read %s from partition %d%n", record.value().toUtf8(), partition);
}
long next = f.nextOffset(offset);
if (next == offset) {
break; // nothing more for now
}
offset = next;
}
}
}
}
}
Points to note:
createStream(name, partitions, replication)throwsClientException.ResourceExistswhen the name is taken in your tenant. Catching it makes a start-up step safe to run twice.- You choose the partition. The client does not hash keys for you. Pick it from the key, as the example does, so that messages with one key stay in order.
producereturns once the write is committed. With an idempotent producer (the default), a retry after a lost answer does not write twice.fetch(stream, partition, offset, maxWait)waits up tomaxWaitfor data.committedMessages()returns only what a reader may see: messages of open or aborted transactions are left out (read_committed).nextOffset(offset)gives the offset to ask for next.- Offsets are unsigned 64-bit numbers; print them with
Long.toUnsignedString.
An offset below the start of the log, after retention removed it, throws ClientException.OffsetOutOfRange, which carries logStart(). To find a position without reading, listOffset(stream, partition, at) takes OffsetAt.EARLIEST, OffsetAt.LATEST or OffsetAt.time(epochMillis) (ADR 0059).
Kotlin
records(stream, partition, from) in the Kotlin layer is a Flow of records, so a stream reads like a collection:
lotse.records("example-invoices-kt", 0, from = 0).take(10).collect { println(it.value().toUtf8()) }
This line is taken from KotlinExample.kt, shown in full in Transactions and exactly-once.
Settings per stream
createStream has a fourth argument, a map of settings; alterConfigs(name, set, reset) changes them later and describeConfigs(name) reads them. Streams made at runtime accept these keys:
| Key | Meaning | Values |
|---|---|---|
retention.ms | Delete segments whose newest record is older than this | A positive number of milliseconds, or -1 for no limit. Default: the node’s, 7 days |
retention.bytes | Delete the oldest segments while the rest still holds at least this many bytes | A positive number of bytes, or -1 for no limit. Default: no limit |
cleanup.policy | What the log does with old data | delete (default), compact, or compact,delete |
delete.retention.ms | How long a tombstone stays in a compacted stream | Milliseconds from 0, or -1 to keep them. Default: 1 day |
A key is reset by naming it in the reset list. Changing settings needs an admin role on the stream.
// Not from the compiled examples: keep a week, at most 10 GiB per partition.
client.createStream("clicks", 4, 3, Map.of(
"retention.ms", Long.toString(Duration.ofDays(7).toMillis()),
"retention.bytes", Long.toString(10L * 1024 * 1024 * 1024)));
Retention
A log is trimmed in whole segments, and never past the last stable offset, so an open transaction’s records stay. The active segment is never deleted. The size limit is a floor: the stream keeps at least retention.bytes. Details: ADR 0046.
Compaction
With cleanup.policy = compact, Lotse keeps at least the newest record of each key. Older records of a key are removed from sealed segments. Records without a key stay. A record with a key and a null value is a tombstone: it deletes the key, and is itself removed once it is older than delete.retention.ms and every replica holds it. Offsets never change; a compacted batch may have gaps. compact alone deletes nothing by age or size; compact,delete does both. The Rust and Java clients read compacted batches. Details: ADR 0067.
Consumer groups
A group reads a stream, or several, as a team and remembers where it is. ExactlyOnceExample in Transactions and exactly-once joins one:
// Not from the compiled examples.
GroupConfig config = GroupConfig.defaults(); // auto-commit on: a revoked partition's position is committed first
try (GroupConsumer group = client.joinGroup("billing", List.of("orders"), config)) {
while (running) {
for (GroupRecord r : group.poll(Duration.ofMillis(500))) {
handle(r.record());
}
}
}
- Call
polloften. It also sends the heartbeat; a member that stops polling loses its partitions when its session times out. GroupConfigsets the session and rebalance timeouts, auto-commit (on by default: the position of a partition taken away is committed first),startFrom(OffsetAt)for partitions without a committed offset, and aninstanceIdfor a member that keeps its place across restarts.joinGroupMatching(group, pattern, config)subscribes by regular expression.- If a member’s position was deleted by retention, it goes on from the start of the log.
- The group id lives in your tenant and needs the consumer role.
When a write fails
| Exception | Meaning |
|---|---|
ClientException.Unknown | The answer was lost. The write may or may not be in the log. The client does not guess; an idempotent producer lets you try again without a duplicate |
ClientException.GaveUp | The client retried for retryFor and stopped |
ClientException.UnknownStream | No such stream or partition in your tenant, or you may not see it |
ClientException.Broker | The node refused; code() names the reason |
Queues
A queue delivers each message to one consumer at a time and keeps it until a consumer acknowledges it. Use it for work that must be done once by one of several workers, and retried when a worker fails.
A work queue with a dead-letter queue
The example creates two queues, writes five jobs, works them off, and lets one job fail until the queue retires it.
package dev.lotse.examples;
import dev.lotse.client.ClientException;
import dev.lotse.client.DeadLetter;
import dev.lotse.client.Delivery;
import dev.lotse.client.LotseClient;
import dev.lotse.client.OutMessage;
import dev.lotse.client.QueueConsumer;
import dev.lotse.client.QueueOptions;
import dev.lotse.client.SubscribeOptions;
import java.time.Duration;
/**
* A work queue with a dead-letter queue: jobs that fail are tried again after a delay, and a job that fails three
* times moves to the dead-letter queue.
*/
public final class QueueExample {
private QueueExample() {}
public static void main(String[] args) {
try (LotseClient client = new LotseClient(Env.config().build())) {
createIfMissing(client, "example-jobs-dead", QueueOptions.defaults());
createIfMissing(client, "example-jobs", QueueOptions.builder()
.maxDeliveries(3)
.deadLetterQueue("example-jobs-dead")
.ackTimeout(Duration.ofSeconds(30))
.build());
for (int i = 0; i < 5; i++) {
client.produce("example-jobs", 0, OutMessage.value("job-" + i));
}
try (QueueConsumer jobs = client.subscribe("example-jobs", SubscribeOptions.withCredit(16))) {
while (true) {
Delivery job = jobs.poll(Duration.ofSeconds(5));
if (job == null) {
break; // nothing for 5 s: done
}
String name = job.value().toUtf8();
if (name.equals("job-3")) {
// Fails every time: back in one second, until max.deliveries retires it.
System.out.printf("%s failed (delivery %d)%n", name, job.deliveryCount());
jobs.nackAfter(job, Duration.ofSeconds(1));
continue;
}
System.out.println("done " + name);
jobs.ack(job);
}
}
// The retired job, with where it came from and why.
try (QueueConsumer dead = client.subscribe("example-jobs-dead")) {
Delivery job = dead.poll(Duration.ofSeconds(5));
if (job != null) {
System.out.printf("dead letter %s: %s after %d deliveries%n", job.value().toUtf8(),
job.deadLetter().map(DeadLetter::reason).orElse("?"),
job.deadLetter().map(DeadLetter::deliveries).orElse(0));
dead.ack(job);
}
}
}
}
private static void createIfMissing(LotseClient client, String name, QueueOptions options) {
try {
client.createQueue(name, options);
} catch (ClientException.ResourceExists e) {
// made by an earlier run
}
}
}
Points to note:
- You write to a queue with
produce(queue, 0, message): a queue is partition 0. Messages can also arrive through an exchange. subscribe(queue, SubscribeOptions.withCredit(16))opens a subscription. Credit is how many messages the broker may have out with you at once. Each ack gives credit back; the default is 64.poll(wait)returns the next delivery, ornullafterwait.next()waits without limit.ack(delivery)settles a message.nack(delivery, true)gives it back to be delivered again;nack(delivery, false)discards it.nackAfter(delivery, delay)gives it back after a delay, for a retry with backoff.delivery.deliveryCount()says how many times the broker has sent the message.- A message you neither ack nor nack goes out again when the ack timeout passes (
ack.timeout.ms) or when you close the consumer. After a leader change or a broken connection the consumer subscribes again and keeps its messages, also within a transaction; if it does not come back within the resume window (resume.window.ms, 30 s by default), they go to others (ADR 0109). Process messages so that doing one twice is harmless, or use a transaction.
Settings
Set at creation with QueueOptions, or later with alterConfigs(name, set, reset). All limits are off by default. describeConfigs(name) reads them.
| Key | Meaning | Values | QueueOptions |
|---|---|---|---|
ack.timeout.ms | Time a delivered message may stay unacknowledged | Milliseconds above 0 | ackTimeout |
resume.window.ms | After a leader change, how long a consumer’s messages wait for it to subscribe again | Milliseconds above 0; default 30,000 | alterConfigs |
max.deliveries | A message goes out at most this often; the next time it is retired | 1 to 65,535 | maxDeliveries |
message.ttl.ms | A message not delivered within this time after the broker stored it is retired | Milliseconds above 0 | messageTtl |
dead.letter.queue | Where retired messages go | Name of a queue in the same tenant | deadLetterQueue |
single.active.consumer | Only one subscription gets messages | true or false | alterConfigs |
priority.levels | Number of priority levels | 1 to 10; 1 means none | alterConfigs |
max.length | Messages that may wait for delivery | Messages above 0 | maxLength |
max.length.bytes | Bytes of keys, values and headers that may wait for delivery | Bytes above 0 | maxLengthBytes |
overflow | What a full queue does | reject-publish (default) or drop-head | overflow |
resume.window.ms and the last two are set with alterConfigs in the Java SDK:
// Not from the compiled examples.
client.alterConfigs("example-jobs", Map.of("single.active.consumer", "true", "priority.levels", "3"), List.of());
Changing settings, purging and requeueing need the tenant-admin role on the queue.
Retries and dead letters
A delivery attempt is a delivery that did not end in an ack: a nack with requeue, an ack timeout, or a lost connection. With max.deliveries set, the broker retires a message when it would otherwise go out once more. With message.ttl.ms set, it retires a message that waited too long. A message a consumer holds does not expire.
Retired messages move to the dead.letter.queue if there is one. Without one, they are dropped. Set a dead-letter queue when messages matter.
The move happens once, also across failovers. The copy keeps key, value and headers, and gets these headers, which delivery.deadLetter() reads:
| Header | Content |
|---|---|
x-lotse-dead-letter-reason | max-deliveries, expired, requeued or max-length |
x-lotse-dead-letter-queue | The queue it came from |
x-lotse-dead-letter-queue-id | That queue’s id |
x-lotse-dead-letter-offset | Its offset there |
x-lotse-dead-letter-deliveries | Deliveries before it was retired |
x-lotse-dead-letter-floor | The source queue’s retention floor when it moved |
A queue cannot be its own dead-letter queue, a chain of dead-letter queues that leads back is refused, and a queue that is someone’s dead-letter queue cannot be deleted. Delivery counts are kept in the queue’s log, so they go on across failovers (ADR 0109). See ADR 0055.
Length limits
A queue may limit what waits for delivery (ADR 0118): max.length messages, max.length.bytes bytes of keys, values and headers, or both. Every message a consumer could still get counts, also one not yet committed or in an open transaction. A message out with a consumer does not count, as in RabbitMQ, so a consumer with a large credit lets the queue hold more.
What a full queue does is its overflow:
reject-publish, the default: a produce, a publish or a transaction’s write that would pass the limit fails withQUEUE_FULL, and nothing is written. The client does not retry it; try again once consumers have made room. A dead letter for a full queue waits in its source queue until there is room.drop-head: the queue takes every write and retires its oldest waiting messages until it fits, into its dead-letter queue with the reasonmax-lengthif it has one. Without one, they are dropped.
If the answer to an earlier attempt was lost, QUEUE_FULL does not say whether that attempt was written. The client then reports the outcome as unknown.
The limits can change at any time with alterConfigs; a queue already over a new limit refuses, or drops down to it. After a leader change, the new leader reads the queue’s backlog first and may refuse while it does.
// Not from the compiled examples.
client.createQueue("orders", QueueOptions.builder().maxLength(100_000).maxLengthBytes(1L << 30).build());
client.alterConfigs("orders", Map.of("overflow", "drop-head"), List.of());
Purge and requeue
purge(queue) settles every message that is ready: not held by a consumer, not in an open transaction. Messages written while it runs are left alone. It returns the number purged.
requeue(deadLetterQueue) moves the ready messages of a dead-letter queue back to the queue named in each message’s dead-letter headers, once each. A message without those headers, or whose queue is gone, stays. It returns the number moved. The copy gets the reason requeued. See ADR 0064. The console has both as buttons.
Delayed delivery
Two ways to make a message wait (ADR 0077):
- On publish, set the header
x-lotse-delay-msto a decimal number of milliseconds, 0 to 4,294,967,295 (about 49 days). The delay counts from the time the queue’s leader stored the message, so the producer’s clock does not matter. Any other value means no delay. - On nack,
nackAfter(delivery, delay). It gives the message back to any consumer once the delay has passed.
// Not from the compiled examples: a reminder in ten minutes.
client.produce("reminders", 0,
OutMessage.value("call back").withHeader("x-lotse-delay-ms", "600000"));
A delayed message never blocks the queue: others keep flowing, and the delayed one is delivered first once it is due. A dead-lettered or requeued copy does not wait again.
Priorities
With priority.levels from 2 to 10, set the header x-lotse-priority to a decimal number from 0. Higher goes first. A value above the queue’s highest level counts as the highest; anything that is not a number from 0 to 255 is level 0. Within a level, the oldest message goes first. Dead letters and requeued messages are level 0 (ADR 0078). The console’s queue page and DescribeMetrics show how many messages are ready at each level; after a failover, messages whose level the new leader has not read yet are counted apart until it has. They also show which subscription of a single active consumer queue is active and which ones are standbys (ADR 0120).
// Not from the compiled examples.
client.produce("jobs", 0, OutMessage.value("urgent").withHeader("x-lotse-priority", "2"));
Single active consumer
With single.active.consumer = true, one subscription gets all messages and the others wait as standbys in the order they subscribed. consumer.isStandby() says whether yours still waits. When the active one ends, the oldest standby takes over only once nothing is out with anyone, so the queue’s order holds across the handover.
This holds per leader term. After a leader change, a deposed leader that has not noticed may still deliver to the old active consumer for a short time; that consumer’s acks then fail (ADR 0076).
Kotlin
lotse.subscribe(queue) returns a LotseQueue with ack, nack, nackAfter as suspend functions, and deliveries() as a Flow:
// Not from the compiled examples.
lotse.subscribe("example-jobs").use { jobs ->
jobs.deliveries().collect { job ->
process(job.value().toUtf8())
jobs.ack(job)
}
}
Errors you may meet
| Error | When |
|---|---|
ClientException.ResourceExists | The queue name is taken in your tenant |
ClientException.Broker with INVALID_REQUEST | A setting is out of range, or would make a dead-letter loop |
ClientException.Broker with MESSAGE_NOT_HELD | You acked a message inside a transaction that is not in flight with your subscription, or another transaction holds it. Abort |
ClientException.Broker with UNAUTHORIZED | Your roles do not allow it |
Exchanges and routing
An exchange takes a message from a publisher and passes it to the queues bound to it. The publisher need not know the queues. You create exchanges and bindings at runtime.
Declare, bind, publish
// Not from the compiled examples.
client.createQueue("invoices-eu");
client.createQueue("invoices-all");
client.declareExchange("invoices", Messages.ExchangeType.TOPIC);
client.bind("invoices", "invoices-eu", "invoice.eu.*");
client.bind("invoices", "invoices-all", "invoice.#");
Published p = client.publish("invoices", "invoice.eu.created", OutMessage.value("#4711"), true);
declareExchange returns the exchange’s id and throws ClientException.ResourceExists if the name is taken. bind and unbind take the exchange, the queue and a binding key. bindHeaders binds by headers.
publish(exchange, routingKey, message, mandatory) goes to any node. The node routes the message by the bindings it has applied, checks that you may produce to the exchange and to every routed queue, and writes the message to each queue. It returns once every routed queue holds the message. Published.routed() lists the queues and the offset in each.
The four kinds
| Kind | A message goes to a queue when |
|---|---|
| direct | The binding key equals the routing key |
| topic | The routing key matches the binding key. Keys are words separated by .; * matches exactly one word, # zero or more. An empty key has no words |
| fanout | Always: every bound queue gets it, and the routing key is ignored |
| headers | The message headers match the binding’s headers: all of them if the binding asks for all, any one if it asks for any. A header with a null value matches only a null value. A binding without headers matches everything when it asks for all, and nothing when it asks for any |
A queue bound twice to one exchange gets a message once.
Headers binding:
// Not from the compiled examples.
client.declareExchange("events", Messages.ExchangeType.HEADERS);
client.bindHeaders("events", "audit-eu",
List.of(new Header(ByteString.utf8("region"), ByteString.utf8("eu"))), true);
client.publish("events", "", OutMessage.value("login").withHeader("region", "eu"), false);
The routing rules are those of the lotse-route crate; the Java SDK is checked against the same test vectors.
What can go wrong
- Not routed. With
mandatory = true, a message that matches no queue throwsClientException.Unroutable(errorUNROUTABLE). Withmandatory = falseit is dropped without an error. - Partly written. If some queues could not be written,
ClientException.NotAllWrittenlists the queues that hold the message and the failures. The message is in those queues; do not assume it is in none. - Lost answer.
ClientException.Unknown. The publish is not sent again, because the node may have written it and a second try could write it twice. Decide whether a duplicate matters. - Not allowed. If you lack the producer role on any routed queue, nothing is written and no queue is named (
UNAUTHORIZED). - Unknown exchange. The name is looked up once more, in case it was deleted and made again with a new id.
- Exchange declared a moment ago. A node that has not heard of it yet refuses the publish before writing anything, and the client sends it again to another node or after a pause, for up to
retry_for.
Inside a transaction, the client routes and writes to each queue itself, so the message reaches all its queues or none.
Transactions and exactly-once
Delivery over a network is at least once: if an answer is lost, the sender asks again. Lotse turns that into exactly once in two steps. Idempotent producers remove duplicates on the way in. Transactions make a read, a change and a write happen all together or not at all.
Idempotent producers
Every producer has an id from the cluster and numbers its batches. A leader that sees a number twice answers with the earlier result and writes nothing. The Java and Rust clients do this by default, so a retried produce does not duplicate. A producer with a name (producerName) keeps its identity across restarts, and a second instance with the same name fences the first (ADR 0028, ADR 0071).
When the outcome of a write is unknown and the client cannot find out, it throws ClientException.Unknown. That is the one case where you decide.
Consume, transform, produce
The example reads orders as a consumer group member and writes one invoice per order. Each poll’s invoices and the group’s offsets commit in one transaction. If the process stops in between, neither the invoices nor the new offsets exist, and the next instance reads the same orders again.
package dev.lotse.examples;
import dev.lotse.client.ClientException;
import dev.lotse.client.GroupConfig;
import dev.lotse.client.GroupConsumer;
import dev.lotse.client.GroupRecord;
import dev.lotse.client.LotseClient;
import dev.lotse.client.OutMessage;
import dev.lotse.client.Transaction;
import java.time.Duration;
import java.util.List;
/**
* Consume, transform, produce, exactly once: a group member reads orders, and each batch of invoices it writes
* commits in one transaction with the group's offsets. A crash between the two leaves neither.
*/
public final class ExactlyOnceExample {
private ExactlyOnceExample() {}
public static void main(String[] args) {
// One running client per transactional id: a second one fences the first.
try (LotseClient client = new LotseClient(Env.config().transactionalId("example-billing-1").build())) {
try {
client.createStream("example-invoices", 1, 3);
} catch (ClientException.ResourceExists e) {
// made by an earlier run
}
GroupConfig config = GroupConfig.builder().autoCommit(false).build(); // offsets go with the transaction
try (GroupConsumer group = client.joinGroup("example-billing", List.of("example-orders"), config)) {
long deadline = System.nanoTime() + Duration.ofSeconds(20).toNanos();
while (System.nanoTime() < deadline) {
List<GroupRecord> orders = group.poll(Duration.ofMillis(500)); // heartbeats too: call it often
if (orders.isEmpty()) {
continue;
}
try (Transaction txn = client.beginTransaction()) {
for (GroupRecord order : orders) {
txn.produce("example-invoices", 0,
OutMessage.value("invoice for " + order.record().value().toUtf8()));
}
txn.sendOffsets(group);
txn.commit();
System.out.printf("invoiced %d orders%n", orders.size());
} catch (ClientException e) {
// Not committed: read the same orders again from the group's committed offsets.
System.out.println("transaction failed, reading again: " + e.getMessage());
group.rewind();
}
}
}
}
}
}
Points to note:
- One client per transactional id.
transactionalId("example-billing-1")names the client’s identity. Starting a second client with the same id fences the first. - Turn auto-commit off for the group (
autoCommit(false)). The offsets belong to the transaction:txn.sendOffsets(group). - Call
group.polloften. It sends the group heartbeat. commit()ends the transaction.close()withoutcommit()orabort()leaves the decision to the coordinator, which aborts it, and the client’s next transaction starts with a fresh producer id; callabort()when you want it gone at once. OnClientExceptionthe example callsgroup.rewind()so the next poll reads from the committed offsets again.ClientException.MustAbortsays the transaction can no longer commit; abort it.
The same in Kotlin
transaction { } commits when its block returns and aborts on an exception or cancellation (ADR 0080).
package dev.lotse.examples
import dev.lotse.client.ClientException
import dev.lotse.client.GroupRecord
import dev.lotse.kotlin.groupConfig
import dev.lotse.kotlin.lotse
import dev.lotse.kotlin.message
import java.time.Duration
import kotlinx.coroutines.flow.take
import kotlinx.coroutines.runBlocking
import kotlinx.coroutines.withTimeoutOrNull
/**
* Writes orders, then invoices them exactly once from a consumer group, each poll's orders in one transaction with
* the group's offsets. Brokers from `LOTSE_BOOTSTRAP` (default `127.0.0.1:9092`), plain TCP on loopback.
*/
fun main(): Unit = runBlocking {
val bootstrap = (System.getenv("LOTSE_BOOTSTRAP") ?: "127.0.0.1:9092").split(",").toTypedArray()
lotse(*bootstrap) {
plaintextOnLoopback()
transactionalId("example-billing-kt")
}.use { lotse ->
for (stream in listOf("example-orders-kt", "example-invoices-kt")) {
try {
lotse.createStream(stream, partitions = 1, replication = 3)
} catch (e: ClientException.ResourceExists) {
// made by an earlier run
}
}
repeat(10) { lotse.produce("example-orders-kt", 0, message("order-$it", key = "customer-${it % 3}")) }
lotse.joinGroup("example-billing-kt", listOf("example-orders-kt"), groupConfig { autoCommit(false) }).use { group ->
withTimeoutOrNull(Duration.ofSeconds(20).toMillis()) {
var invoiced = 0
while (invoiced < 10) {
val orders: List<GroupRecord> = group.poll(Duration.ofMillis(500))
if (orders.isEmpty()) continue
try {
lotse.transaction {
for (order in orders) {
produce("example-invoices-kt", 0, message("invoice for ${order.record().value().toUtf8()}"))
}
sendOffsets(group)
}
invoiced += orders.size
println("invoiced $invoiced orders")
} catch (e: ClientException) {
group.rewind() // not committed: read them again
}
}
}
}
lotse.records("example-invoices-kt", 0, from = 0).take(10).collect { println(it.value().toUtf8()) }
}
}
What a transaction can hold
Inside one transaction you can:
produceto any number of streams and queues;publishthrough an exchange (the client routes, and the message reaches all its queues or none);sendOffsets(group)to commit a consumer group’s offsets;queue.ackIn(txn, delivery)to acknowledge queue messages.
A transaction that is neither committed nor aborted is aborted by the coordinator after txnTimeout.
Consume from a queue, produce, exactly once
ackIn ties a queue acknowledgement to the transaction. The commit settles the message; an abort gives it back to the queue.
// Not from the compiled examples.
Delivery job = jobs.poll(Duration.ofSeconds(5));
try (Transaction txn = client.beginTransaction()) {
txn.produce("results", 0, OutMessage.value(compute(job.value().toUtf8())));
jobs.ackIn(txn, job);
txn.commit();
}
Rules to know (ADR 0075):
- Ack on the connection the message came on; the client does this for you. After a leader change it subscribes again as the same consumer and sends the ack there, so the transaction goes on. A message that is not in flight with your subscription gets
MESSAGE_NOT_HELD; abort. - Of two transactions that ack the same message, the first in the log holds it; the other gets
MESSAGE_NOT_HELD. - A held message is not ready: purge leaves it, and expiry and the delivery limit do not touch it.
What readers see
A fetch is read_committed by default: records of open and aborted transactions are not returned. A reader never sees a half-written transaction. Messages in a queue that a transaction wrote become deliverable when it commits.
Limits of the guarantee
- Exactly once holds for effects inside Lotse: streams, queues, offsets, acks. A call to a database or an e-mail service inside your code is outside it; make that call idempotent.
- A plain
publishoutside a transaction that ends inUnknownis not retried by the client (see Exchanges and routing). - A stream consumer that commits offsets outside a transaction gives at-least-once.
See also ADR 0029 and ADR 0057.
The management console
The console is a web app for looking at a cluster and, if your roles allow, changing it. It shows streams, queues, nodes and tenants, with live numbers and charts. It runs on a node of its own and talks to the cluster as you.
Ask your administrator for its address. The setup of the console itself is in the administration guide.
Sign in
Users sign in with OpenID Connect at the identity provider the cluster trusts. On a development machine, the console may instead be in dev mode with one configured user and an optional password. The console passes your credential on to the cluster, so every screen shows only what your roles allow (ADR 0040). A queue you may not see is not listed. Sessions survive a console restart (ADR 0042).
Overview
The overview shows the cluster at a glance: nodes, groups, throughput and latency. Charts add up only the streams and queues you may see, and history gets coarser with age (ADR 0045). What a real cluster does not report yet, such as zones, disk limits or quotas, shows as a dash.
Nodes
The nodes page lists each node with its state, replicas and leaders, and shows a rolling upgrade while one runs.
Streams
The list shows each stream with its partitions, replication and rates. The detail page shows its partitions, where their leaders are, retention, and a message browser.
A cluster admin can pin a partition’s leader to one of its voters from the detail page (“Pin to”). The spreading of leaders then leaves that partition alone (ADR 0065).
Queues
The list shows the messages per queue and how many were dead-lettered. The detail page shows tags for the queue’s priority levels, single active consumer and dead-letter setting, the dead-lettered count, the dead letters held in the dead-letter queue, and charts.
Purge
Purge removes the ready messages of a queue. Messages a consumer holds, and messages in open transactions, stay. The console asks you to type the queue’s name again before it acts. You need the tenant-admin role on the queue.
Requeue
On a queue that is another queue’s dead-letter queue, Requeue all moves its messages back to the queues they came from. A message moves once. Messages whose origin is unknown or gone stay where they are. Same role as for purge. Both actions are done as you (ADR 0066, ADR 0064).
Messages
The message browser shows messages of a stream or queue without taking them from it.
Tenants and access
The tenant page lists tenants and the role bindings that apply to them: who holds which of the six roles, on which name prefix.
Users and access
The Users and access page manages who may sign in and what they may do, as you: every change is a request the cluster checks against your roles, and the console adds no rights of its own (ADR 0089). Only cluster admins can use it.
Users lists the SCRAM users and where each comes from. Add user and Set password turn the password into a salted key in your browser and send only the key; the console never sees the password. Delete asks you to type the name again.
Role bindings shows every binding over all tenants with its source, and adds or removes bindings for users and identity provider groups. Bindings from the node configuration are read only.
Sessions ends the sessions of a user or an identity provider group. Type the name again to confirm.
Client certificates
Certificates lists the client certificates the nodes accepted, with the principal, when they expire and their state: valid, expiring within 30 days, expired or revoked. Revoke refuses a certificate on every node at once and ends the connections that showed it; type the serial again to confirm (ADR 0090, ADR 0091).
Request certificate asks the CA holder for a new client certificate. Your browser makes the key and the signing request; the private key is saved as a file once and never reaches the console. The CA holder signs the request with lotse certs --sign, and the signed certificate can then be downloaded from the list (ADR 0092).
You need not be a cluster admin to ask for a certificate in your own name. The tab then lists only your requests, and the dialog fills in your name. Asking again replaces your request that is still waiting, and the key file of the old one will not fit a certificate signed from then on. If no role is bound to your name itself, the page says so: a certificate signs in without your identity provider’s groups, so ask a cluster admin to bind a role to your name. Cluster admins see every request, and the tab shows how many wait for the CA holder (ADR 0119).
When something is missing
If a stream or queue you expect is absent, check the tenant you are looking at and your role bindings first. What a principal may not see is unknown to it, so a missing permission looks like a missing resource.