The native protocol
This page is sdk/PROTOCOL.md from the repository, included as it stands, so it cannot drift from the file. It is written for people who build a client in another language.
The Rust code is the authority. docs/design.md (German) gives the design, and the golden vectors in sdk/conformance/ pin the layouts below to the broker’s bytes.
All of it is version 0 of every API key.
Integers and fields
Fixed-width integers are little-endian. The one exception is the frame length prefix.
| Type | Wire | Notes |
|---|---|---|
u8, u16, u32, u64, i32, i64 | 1, 2, 4, 8, 4, 8 bytes | booleans are a u8: write 0 or 1, read any non-zero as true |
varint | ZigZag, then LEB128, at most 10 bytes | array counts, tag numbers, record fields |
string | u16 length, UTF-8 | length 0xFFFF is null; at most 65,534 bytes |
bytes | u32 length, data | no null form |
array<T> | varint count, elements | the count is ZigZag: 2 elements is the byte 04 |
| optional | u8 0 or 1, then the value if 1 | a different form from a null string |
| tagged fields | varint count, then {tag varint, bytes} | see below |
Unsigned types in Java. u64 travels in a long as its bit pattern; compare, divide and print it with the Long.*Unsigned methods. u32 fits a long, u16 an int. NO_OFFSET (a group’s “no committed offset”) is u64::MAX, which is -1 as a long. Offsets are u64 in the protocol but the first offset of a log is 0: base_offset is the Raft index of the first record minus one (the broker stamps index - 1), although the doc comment in lotse-record says “index of the first record”.
Counts are bounded before allocating. A reader must refuse an array whose count times the smallest encoding of one element exceeds the bytes left in the frame. The minimum element sizes are part of the broker’s decoder (the array(min_size, ..) arguments in messages.rs); a decoder that guesses a smaller minimum is merely more permissive, one that guesses a larger minimum refuses valid frames of nothing but minimal elements.
Varints. The reader accepts overlong encodings such as 80 00 for 0 and does not tell a truncated varint from an overlong one in a frame (both are “truncated”); inside a record they are different errors. Writers never produce overlong forms. The tenth byte of a 10-byte varint may only be 00 or 01.
Frames
On a connection every frame is preceded by its length, a big-endian u32, at most 16 MiB (MAX_FRAME). A larger length ends the connection.
Request: api_key u16 | api_version u16 | correlation_id u32 | flags u16 | tagged_fields | body
Response: correlation_id u32 | flags u16 | tagged_fields | body
Push: correlation_id u32 (0) | flags u16 (PUSH) | tagged_fields | api_key u16 | api_version u16 | body
- Flags:
PUSH= 1,ERROR= 2. - Correlation ids of requests start at 1 and are never 0; 0 marks pushes. Requests are pipelined and responses may arrive in another order: match them by id.
- A response has no API key. The decoder must be told which request it answers.
- Check
ERRORbefore decoding a typed body. A response with that flag has the bodyerror_code u16 | tagged_fields, whatever the request was (unsupported version, connection not authenticated). If tagged field 0 is present, its bytes are the reason, in UTF-8. Decoding such a frame as aProduceresponse misreads it. - Push detection: the first four bytes are zero and bit 0 of the flags is set. A response with correlation id 0 cannot exist.
- A push’s body begins with the subscription id;
docs/design.mdused to say the id sat in the header. Creditis never answered. No other request is unanswered.- After the header’s tagged fields,
ApiVersionsstill has a body: one more, empty block of tagged fields (00).
Tagged fields: not at the same place everywhere
Most structures end with a block of tagged fields, written empty (00) and skipped by readers (unknown tags come from newer versions and are dropped, not preserved). That is not uniform, and a generic “every struct ends in tags” decoder is wrong. The leader hint is tag 0 of the block it sits in: leader u32 | term u64, present only after NOT_LEADER.
The tenant of a transactional or group id is tag 1 of a request’s trailing block: one string (ADR 0062). It is written only for a tenant other than default; a reader that finds none takes default. The requests that carry it are Produce (it applies to its transactional id), InitProducer, AddPartitionsToTxn, EndTxn, GroupHeartbeat, CommitOffsets, OffsetFetch and AddOffsetsToTxn. A client sends the tenant of its configuration. DescribeMetrics gives each group’s tenant as a plain string after its id.
| Where the block is | Messages |
|---|---|
| Hint is the trailing block | InitProducer, Subscribe, QueueAction (answer to PurgeQueue, Requeue), Outcome (answer to Ack, Nack, Unsubscribe, Bind, Unbind), Create (CreateStream, CreateQueue, DeclareExchange), Admin (DeleteResource, AlterConfigs, AlterUser, CreateTenant, DeleteTenant, AlterRoleBindings, AddNode, RemoveNode, MoveLeader, ReassignPartition, RevokeSessions, RevokeCertificate, RecordCertificates, RequestCertificate, RecordCertificate), RenewCertificate |
| Hint in the middle, no trailing block | DescribeConfigs, DescribeUsers, DescribeTenants, DescribeRoleBindings, DescribeNodes (after the error code, before the array) |
| Hint is the element’s block, one per partition | Produce and Fetch partitions (the resource around them still ends in an empty block) |
| Hint is the last field of a push | Deliver |
| No block at all | SaslHandshake and SaslAuthenticate responses, WhoAmI response |
| Empty block after the body fields | everything else |
Elements of arrays carry their own empty block in most requests (TxnResource, produce and fetch partitions, offset commits) and none in a few (ApiVersions entries, queue ranges, config pairs, role bindings, metrics sub-structures). Do not infer: the layouts are in Request.java and Response.java of the Java SDK, and each is covered by a vector.
Responses worth a second look
Metadata: a resource carries its ownerror. An unknown name is a resource withUNKNOWN_RESOURCE, not a frame error. Leader0means unknown. A partition’s tag block may hold tag 0,pinned_leader(u32, the node a cluster admin pinned its leader to), and tag 1,placed(empty: a cluster admin placed it on named nodes); each is present only when set (ADR 0121).OffsetFetch: an offset ofu64::MAXmeans none.DescribeMetrics:connectionsis sent as au64but must fit au32. Between nodes the answer may carry tag 1,removed_through(oneu64, ADR 0100), and tag 2,responsive(an array of: groupu64, termu64, an array of node idsu32; ADR 0117), which a principal that sees the whole cluster gets too; a client skips both like any unknown tag. Tag 3,queue_details(ADR 0120), goes to every principal: an array of entries, each a queue idu64, its ready messages per priority level as an array ofu64(level 0 first, empty without levels), the ready messages whose level the leader has not read yetu64, and an array of subscription idu32and stateu8(0 shared, 1 the single active consumer, 2 a standby; read an unknown code as shared). It lists only queues with levels or a subscription that is not shared, and a client reads it into the queue entries of the same id; a queue or subscription it does not list has no levels and is shared. ADR 0122 adds three tags, each read into the entries of the same id, and each left out when it has nothing to say: tag 4, an array of groupu64, the oldest data record’s timestampi64(ms since the epoch, -1 for none) and the open transactionsu32, read into the partitions; tag 5, an array of a queue idu64and an array of subscription idu32and the bytes it holdsu64, read into the subscriptions; tag 6, the node’s free disk limit in bytesu64(0 for none) and whether it is below itu8. A partition or subscription a tag does not list has no oldest record, no open transactions and holds 0 bytes; without tag 6 the node did not say.DescribeNodes: between nodes the request may carry tag 0,applied_at_least(oneu64, ADR 0105), and the answer is thenNOT_LEADERfrom a leader that has applied less. Clients do not send it.ProduceandFetch: every partition has its own error; a request can partly succeed.
Error codes
Codes 0 to 34, with names and retry classes, are in ErrorCode.java and in conformance/error-codes.txt. Retriable: NOT_LEADER, FENCED_LEADER_TERM, NOT_ENOUGH_REPLICAS, REQUEST_TIMED_OUT, COORDINATOR_NOT_AVAILABLE, CONCURRENT_TRANSACTIONS, UNSTABLE_OFFSET_COMMIT, CONCURRENT_GROUP_CHANGE, PRODUCER_LIMIT, STORAGE_FULL. STORAGE_FULL (11, ADR 0122) refuses a new batch while a voter of the partition is below its free disk limit: nothing was written, so send the same batch again, with the same sequence, after a backoff, and when the time for retries runs out, fail with STORAGE_FULL itself rather than a timeout, the sequence still free; after an attempt whose answer was lost the outcome stays unknown. PRODUCER_LIMIT (ADR 0110) refuses a produce from a producer the partition does not know while it holds as many as it admits: nothing was written, so send the same batch again, with the same sequence, after a backoff. QUEUE_FULL (34, ADR 0118) refuses a write to a queue at its length limit with overflow = reject-publish: nothing was written, it is not retried, and the caller decides; after an attempt whose answer was lost it says nothing about that attempt, so report the outcome as unknown and start a new producer, as after UNAUTHORIZED then. REQUEST_TIMED_OUT means the outcome is unknown: retry only what is idempotent. A code a client does not know is kept as it came and named UNKNOWN.
The record batch
A batch is a 65-byte header and a block of records. All fixed-width header fields are little-endian.
offset size field
0 8 base_offset stamped by the leader: Raft index of the first record, minus one
8 4 batch_len bytes after this field, so the batch is batch_len + 12 bytes
12 8 term stamped by the leader
20 1 magic 1
21 4 crc32c CRC32C (Castagnoli) over bytes 25 up to the end of this batch
25 2 attributes compression(3) | transactional(1) | control(1) | kind(3) | compacted(1), bit 0 first
27 4 last_offset_delta record_count - 1
31 8 base_ts milliseconds since the epoch
39 8 max_ts
47 8 producer_id 0 = not idempotent
55 2 producer_epoch
57 4 base_sequence i32, -1 when there is none
61 4 record_count
65 . records
base_offset and term sit before the checksummed range: the leader stamps them without invalidating the CRC. A client sends both as zero.
Kinds: 0 data, 1 ack, 2 queue state, 3 timer, 4 dead letter, 5 retire, 6 checkout (ADR 0109: which consumer a leader handed which messages to; clients never read or write one). Compression: 0 none, 1 zstd, 2 lz4. Bits 8 to 15 must be zero.
Compression (ADR 0058) covers the record block only; the header stays as it is, and the CRC is over the compressed bytes. With zstd the block is one zstd frame; with lz4 it is one LZ4 frame (the frame format with its magic 04 22 4d 18, not the bare block format). Once decompressed the block reads exactly like an uncompressed one, and it is at most 16 MiB: a reader stops decompressing past that and treats the batch as damaged. The batch itself, compressed, stays within 1 MiB. A node checks a compressed batch’s records when it is produced and refuses one that does not decode (CORRUPT_BATCH); it stores, replicates and serves the batch compressed, so a consumer must read every codec its producers use. Control batches are never compressed.
A record, every varint ZigZag:
length varint | attributes i8 (0) | ts_delta varint | offset_delta varint
| key_len varint (-1 = null) | key | value_len varint (-1 = null) | value
| header_count varint | [key_len varint, key, value_len varint, value]*
ts_delta is relative to base_ts and wraps like a signed 64-bit subtraction; offset_delta is the record’s index and must run 0, 1, 2, and so on, except in a compacted batch, where records that compaction removed leave gaps (vectors in compacted.hex). A header key is never null.
Checks a parser makes, in this order: the buffer holds the 12-byte prefix; the length is at most 1 MiB (the length is a u32, so test it without overflow); it is at least 65; the whole batch is present; magic is 1; the CRC matches; the attributes are valid; record_count is not 0; last_offset_delta == record_count - 1; max_ts >= base_ts; a control batch holds exactly one record. A compacted batch (bit 8, ADR 0067) replaces the two record-count checks: it holds at most last_offset_delta + 1 records, possibly none, and is never a control batch. Decoding the records adds: the block decompresses, to at most 16 MiB; consecutive offset deltas (in a compacted batch: strictly rising and at most last_offset_delta; an emptied one has no block at all, whatever its codec), no bytes left inside a record, no bytes left after the last record, lengths and counts not negative, and every length within the record.
Editing in place. A client sets producer fields (set_producer), the transactional bit (set_transactional) and, for the leader’s TTL stamping, the base timestamp, on the encoded bytes and recomputes the CRC over bytes 25 to the end of that batch. BatchBuilder takes the first record’s timestamp as base_ts and the largest as max_ts; it does not read a clock.
Several batches in a row (the records of a Fetch partition, a log segment): read batch_len, add 12, cut, repeat. A fetch response holds whole batches.
Control batches. One record. Key: version u16 (0) | type u16. Types 0 abort and 1 commit are transaction markers whose value is the coordinator epoch as a u32. Types 2 and 3 are Raft’s own entries, 4 to 8 the coordinator and metadata groups’ records. Consumers skip all of them. A read_committed consumer uses the fetch response’s aborted-transaction list and the markers to hide aborted data.
Connections
One TCP connection per broker address, kept open. Every connection is TLS 1.3 with the cluster’s CA (ADR 0037), or plain TCP on loopback when a node was set up for development; a client never falls back from one to the other. The client checks the node’s certificate against the host it dialled, including IP entries: lotse certs writes the node’s addresses from the cluster file as IP entries, so dial the same address the file lists. A client certificate, if the node asks for one, names the principal by its common name. There is no hello, no ALPN and no version byte: the first bytes after the handshake are the first frame.
- Set
TCP_NODELAY. Frames are written whole; writes from several threads must not interleave. - One reader per connection decodes frames and completes requests by correlation id. A response nobody waits for (its request timed out) is dropped without ending the connection.
- A frame whose length exceeds 16 MiB, one that does not decode, or end of stream ends the connection: fail every pending request and every subscription. A refused request (
ERRORflag) does not end it. - A request that timed out has an unknown outcome. Retry only what is idempotent.
ApiVersionsis answered before authentication. The Rust client never sends it; the broker does not require it.- A request on a connection that is not authenticated, or whose session ended, gets an
ERRORresponse withUNAUTHORIZEDand no reason. The connection stays open. - Pushes of a subscription can arrive before the response to
Subscribehas been handled, so the client keeps pushes for subscriptions nobody has registered yet. The Rust client keeps up to 64 such subscriptions and drops the lowest id for another.
Authentication
Two SASL mechanisms, chosen by name in SaslHandshake. A node offers the ones its configuration enables; asking for another gives UNSUPPORTED_SASL_MECHANISM (28) with the node’s list in the same response.
SaslHandshake(mechanism) -> error u16 | mechanisms array<string> (no tagged fields)
SaslAuthenticate(bytes) -> error u16 | error_msg optional string | auth_bytes bytes | session_lifetime_ms u64
An unsuccessful SaslAuthenticate carries AUTHENTICATION_FAILED (27) and a message, and the node closes the connection after sending it. It does not say whether the user or the password was wrong.
OAUTHBEARER is one round: n,,\x01auth=Bearer <token>\x01\x01. Ask for the token on every authentication, including renewals.
SCRAM-SHA-512 (RFC 5802, RFC 7677) is two rounds, with these particulars that differ from what a library might assume:
| Step | Detail |
|---|---|
| client first | n,,n=<user>,r=<nonce>; in the user name = becomes =3D and , becomes =2C. The nonce is 24 random bytes in hex |
| server first | r=<nonce>,s=<base64 salt>,i=<iterations>. The nonce must strictly extend the client’s. Refuse iterations below 4096 or above 1,048,576 |
| password | the raw UTF-8 bytes, no SASLprep, as PBKDF2-HMAC-SHA512 input with a 64-byte output |
| client final | c=biws,r=<nonce>,p=<base64 proof>; biws is base64 of n,, (no channel binding) |
| AuthMessage | <client first without n,,>,<server first>,<client final without ,p=...> |
| proof | ClientKey xor HMAC(StoredKey, AuthMessage), where ClientKey = HMAC(SaltedPassword, "Client Key") and StoredKey = SHA-512(ClientKey) |
| server final | v=<base64 HMAC(ServerKey, AuthMessage)>; the client must verify it, it is the server’s proof that it knows the credential |
An empty password is valid. HMAC pads a short key with zeros, so an implementation whose library refuses an empty key can use 128 zero bytes.
conformance/scram.txt holds whole exchanges with fixed nonces, written by the broker’s own code, and the credential computed for each.
Setting a password (AlterUser, admin) sends a credential, never the password: a 16-byte random salt, 4096 iterations, StoredKey and ServerKey = HMAC(SaltedPassword, "Server Key") (each 64 bytes). A removal sends no credential.
Sessions. session_lifetime_ms of the last response is how long the session lasts, counted by the node from when it completes the login (its last answer); 0 means as long as the connection (a certificate login has no SASL session at all). The node caps it at its max_session_s and, for OAUTHBEARER, at the token’s expiry. Authenticate again on the same connection (SaslHandshake, then SaslAuthenticate) before it ends: the Rust client does so at four fifths of the lifetime, counted from before its first request (so it never believes in a session longer than the node does), and the node insists on the same principal. Requests in flight meanwhile are served. After the end the node answers UNAUTHORIZED to everything but SASL and ApiVersions; the Rust client does not try to renew then but opens a new connection and logs in again, and so does the Java one.
Client behaviour: streams
The bytes above are enough to talk to a node. These rules are what makes a client behave like lotse-client when nodes fail; they come from crates/lotse-client/src/lib.rs and the Java client follows them rule for rule.
Errors of a call
Every request ends in one of four ways, and the retry rules depend on which:
| Outcome | Meaning | Retried by |
|---|---|---|
| not sent | the connection was not there, or the node refused the whole frame without a reason | everything |
| lost | the request may have reached the node (timeout, broken connection after sending, an answer that does not decode) | idempotent produce, and requests that can be repeated |
| denied | the node requires authentication, or login failed | nothing: Authentication at once |
| refused | the node answered with ERROR and a reason (a role is missing) | nothing |
A connection that failed in any of these ways is closed and not trusted again. A failed login is remembered until a new connection logs in, so that a retry loop whose last attempt only saw a metadata lookup fail still reports the login. If an earlier attempt of a produce was lost and a later one is denied, the answer is Unknown, not Authentication: the first attempt may have been written.
The retry loop tries again until retry_for (default 30 s) has passed, pausing backoff (50 ms) between attempts. When time runs out it reports GaveUp if every attempt was certainly not carried out, and Unknown if one may have been.
Routes
The client keeps (stream, partition) -> {resource_id, leader, term}, learnt from Metadata, and uses a route until a request fails. Routes do not expire by age; metadata_max_age applies to exchange bindings only.
Metadatafor a name a node does not know returns a resource withUNKNOWN_RESOURCE. The client then asks every broker it knows (sorted by address, the bootstrap list included) and keeps, per name, the resource with the highestresource_id(compared asu64): a name deleted and created again has a higher id, and a node that lags may still report the old one. A resource with an error names nothing and is never merged.leader == 0means unknown here, not that there is none. A partition created at runtime lives on some nodes only; the client asks the partition’s voters, one after the other, forMetadataof that name and takes the first non-zero leader. If none knows, the attempt is retried.NOT_LEADERandFENCED_LEADER_TERMcarry a leader hint (leader,term) when the node knows one. The client switches the route to the hint only ifhint.term >= route.term(unsigned) andhint.leader != route.leader. Otherwise (no hint, an older term, the same leader) it drops the route, and the next attempt asks for metadata again. An older term comes from a node that lags; following it would send the client back to a leader that has been replaced.- A leader whose node id has no known address drops the route and retries.
UNKNOWN_RESOURCEin aProduceorFetchdrops the route too: the name may come back as a new resource.
Idempotent produce
- On the first produce the client asks any broker
FindCoordinator(key_type = PRODUCER, key = "")and then the coordinatorInitProducer(txn_id = null). The answer isproducer_idandepoch; both are kept for the client’s session. - Per
(resource_id, partition)the client counts the next sequence, starting at 0 (ADR 0107): a stream deleted and created again under its name has a new resource id and sequences of its own. Note the resource id the partition routes to before stamping; if a retry routes to another resource id, end the produce with an unknown-stream error rather than send the batch there, and if an earlier attempt’s outcome was unknown, drop the session as in step 6. It stamps the batch (set_producer(id, epoch, sequence), which recomputes the CRC) and sends it. - Produces to one partition go out one at a time. A second batch is not sent until the first has an answer, so the leader sees sequences in order. Different partitions run in parallel.
- Retrying a batch whose outcome is unknown (lost connection,
REQUEST_TIMED_OUT) sends the same bytes: same id, epoch and sequence. The leader recognises a batch it has and answers with its offset.PRODUCER_LIMIT(ADR 0110) is retried the same way after a backoff, though nothing was written: the partition takes the producer once it has forgotten an idle one. - On success the sequence advances by the batch’s record count (a wrapping
i32). - When the call ends with
Unknown,OUT_OF_ORDER_SEQUENCEorPRODUCER_FENCED, the whole session is dropped; the next produce gets a new producer id and starts every partition at 0. A gap in the sequence would otherwise block everything behind it. A call abandoned before its outcome is known (a caller’s timeout, a cancelled task) counts asUnknowntoo: reusing its sequence would let the leader take the next batch for the abandoned one, answer with that one’s offset and write nothing. Any other failure means the batch was not written, and its sequence is free. REQUEST_TIMED_OUTmeans the quorum did not confirm in time: the batch may or may not be written.- Without idempotence nothing is stamped and an unknown outcome fails at once, because a retry could write the batch twice. Errors that prove the batch was not written (
NOT_LEADER, a connection that was never opened) are still retried.
The timestamp of a record is the producer’s. A stream’s retention judges a batch’s age by it, so a record stamped 0 is old at once and may be deleted at the next retention pass. The Rust helper OutMessage defaults to 0; the Java client uses the time of the call.
Fetch
Fetch asks for min_bytes = 1, max_bytes 4 MiB, 1 MiB per partition, isolation = 1 (committed) and leader_term = the route’s term; the broker holds the request up to max_wait_ms when there is nothing to return. Keep max_wait well below the request timeout.
Fetching in a zone (ADR 0104). Metadata names each broker’s zone as rack (empty: none). A client configured with a zone sends a partition’s Fetch to the first voter whose rack is that zone, unless the leader is in it, and sets the request’s tag 0, allow_follower (one u8, 1). A follower then serves the committed data it has: it may lag the leader, it ignores leader_term, and past its log it answers with no records and no error. On any error answer from that replica, or a failed connection, the client reads the partition from the leader until it next learns metadata. Without the tag a follower answers NOT_LEADER as before. A follower also answers NOT_LEADER when it lags (ADR 0116): it has not heard its leader within an election timeout, or not applied what the leader said was committed within follower_max_lag (10 s by default). The records are whole batches; a response may contain only a control batch, so a reader moves on by the offset after the last batch, not by the number of messages it got.
A read_committed reader drops what it must not show:
- Sort the
aborted_txnsby first offset (asu64). - For each batch in order, every aborted entry whose first offset is at or below the batch’s base offset puts its producer into the skipping set.
- Control batches are never shown. A control batch that is an abort marker (type 0) takes its producer out of the skipping set; commit markers and the broker’s own control entries do nothing.
- A transactional data batch of a producer in the skipping set is dropped. A non-transactional batch of the same producer is kept.
The first record of a fresh partition is not at offset 0: the log begins with entries of the broker’s own. Do not assume the offset of the first message.
OFFSET_OUT_OF_RANGE is not retried. The partition response’s log_start_offset says where the log starts now; a reader that was overtaken by retention continues from there.
ListOffsets
ListOffsets (key 5, ADR 0059) goes to the partition’s leader, like Fetch, and answers per partition with an offset and a timestamp. With request tag 0 allow_follower (one u8, 1) a follower answers too, from its own log start, commit index and last stable offset, which may lag the leader’s; a client with a zone sends it where it sends Fetch, and leaves a replica that answers with an error for the leader until its next metadata (ADR 0116). timestamp = -2 asks for the first offset still stored, -1 for the offset after the last one the reader may see (the high watermark, or the last stable offset with isolation = 1), and a time in milliseconds since the epoch for the first offset of the first data batch whose newest record is at or after it; that batch’s newest timestamp comes back too. If no batch qualifies, the answer is the latest offset with timestamp -1. Other negative timestamps are INVALID_REQUEST. A time lookup goes by batches: the batch it finds may begin with records older than the time asked for. A consumer group member uses it for a partition the group has committed no offset for, when it is told to start at the end or at a time.
Client behaviour: queues
These rules come from crates/lotse-client/src/consumer.rs and the push handling in lib.rs. A queue is a resource of kind 1 with one partition, 0, so routes, leader hints and Produce work as for a stream; messages are produced with a normal Produce to partition 0.
Subscribing
- Look up the route of
(queue, 0)and open (or reuse) the connection to its leader. - Send
Subscribe(queue_id, consumer_tag = "", credit_msgs, credit_bytes = 0xFFFFFFFF, exclusive = false).exclusiveis refused withINVALID_REQUEST; do not offer it. On a queue with a single active consumer (ADR 0076) the answer may carry tag 1standby: the subscription is accepted but gets nothing until it is its turn. A client that sends tag 1notify_active = 1gets a push when that happens: errorNONE, an empty batch and no deliveries; one that does not send it never gets such a push. The Rust and Java clients send it and reportisStandbyuntil the activation push or the first message. The answer carries tag 2consumer_id(oneu64, ADR 0109), which the leader gives a new consumer; keep the latest nonzero one, and send it as tag 2resume(oneu64) whenever the consumer subscribes again, so its messages stay its own at the new subscription. Send nothing in tag 2 before a consumer has an id. Withresume, send tag 3held(ranges[from, to), built like an ack’s, ADR 0112): the messages the consumer holds, which are those handed out and not yet acknowledged, nacked or acknowledged in a transaction, those received and not handed out, and those whose requeue waits to be sent. The leader takes back only these, and hands the consumer’s other messages out again at once: their pushes never reached it, because the leader went or the connection broke after it wrote their checkout. An empty list is sent as an empty array, not left out: without the tag the leader takes back everything handed to the consumer, as an older client expects. - On
NONE, register thesubscription_idon that connection. Subscription ids are per node, andCredit,Unsubscribeand a requeueingNackare honoured only on the connection that owns the subscription (UNKNOWN_SUBSCRIPTIONotherwise). Pushes can arrive before the answer toSubscribehas been handled: the connection keeps them (see Connections) and hands them over on registration. - A retriable error (
NOT_LEADER) follows the leader hint and retries.UNKNOWN_RESOURCEdrops the route and fails the call. A lost answer drops the connection (a subscription the broker may have made ends with it), drops the route and retries.
Receiving
A push holds one batch exactly as stored and deliveries, the offsets in it that are meant for this subscription, each with its delivery count. Only records whose offset is in deliveries are the consumer’s; the rest of the batch belongs to others. Read the first batch of the push only.
The consumer hands out what it holds first and reads the next push (or the end of the subscription) only when its buffer is empty. A push with a non-zero error ends the subscription:
| Error | Then |
|---|---|
NOT_LEADER | follow the leader hint (as for a request), subscribe again |
UNKNOWN_RESOURCE | drop the route, subscribe again |
ACK_TIMED_OUT, any other | subscribe again; the broker took the unacknowledged messages back |
A connection that breaks ends the subscription in the same way. The next read subscribes again, on the next call and not before, resuming with the consumer id. After NOT_LEADER or a broken connection a consumer with an id keeps what it received, both the messages parsed into the buffer and the pushes that arrived and were not read yet: the leader writes each delivery to the log before it pushes it (ADR 0109), and a new leader keeps the messages for their consumer until it resumes, for the queue’s resume window (30 s by default). After ACK_TIMED_OUT or any other error, or without an id, drop the buffer and the unread pushes: the broker gave them to others. Messages handed out before can still arrive twice, and a delivery count that rose by one for each time says so; counts are in the log, so they go on across failovers.
Credit
The consumer keeps two counters: messages handed out and bytes received (the length of each pushed batch, added when it arrives, saturating at u32::MAX). After handing out a message, if messages used is at least ceil(credit / 2), it sends Credit(subscription_id, add_msgs = used, add_bytes = bytes) on the subscription’s connection and sets both counters to 0. Credit is never answered; a lost one shows as a broken connection on the next read.
Settling
- Ack goes to the queue’s leader, on any connection, with ranges
[from, to)built from the offsets: sorted asu64, duplicates dropped, neighbours joined.txn_idis null outside a transaction (see below for one inside). An ack names an offset, not a delivery, so it settles a message that went to someone else meanwhile, and settling twice is harmless: a lost answer is retried, and so isNOT_LEADERafter following the hint. - Nack without requeue discards the message (or dead-letters it, if the queue’s limits say so) and is sent like an ack.
- Nack with requeue is sent on the subscription’s own connection and names the subscription its messages came on in tag 0,
subscription_id(oneu32, ADR 0106): the broker takes back only what that subscription holds. Without the tag it takes back what any subscription of the connection holds, so a lateNackfrom a consumer that ended could take a message a newer subscription on the connection received again since; always send it.delay_msis the ms the message waits before it is delivered again (ADR 0077; refused withoutrequeue); the Java client’snacksends 0 andnackAfterthe delay asked for. A producer delays a message with the headerx-lotse-delay-ms(decimal ms up to 4294967295), counted from the leader’s append time; messages of a queue carry that time as their batch’s base timestamp. A consumer with an id sends it on its current subscription, also for a message an earlier subscription received: the current one took it over. If there is no current subscription, or theNackfails with a retriable error or a broken connection, keep the offset and delay and send them once the consumer has subscribed again. Without an id the message has gone back already when the subscription ended, and the call does nothing; a message of an earlier subscription is then not asked for on the new connection, which does not hold it.
Closing
Send Unsubscribe(subscription_id) and wait for its answer (up to the request timeout). Then, for the messages received but not handed out (the ones parsed into the buffer and those in pushes that arrived until the answer, which the node holds in flight just the same), send one Nack(requeue = true) with their offsets as ranges and the ending subscription in tag 0, on the subscription’s connection and without waiting for its answer. Pushes and answers keep their order on a connection, and a push waits for its checkout to commit (ADR 0109), so a push can still arrive after the Unsubscribe was sent; waiting for the answer catches it. Messages handed out stay with the consumer until it acknowledges them or the connection closes.
If the answer is not NONE (the node no longer knows the subscription, a lost answer, a broken connection), or the subscription had ended already after NOT_LEADER or a broken connection and the consumer did not subscribe again, a new leader may keep the consumer’s messages for it until its resume window is over (ADR 0112). A consumer with an id then subscribes once more (retried like any subscribe, within retry_for) with resume, credit_msgs = 0 and held naming only the messages handed out and not settled, and unsubscribes that subscription as above: the leader hands the rest out again at once. The Rust client does this in close, not when a consumer is dropped.
Publishing through an exchange
Publish (key 9, ADR 0079) goes to any node, with entries of exchange id, routing key, mandatory and a batch of exactly one non-transactional data record; txn_id must be null. The node routes by the bindings it has applied and writes each queue in turn. An entry’s error is UNKNOWN_RESOURCE (exchange unknown: look the name up again, it may have a new id), NOT_LEADER (the node has not heard of the exchange yet, ADR 0101: nothing was written, so send the publish again to another node or after a pause; this holds for every retriable entry error), INVALID_REQUEST, UNROUTABLE (mandatory and no queue) or UNAUTHORIZED (no producer role on a routed queue; nothing written); otherwise routed lists each queue by id and name (the publisher may never have seen a queue bound and deleted between two of its lookups; the name is empty only when the node no longer knows the queue), with NONE and the offset, REQUEST_TIMED_OUT when its outcome is unknown, or the code that refused it. A publish that is not mandatory and routes nowhere has NONE and no queues. When the answer is lost, do not send the publish again: the node may have written it, and a second one could write it twice. Report the outcome as unknown.
Dead letters
A queue with limits (max.deliveries, message.ttl.ms) and a dead.letter.queue moves a message there as a one-record batch with the original key, value, headers and timestamp, and these headers (UTF-8 text, decimal numbers; they describe the latest move and replace those a message arrived with): x-lotse-dead-letter-reason (max-deliveries, expired, requeued or max-length, ADR 0118), -queue (source queue name), -queue-id, -offset, -deliveries and -floor (the source’s ack floor when it was sent). A client reads them like any header; the Java client offers them typed as Delivery.deadLetter().
Client behaviour: transactions
A client configured with a transactional id runs one transaction at a time (ADR 0029):
- Begin. The first transaction asks the coordinator
InitProducer(txn_id)for a producer id and epoch. That fences an earlier client with the same id and aborts what it left open. Later transactions reuse the id with the epoch the lastEndTxnanswer gave. After a transaction that was left open, or anEndTxnthat failed, the next one asksInitProduceragain. - Add. Before the first batch to a partition (a queue is partition 0), send
AddPartitionsToTxnto the coordinator, once per partition per transaction. A retriable error means: forget the coordinator and try again. - Produce. Stamp the batch with the producer id, epoch and a sequence per partition starting at 0 in each transaction, set the transactional bit, and send it with
txn_idset. Retry an unknown outcome with the same bytes, as for an idempotent produce. On success the sequence advances by the record count. - Must abort. After any error except “no broker in time” (
GaveUp) and “no such stream” the transaction may hold a write nobody knows, so only an abort is left. That includes a call that ends before its outcome is known. A client refuses further writes and the commit, without asking the coordinator. - End.
EndTxn(commit)to the coordinator. A lost request or a retriable error may have been decided already: retry, and reportUnknownif time runs out. A transaction that added no partition and sent no offsets ends withoutEndTxn. One that sent offsets (AddOffsetsToTxn, even if the answer was lost or refused) is ended all the same, with the sameEndTxn: the coordinator holds the offsets as pending until it hears, and every reader of the group’s offsets waits for them until the transaction times out.
Publishing in a transaction is routed by the client (ADR 0032): read the exchange’s type and bindings from Metadata (kept for metadata_max_age), pick the queues as routing.txt describes, and produce the message to each of them in the transaction. If a write fails after some queues have it, only an abort is left.
Acking a queue message in a transaction (ADR 0075): add the queue to the transaction, then send Ack with txn_id, the producer id and the epoch on the subscription’s own connection, since only the connection that holds a message can ack it so. A lost answer is sent again on the same connection, up to twice. A consumer with an id (ADR 0109) that then meets a retriable error, a broken connection or a third lost answer subscribes again, resuming with the id, and sends the ack on the new subscription’s connection, within retry_for; the new subscription holds the message. Without an id a broken connection means the message went back: MESSAGE_NOT_HELD, which also comes when another transaction holds it, or when the consumer did not come back within the resume window. Any other error leaves only an abort.
Offsets in a transaction: AddOffsetsToTxn to the coordinator, with the group, member id, member epoch and positions; it answers like CommitOffsets. A refusal (a fenced member, say) or an unknown outcome leaves only an abort.
Client behaviour: consumer groups
A member heartbeats with GroupHeartbeat to the coordinator (ADR 0033). The Rust and Java clients heartbeat only from their poll: a caller that takes longer than the session timeout between polls loses its partitions.
- Join with member id
"", epoch 0, and the subscribed stream ids, or the pattern instead (ADR 0073), and the instance id if any (ADR 0072). Later heartbeats carry the member id and epoch the answer gave, the owned partitions, and no subscription. - When. The next heartbeat is due after
min(heartbeat_interval_ms, session_timeout / 3), counted from when the last one was sent; after a failure, after one backoff. - Assignment. An answer with an assignment replaces the target. A partition no longer in it is not read from that moment on. With auto-commit its position is committed, and the next heartbeat goes out at once, reporting it released. Only then may another member get it. New partitions start without a position.
- Positions. A partition without one asks
OffsetFetchwithrequire_stable = 1, so an offset an open transaction holds waits.NO_OFFSETstarts at 0, or where the member was told to start (ListOffsetswith-1or a time). AnOFFSET_OUT_OF_RANGEwhose log start is past the position goes on from the log start. - Lost. When no heartbeat has succeeded for the session timeout, counted from when the last successful one was sent, every partition is dropped at once, without a commit, and the member joins again.
UNKNOWN_MEMBER_IDandFENCED_MEMBER_EPOCHmean the same.FENCED_INSTANCE_IDmeans a newer instance took the place: stop, and do not join again. - Commit sends
CommitOffsetswith the member id and epoch; a retriable error is retried as maybe done. - Leave with epoch
-1after committing, if auto-commit is on.
Client behaviour: admin
Admin requests (creates, deletes, bindings, settings, users, tenants, roles, nodes, pins, placements, partitions, revoking sessions) go to the metadata group’s leader (ADR 0034). Until a client knows it, it asks the brokers in turn. A retriable error carries a leader hint, which becomes the address for the next attempt. A node named in a hint whose address is unknown means: ask for metadata first.
-
Every admin request may be retried: a lost one may have been carried out. A create retried after a lost answer may find what its first attempt made and answer
RESOURCE_EXISTSwith the resource’s id; report it so, with the id, rather than as an error of the create. A delete answeredUNKNOWN_RESOURCEis done. -
Names are turned into ids with
Metadataasked of every known broker, keeping the newest resource per name, so a name made a moment ago is found even on a broker that lags. -
A bind or unbind changes routing: forget the exchange’s cached bindings.
-
PurgeQueueandRequeuego to the queue’s leader and work in rounds: sendup_to0, then theup_tothe answer gave, untilmoreis false, adding upcount. -
RevokeCertificate(key 52) carries the issuer (32 bytes: the SHA-256 of the issuer’s distinguished name in DER form followed by the certificate’s authority key identifier, if it has one; ADR 0096), the serial, the certificate’snot_after_msand a reason. A cluster admin sends it; every node then ends connections that showed that certificate and refuses it until it expires (ADR 0090). -
DescribeCertificates(key 53, cluster admin) lists the client certificates nodes accepted and the revoked ones: per entry issuer, serial, principal, validity, first and last sight, a state (0 valid, 1 expiring within 30 days, 2 expired, 3 revoked), the reason and time of a revocation; the answer carries the leader’s clock the states are computed against.RecordCertificates(key 54) is internal: only broker nodes send it, and a client never does (ADR 0091). So isReportDisk(key 61), by which a node tells the metadata leader that it went low on disk or back (ADR 0124); it is not in the vectors. -
RequestCertificate(key 55) carries a principal and a PKCS#10 signing request in DER form; its id is the SHA-256 of the request, so the same request twice is one. A cluster admin asks in any name; any other user only in their own (another name isUNAUTHORIZED), and their new request replaces their pending one; users’ own requests take at most 192 of the 256 places (ADR 0119).DescribeCertificateRequests(key 56) lists them, newest first, all of them to a cluster admin and to anyone else what they asked for: id, principal, request, who asked, when, when it lapses (7 days), a state (0 pending, 1 signed, 2 expired) and, once signed, the certificate (DER) and the time.RecordCertificate(key 57, cluster admin) is what the CA holder’slotse certs --signsends after signing: the request id, the certificate, the issuer fingerprint, serial and validity beside it (ADR 0092). The broker refuses it withINVALID_REQUESTunless the certificate has the request’s key and principal as its only common name, is no CA or node certificate, is signed by one of the cluster’s CA certificates, and says of itself what the fields beside it say (ADR 0096). -
DescribeSessions(key 59) is answered by the node it is sent to, for itself: an erroru16, the node’s idu32, its clocku64(ms since the Unix epoch) and an array of sessions, each the principal’s kindu8(0 anonymous, 1 user, 2 node), its name (a node’s id in decimal), the IdP groups (array of strings), the mechanismu8(0 none, 1 certificate, 2 SCRAM-SHA-512, 3 OAUTHBEARER), the peer address, when the connection was made, when its credential dates from and when it ends unless renewed (eachu64ms; 0: with the connection), then an empty tag block. Cluster admins and viewers get every session that has signed in, anyone else only those made with their own name (ADR 0121). A client asks every broker, as forDescribeMetrics. -
RenewCertificate(key 60, ADR 0123) renews the client certificate the connection was made with. The request is one field, a PKCS#10 signing request in DER form (at most 4,096 bytes) signed with the new key, then an empty tag block; its subject and extensions are ignored. The answer is the erroru16, the reason (optional string), the chain (array of bytes: the new certificate, then the renewal CA’s, DER), the new certificate’s end (u64, ms since the Unix epoch) and the metadata leader’s hint as the trailing block. It goes to the metadata leader like an admin change. It is refused withINVALID_REQUEST:- on a connection that showed no client certificate;
- outside the certificate’s window, which opens at the earlier of its last 30 days and two thirds of its validity, as the leader’s clock sees it;
- for a request that does not verify;
- on a cluster without a renewal CA;
- when 4,096 renewed certificates are held and this one replaces none.
A connection that signed in again with SASL as someone else gets
UNAUTHORIZED. The new certificate names the same principal and is valid for 90 days. A later renewal of the same certificate replaces it, so a client that retries after a lost answer keeps only the last chain it was given. A client makes a new P-256 key for each call and writes the key before the chain. -
ReassignPartition(key 58, cluster admin) carries a resource id, a partition and the voters (an array ofu64node ids, each once, active nodes); an empty array releases the partition to the planner (ADR 0115). It is refused withINVALID_REQUESTwhile the partition is moving: ask again later. The replica count of every partition is the settingreplication.factor, changed withAlterConfigson its own (no other key in the same request). -
AlterUsercarries a SCRAM-SHA-512 credential: a random 16-byte salt, at least 4096 iterations, the stored and server keys. The password never leaves the client.
Conformance vectors
sdk/conformance/* is written by cargo run -p lotse-proto --features samples --example vectors (make sdk-vectors). One vector per line, # for comments.
| File | Line |
|---|---|
requests.hex | label hex (frame without the length prefix; correlation id is index + 1) |
responses.hex | label api_key hex (correlation id is the index in the sample list) |
pushes.hex | label hex |
batches.hex | label hex |
batch-fields.txt, batch-records.txt | the header fields and records of each batch, as the Rust parser reads them |
batch-ops.txt | op arguments input output: an in-place edit and its result |
fetch-records.hex | several batches in a row |
varints.txt, crc32c.txt, error-codes.txt | fixed cases, including CRC32C of 123456789 = E3069283 |
routing.txt | label type bindings routing_key headers queues: exchange routing as lotse-route does it; the header of the file explains the fields |
scram.txt | label user_hex password salt_hex iterations client_nonce server_nonce and the five messages, stored_key server_key; the password is = and its hex |
A decoder passes when it reads every vector and writes it again to the same bytes, refuses each vector cut at every length and with one byte added, and agrees with the field files.