Exactly-once recipes
Lotse gives exactly-once effects from three pieces that build on each other. Use the smallest one that covers your case.
| Need | Use |
|---|---|
| A retry must not write a message twice | An idempotent producer |
| Several writes must appear together or not at all | A transaction |
| Read, transform and write must happen once as a unit | A transaction that also commits the consumer’s position |
1. Idempotent producer
Idempotent produce is on by default in both clients. The client gets a producer id from the coordinator and stamps each batch with a sequence per partition. If a connection breaks after sending, or a request times out, it sends the batch again with the same sequence. The leader checks the sequence against the whole log and answers with the offset of the batch it already has (ADR 0028).
What you must handle:
- Unknown outcome. When retrying runs out of
retry_for(Rust) orretryFor(Java) with the result unknown, the call fails withClientError::UnknownorClientException.Unknown. The client drops its producer id. Read the partition to see whether the batch is there, or produce again and accept a possible duplicate. - Restarts. Without a name, each client instance gets a fresh producer id, so a retry from before a restart is a different producer. Set a name (
producer_namein Rust,producerNamein Java) and the coordinator hands out the same id with a raised epoch on every start. The earlier instance is fenced: its retries are refused instead of written after the new instance’s batches (ADR 0071). Names share a namespace with transactional ids, and one client instance may use a name at a time. - Idle producers. A partition forgets a producer that has written nothing to it for a day (
[producers] expiration_ms); a retry after that is a new batch. It never forgets a live producer to make room. While a partition knows[producers] maxproducers (10,000), it refuses new ones withPRODUCER_LIMIT, and the client retries until one is forgotten orretry_forruns out (ADR 0110). Many short-lived producers without a name fill that table; give long-running ones a name. - Publishing through an exchange. A node writes each routed queue with an idempotent producer of its own. If its answer is lost, you get
Unknown, not a second publish (ADR 0079).
2. Transactions
Set a transactional id. One client instance per id: starting a transaction fences any older instance with the same id.
A transaction can write to several partitions and queues atomically. A read_committed reader (the default for fetch) sees all of a committed transaction and nothing of an aborted one (ADR 0029).
let mut txn = client.begin_transaction().await?;
txn.produce("payments", 0, payment_batch).await?;
txn.produce("audit", 1, audit_batch).await?;
txn.commit().await?; // or txn.abort().await?
try (Transaction txn = client.beginTransaction()) {
txn.produce("payments", 0, OutMessage.value("paid"));
txn.produce("audit", 1, OutMessage.value("payment recorded"));
txn.commit();
}
lotse.transaction {
produce("payments", 0, message("paid"))
produce("audit", 1, message("payment recorded"))
} // commits when the block returns; aborts on an exception or cancellation
Rules to keep in mind:
- A transaction that fails part way reports
MustAbort(Rust) and can only be aborted. Rust’sTransaction::publishdoes this when some queues hold a message and others do not. - A transaction left open times out after
txn_timeout(60 s by default) and the coordinator aborts it. The next transaction of the same client starts with a fresh producer id and aborts it sooner. - An open transaction holds back the last stable offset of its partitions, so read_committed readers wait behind it. Keep transactions short.
- A commit that fails with
Unknownmay have committed. The outcome stays unknown to the caller; check the data before you repeat the work.
3. Consume, transform, produce
From a stream
A consumer group member reads records; the transaction writes the results and commits the member’s positions together. A crash between the two leaves neither.
- Join the group with
auto_commitoff. Offsets then go with each transaction, not on a timer. pollfor records. It also heartbeats, so call it often.- Begin a transaction, produce the results, call
send_offsets(Rust) orsendOffsets(Java) with the group member, commit. - If the transaction fails, call
rewind(). The member reads the same records again from the group’s committed offsets.
let cfg = GroupConfig { auto_commit: false, ..GroupConfig::default() };
let mut group = client.join_group("billing", &["orders"], cfg).await?;
loop {
let orders = group.poll(Duration::from_millis(500)).await?;
if orders.is_empty() { continue; }
let mut txn = client.begin_transaction().await?;
for order in &orders {
txn.produce("invoices", 0, invoice_batch(order)).await?;
}
txn.send_offsets(&group).await?;
match txn.commit().await {
Ok(()) => {}
Err(_) => group.rewind(),
}
}
A member that no longer owns a partition is fenced: send_offsets fails and the transaction can only abort. That is the point: the new owner reads from the last committed offset, and nothing is written twice.
This is the complete Java version, compiled by gradlew check:
package dev.lotse.examples;
import dev.lotse.client.ClientException;
import dev.lotse.client.GroupConfig;
import dev.lotse.client.GroupConsumer;
import dev.lotse.client.GroupRecord;
import dev.lotse.client.LotseClient;
import dev.lotse.client.OutMessage;
import dev.lotse.client.Transaction;
import java.time.Duration;
import java.util.List;
/**
* Consume, transform, produce, exactly once: a group member reads orders, and each batch of invoices it writes
* commits in one transaction with the group's offsets. A crash between the two leaves neither.
*/
public final class ExactlyOnceExample {
private ExactlyOnceExample() {}
public static void main(String[] args) {
// One running client per transactional id: a second one fences the first.
try (LotseClient client = new LotseClient(Env.config().transactionalId("example-billing-1").build())) {
try {
client.createStream("example-invoices", 1, 3);
} catch (ClientException.ResourceExists e) {
// made by an earlier run
}
GroupConfig config = GroupConfig.builder().autoCommit(false).build(); // offsets go with the transaction
try (GroupConsumer group = client.joinGroup("example-billing", List.of("example-orders"), config)) {
long deadline = System.nanoTime() + Duration.ofSeconds(20).toNanos();
while (System.nanoTime() < deadline) {
List<GroupRecord> orders = group.poll(Duration.ofMillis(500)); // heartbeats too: call it often
if (orders.isEmpty()) {
continue;
}
try (Transaction txn = client.beginTransaction()) {
for (GroupRecord order : orders) {
txn.produce("example-invoices", 0,
OutMessage.value("invoice for " + order.record().value().toUtf8()));
}
txn.sendOffsets(group);
txn.commit();
System.out.printf("invoiced %d orders%n", orders.size());
} catch (ClientException e) {
// Not committed: read the same orders again from the group's committed offsets.
System.out.println("transaction failed, reading again: " + e.getMessage());
group.rewind();
}
}
}
}
}
}
And the same in Kotlin with transaction { }:
package dev.lotse.examples
import dev.lotse.client.ClientException
import dev.lotse.client.GroupRecord
import dev.lotse.kotlin.groupConfig
import dev.lotse.kotlin.lotse
import dev.lotse.kotlin.message
import java.time.Duration
import kotlinx.coroutines.flow.take
import kotlinx.coroutines.runBlocking
import kotlinx.coroutines.withTimeoutOrNull
/**
* Writes orders, then invoices them exactly once from a consumer group, each poll's orders in one transaction with
* the group's offsets. Brokers from `LOTSE_BOOTSTRAP` (default `127.0.0.1:9092`), plain TCP on loopback.
*/
fun main(): Unit = runBlocking {
val bootstrap = (System.getenv("LOTSE_BOOTSTRAP") ?: "127.0.0.1:9092").split(",").toTypedArray()
lotse(*bootstrap) {
plaintextOnLoopback()
transactionalId("example-billing-kt")
}.use { lotse ->
for (stream in listOf("example-orders-kt", "example-invoices-kt")) {
try {
lotse.createStream(stream, partitions = 1, replication = 3)
} catch (e: ClientException.ResourceExists) {
// made by an earlier run
}
}
repeat(10) { lotse.produce("example-orders-kt", 0, message("order-$it", key = "customer-${it % 3}")) }
lotse.joinGroup("example-billing-kt", listOf("example-orders-kt"), groupConfig { autoCommit(false) }).use { group ->
withTimeoutOrNull(Duration.ofSeconds(20).toMillis()) {
var invoiced = 0
while (invoiced < 10) {
val orders: List<GroupRecord> = group.poll(Duration.ofMillis(500))
if (orders.isEmpty()) continue
try {
lotse.transaction {
for (order in orders) {
produce("example-invoices-kt", 0, message("invoice for ${order.record().value().toUtf8()}"))
}
sendOffsets(group)
}
invoiced += orders.size
println("invoiced $invoiced orders")
} catch (e: ClientException) {
group.rewind() // not committed: read them again
}
}
}
}
lotse.records("example-invoices-kt", 0, from = 0).take(10).collect { println(it.value().toUtf8()) }
}
}
From a queue
A consumer acknowledges messages inside the transaction (ADR 0075). If it commits, the messages are settled; if it aborts, they are delivered again.
let mut jobs = client.subscribe("jobs", 16).await?;
while let Some(job) = jobs.next(Duration::from_secs(5)).await? {
let mut txn = client.begin_transaction().await?;
txn.produce("results", 0, result_batch(&job)).await?;
jobs.ack_in(&mut txn, &job).await?; // adds the queue to the transaction first
txn.commit().await?;
}
In Java, queue.ackIn(txn, delivery); in Kotlin, ack(queue, delivery) inside transaction { }.
- Acknowledge on the subscription that holds the message. The ack goes on the connection the message came on.
- Of two transactions that try to hold one message, the first in the log holds it and the other gets
MESSAGE_NOT_HELD. Any error fromack_inleaves the transaction to abort. - Acks alone do not hold the last stable offset back.
- Messages that stay unsettled past the queue’s ack timeout are delivered again; keep the transaction shorter than that.
What exactly-once does not cover
The guarantee holds for effects inside Lotse: the log, the queue ack index and the group offsets. A side effect outside it, such as a payment or an e-mail, can run twice if your process crashes after the effect and before the commit. Make such effects idempotent on their own side, with a key derived from the stream, partition and offset or from the queue message.