Java and Kotlin
The Java client is in sdk/java (Java 17, Gradle). It follows lotse-client rule for rule; the Kotlin layer in sdk/kotlin wraps it for coroutines. The design is in ADR 0056 and ADR 0080. The artifact group is dev.lotse, version 0.0.1; nothing is published to a repository yet, so build it from the source tree.
Packages
| Package | What |
|---|---|
dev.lotse.proto | Field codec, frames, every client-facing request and response, error codes. |
dev.lotse.record | Record batch, BatchBuilder, in-place edits, control batches. |
dev.lotse.net | Connections, TLS 1.3 (TlsConfig), pipelining, pushes, session renewal. |
dev.lotse.auth | Credentials: SCRAM-SHA-512 and OAUTHBEARER. |
dev.lotse.client | LotseClient, QueueConsumer, GroupConsumer, Transaction and the admin calls. |
The client has no runtime dependencies. Reading and writing zstd and lz4 batches needs zstd-jni and lz4-java, which are optional; their absence is named in the error (ADR 0061).
Connecting
ClientConfig.builder(bootstrap...) builds a configuration. There is no transport default: call tls(TlsConfig.fromFiles(ca, cert, key)) or plaintextOnLoopback().
var config = ClientConfig.builder("127.0.0.1:9092", "127.0.0.1:9093")
.plaintextOnLoopback()
.build();
try (var client = new LotseClient(config)) {
// ...
}
Other builder methods: requestTimeout, retryFor, backoff, produceTimeout, idempotent, txnTimeout, producerName, transactionalId, metadataMaxAge, tenant, zone, credentials. They mean what the Rust fields mean (see Rust client).
Streams
LotseClient offers createStream, produce, fetch, fetchUncommitted and listOffset. Every call has an ...Async form that returns a CompletableFuture. Two async produces to one partition run one after the other in no promised order: await the first if order matters.
- You choose the partition; the client does not hash keys.
u64values such as offsets travel in alongas their bit pattern. Print and compare them withLong.toUnsignedStringand the otherLong.*Unsignedmethods.- A message without a timestamp is stamped with the time of the call. Retention judges data by producers’ timestamps, so a message stamped 0 would be deleted at the next pass.
ClientException.Unknownis the one error that needs a decision: afterretryForthe client cannot tell whether the batch was written. Read the partition, or produce again and accept a possible duplicate.ClientException.OffsetOutOfRangecarrieslogStart().
The example below creates a stream, writes ten messages and reads them back. It is compiled by gradlew check.
package dev.lotse.examples;
import dev.lotse.client.ClientException;
import dev.lotse.client.Fetched;
import dev.lotse.client.LotseClient;
import dev.lotse.client.OutMessage;
import dev.lotse.client.ProduceResult;
import dev.lotse.record.BatchRecord;
import java.time.Duration;
/** Creates a stream, writes ten messages to it and reads them back. */
public final class StreamExample {
private StreamExample() {}
public static void main(String[] args) {
try (LotseClient client = new LotseClient(Env.config().build())) {
try {
client.createStream("example-orders", 2, 3);
} catch (ClientException.ResourceExists e) {
// made by an earlier run
}
for (int i = 0; i < 10; i++) {
// The partition is the caller's to choose; here, by key.
String customer = "customer-" + (i % 3);
int partition = Math.floorMod(customer.hashCode(), 2);
ProduceResult r = client.produce(
"example-orders", partition, OutMessage.value("order-" + i).withKey(customer));
System.out.printf("order-%d at partition %d, offset %s%n", i, partition,
Long.toUnsignedString(r.baseOffset()));
}
for (int partition = 0; partition < 2; partition++) {
long offset = 0;
while (true) {
Fetched f = client.fetch("example-orders", partition, offset, Duration.ofMillis(200));
for (BatchRecord record : f.committedMessages()) {
System.out.printf("read %s from partition %d%n", record.value().toUtf8(), partition);
}
long next = f.nextOffset(offset);
if (next == offset) {
break; // nothing more for now
}
offset = next;
}
}
}
}
}
Queues
createQueue(name, QueueOptions) and subscribe(queue, SubscribeOptions) return a QueueConsumer. poll(Duration) returns a Delivery or null; settle with ack, nack(delivery, requeue) or nackAfter(delivery, delay). isStandby() is true while a subscription waits behind another on a single active consumer queue. The consumer resubscribes by itself after a leader change, as the same consumer, and keeps the messages it received, also those acked in a transaction (ADR 0109).
QueueOptions.builder() sets maxDeliveries, deadLetterQueue, ackTimeout and the other queue settings. A retired message arrives in the dead-letter queue with Delivery.deadLetter() describing where it came from and why.
package dev.lotse.examples;
import dev.lotse.client.ClientException;
import dev.lotse.client.DeadLetter;
import dev.lotse.client.Delivery;
import dev.lotse.client.LotseClient;
import dev.lotse.client.OutMessage;
import dev.lotse.client.QueueConsumer;
import dev.lotse.client.QueueOptions;
import dev.lotse.client.SubscribeOptions;
import java.time.Duration;
/**
* A work queue with a dead-letter queue: jobs that fail are tried again after a delay, and a job that fails three
* times moves to the dead-letter queue.
*/
public final class QueueExample {
private QueueExample() {}
public static void main(String[] args) {
try (LotseClient client = new LotseClient(Env.config().build())) {
createIfMissing(client, "example-jobs-dead", QueueOptions.defaults());
createIfMissing(client, "example-jobs", QueueOptions.builder()
.maxDeliveries(3)
.deadLetterQueue("example-jobs-dead")
.ackTimeout(Duration.ofSeconds(30))
.build());
for (int i = 0; i < 5; i++) {
client.produce("example-jobs", 0, OutMessage.value("job-" + i));
}
try (QueueConsumer jobs = client.subscribe("example-jobs", SubscribeOptions.withCredit(16))) {
while (true) {
Delivery job = jobs.poll(Duration.ofSeconds(5));
if (job == null) {
break; // nothing for 5 s: done
}
String name = job.value().toUtf8();
if (name.equals("job-3")) {
// Fails every time: back in one second, until max.deliveries retires it.
System.out.printf("%s failed (delivery %d)%n", name, job.deliveryCount());
jobs.nackAfter(job, Duration.ofSeconds(1));
continue;
}
System.out.println("done " + name);
jobs.ack(job);
}
}
// The retired job, with where it came from and why.
try (QueueConsumer dead = client.subscribe("example-jobs-dead")) {
Delivery job = dead.poll(Duration.ofSeconds(5));
if (job != null) {
System.out.printf("dead letter %s: %s after %d deliveries%n", job.value().toUtf8(),
job.deadLetter().map(DeadLetter::reason).orElse("?"),
job.deadLetter().map(DeadLetter::deliveries).orElse(0));
dead.ack(job);
}
}
}
}
private static void createIfMissing(LotseClient client, String name, QueueOptions options) {
try {
client.createQueue(name, options);
} catch (ClientException.ResourceExists e) {
// made by an earlier run
}
}
}
Publishing through an exchange
declareExchange, bind, bindHeaders and unbind set up routing; LotseClient.publish(exchange, routingKey, message, mandatory) goes to a node, which routes it (ADR 0079). Inside a transaction Transaction.publish routes in the client, checked against sdk/conformance/routing.txt.
Transactions and groups
beginTransaction() returns a Transaction; set transactionalId first. It has produce, produceBatch, publish, sendOffsets(GroupConsumer), commit and abort; QueueConsumer.ackIn(txn, delivery) acknowledges within it. close() without a decision ends the handle; the coordinator aborts what is left when the next transaction starts with a fresh producer id.
joinGroup(group, streams, GroupConfig) and joinGroupMatching(group, pattern, config) return a GroupConsumer. poll(Duration) heartbeats too, so call it often. commit(), rewind(), assignment(), positions() and close() match the Rust client. Exactly-once recipes shows the combination.
package dev.lotse.examples;
import dev.lotse.client.ClientException;
import dev.lotse.client.GroupConfig;
import dev.lotse.client.GroupConsumer;
import dev.lotse.client.GroupRecord;
import dev.lotse.client.LotseClient;
import dev.lotse.client.OutMessage;
import dev.lotse.client.Transaction;
import java.time.Duration;
import java.util.List;
/**
* Consume, transform, produce, exactly once: a group member reads orders, and each batch of invoices it writes
* commits in one transaction with the group's offsets. A crash between the two leaves neither.
*/
public final class ExactlyOnceExample {
private ExactlyOnceExample() {}
public static void main(String[] args) {
// One running client per transactional id: a second one fences the first.
try (LotseClient client = new LotseClient(Env.config().transactionalId("example-billing-1").build())) {
try {
client.createStream("example-invoices", 1, 3);
} catch (ClientException.ResourceExists e) {
// made by an earlier run
}
GroupConfig config = GroupConfig.builder().autoCommit(false).build(); // offsets go with the transaction
try (GroupConsumer group = client.joinGroup("example-billing", List.of("example-orders"), config)) {
long deadline = System.nanoTime() + Duration.ofSeconds(20).toNanos();
while (System.nanoTime() < deadline) {
List<GroupRecord> orders = group.poll(Duration.ofMillis(500)); // heartbeats too: call it often
if (orders.isEmpty()) {
continue;
}
try (Transaction txn = client.beginTransaction()) {
for (GroupRecord order : orders) {
txn.produce("example-invoices", 0,
OutMessage.value("invoice for " + order.record().value().toUtf8()));
}
txn.sendOffsets(group);
txn.commit();
System.out.printf("invoiced %d orders%n", orders.size());
} catch (ClientException e) {
// Not committed: read the same orders again from the group's committed offsets.
System.out.println("transaction failed, reading again: " + e.getMessage());
group.rewind();
}
}
}
}
}
}
Admin
The admin calls are methods of LotseClient: delete, deleteIfEmpty, purge, requeue, alterConfigs, describeConfigs, setUserPassword, deleteUser, describeUsers, createTenant, deleteTenant, describeTenants, alterRoleBindings, describeRoleBindings, whoAmI, revokeSessions, addNode, removeNode, describeNodes, moveLeader, reassignPartition, createPartitions and nodeMetrics.
Renewing the client certificate
On a cluster with a renewal CA, a client that signs in with a certificate renews it in its window: its last 30 days, or past two thirds of its validity (ADR 0123).
TlsConfig tls = TlsConfig.fromFiles(ca, cert, key); // reads the files again when they change
LotseClient client = new LotseClient(ClientConfig.builder("10.0.0.1:9000")
.tls(tls)
.renewCertificate(cert, key) // renews by itself in the window; looks every hour
.build());
renewCertificate(cert, key) on the client renews once and writes the new key, then the new chain, over the files; renewCertificate() only returns them. CertificateRenewal.due(cert, now) says whether the window is open. renewalFailure() tells why the last automatic renewal failed. Connections made after the files changed show the new certificate; the old one keeps working until it expires.
Kotlin
dev.lotse.kotlin wraps the Java client: every call is a suspend function that runs the blocking Java call on Dispatchers.IO.
lotse(vararg bootstrap) { ... }builds aLotse(the block configuresClientConfig.Builder). BuildersgroupConfig { }andqueueOptions { }do the same for their types, andmessage(value, key, headers)makes anOutMessage.Lotse.records(stream, partition, from)andLotseQueue.deliveries()andLotseGroup.records()areFlows.Lotse.transaction { }begins a transaction, runs the block with aLotseTransactionreceiver, commits when the block returns, and aborts on an exception or cancellation.LotseTransaction,LotseQueueandLotseGroupexpose the Java object asjava.
package dev.lotse.examples
import dev.lotse.client.ClientException
import dev.lotse.client.GroupRecord
import dev.lotse.kotlin.groupConfig
import dev.lotse.kotlin.lotse
import dev.lotse.kotlin.message
import java.time.Duration
import kotlinx.coroutines.flow.take
import kotlinx.coroutines.runBlocking
import kotlinx.coroutines.withTimeoutOrNull
/**
* Writes orders, then invoices them exactly once from a consumer group, each poll's orders in one transaction with
* the group's offsets. Brokers from `LOTSE_BOOTSTRAP` (default `127.0.0.1:9092`), plain TCP on loopback.
*/
fun main(): Unit = runBlocking {
val bootstrap = (System.getenv("LOTSE_BOOTSTRAP") ?: "127.0.0.1:9092").split(",").toTypedArray()
lotse(*bootstrap) {
plaintextOnLoopback()
transactionalId("example-billing-kt")
}.use { lotse ->
for (stream in listOf("example-orders-kt", "example-invoices-kt")) {
try {
lotse.createStream(stream, partitions = 1, replication = 3)
} catch (e: ClientException.ResourceExists) {
// made by an earlier run
}
}
repeat(10) { lotse.produce("example-orders-kt", 0, message("order-$it", key = "customer-${it % 3}")) }
lotse.joinGroup("example-billing-kt", listOf("example-orders-kt"), groupConfig { autoCommit(false) }).use { group ->
withTimeoutOrNull(Duration.ofSeconds(20).toMillis()) {
var invoiced = 0
while (invoiced < 10) {
val orders: List<GroupRecord> = group.poll(Duration.ofMillis(500))
if (orders.isEmpty()) continue
try {
lotse.transaction {
for (order in orders) {
produce("example-invoices-kt", 0, message("invoice for ${order.record().value().toUtf8()}"))
}
sendOffsets(group)
}
invoiced += orders.size
println("invoiced $invoiced orders")
} catch (e: ClientException) {
group.rewind() // not committed: read them again
}
}
}
}
lotse.records("example-invoices-kt", 0, from = 0).take(10).collect { println(it.value().toUtf8()) }
}
}
Running the examples
The examples read LOTSE_BOOTSTRAP (default 127.0.0.1:9092). The Java examples use plain TCP on loopback unless LOTSE_CA is set, with optional LOTSE_CERT, LOTSE_KEY, LOTSE_USER and LOTSE_PASSWORD. make dev starts a local three-node cluster.