Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Java and Kotlin

The Java client is in sdk/java (Java 17, Gradle). It follows lotse-client rule for rule; the Kotlin layer in sdk/kotlin wraps it for coroutines. The design is in ADR 0056 and ADR 0080. The artifact group is dev.lotse, version 0.0.1; nothing is published to a repository yet, so build it from the source tree.

Packages

PackageWhat
dev.lotse.protoField codec, frames, every client-facing request and response, error codes.
dev.lotse.recordRecord batch, BatchBuilder, in-place edits, control batches.
dev.lotse.netConnections, TLS 1.3 (TlsConfig), pipelining, pushes, session renewal.
dev.lotse.authCredentials: SCRAM-SHA-512 and OAUTHBEARER.
dev.lotse.clientLotseClient, QueueConsumer, GroupConsumer, Transaction and the admin calls.

The client has no runtime dependencies. Reading and writing zstd and lz4 batches needs zstd-jni and lz4-java, which are optional; their absence is named in the error (ADR 0061).

Connecting

ClientConfig.builder(bootstrap...) builds a configuration. There is no transport default: call tls(TlsConfig.fromFiles(ca, cert, key)) or plaintextOnLoopback().

var config = ClientConfig.builder("127.0.0.1:9092", "127.0.0.1:9093")
        .plaintextOnLoopback()
        .build();
try (var client = new LotseClient(config)) {
    // ...
}

Other builder methods: requestTimeout, retryFor, backoff, produceTimeout, idempotent, txnTimeout, producerName, transactionalId, metadataMaxAge, tenant, zone, credentials. They mean what the Rust fields mean (see Rust client).

Streams

LotseClient offers createStream, produce, fetch, fetchUncommitted and listOffset. Every call has an ...Async form that returns a CompletableFuture. Two async produces to one partition run one after the other in no promised order: await the first if order matters.

  • You choose the partition; the client does not hash keys.
  • u64 values such as offsets travel in a long as their bit pattern. Print and compare them with Long.toUnsignedString and the other Long.*Unsigned methods.
  • A message without a timestamp is stamped with the time of the call. Retention judges data by producers’ timestamps, so a message stamped 0 would be deleted at the next pass.
  • ClientException.Unknown is the one error that needs a decision: after retryFor the client cannot tell whether the batch was written. Read the partition, or produce again and accept a possible duplicate.
  • ClientException.OffsetOutOfRange carries logStart().

The example below creates a stream, writes ten messages and reads them back. It is compiled by gradlew check.

package dev.lotse.examples;

import dev.lotse.client.ClientException;
import dev.lotse.client.Fetched;
import dev.lotse.client.LotseClient;
import dev.lotse.client.OutMessage;
import dev.lotse.client.ProduceResult;
import dev.lotse.record.BatchRecord;
import java.time.Duration;

/** Creates a stream, writes ten messages to it and reads them back. */
public final class StreamExample {
    private StreamExample() {}

    public static void main(String[] args) {
        try (LotseClient client = new LotseClient(Env.config().build())) {
            try {
                client.createStream("example-orders", 2, 3);
            } catch (ClientException.ResourceExists e) {
                // made by an earlier run
            }
            for (int i = 0; i < 10; i++) {
                // The partition is the caller's to choose; here, by key.
                String customer = "customer-" + (i % 3);
                int partition = Math.floorMod(customer.hashCode(), 2);
                ProduceResult r = client.produce(
                        "example-orders", partition, OutMessage.value("order-" + i).withKey(customer));
                System.out.printf("order-%d at partition %d, offset %s%n", i, partition,
                        Long.toUnsignedString(r.baseOffset()));
            }
            for (int partition = 0; partition < 2; partition++) {
                long offset = 0;
                while (true) {
                    Fetched f = client.fetch("example-orders", partition, offset, Duration.ofMillis(200));
                    for (BatchRecord record : f.committedMessages()) {
                        System.out.printf("read %s from partition %d%n", record.value().toUtf8(), partition);
                    }
                    long next = f.nextOffset(offset);
                    if (next == offset) {
                        break; // nothing more for now
                    }
                    offset = next;
                }
            }
        }
    }
}

Queues

createQueue(name, QueueOptions) and subscribe(queue, SubscribeOptions) return a QueueConsumer. poll(Duration) returns a Delivery or null; settle with ack, nack(delivery, requeue) or nackAfter(delivery, delay). isStandby() is true while a subscription waits behind another on a single active consumer queue. The consumer resubscribes by itself after a leader change, as the same consumer, and keeps the messages it received, also those acked in a transaction (ADR 0109).

QueueOptions.builder() sets maxDeliveries, deadLetterQueue, ackTimeout and the other queue settings. A retired message arrives in the dead-letter queue with Delivery.deadLetter() describing where it came from and why.

package dev.lotse.examples;

import dev.lotse.client.ClientException;
import dev.lotse.client.DeadLetter;
import dev.lotse.client.Delivery;
import dev.lotse.client.LotseClient;
import dev.lotse.client.OutMessage;
import dev.lotse.client.QueueConsumer;
import dev.lotse.client.QueueOptions;
import dev.lotse.client.SubscribeOptions;
import java.time.Duration;

/**
 * A work queue with a dead-letter queue: jobs that fail are tried again after a delay, and a job that fails three
 * times moves to the dead-letter queue.
 */
public final class QueueExample {
    private QueueExample() {}

    public static void main(String[] args) {
        try (LotseClient client = new LotseClient(Env.config().build())) {
            createIfMissing(client, "example-jobs-dead", QueueOptions.defaults());
            createIfMissing(client, "example-jobs", QueueOptions.builder()
                    .maxDeliveries(3)
                    .deadLetterQueue("example-jobs-dead")
                    .ackTimeout(Duration.ofSeconds(30))
                    .build());
            for (int i = 0; i < 5; i++) {
                client.produce("example-jobs", 0, OutMessage.value("job-" + i));
            }
            try (QueueConsumer jobs = client.subscribe("example-jobs", SubscribeOptions.withCredit(16))) {
                while (true) {
                    Delivery job = jobs.poll(Duration.ofSeconds(5));
                    if (job == null) {
                        break; // nothing for 5 s: done
                    }
                    String name = job.value().toUtf8();
                    if (name.equals("job-3")) {
                        // Fails every time: back in one second, until max.deliveries retires it.
                        System.out.printf("%s failed (delivery %d)%n", name, job.deliveryCount());
                        jobs.nackAfter(job, Duration.ofSeconds(1));
                        continue;
                    }
                    System.out.println("done " + name);
                    jobs.ack(job);
                }
            }
            // The retired job, with where it came from and why.
            try (QueueConsumer dead = client.subscribe("example-jobs-dead")) {
                Delivery job = dead.poll(Duration.ofSeconds(5));
                if (job != null) {
                    System.out.printf("dead letter %s: %s after %d deliveries%n", job.value().toUtf8(),
                            job.deadLetter().map(DeadLetter::reason).orElse("?"),
                            job.deadLetter().map(DeadLetter::deliveries).orElse(0));
                    dead.ack(job);
                }
            }
        }
    }

    private static void createIfMissing(LotseClient client, String name, QueueOptions options) {
        try {
            client.createQueue(name, options);
        } catch (ClientException.ResourceExists e) {
            // made by an earlier run
        }
    }
}

Publishing through an exchange

declareExchange, bind, bindHeaders and unbind set up routing; LotseClient.publish(exchange, routingKey, message, mandatory) goes to a node, which routes it (ADR 0079). Inside a transaction Transaction.publish routes in the client, checked against sdk/conformance/routing.txt.

Transactions and groups

beginTransaction() returns a Transaction; set transactionalId first. It has produce, produceBatch, publish, sendOffsets(GroupConsumer), commit and abort; QueueConsumer.ackIn(txn, delivery) acknowledges within it. close() without a decision ends the handle; the coordinator aborts what is left when the next transaction starts with a fresh producer id.

joinGroup(group, streams, GroupConfig) and joinGroupMatching(group, pattern, config) return a GroupConsumer. poll(Duration) heartbeats too, so call it often. commit(), rewind(), assignment(), positions() and close() match the Rust client. Exactly-once recipes shows the combination.

package dev.lotse.examples;

import dev.lotse.client.ClientException;
import dev.lotse.client.GroupConfig;
import dev.lotse.client.GroupConsumer;
import dev.lotse.client.GroupRecord;
import dev.lotse.client.LotseClient;
import dev.lotse.client.OutMessage;
import dev.lotse.client.Transaction;
import java.time.Duration;
import java.util.List;

/**
 * Consume, transform, produce, exactly once: a group member reads orders, and each batch of invoices it writes
 * commits in one transaction with the group's offsets. A crash between the two leaves neither.
 */
public final class ExactlyOnceExample {
    private ExactlyOnceExample() {}

    public static void main(String[] args) {
        // One running client per transactional id: a second one fences the first.
        try (LotseClient client = new LotseClient(Env.config().transactionalId("example-billing-1").build())) {
            try {
                client.createStream("example-invoices", 1, 3);
            } catch (ClientException.ResourceExists e) {
                // made by an earlier run
            }
            GroupConfig config = GroupConfig.builder().autoCommit(false).build(); // offsets go with the transaction
            try (GroupConsumer group = client.joinGroup("example-billing", List.of("example-orders"), config)) {
                long deadline = System.nanoTime() + Duration.ofSeconds(20).toNanos();
                while (System.nanoTime() < deadline) {
                    List<GroupRecord> orders = group.poll(Duration.ofMillis(500)); // heartbeats too: call it often
                    if (orders.isEmpty()) {
                        continue;
                    }
                    try (Transaction txn = client.beginTransaction()) {
                        for (GroupRecord order : orders) {
                            txn.produce("example-invoices", 0,
                                    OutMessage.value("invoice for " + order.record().value().toUtf8()));
                        }
                        txn.sendOffsets(group);
                        txn.commit();
                        System.out.printf("invoiced %d orders%n", orders.size());
                    } catch (ClientException e) {
                        // Not committed: read the same orders again from the group's committed offsets.
                        System.out.println("transaction failed, reading again: " + e.getMessage());
                        group.rewind();
                    }
                }
            }
        }
    }
}

Admin

The admin calls are methods of LotseClient: delete, deleteIfEmpty, purge, requeue, alterConfigs, describeConfigs, setUserPassword, deleteUser, describeUsers, createTenant, deleteTenant, describeTenants, alterRoleBindings, describeRoleBindings, whoAmI, revokeSessions, addNode, removeNode, describeNodes, moveLeader, reassignPartition, createPartitions and nodeMetrics.

Renewing the client certificate

On a cluster with a renewal CA, a client that signs in with a certificate renews it in its window: its last 30 days, or past two thirds of its validity (ADR 0123).

TlsConfig tls = TlsConfig.fromFiles(ca, cert, key);   // reads the files again when they change
LotseClient client = new LotseClient(ClientConfig.builder("10.0.0.1:9000")
        .tls(tls)
        .renewCertificate(cert, key)                   // renews by itself in the window; looks every hour
        .build());

renewCertificate(cert, key) on the client renews once and writes the new key, then the new chain, over the files; renewCertificate() only returns them. CertificateRenewal.due(cert, now) says whether the window is open. renewalFailure() tells why the last automatic renewal failed. Connections made after the files changed show the new certificate; the old one keeps working until it expires.

Kotlin

dev.lotse.kotlin wraps the Java client: every call is a suspend function that runs the blocking Java call on Dispatchers.IO.

  • lotse(vararg bootstrap) { ... } builds a Lotse (the block configures ClientConfig.Builder). Builders groupConfig { } and queueOptions { } do the same for their types, and message(value, key, headers) makes an OutMessage.
  • Lotse.records(stream, partition, from) and LotseQueue.deliveries() and LotseGroup.records() are Flows.
  • Lotse.transaction { } begins a transaction, runs the block with a LotseTransaction receiver, commits when the block returns, and aborts on an exception or cancellation.
  • LotseTransaction, LotseQueue and LotseGroup expose the Java object as java.
package dev.lotse.examples

import dev.lotse.client.ClientException
import dev.lotse.client.GroupRecord
import dev.lotse.kotlin.groupConfig
import dev.lotse.kotlin.lotse
import dev.lotse.kotlin.message
import java.time.Duration
import kotlinx.coroutines.flow.take
import kotlinx.coroutines.runBlocking
import kotlinx.coroutines.withTimeoutOrNull

/**
 * Writes orders, then invoices them exactly once from a consumer group, each poll's orders in one transaction with
 * the group's offsets. Brokers from `LOTSE_BOOTSTRAP` (default `127.0.0.1:9092`), plain TCP on loopback.
 */
fun main(): Unit = runBlocking {
    val bootstrap = (System.getenv("LOTSE_BOOTSTRAP") ?: "127.0.0.1:9092").split(",").toTypedArray()
    lotse(*bootstrap) {
        plaintextOnLoopback()
        transactionalId("example-billing-kt")
    }.use { lotse ->
        for (stream in listOf("example-orders-kt", "example-invoices-kt")) {
            try {
                lotse.createStream(stream, partitions = 1, replication = 3)
            } catch (e: ClientException.ResourceExists) {
                // made by an earlier run
            }
        }
        repeat(10) { lotse.produce("example-orders-kt", 0, message("order-$it", key = "customer-${it % 3}")) }

        lotse.joinGroup("example-billing-kt", listOf("example-orders-kt"), groupConfig { autoCommit(false) }).use { group ->
            withTimeoutOrNull(Duration.ofSeconds(20).toMillis()) {
                var invoiced = 0
                while (invoiced < 10) {
                    val orders: List<GroupRecord> = group.poll(Duration.ofMillis(500))
                    if (orders.isEmpty()) continue
                    try {
                        lotse.transaction {
                            for (order in orders) {
                                produce("example-invoices-kt", 0, message("invoice for ${order.record().value().toUtf8()}"))
                            }
                            sendOffsets(group)
                        }
                        invoiced += orders.size
                        println("invoiced $invoiced orders")
                    } catch (e: ClientException) {
                        group.rewind() // not committed: read them again
                    }
                }
            }
        }

        lotse.records("example-invoices-kt", 0, from = 0).take(10).collect { println(it.value().toUtf8()) }
    }
}

Running the examples

The examples read LOTSE_BOOTSTRAP (default 127.0.0.1:9092). The Java examples use plain TCP on loopback unless LOTSE_CA is set, with optional LOTSE_CERT, LOTSE_KEY, LOTSE_USER and LOTSE_PASSWORD. make dev starts a local three-node cluster.