Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

A first message

This chapter writes ten records to a stream and reads them back, then looks at them in the console. It assumes the one-machine trial is running. Against a TLS cluster you set the CA and a client certificate instead; see below.

With the Java client

The Java 17 client is in sdk/java of the repository. The example StreamExample creates the stream example-orders with 2 partitions and 3 replicas, writes ten records and reads them back. Its source is compiled by ./gradlew check, so it keeps up with the API:

package dev.lotse.examples;

import dev.lotse.client.ClientException;
import dev.lotse.client.Fetched;
import dev.lotse.client.LotseClient;
import dev.lotse.client.OutMessage;
import dev.lotse.client.ProduceResult;
import dev.lotse.record.BatchRecord;
import java.time.Duration;

/** Creates a stream, writes ten messages to it and reads them back. */
public final class StreamExample {
    private StreamExample() {}

    public static void main(String[] args) {
        try (LotseClient client = new LotseClient(Env.config().build())) {
            try {
                client.createStream("example-orders", 2, 3);
            } catch (ClientException.ResourceExists e) {
                // made by an earlier run
            }
            for (int i = 0; i < 10; i++) {
                // The partition is the caller's to choose; here, by key.
                String customer = "customer-" + (i % 3);
                int partition = Math.floorMod(customer.hashCode(), 2);
                ProduceResult r = client.produce(
                        "example-orders", partition, OutMessage.value("order-" + i).withKey(customer));
                System.out.printf("order-%d at partition %d, offset %s%n", i, partition,
                        Long.toUnsignedString(r.baseOffset()));
            }
            for (int partition = 0; partition < 2; partition++) {
                long offset = 0;
                while (true) {
                    Fetched f = client.fetch("example-orders", partition, offset, Duration.ofMillis(200));
                    for (BatchRecord record : f.committedMessages()) {
                        System.out.printf("read %s from partition %d%n", record.value().toUtf8(), partition);
                    }
                    long next = f.nextOffset(offset);
                    if (next == offset) {
                        break; // nothing more for now
                    }
                    offset = next;
                }
            }
        }
    }
}

The examples read their connection settings from the environment (sdk/java/src/examples/java/dev/lotse/examples/Env.java):

VariableMeaning
LOTSE_BOOTSTRAPbrokers as host:port, separated by commas. The default is 127.0.0.1:9092, so set it.
LOTSE_CAthe CA certificate, to use TLS. Without it the example uses plain TCP, which the client allows on loopback only.
LOTSE_CERT, LOTSE_KEYa client certificate and its key, with LOTSE_CA.
LOTSE_USER, LOTSE_PASSWORDa SCRAM-SHA-512 login, if the cluster asks for one.

For the trial:

export LOTSE_BOOTSTRAP=127.0.0.1:9001,127.0.0.1:9002,127.0.0.1:9003

For the TLS cluster of A first cluster:

export LOTSE_BOOTSTRAP=10.0.0.1:9000,10.0.0.2:9000,10.0.0.3:9000
export LOTSE_CA=/etc/lotse/ca.pem LOTSE_CERT=admin/admin.pem LOTSE_KEY=admin/admin.key

The expected output has ten lines of the form order-3 at partition 1, offset 0, then ten lines of the form read order-3 from partition 1. The offsets start at 0 on a new stream and the order across partitions is not defined; within a partition it is the order written.

This guide has not run the example end to end. The repository’s Gradle build has no task that starts an example; it only compiles them. Put the compiled classes of the examples source set and the main classes on the class path and run dev.lotse.examples.StreamExample. The developer guide covers the clients in full.

With the Rust client

lotse-client is the Rust client over the native protocol. Client::new(rt, ClientConfig::new(bootstrap)) connects, create_stream(name, partitions, replication) creates a stream, produce(stream, partition, batch) writes a record batch and returns its offset, and fetch reads from an offset. The client is generic over the runtime in lotse-rt; on a real machine that is TokioRuntime from lotse-rt-tokio with .plaintext_on_loopback() for the trial or .with_tls(...) for a TLS cluster. The working code to start from is crates/lotse-console/examples/local_cluster.rs, which produces to a stream and a queue in a loop.

Check it in the console

The console shows what the cluster holds. For the trial cluster, write a console file lotse-console.toml:

listen = "127.0.0.1:8080"

[auth]
mode = "dev"
user = "trial"

[[clusters]]
id = "trial"
kind = "native"
bootstrap = ["127.0.0.1:9001", "127.0.0.1:9002", "127.0.0.1:9003"]

Build the web app once, then start the console:

cd console/web && npm ci && npm run build && cd ../..
cargo run -p lotse-console -- --config lotse-console.toml

Open http://127.0.0.1:8080. The stream example-orders is listed with its partitions, replicas and offsets, and the charts show produce and fetch rates. This guide has not run this exact console file; the keys follow console/lotse-console.example.toml and the comments in crates/lotse-console/src/config.rs, where a native cluster’s ca_file is optional and plain TCP is allowed on loopback only.

Against a TLS cluster, add ca_file = "/etc/lotse/ca.pem" to the cluster entry. The console passes the signed-in user’s identity on to the cluster, so each user sees only what their roles allow (ADR 0040). In the dev mode used here the console acts as the one configured user; use it only on a machine only you can reach.

With make dev you can skip all this: the console is already running at http://127.0.0.1:8080 with the cluster local, and your Java client can connect to the client addresses in the log line three nodes running, because that cluster picks free ports at random.