Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

The cluster file

One TOML file describes the whole cluster. It is the same file on every node; a node learns which entry is its own from --node (or LOTSE_NODE). Streams, queues and exchanges are not in the file: clients create them at runtime (ADR 0047).

Unknown keys are refused. So is a file that could not run: no [[node]], a duplicate id or address, missing certificates, plain TCP off loopback, authentication without TLS, or authentication required with no admin. lotse check --config cluster.toml reads the file and says what it describes.

Example

[cluster]
data_dir = "/var/lib/lotse"

[[node]]
id = 1
peer = "10.0.0.1:7000"
client = "10.0.0.1:9000"

[[node]]
id = 2
peer = "10.0.0.2:7000"
client = "10.0.0.2:9000"

[[node]]
id = 3
peer = "10.0.0.3:7000"
client = "10.0.0.3:9000"

[tls]
ca = "/etc/lotse/ca.pem"
cert = "/etc/lotse/node-{id}.pem"
key = "/etc/lotse/node-{id}.key"
crl = "/etc/lotse/crl.pem"

[retention]
ms = 604800000

[auth]
admins = ["admin"]

The Debian package installs the same example as /usr/share/doc/lotse/cluster.toml.example.

{id} in paths

In cluster.data_dir, tls.cert and tls.key, the text {id} becomes the node’s id. Several nodes can then share one machine and one file. tls.ca and tls.crl are taken as written.

[cluster]

KeyType, unitDefaultMeaning
data_dirstring, requirednoneWhere a node keeps its data.
tick_msinteger, ms10The Raft tick. Must be at least 1.
segment_bytesinteger, bytes134217728 (128 MiB)Log segment size. Retention deletes whole segments. Must be at least 4096.
group_session_timeout_msinteger, ms45000A consumer group member without a heartbeat for this long is removed.
group_heartbeat_msinteger, ms5000How often group members send heartbeats.
follower_max_lag_msinteger, ms10000How long a follower may go without having applied what its leader said was committed and still serve reads to clients in its zone; after that it sends them to the leader, like Kafka’s replica.lag.time.max.ms (ADR 0116).

[[node]]

One entry per node, at least one.

KeyTypeDefaultMeaning
idinteger, requirednoneThe node’s id, 1 or more, unique.
peerhost:port, requirednoneThe address other nodes dial.
clienthost:port, requirednoneThe address clients dial.
peer_bindhost:portthe peer addressWhere the node listens for nodes, if not on peer, such as 0.0.0.0:7000.
client_bindhost:portthe client addressWhere the node listens for clients, if not on client.
zonestringnoneThe node’s zone (a data centre or availability zone). Clients that name the same zone fetch from its replicas (ADR 0104). A node added later takes it with lotse node add --zone.
metricshost:portnoneWhere the node serves /metrics, /healthz, /readyz and /safe-to-stop over HTTP (ADR 0117). Plain HTTP on a loopback address, TLS with the node’s certificate anywhere else. Left out, nothing listens. See Monitoring.

Every peer, client and metrics address must be unique in the file.

[tls]

Either name ca, cert and key, or set plaintext_on_loopback. Naming both is refused. See Certificates.

KeyTypeDefaultMeaning
capathnoneThe CA certificate, PEM.
certpath with {id}noneThe node’s certificate chain, PEM.
keypath with {id}noneThe node’s private key, PEM.
crlpathnoneThe certificate revocation list, PEM. Optional. A certificate it names is refused in every handshake (ADR 0060).
renewal_capathnoneThe renewal CA’s certificate, PEM, made by lotse certs --renewal-ca. Optional. With it, a client certificate is renewed over a connection made with it (ADR 0123). Not with plaintext_on_loopback.
renewal_ca_keypathrenewal_ca ending in .keyThe renewal CA’s private key, PEM. Only with renewal_ca.
plaintext_on_loopbackbooleanfalseNo TLS. Allowed only when every address, including peer_bind and client_bind, is on loopback. Needs auth.required = false. For trying a cluster on one machine.

Plain TCP is refused off loopback (ADR 0037).

[retention]

What streams keep unless they say otherwise. A stream created at runtime can override them; see Retention and compaction.

KeyType, unitDefaultMeaning
msinteger, ms604800000 (7 days)Records older than this go. -1 for no limit.
bytesinteger, bytesno limitEach partition keeps at least this much. -1 for no limit.

Any other value must be positive.

[compaction]

How compacted streams are compacted; see Retention and compaction.

KeyType, unitDefaultMeaning
map_bytesinteger, bytes67108864 (64 MiB)Memory the key map of the node’s one compaction sweep may take, 48 bytes per key.

[disk]

How much free space each node keeps on its data volume (ADR 0122).

KeyType, unitDefaultMeaning
free_limitbytes or a sharethe larger of 1 GiB and 5 %Below this much free space a node takes no new data in the groups it votes in. Bytes as a number (2147483648) or text with a unit ("2GiB", "500MB"; KB, MB, GB, TB, KiB, MiB, GiB, TiB), a share of the volume ("5%", "2.5%"), or 0 for no limit.
check_interval_msinteger1000How often a node reads its volume’s free space, 10 to 60000. A node that crosses the limit reports it at its next read (ADR 0124).

A node reads its volume every check_interval_ms. Below the limit it is low: the leaders of every group with a replica on it refuse produces, publishes and dead letters with STORAGE_FULL, which clients retry; reads, acks and metadata changes go on. It is no longer low once its free space is a tenth above the limit. The node tells the metadata group at once, so other nodes learn of it within a few seconds; the node itself refuses at once for the groups it leads. See Nodes.

[producers]

How each partition keeps idempotent producers (ADR 0110). Use the same values on every node.

KeyType, unitDefaultMeaning
maxinteger10000Producers a partition admits. While it knows this many, a new producer is refused with PRODUCER_LIMIT and its client retries. A known producer is never refused or forgotten to make room.
expiration_msinteger, ms86400000 (a day)How long a producer may write nothing to a partition before its leader forgets it. A retry after that is taken as a new batch.

Both must be at least 1.

[auth]

KeyType, unitDefaultMeaning
requiredbooleantrueClients must authenticate. With false, everyone may do everything.
mtlsbooleantrueA client certificate names its principal.
scrambooleantrueSASL SCRAM-SHA-512, with users set at runtime.
adminslist of stringsemptyPrincipals that are cluster admins from the start. Must not be empty while required is true.
tenantslist of stringsemptyTenants besides default that exist from the start.
max_session_sinteger, s3600A session ends after this at the latest.

With required true, at least one of mtls, scram and oidc must be on, and TLS is required: nodes prove who they are by their certificates.

[auth.oidc]

Present only when tokens from an identity provider are accepted (SASL OAUTHBEARER).

KeyTypeDefaultMeaning
issuerstring, requirednoneThe issuer, as in the tokens’ iss.
audiencestring, requirednoneThe audience tokens must be for.
principal_claimstringsubThe claim that names the principal.
groups_claimstringgroupsThe claim that lists the user’s groups.
jwks_urlURLthe issuer’s discovery documentThe key set’s URL.

See Authentication and roles.

Environment variables

VariableUsed byMeaning
LOTSE_CONFIGevery command that reads the fileThe cluster file, when --config is not given.
LOTSE_NODElotse runThe node’s id, when --node is not given.
LOTSE_JOINlotse run1 or true makes the node join a running cluster, like --join.
LOTSE_LOGlotse runThe log filter, in the syntax of the tracing env filter. Default info.
LOTSE_PASSWORDlotse nodeThe password for --user.

A flag wins over the variable. The systemd unit and the container image use these variables (ADR 0049).

Commands

lotse run   --config <cluster.toml> --node <id> [--join]
lotse certs --config <cluster.toml> [--days <n>]
lotse certs --config <cluster.toml> --client <principal> --out <dir> [--days <n>]
lotse certs --config <cluster.toml> --revoke <cert.pem>
lotse check --config <cluster.toml>
lotse node add    --config <cluster.toml> --node <id> [--peer <host:port> --client-addr <host:port>] <admin>
lotse node remove --config <cluster.toml> --node <id> <admin>
lotse node list   --config <cluster.toml> <admin>
lotse --version

<admin> is --cert <file> --key <file> (a client certificate) or --user <name> with the password in LOTSE_PASSWORD.

lotse run stops cleanly on Ctrl-C or SIGTERM, after handing the groups its node leads to other voters (at most 30 s). lotse check prints the node count, the TLS mode, the certificates on this machine that expire within 30 days, the retention defaults and the sign-in methods.