A first cluster
This chapter sets up three nodes with TLS 1.3. Every node reads the same cluster file. Streams, queues and exchanges are not in the file; clients create them.
The cluster file
Copy the example to /etc/lotse/cluster.toml and change the addresses. The Debian package ships it as /usr/share/doc/lotse/cluster.toml.example.
# A Lotse cluster file (ADR 0047): the same on every node. Copy it to
# /etc/lotse/cluster.toml, name your nodes, make the certificates with
#
# lotse certs --config /etc/lotse/cluster.toml
#
# and start each node with `systemctl enable --now lotse@<id>`.
[cluster]
data_dir = "/var/lib/lotse"
[[node]]
id = 1
peer = "10.0.0.1:7000"
client = "10.0.0.1:9000"
metrics = "10.0.0.1:9464" # /metrics, /healthz, /readyz, /safe-to-stop (ADR 0117); leave out to serve none
[[node]]
id = 2
peer = "10.0.0.2:7000"
client = "10.0.0.2:9000"
metrics = "10.0.0.2:9464"
[[node]]
id = 3
peer = "10.0.0.3:7000"
client = "10.0.0.3:9000"
metrics = "10.0.0.3:9464"
[tls]
ca = "/etc/lotse/ca.pem"
cert = "/etc/lotse/node-{id}.pem"
key = "/etc/lotse/node-{id}.key"
crl = "/etc/lotse/crl.pem" # `lotse certs --revoke <cert>` adds to it; same file on every node
[retention]
ms = 604800000 # 7 days; -1 keeps everything
[auth]
admins = ["admin"] # signs in with `lotse certs --client admin`
What the keys mean:
| Key | Meaning |
|---|---|
[cluster] data_dir | where a node keeps its data. {id} becomes the node’s id, so /var/lib/lotse/node-{id} gives each node its own directory on a shared machine. |
[[node]] id | a number from 1 up, unique. |
[[node]] peer | host:port where the other nodes reach this node. |
[[node]] client | host:port where clients reach this node. |
[tls] ca, cert, key | the CA certificate and the node’s certificate and key. {id} is replaced as in data_dir. |
[tls] crl | the revocation list. Optional. |
[retention] ms | how long streams keep records, in milliseconds. -1 keeps everything. The default is 7 days. |
[auth] admins | the principals that may create tenants and grant roles. Required while auth.required is on, which is the default. |
The file is checked strictly: an unknown key is an error. Administration will list every key; this guide uses only those above.
Check the file
lotse check --config /etc/lotse/cluster.toml
For the example it prints:
3 nodes: 1, 2, 3
TLS: certificates from [tls]
retention: 604800000 ms by age, no limit per partition
clients sign in with: certificate, SCRAM-SHA-512; admins: admin
Once certificates exist, lotse check also names any that expire within 30 days.
Make the certificates
lotse certs --config /etc/lotse/cluster.toml
lotse certs --config /etc/lotse/cluster.toml --client admin --out ./admin
The first command writes a CA (ca.pem, ca.key), a certificate and key per node, and an empty revocation list where [tls] names them. Node certificates last one year, the CA ten; --days sets another length. lotse certs keeps files that exist, so you can run it again after you add a node to the file.
The second command writes a client certificate ./admin/admin.pem and key ./admin/admin.key for the principal admin, the name auth.admins lists. Use this certificate to sign in as the first administrator.
ca.key signs every certificate of the cluster. Move it off the nodes after you have made the certificates. Copy ca.pem, crl.pem and each node’s own node-<id>.pem and node-<id>.key to its machine, readable by the user that runs lotse.
Start the nodes
On each machine, with its own id:
lotse run --config /etc/lotse/cluster.toml --node 1
A node stops cleanly on Ctrl-C or SIGTERM. LOTSE_LOG sets the log filter, for example LOTSE_LOG=debug; the default is info.
Run under systemd
The Debian package installs lotse@.service. The text after @ is the node’s id. The unit reads /etc/lotse/cluster.toml, runs as the user lotse and restarts the node on failure.
sudo install -d /etc/lotse
sudo cp /usr/share/doc/lotse/cluster.toml.example /etc/lotse/cluster.toml # then edit it
sudo lotse certs --config /etc/lotse/cluster.toml
sudo chown -R lotse: /etc/lotse
sudo systemctl enable --now lotse@1
Run the last command on each machine with its own id (lotse@2, lotse@3). Keep data_dir at /var/lib/lotse or below: the unit lets the node write nowhere else. Follow a node with journalctl -u lotse@1 -f.
This guide ran lotse check and lotse certs on the example file’s shape and started three nodes on one machine (the trial). It did not run the systemd unit or a cluster across three machines.
Next
A first message sends and reads a record, and shows the cluster in the console. Certificates, renewal, revocation, authentication, roles, and adding and draining nodes belong to the administration guide.