Queues
A queue delivers each message to one consumer at a time and keeps it until a consumer acknowledges it. Use it for work that must be done once by one of several workers, and retried when a worker fails.
A work queue with a dead-letter queue
The example creates two queues, writes five jobs, works them off, and lets one job fail until the queue retires it.
package dev.lotse.examples;
import dev.lotse.client.ClientException;
import dev.lotse.client.DeadLetter;
import dev.lotse.client.Delivery;
import dev.lotse.client.LotseClient;
import dev.lotse.client.OutMessage;
import dev.lotse.client.QueueConsumer;
import dev.lotse.client.QueueOptions;
import dev.lotse.client.SubscribeOptions;
import java.time.Duration;
/**
* A work queue with a dead-letter queue: jobs that fail are tried again after a delay, and a job that fails three
* times moves to the dead-letter queue.
*/
public final class QueueExample {
private QueueExample() {}
public static void main(String[] args) {
try (LotseClient client = new LotseClient(Env.config().build())) {
createIfMissing(client, "example-jobs-dead", QueueOptions.defaults());
createIfMissing(client, "example-jobs", QueueOptions.builder()
.maxDeliveries(3)
.deadLetterQueue("example-jobs-dead")
.ackTimeout(Duration.ofSeconds(30))
.build());
for (int i = 0; i < 5; i++) {
client.produce("example-jobs", 0, OutMessage.value("job-" + i));
}
try (QueueConsumer jobs = client.subscribe("example-jobs", SubscribeOptions.withCredit(16))) {
while (true) {
Delivery job = jobs.poll(Duration.ofSeconds(5));
if (job == null) {
break; // nothing for 5 s: done
}
String name = job.value().toUtf8();
if (name.equals("job-3")) {
// Fails every time: back in one second, until max.deliveries retires it.
System.out.printf("%s failed (delivery %d)%n", name, job.deliveryCount());
jobs.nackAfter(job, Duration.ofSeconds(1));
continue;
}
System.out.println("done " + name);
jobs.ack(job);
}
}
// The retired job, with where it came from and why.
try (QueueConsumer dead = client.subscribe("example-jobs-dead")) {
Delivery job = dead.poll(Duration.ofSeconds(5));
if (job != null) {
System.out.printf("dead letter %s: %s after %d deliveries%n", job.value().toUtf8(),
job.deadLetter().map(DeadLetter::reason).orElse("?"),
job.deadLetter().map(DeadLetter::deliveries).orElse(0));
dead.ack(job);
}
}
}
}
private static void createIfMissing(LotseClient client, String name, QueueOptions options) {
try {
client.createQueue(name, options);
} catch (ClientException.ResourceExists e) {
// made by an earlier run
}
}
}
Points to note:
- You write to a queue with
produce(queue, 0, message): a queue is partition 0. Messages can also arrive through an exchange. subscribe(queue, SubscribeOptions.withCredit(16))opens a subscription. Credit is how many messages the broker may have out with you at once. Each ack gives credit back; the default is 64.poll(wait)returns the next delivery, ornullafterwait.next()waits without limit.ack(delivery)settles a message.nack(delivery, true)gives it back to be delivered again;nack(delivery, false)discards it.nackAfter(delivery, delay)gives it back after a delay, for a retry with backoff.delivery.deliveryCount()says how many times the broker has sent the message.- A message you neither ack nor nack goes out again when the ack timeout passes (
ack.timeout.ms) or when you close the consumer. After a leader change or a broken connection the consumer subscribes again and keeps its messages, also within a transaction; if it does not come back within the resume window (resume.window.ms, 30 s by default), they go to others (ADR 0109). Process messages so that doing one twice is harmless, or use a transaction.
Settings
Set at creation with QueueOptions, or later with alterConfigs(name, set, reset). All limits are off by default. describeConfigs(name) reads them.
| Key | Meaning | Values | QueueOptions |
|---|---|---|---|
ack.timeout.ms | Time a delivered message may stay unacknowledged | Milliseconds above 0 | ackTimeout |
resume.window.ms | After a leader change, how long a consumer’s messages wait for it to subscribe again | Milliseconds above 0; default 30,000 | alterConfigs |
max.deliveries | A message goes out at most this often; the next time it is retired | 1 to 65,535 | maxDeliveries |
message.ttl.ms | A message not delivered within this time after the broker stored it is retired | Milliseconds above 0 | messageTtl |
dead.letter.queue | Where retired messages go | Name of a queue in the same tenant | deadLetterQueue |
single.active.consumer | Only one subscription gets messages | true or false | alterConfigs |
priority.levels | Number of priority levels | 1 to 10; 1 means none | alterConfigs |
max.length | Messages that may wait for delivery | Messages above 0 | maxLength |
max.length.bytes | Bytes of keys, values and headers that may wait for delivery | Bytes above 0 | maxLengthBytes |
overflow | What a full queue does | reject-publish (default) or drop-head | overflow |
resume.window.ms and the last two are set with alterConfigs in the Java SDK:
// Not from the compiled examples.
client.alterConfigs("example-jobs", Map.of("single.active.consumer", "true", "priority.levels", "3"), List.of());
Changing settings, purging and requeueing need the tenant-admin role on the queue.
Retries and dead letters
A delivery attempt is a delivery that did not end in an ack: a nack with requeue, an ack timeout, or a lost connection. With max.deliveries set, the broker retires a message when it would otherwise go out once more. With message.ttl.ms set, it retires a message that waited too long. A message a consumer holds does not expire.
Retired messages move to the dead.letter.queue if there is one. Without one, they are dropped. Set a dead-letter queue when messages matter.
The move happens once, also across failovers. The copy keeps key, value and headers, and gets these headers, which delivery.deadLetter() reads:
| Header | Content |
|---|---|
x-lotse-dead-letter-reason | max-deliveries, expired, requeued or max-length |
x-lotse-dead-letter-queue | The queue it came from |
x-lotse-dead-letter-queue-id | That queue’s id |
x-lotse-dead-letter-offset | Its offset there |
x-lotse-dead-letter-deliveries | Deliveries before it was retired |
x-lotse-dead-letter-floor | The source queue’s retention floor when it moved |
A queue cannot be its own dead-letter queue, a chain of dead-letter queues that leads back is refused, and a queue that is someone’s dead-letter queue cannot be deleted. Delivery counts are kept in the queue’s log, so they go on across failovers (ADR 0109). See ADR 0055.
Length limits
A queue may limit what waits for delivery (ADR 0118): max.length messages, max.length.bytes bytes of keys, values and headers, or both. Every message a consumer could still get counts, also one not yet committed or in an open transaction. A message out with a consumer does not count, as in RabbitMQ, so a consumer with a large credit lets the queue hold more.
What a full queue does is its overflow:
reject-publish, the default: a produce, a publish or a transaction’s write that would pass the limit fails withQUEUE_FULL, and nothing is written. The client does not retry it; try again once consumers have made room. A dead letter for a full queue waits in its source queue until there is room.drop-head: the queue takes every write and retires its oldest waiting messages until it fits, into its dead-letter queue with the reasonmax-lengthif it has one. Without one, they are dropped.
If the answer to an earlier attempt was lost, QUEUE_FULL does not say whether that attempt was written. The client then reports the outcome as unknown.
The limits can change at any time with alterConfigs; a queue already over a new limit refuses, or drops down to it. After a leader change, the new leader reads the queue’s backlog first and may refuse while it does.
// Not from the compiled examples.
client.createQueue("orders", QueueOptions.builder().maxLength(100_000).maxLengthBytes(1L << 30).build());
client.alterConfigs("orders", Map.of("overflow", "drop-head"), List.of());
Purge and requeue
purge(queue) settles every message that is ready: not held by a consumer, not in an open transaction. Messages written while it runs are left alone. It returns the number purged.
requeue(deadLetterQueue) moves the ready messages of a dead-letter queue back to the queue named in each message’s dead-letter headers, once each. A message without those headers, or whose queue is gone, stays. It returns the number moved. The copy gets the reason requeued. See ADR 0064. The console has both as buttons.
Delayed delivery
Two ways to make a message wait (ADR 0077):
- On publish, set the header
x-lotse-delay-msto a decimal number of milliseconds, 0 to 4,294,967,295 (about 49 days). The delay counts from the time the queue’s leader stored the message, so the producer’s clock does not matter. Any other value means no delay. - On nack,
nackAfter(delivery, delay). It gives the message back to any consumer once the delay has passed.
// Not from the compiled examples: a reminder in ten minutes.
client.produce("reminders", 0,
OutMessage.value("call back").withHeader("x-lotse-delay-ms", "600000"));
A delayed message never blocks the queue: others keep flowing, and the delayed one is delivered first once it is due. A dead-lettered or requeued copy does not wait again.
Priorities
With priority.levels from 2 to 10, set the header x-lotse-priority to a decimal number from 0. Higher goes first. A value above the queue’s highest level counts as the highest; anything that is not a number from 0 to 255 is level 0. Within a level, the oldest message goes first. Dead letters and requeued messages are level 0 (ADR 0078). The console’s queue page and DescribeMetrics show how many messages are ready at each level; after a failover, messages whose level the new leader has not read yet are counted apart until it has. They also show which subscription of a single active consumer queue is active and which ones are standbys (ADR 0120).
// Not from the compiled examples.
client.produce("jobs", 0, OutMessage.value("urgent").withHeader("x-lotse-priority", "2"));
Single active consumer
With single.active.consumer = true, one subscription gets all messages and the others wait as standbys in the order they subscribed. consumer.isStandby() says whether yours still waits. When the active one ends, the oldest standby takes over only once nothing is out with anyone, so the queue’s order holds across the handover.
This holds per leader term. After a leader change, a deposed leader that has not noticed may still deliver to the old active consumer for a short time; that consumer’s acks then fail (ADR 0076).
Kotlin
lotse.subscribe(queue) returns a LotseQueue with ack, nack, nackAfter as suspend functions, and deliveries() as a Flow:
// Not from the compiled examples.
lotse.subscribe("example-jobs").use { jobs ->
jobs.deliveries().collect { job ->
process(job.value().toUtf8())
jobs.ack(job)
}
}
Errors you may meet
| Error | When |
|---|---|
ClientException.ResourceExists | The queue name is taken in your tenant |
ClientException.Broker with INVALID_REQUEST | A setting is out of range, or would make a dead-letter loop |
ClientException.Broker with MESSAGE_NOT_HELD | You acked a message inside a transaction that is not in flight with your subscription, or another transaction holds it. Abort |
ClientException.Broker with UNAUTHORIZED | Your roles do not allow it |