Are you an LLM? Read llms.txt for a summary of the docs, or llms-full.txt for the full context.
Skip to content

Backend Ops Notes

Setup is covered by the per-backend overview pages. This page covers what tends to bite in production: which knobs to set before you go live, which surprises to anticipate, and which monitoring to put in place. One section per backend.

RabbitMQ

Management plugin required for autoscaler. The autoscaler queries the RabbitMQ HTTP management API (/api/queues) for queue depth. Without the plugin, the autoscaler errors at startup. Enable it with:

rabbitmq-plugins enable rabbitmq_management

The management API is available on port 15672 by default. If you run RabbitMQ behind a network policy that blocks that port, the autoscaler will not start.

Auto-reconnect is automatic at the consumer level. Transient broker failures — network blips, broker restarts — recover without operator intervention. The consumer reconnects and resumes processing. Don't add wrapping retry loops at the application level; they interfere with shove's internal reconnect logic and can cause duplicate deliveries.

TLS via rustls-ring is the default lapin feature used by shove. For environments that require OpenSSL (e.g. FIPS-compliant infrastructure) or vendored alternatives, configure the TLS backend downstream in your own Cargo.toml feature selection.

Cluster setups. Point RabbitMqConfig at a load-balanced cluster URL (e.g. via HAProxy or a DNS-based round-robin). shove uses a single AMQP connection per Broker\<RabbitMq\>. For HA with automatic failover, use a clustered broker with quorum queues — the connection layer handles reconnect, but the queue must be replicated to survive a node failure.

Hold queue dead-letter exchange (DLX) is configured at topology declaration. shove sets queue arguments (including x-dead-letter-exchange and x-message-ttl) when it declares hold queues. RabbitMQ does not allow modifying queue arguments in place after creation. If you need to change hold-queue delays, you must delete and recreate the queues — or use a different queue name to force a fresh declaration.

AWS SNS + SQS

Visibility timeout MUST exceed handler timeout. If your handler can take 30 s, set the SQS visibility timeout to at least 60 s. If the handler is still running when the visibility timeout expires, the message becomes visible again and another consumer picks it up — producing duplicate work. Configure via the SQS console, Terraform, or CloudFormation:

VisibilityTimeout: 60   # seconds — set to 2× your worst-case handler time

FIFO queues have a 300 msg/s default cap. High-throughput sequenced topics that need sequenced delivery via FIFO will hit this limit before exhausting hardware capacity. Request a quota increase via AWS Support, or switch to standard SQS queues (no ordering guarantee) if ordering is not required.

endpoint_url is for local dev only. The endpoint_url field on SnsConfig points shove at a custom SQS/SNS endpoint (typically LocalStack on http://localhost:4566). Omit it in production, or you will point all traffic at localhost.

Region matters. SQS queues are regional. Consuming a queue in a different region than your application's default region requires explicit configuration. Cross-region fanout is handled at the SNS topic ARN level, not in shove.

Dead-letter queues are wired at topology-declaration time. When you declare a topic with .dlq(), shove creates the DLQ queue (or its FIFO equivalent for sequenced topics), attaches a RedrivePolicy with maxReceiveCount set to shove's default (DEFAULT_MAX_RECEIVE_COUNT), and points the main queue at it. You don't need to provision the DLQ separately in Terraform/CloudFormation. If you want a different maxReceiveCount or a pre-existing DLQ ARN, configure it via the SQS console or IaC tooling AFTER declare runs — but the default wiring is automatic.

IAM permissions. The consumer role needs the following permissions on each queue it reads from:

sqs:ReceiveMessage
sqs:DeleteMessage
sqs:ChangeMessageVisibility
sqs:GetQueueUrl
sqs:GetQueueAttributes

If the same role publishes, add sns:Publish on each SNS topic it publishes to.

NATS JetStream

Stream provisioning happens at topology declaration. When declare_topology runs, shove creates the JetStream stream if it does not already exist. For policy-driven setups — limits-based retention, interest-based retention, work-queue retention — pre-create the stream with the correct policy before starting the application. shove will detect the existing stream and use it as-is without overwriting configuration.

Nats-Msg-Id dedup window is 120 s. shove stamps this header on every publish (a fresh UUID per call). JetStream de-duplicates within the stream's duplicate_window (default 120 seconds), so a publisher that retries the same logical message inside that window will not produce a duplicate delivery. If you need deduplication across application-level retries, pass a stable id via publish_with_headers so each retry sets the same Nats-Msg-Id. If your handler routinely takes longer than 120 s, raise the stream's duplicate_window to match:

duplicate_window: 600s  # example: 10-minute dedup window

Stream-side vs consumer-side limits. Storage size and per-message size are set on the JetStream stream itself (configured outside shove). max_ack_pending, however, is set by shove on the durable pull consumer at consumer-creation time, derived from prefetch_count (and from max_consumers × prefetch_count for groups). To override per-consumer, use ConsumerOptions::<Nats>::with_max_ack_pending(...).

Subject naming. shove uses {queue}.shard.{N} subjects for sequenced topics. Do not reuse these subjects for other applications or publishers outside of shove — you will interfere with sequencing semantics.

Consumer durability. shove uses durable pull consumers. The consumer name is derived from the topic and group configuration. On restart, the NATS server remembers the consumer's last acknowledged sequence number, so processing resumes where it left off rather than re-reading from the start of the stream.

Apache Kafka

Partition count is set at provisioning time and is rarely changed after the fact. It caps the maximum group concurrency: a 4-partition topic can have at most 4 active consumers in a consumer group simultaneously. Over-partitioning wastes broker resources; under-partitioning limits scale. Plan partition count for peak load plus headroom before creating the topic.

Consumer-group rebalances during scale events cause brief delivery pauses. When a consumer joins or leaves the group — including during autoscaling — Kafka triggers a group rebalance, during which no consumer in the group processes messages. Rebalances typically complete in under 1 s, but aggressive autoscaling that adds and removes consumers rapidly can cause near-continuous rebalance churn. Set scale cooldowns to allow the group to stabilize between scale events.

cmake is required on Windows. The rdkafka crate links librdkafka, which requires cmake during the build on Windows. Linux and macOS use the standard librdkafka build path without additional prerequisites.

SASL/TLS. Enable the kafka-ssl feature for TLS transport. KafkaSasl configuration supports SCRAM-SHA-256 and SCRAM-SHA-512. GSSAPI/Kerberos requires activating the rdkafka/gssapi feature downstream in your own Cargo.toml. For Amazon MSK with IAM authentication, enable the kafka-msk-iam feature instead — see the Kafka backend page for setup details.

librdkafka system dependency. On Debian/Ubuntu CI runners, install the SASL development headers before building:

apt-get install -y libsasl2-dev

On macOS, the dependency is typically satisfied via Homebrew. librdkafka is a runtime dependency — it is not statically bundled.

Consumer offsets are committed asynchronously. librdkafka commits offsets in the background. On clean shutdown, shove flushes pending commits before exiting. On a crash or SIGKILL, expect at-least-once redelivery from the last committed offset when the consumer restarts. Design handlers to be idempotent or use the dedup layer to handle this.

Redis / Valkey Streams

Redis 6.2 or newer is required. shove uses ZRANGE … BYSCORE for hold-queue polling, a command introduced in Redis 6.2. The version is checked at connection time — if the server is older, Broker::new returns an error with a clear message rather than failing later with a cryptic protocol error. Any Valkey release is compatible (Valkey reports a 7.x-equivalent redis_version for backwards compatibility).

Standalone, TLS, and cluster modes are all supported. Use RedisMode::Standalone with a redis:// or rediss:// URL for single-node and Sentinel setups. Use RedisMode::Cluster with a list of seed node URLs for Redis Cluster. shove opens one multiplexed connection for non-blocking operations (XADD, XACK, ZADD) and a separate dedicated connection per consumer loop for blocking reads (XREADGROUP with BLOCK).

Hold queues use Sorted Sets, not Streams. Messages waiting for delayed retry are stored in a Redis Sorted Set with the redeliver-at timestamp as the score. A background requeuer task polls the set on each node and XADDs due entries back to the appropriate shard stream. If two requeuer instances run concurrently (e.g. during a rolling restart), the same entry may be XADDed twice — this is expected at-least-once behaviour; handlers must be idempotent.

Consumer groups are created automatically. At topology declaration, shove creates an XGROUP on each shard stream if one does not already exist. The group name defaults to "shove" and can be overridden via RedisConfig::group. Do not reuse the same group name across unrelated applications that share a Redis instance.

Stream memory is bounded by outstanding work. While consumers run, shove trims entries that every consumer group on the stream has acknowledged, so a stream's size tracks its backlog rather than its lifetime throughput — no external XTRIM policy is needed. Trimming waits for the slowest group, never touches pending (in-flight) entries, and never touches DLQ streams. The same background task reclaims entries left pending past the handler timeout (e.g. after a consumer crash) and redelivers them; keep the handler timeout consistent across every consumer of a stream and group, including other processes.

In-process

No durability. Messages live entirely in process memory. Process exit — whether clean or via a crash — means message loss. State this explicitly on every internal review of "should we use the in-process backend in production?" — the answer is almost always no for distributed or stateful workloads.

No cross-process delivery. Only consumers running in the same OS process see messages published to an in-process broker. There is no IPC, no network transport, no persistence layer. Two separate service instances cannot share an in-process topic.

Use cases: integration tests (fast, no broker required), single-process CLI tools that use pub/sub internally for decoupling, and prototyping before wiring up a real broker. Not suitable for distributed services or anything that must survive process restart.

Memory footprint. Each topic holds undelivered messages in memory. A publisher that outpaces its consumers will grow the backlog without bound. shove does not enforce backpressure on publishers in the in-process backend — if this is a concern, add an external rate limiter or switch to a broker-backed backend.

What's next