Troubleshooting

Common failure modes and how to read them. When something “silently does nothing”, the cause is almost always one of the drop/drain behaviours below.

Messages disappear and the route still ends completed

By design, a message that fails permanently (a type/data error the sink rejects, a poison payload a handler rejects) is logged at error level and dropped — the route keeps processing the rest of the batch. So a systematic failure (e.g. every row hitting a column-type mismatch) drains the input while committing nothing and still ends completed.

• Watch for a burst of Dropping message … due to non-retryable error in the logs — that’s the signal.
• Add a dlq to capture the failures for inspection/replay.
• retry alone does not retain a failed message — pair it with dlq (dlq last). See Dead-letter queues.

--drain / exit_on_empty behaviour

• --drain (CLI) / exit_on_empty (MCP/route) exits once the source yields an empty batch — right for finite sources (a file, a full-table read). For brokers and CDC sources that never “end”, omit it and run continuously.
• A drained Redis Streams source reports an error at the end: "healthy": false with Redis XREAD failed: timed out, even though every message moved. Cosmetic — verify the sink, not the final status. NATS does not behave this way.
• Finished MCP routes are not reaped: a route started with exit_on_empty stays in list_routes after draining. Check "finished" / "outcome" on the entry, or call stop_route to clear it. See MCP known limitations.

IPC: publisher fails to connect or blocks

The memory endpoint’s IPC transport is unidirectional, publisher → consumer, and the consumer is the server:

• The consumer must be running before the publisher connects — otherwise the publisher fails with a connection error. Start the consumer-side route first.
• send_batch blocking is normal backpressure, not a fault: socket buffers are small (8 KiB on macOS), so a batch larger than the buffer only completes once the consumer drains it. After 5 s blocked the publisher logs a warning naming the socket and keeps waiting.
• A consumer that has accepted but stopped draining stalls the publisher indefinitely — make sure the consumer side is actually reading.
• Named ipc://name resolves to different paths for different users/services (the /run → $XDG_RUNTIME_DIR → /tmp fallback). When both sides must agree, use an explicit path (ipc:///run/myapp/queue.sock).
• IPC redelivery is consumer-local and does not survive a consumer crash; use a real broker for durable redelivery. See Cross-process IPC bridge.

Request/reply timeouts and dropped responses

• The response output requires an input that carries a reply channel (http, websocket, grpc, or a request/reply nats/mongodb/memory). If the input does not support responses (File, SQLx, …), the message sent to response is dropped.
• Configure a generous timeout on the requester side — bridge processing adds latency.
• Middleware that drops metadata (like correlation_id) breaks the response chain. Keep the correlation metadata intact through the route.
• MongoDB’s reply pattern uses emulation that waits for messages; misconfigured timeouts can cause severe stalls. Test it before relying on it.

Ordering looks wrong

With concurrency > 1, batches process in parallel, so strict per-key ordering across the route is not guaranteed — side effects at the sink can interleave. Commits are still sequenced for cumulative-ack brokers. If you need ordering, set concurrency: 1 or partition upstream so each key lands on one route. See Performance tuning.

NATS JetStream: “no stream found for given subject”

When mq-bridge auto-creates a JetStream stream it scopes it to {stream}.>, so the subject must be prefixed with the stream name: stream: "orders_stream" needs a subject like orders_stream.foo. Also, stream is required even in Core NATS mode (no_jetstream: true) — pass any placeholder string. See NATS JetStream notes.

SQLx source: “reconnecting forever” or rejects the table

An SQLx source with no cursor_column is a competing-consumers work queue and requires the queue schema (id, payload, locked_until). A plain table without locked_until fails fast with a permanent error. To read an arbitrary table non-destructively, set cursor_column to switch to cursor polling. See Endpoints.

Encrypted / compressed file source ends immediately or failed

A file source must declare the same compression/encryption the data was written with. A mismatch (wrong key, wrong codec, missing field) is a permanent decode failure — the route ends failed with the error in its status. Reading a compressed file with no compression set is rejected up front by magic-byte sniffing. See Encryption at rest.

Postgres bulk copy is slow / near-quadratic

The Postgres keyset-cursor reader does WHERE id > $cursor ORDER BY id LIMIT batch. Without an index on the cursor column it does a full scan per batch. Create one: CREATE INDEX ON <table>(id). See Performance tuning.

Throughput numbers look wrong

Measure on a release build — a debug build reports dramatically slower rates. Through the MCP server, call server_info first to confirm the build profile. See Measuring.

Float values change in the last bit

Numbers move through payloads as JSON, and serde_json’s default parser shifts ~1 ULP on ~19% of 17-significant-digit doubles. For bit-exact float round-tripping across every endpoint, build with the float-roundtrip feature. See middleware encryption notes.