Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

How to test Kafka consumers and message handlers without a broker

Build the route from endpoints passed in by the caller: Kafka (or NATS, RabbitMQ, MQTT) in production, mq-bridge’s in-process memory endpoint in tests. The handler, the route and the middleware under test are the production code; only the transport is swapped, so the tests need no Docker and wait for no broker to start.

Runnable example: examples/test-without-a-broker.

The problem: slow, flaky or shallow tests for messaging code

Code that consumes from a broker is awkward to test:

  • A real broker in every test run (Docker Compose or Testcontainers) costs seconds to minutes of startup, needs Docker on every developer machine and CI runner, and fails for reasons that have nothing to do with the change under test.
  • Mocking the client library tests the mock. Hand-written fakes of a Kafka consumer drift from the real API and skip the code between the client and the handler: deserialization, routing by message type, acknowledgement.
  • Testing the handler function alone is fast but leaves that same glue untested.

The solution: make the transport a parameter

flowchart LR
  subgraph Production
    K1[[Kafka topic]] --> R1[route + handler] --> K2[[Kafka topic]]
  end
  subgraph Test
    M1[memory channel] --> R2[same route + handler] --> M2[memory channel]
  end

A route is an input endpoint, an output endpoint and an optional handler. The handler works with mq-bridge’s message type and never sees a Kafka record, so the same route runs over any endpoint. In production the two endpoints are a broker:

orders:
  input:
    kafka: { url: "localhost:9092", topic: "orders.in", group_id: "orders" }
  output:
    kafka: { url: "localhost:9092", topic: "orders.out" }

In a test they are in-process channels, which need no Cargo feature and no service:

orders:
  input:
    memory: { topic: "orders.in", capacity: 100 }
  output:
    memory: { topic: "orders.out", capacity: 100 }

Rust

The route takes its endpoints as arguments:

use mq_bridge::{models::Endpoint, CanonicalMessage, Handled, HandlerError, Route};
use serde_json::{json, Value};

/// Business logic: reject orders without a positive amount, mark the rest as checked.
async fn check_order(msg: CanonicalMessage) -> Result<Handled, HandlerError> {
    let mut order: Value = msg
        .parse()
        .map_err(|e| HandlerError::NonRetryable(e.into()))?;
    if order["amount"].as_f64().unwrap_or(0.0) <= 0.0 {
        return Ok(Handled::Ack);
    }
    order["checked"] = json!(true);
    let out = CanonicalMessage::from_json(order).map_err(|e| HandlerError::NonRetryable(e.into()))?;
    Ok(Handled::Publish(out))
}

/// The route used in production and in tests. Only the endpoints differ.
pub fn orders_route(input: Endpoint, output: Endpoint) -> Route {
    Route::new(input, output).with_handler(check_order)
}

The test passes memory endpoints, sends messages into one channel and reads the other:

use std::time::Duration;

#[tokio::test(flavor = "multi_thread")]
async fn valid_orders_are_checked_and_invalid_ones_dropped() {
    let input = Endpoint::new_memory("orders.in", 100);
    let output = Endpoint::new_memory("orders.out", 100);
    let (orders_in, orders_out) = (input.channel().unwrap(), output.channel().unwrap());

    orders_route(input, output).deploy("orders").await.unwrap();

    for order in [json!({"id": 1, "amount": 25.0}), json!({"id": 2, "amount": 0})] {
        let msg = CanonicalMessage::from_json(order).unwrap();
        orders_in.send_message(msg).await.unwrap();
    }

    let mut received = Vec::new();
    for _ in 0..100 {
        received.extend(orders_out.drain_messages());
        if !received.is_empty() && orders_in.is_empty() {
            break;
        }
        tokio::time::sleep(Duration::from_millis(20)).await;
    }
    Route::stop("orders").await;

    assert_eq!(received.len(), 1);
    let order: Value = received[0].parse().unwrap();
    assert_eq!(order, json!({"id": 1, "amount": 25.0, "checked": true}));
}
[dependencies]
mq-bridge = { version = "=0.4.18", default-features = false }   # production adds e.g. features = ["kafka"]
serde_json = "1"

[dev-dependencies]
tokio = { version = "1", features = ["macros", "rt-multi-thread", "time"] }

Python

The route is built from a config dict, so the test passes a different dict:

from mq_bridge import Route


def check_order(order):
    """Business logic: drop orders without a positive amount, mark the rest as checked."""
    if order.get("amount", 0) <= 0:
        return None
    return {**order, "checked": True}


def orders_route(config: dict) -> Route:
    """The route used in production and in tests. Only `config` differs."""
    return Route.from_config(config).add_handler("order.created", check_order)
from mq_bridge import Consumer, Publisher

from orders import orders_route

INPUT = {"memory": {"topic": "orders.in", "capacity": 100}}
OUTPUT = {"memory": {"topic": "orders.out", "capacity": 100}}


def test_valid_orders_are_checked_and_invalid_ones_dropped():
    publisher = Publisher.from_config(INPUT)
    results = Consumer.from_config(OUTPUT)

    with orders_route({"input": INPUT, "output": OUTPUT}):
        publisher.send_json({"id": 1, "amount": 25.0}, {"kind": "order.created"})
        publisher.send_json({"id": 2, "amount": 0}, {"kind": "order.created"})
        publisher.send_json({"id": 3, "amount": 5.0}, {"kind": "order.created"})
        received = []
        while len(received) < 2:
            batch = results.poll(max=10, timeout_ms=5000)
            assert batch, "timed out waiting for the route"
            received.extend(batch)

    assert [m.json() for m in received] == [
        {"id": 1, "amount": 25.0, "checked": True},
        {"id": 3, "amount": 5.0, "checked": True},
    ]

The Node.js binding has the same memory endpoint; this page’s example covers Rust and Python.

Runnable example

This one needs no Docker:

git clone https://github.com/marcomq/mq-bridge && cd mq-bridge/examples/test-without-a-broker
cargo test                                             # Rust
cd python && pip install -r requirements.txt && pytest  # Python

Both pin mq-bridge 0.4.18 and run in CI.

Failure semantics: what an in-memory test does and does not prove

AspectIn the memory testAgainst a real broker
Handler logic, routing by kind, middleware in the routeExercised as in productionSame code
RestartsA memory channel lives in the process. Stopping the process loses its content.The broker keeps messages and redelivers from the committed position
DuplicatesNone occur by themselves. To test idempotency, send the same message twice.At-least-once delivery produces them after crashes and rebalances
OrderingOne channel, in send orderPer partition, subject or queue, depending on the broker
Sink unavailableNot simulated by the memory endpointNetwork errors, timeouts, throttling
Broker configurationNot covered: topics, ACLs, TLS, consumer groups, serialization of headersCovered

A test over memory channels shares global state by topic name within one process. Give each test its own topic names (or route name) when tests run in parallel.

When not to use this

  • You are testing the broker integration itself: consumer-group rebalancing, offset commits, partition assignment, authentication, schema-registry interaction. Use a real broker.
  • Your code uses a broker client directly and you do not want mq-bridge in production. The memory endpoint replaces a transport behind mq-bridge’s route; it is not a fake Kafka server that other client libraries can connect to.
  • You need wire compatibility checks between producers and consumers written in different stacks. Contract tests or a shared broker environment cover that.

Alternatives

mq-bridge memory endpointTestcontainersMocking the client
What runsYour route and handler, in-processA real broker in a container, started by the testYour handler against a hand-written or generated fake
Needs DockerNoYesNo
Covers broker behaviourNoYesNo
Covers the code between client and handlerYes, if that code is the mq-bridge routeYesUsually not
Requiresmq-bridge as the messaging layer in productionNothing about your production stackNothing about your production stack

Why mq-bridge can be the better fit here

  • The test runs the production route. Nothing is faked: the handler, the dispatch on the message type, the middleware and the acknowledgement path are the same objects, with a different endpoint underneath.
  • No Docker in the inner loop. The tests start no container, so they run wherever cargo test or pytest runs.
  • The transport stays a configuration choice afterwards. The same handler can move from Kafka to NATS or RabbitMQ by changing the endpoint, with the tests unchanged.

Measured with the example on an Apple M1 (2026-10-05, three runs each): the Rust test finished in 0.02 s (cargo test 0.16 s wall time after the build) and the Python test in 0.07 s. An apache/kafka:3.9.0 container with the image already pulled took 4.3 to 4.8 s to pass its health check, before any test could run.

Where the alternatives are the better fit: Testcontainers is independent of the library you use and is the only option in this table that exercises a real broker. The two combine well: most tests on memory endpoints, a few integration tests against a container.