Encryption at rest

Two ways to encrypt data, depending on whether you want it encrypted in transit between endpoints or at rest in a file / object.

The encryption middleware — payloads in transit

The encryption middleware seals each message payload into a self-describing AEAD envelope on the output side and decrypts it on the input side. Metadata and routing keys stay in the clear. Requires the encryption feature.

output:
  middlewares:
    - encryption: { key: "${env:MQB_ENC_KEY}" }
  nats: { subject: "secure.orders", url: "nats://localhost:4222" }

The key is a base64-encoded 32-byte key; ${env:VAR} reads it from the environment (see Secrets & interpolation). Cipher defaults to xchacha20poly1305 (aes256gcm also available).

Key rotation: seal with a new key_id/key while listing the old key under decrypt_keys on the consuming side. Each payload is authenticated independently — tampering, a torn frame, or a wrong/missing key is a hard consumer error, not a silent drop.

The AEAD binds only the payload (empty associated data), so a sealed payload can be replayed under different metadata. Use deduplication or a sink uniqueness constraint if that matters.

File / object encryption at rest

To store data compressed and encrypted at rest, use the file / object_store endpoints’ own compression + encryption fields — they apply compress-then-encrypt per batch, which is what you want (ciphertext does not compress):

output:
  file:
    path: "data.enc"
    format: raw
    compression: lz4          # none | gzip | lz4 | zstd  (`compression` feature)
    encryption: { key: "${env:MQB_ENC_KEY}" }

Do not stack the encryption middleware on top of a sink’s batch compression on the same route — ciphertext won’t compress. Use the endpoint fields above instead.

An encrypted file is written as length-prefixed sealed frames (one per batch) and is only readable through a matching consumer. object_store derives its object extension from these fields (e.g. .jsonl.gz / .jsonl.lz4, plus a trailing .enc when encryption is on).

Reading it back

A file source must declare the same compression/encryption the data was written with. A mismatch (wrong key, wrong codec, or a missing field) is a permanent decode failure — the route ends failed with the error in its status, rather than completing as if the file were empty. See Troubleshooting and the Compression recipe.