Connect plugin: processors

The 86 Redpanda Connect processors the Connect plugin links, generated from the component specs of Redpanda Connect v4.110.0. Field descriptions are Redpanda Connect’s own, shortened to one sentence; follow Upstream documentation for the rest. Install and configuration forms are on the plugin page.

16 of them are also exported as their own connect_<name> middleware and run on any endpoint, native ones included. Every other processor that keeps, rewrites or drops each message runs inside a connect middleware; one that splits or merges messages belongs in a connect endpoint’s pipeline. Middleware needs plugin 0.1.1 or newer.

Own middleware: connect_avro, connect_bloblang, connect_branch, connect_cached, connect_dedupe, connect_grok, connect_http, connect_javascript, connect_jmespath, connect_jq, connect_json_schema, connect_log, connect_mapping, connect_msgpack, connect_mutation, connect_parse_log.

input:
  kafka: { url: localhost:9092, topic: orders }
  middlewares:
    - connect_mapping: 'root = this.merge({"received_at": now()})'

Processors in bold are also their own middleware.

archive

Archives all the messages of a batch into a single message according to the selected archive format.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Upstream documentation

avro

Performs Avro based operations on messages based on a schema.

Middleware: connect_avro, or inside a connect middleware.

Upstream documentation

awk

Executes an AWK program on messages. This processor is very powerful as it offers a range of custom functions for querying and mutating message contents and metadata.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Upstream documentation

aws_bedrock_chat

Generates responses to messages in a chat conversation, using the AWS Bedrock API.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Advanced: region, endpoint, tcp, credentials, stop, top_p.

Upstream documentation

aws_bedrock_embeddings

Computes vector embeddings on text, using the AWS Bedrock API.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Advanced: region, endpoint, tcp, credentials.

Upstream documentation

aws_lambda

Invokes an AWS lambda for each message. The contents of the message is the payload of the request, and the result of the invocation will become the new contents of the message.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Advanced: rate_limit, region, endpoint, tcp, credentials, timeout, retries.

Upstream documentation

azure_cosmosdb

Creates or updates messages as JSON documents in Azure CosmosDB.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Advanced: patch_operations, patch_condition, auto_id, enable_content_response_on_write.

Upstream documentation

benchmark

Logs basic throughput statistics of messages that pass through this processor.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Upstream documentation

bloblang

Executes a Bloblang mapping on messages.

Middleware: connect_bloblang, or inside a connect middleware.

Takes a single string value, not a map of fields.

Upstream documentation

bounds_check

Removes messages (and batches) that do not fit within certain size boundaries.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Advanced: max_parts, min_parts.

Upstream documentation

branch

The branch processor allows you to create a new request message via a Bloblang mapping, execute a list of processors on the request messages, and, finally, map the result back into the source message using another mapping.

Middleware: connect_branch, or inside a connect middleware.

Upstream documentation

cache

Performs operations against a cache resource for each message, allowing you to store or retrieve data within message payloads.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Advanced: ttl.

Upstream documentation

cached

Cache the result of applying one or more processors to messages identified by a key. If the key already exists within the cache the contents of the message will be replaced with the cached result instead of applying the processors. This component is therefore useful in situations where an expensive set of processors need only be executed periodically.

Middleware: connect_cached, or inside a connect middleware.

Upstream documentation

catch

Applies a list of child processors only when a previous processing step has failed.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Takes a single list of processor value, not a map of fields.

Upstream documentation

cohere_chat

Generates responses to messages in a chat conversation, using the Cohere API.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Advanced: schema_registry, top_p, frequency_penalty, presence_penalty, seed, stop.

Upstream documentation

cohere_embeddings

Generates vector embeddings to represent input text, using the Cohere API.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Upstream documentation

cohere_rerank

Ranks a list of documents by their relevance to a query, using the Cohere API.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Upstream documentation

command

Executes a command for each message.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Upstream documentation

compress

Compresses messages according to the selected algorithm. Supported compression algorithms are: [flate gzip lz4 pgzip snappy zlib]

Use it inside a connect middleware or a connect endpoint’s pipeline.

Upstream documentation

couchbase

Performs operations against Couchbase for each message, allowing you to store or retrieve data within message payloads.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Advanced: collection, scope, transcoder, timeout, ttl.

Upstream documentation

crash

Beta. Crashes the process using a fatal log message. The log message can be set using function interpolations described in Bloblang queries which allows you to log the contents and metadata of messages.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Takes a single string value, not a map of fields.

Upstream documentation

decompress

Decompresses messages according to the selected algorithm. Supported decompression algorithms are: [bzip2 flate gzip lz4 pgzip snappy zlib]

Use it inside a connect middleware or a connect endpoint’s pipeline.

Upstream documentation

dedupe

Deduplicates messages by storing a key value in a cache using the add operator. If the key already exists within the cache it is dropped.

Middleware: connect_dedupe, or inside a connect middleware.

Upstream documentation

for_each

A processor that applies a list of child processors to messages of a batch as though they were each a batch of one message.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Takes a single list of processor value, not a map of fields.

Upstream documentation

gcp_bigquery_select

Executes a SELECT query against BigQuery and replaces messages with the rows returned.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Upstream documentation

gcp_vertex_ai_chat

Generates responses to messages in a chat conversation, using the Vertex AI API.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Advanced: system_prompt, top_p, top_k, stop, presence_penalty, frequency_penalty, max_tool_calls.

Upstream documentation

gcp_vertex_ai_embeddings

Generates vector embeddings to represent input text, using the Vertex AI API.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Upstream documentation

grok

Parses messages into a structured format by attempting to apply a list of Grok expressions, the first expression to result in at least one value replaces the original message with a JSON object containing the values.

Middleware: connect_grok, or inside a connect middleware.

Advanced: named_captures_only, use_default_patterns, remove_empty_values.

Upstream documentation

group_by

Splits a batch of messages into N batches, where each resulting batch contains a group of messages determined by a Bloblang query.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Upstream documentation

group_by_value

Splits a batch of messages into N batches, where each resulting batch contains a group of messages determined by a function interpolated string evaluated per message.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Upstream documentation

http

Performs an HTTP request using a message batch as the request body, and replaces the original message parts with the body of the response.

Middleware: connect_http, or inside a connect middleware.

Advanced: metadata, dump_request_log_level, oauth, oauth2, basic_auth, jwt, tls, extract_headers, retry_period, max_retry_backoff, retries, follow_redirects, backoff_on, drop_on, successful_on, proxy_url, disable_http2, batch_as_multipart.

Upstream documentation

insert_part

Insert a new message into a batch at an index. If the specified index is greater than the length of the existing batch it will be appended to the end.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Upstream documentation

javascript

Executes a provided JavaScript code block or file for each message.

Middleware: connect_javascript, or inside a connect middleware.

Upstream documentation

jmespath

Executes a JMESPath query on JSON documents and replaces the message with the resulting document.

Middleware: connect_jmespath, or inside a connect middleware.

Upstream documentation

jq

Transforms and filters messages using jq queries.

Middleware: connect_jq, or inside a connect middleware.

Advanced: raw, output_raw.

Upstream documentation

json_schema

Checks messages against a provided JSONSchema definition but does not change the payload under any circumstances. If a message does not match the schema it can be caught using error handling methods.

Middleware: connect_json_schema, or inside a connect middleware.

Upstream documentation

log

Prints a log event for each message. Messages always remain unchanged. The log message can be set using function interpolations described in Bloblang queries which allows you to log the contents and metadata of messages.

Middleware: connect_log, or inside a connect middleware.

Upstream documentation

mapping

Executes a Bloblang mapping on messages, creating a new document that replaces (or filters) the original message.

Middleware: connect_mapping, or inside a connect middleware.

Takes a single string value, not a map of fields.

Upstream documentation

metric

Emit custom metrics by extracting values from messages.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Upstream documentation

mongodb

Performs operations against MongoDB for each message, allowing you to store or retrieve data within message payloads.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Advanced: app_name, aws, json_marshal_mode.

Upstream documentation

msgpack

Converts messages to or from the MessagePack format.

Middleware: connect_msgpack, or inside a connect middleware.

Upstream documentation

mutation

Executes a Bloblang mapping and directly transforms the contents of messages, mutating (or deleting) them.

Middleware: connect_mutation, or inside a connect middleware.

Takes a single string value, not a map of fields.

Upstream documentation

nats_kv

Perform operations on a NATS key-value bucket.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Advanced: max_reconnects, revision, timeout, tls, tls_handshake_first, auth.

Upstream documentation

nats_request_reply

Sends a message to a NATS subject and expects a reply, from a NATS subscriber acting as a responder, back.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Advanced: max_reconnects, inbox_prefix, tls, tls_handshake_first, auth.

Upstream documentation

noop

Noop is a processor that does nothing, the message passes through unchanged. Why? Sometimes doing nothing is the braver option.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Upstream documentation

ollama_chat

Generates responses to messages in a chat conversation, using the Ollama API.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Advanced: system_prompt, num_keep, seed, top_k, top_p, repeat_penalty, presence_penalty, frequency_penalty, stop, max_tool_calls, cache_directory, download_url.

Upstream documentation

ollama_embeddings

Generates vector embeddings from text, using the Ollama API.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Advanced: cache_directory, download_url.

Upstream documentation

ollama_moderation

Classifies an LLM response as safe or unsafe, using the Ollama API.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Advanced: cache_directory, download_url.

Upstream documentation

openai_chat_completion

Generates responses to messages in a chat conversation, using the OpenAI API.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Advanced: schema_registry, top_p, frequency_penalty, presence_penalty, seed, stop.

Upstream documentation

openai_embeddings

Generates vector embeddings to represent input text, using the OpenAI API.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Upstream documentation

openai_image_generation

Generates an image from a text description and other attributes, using OpenAI API.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Advanced: quality, size, style.

Upstream documentation

openai_speech

Generates audio from a text description and other attributes, using OpenAI API.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Advanced: response_format.

Upstream documentation

openai_transcription

Generates a transcription of spoken audio in the input language, using the OpenAI API.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Advanced: language, prompt.

Upstream documentation

openai_translation

Translates spoken audio into English, using the OpenAI API.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Advanced: prompt.

Upstream documentation

parallel

A processor that applies a list of child processors to messages of a batch as though they were each a batch of one message (similar to the for_each processor), but where each message is processed in parallel.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Upstream documentation

parquet

Deprecated. Converts batches of documents to or from Parquet files.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Upstream documentation

parquet_decode

Decodes Parquet files into a batch of structured messages.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Upstream documentation

parquet_encode

Encodes Parquet files from a batch of structured messages.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Advanced: default_encoding, default_timestamp_unit.

Upstream documentation

parse_log

Parses common log formats into structured data. This is easier and often much faster than grok.

Middleware: connect_parse_log, or inside a connect middleware.

Advanced: best_effort, allow_rfc3339, default_year, default_timezone.

Upstream documentation

processors

A processor grouping several sub-processors.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Takes a single list of processor value, not a map of fields.

Upstream documentation

qdrant

Query items within a Qdrant collection.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Advanced: tls.

Upstream documentation

rate_limit

Throttles the throughput of a pipeline according to a specified rate_limit resource. Rate limits are shared across components and therefore apply globally to all processing pipelines.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Upstream documentation

redis

Performs actions against Redis that aren’t possible using a cache processor. Actions are performed for each message and the message contents are replaced with the result. In order to merge the result into the original message compose this processor within a branch processor.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Advanced: kind, master, client_name, tls, retries, retry_period.

Upstream documentation

redis_script

Performs actions against Redis using LUA scripts.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Advanced: kind, master, client_name, tls, retries, retry_period.

Upstream documentation

resource

Resource is a processor type that runs a processor resource identified by its label.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Takes a single string value, not a map of fields.

Upstream documentation

retry

Beta. Attempts to execute a series of child processors until success.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Upstream documentation

select_parts

Cherry pick a set of messages from a batch by their index. Indexes larger than the number of messages are simply ignored.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Upstream documentation

sentry_capture

Captures log events from messages and submits them to Sentry.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Upstream documentation

sleep

Sleep for a period of time specified as a duration string for each message. This processor will interpolate functions within the duration field, you can find a list of functions here.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Upstream documentation

split

Breaks message batches (synonymous with multiple part messages) into smaller batches. The size of the resulting batches are determined either by a discrete size or, if the field byte_size is non-zero, then by total size in bytes (which ever limit is reached first).

Use it inside a connect middleware or a connect endpoint’s pipeline.

Upstream documentation

sql

Deprecated. Runs an arbitrary SQL query against a database and (optionally) returns the result as an array of objects, one for each row returned.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Advanced: unsafe_dynamic_query.

Upstream documentation

sql_insert

Inserts rows into an SQL database for each message, and leaves the message unchanged.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Advanced: prefix, suffix, options, init_files, init_statement, conn_max_idle_time, conn_max_life_time, conn_max_idle, conn_max_open.

Upstream documentation

sql_raw

Runs an arbitrary SQL query against a database and (optionally) returns the result as an array of objects, one for each row returned.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Advanced: unsafe_dynamic_query, init_files, init_statement, conn_max_idle_time, conn_max_life_time, conn_max_idle, conn_max_open.

Upstream documentation

sql_select

Runs an SQL select query against a database and returns the result as an array of objects, one for each row returned, containing a key for each column queried and its value.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Advanced: prefix, suffix, init_files, init_statement, conn_max_idle_time, conn_max_life_time, conn_max_idle, conn_max_open.

Upstream documentation

string_split

Splits a string by a delimiter into an array. Generally, using bloblang’s split method is preferred. In some high performance use cases this processor can be faster than the equivalent bloblang if there is no additional logic.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Advanced: emit_bytes.

Upstream documentation

subprocess

Executes a command as a subprocess and, for each message, will pipe its contents to the stdin stream of the process followed by a newline.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Advanced: max_buffer, codec_send, codec_recv.

Upstream documentation

switch

Conditionally processes messages based on their contents.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Advanced: fallthrough, continue.

Upstream documentation

sync_response

Adds the payload in its current state as a synchronous response to the input source, where it is dealt with according to that specific input type.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Upstream documentation

text_chunker

A processor that allows chunking and splitting text based on some strategy. Usually used for creating vector embeddings of large documents.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Advanced: token_encoding, allowed_special, disallowed_special.

Upstream documentation

try

Executes a list of child processors on messages only if no prior processors have failed (or the errors have been cleared).

Use it inside a connect middleware or a connect endpoint’s pipeline.

Takes a single list of processor value, not a map of fields.

Upstream documentation

try_catch

Beta. Executes a list of child processors on each message and, if any of them fail, executes a separate list of catch processors to recover from or react to the error.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Upstream documentation

unarchive

Unarchives messages according to the selected archive format into multiple messages within a batch.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Upstream documentation

wasm

Executes a function exported by a WASM module for each message.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Upstream documentation

while

A processor that checks a Bloblang query against each batch of messages and executes child processors on them for as long as the query resolves to true.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Advanced: max_loops.

Upstream documentation

workflow

Executes a topology of branch processors, performing them in parallel where possible.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Advanced: branch_resources.

Upstream documentation

xml

Parses messages as an XML document, performs a mutation on the data, and then overwrites the previous contents with the new value.

Use it inside a connect middleware or a connect endpoint’s pipeline.

Upstream documentation