n8n Message Queue RabbitMQ Kafka Event-Driven Architecture

I build event-driven architectures in n8n using message queues. The pattern is simple: workflows publish events to a queue. Other workflows consume those events and process them independently. This decouples system components, handles traffic spikes, and prevents cascading failures.

Message queues sit between producers and consumers. Producers send messages without knowing who will read them. Consumers process messages without knowing where they came from. The queue holds messages until consumers are ready. This buffer absorbs bursts of activity and smooths out processing loads.

RabbitMQ Integration with n8n

RabbitMQ is the most accessible message queue for n8n workflows. The HTTP Request node handles all RabbitMQ interactions through its management API. No custom nodes needed.

Here’s how I publish messages to RabbitMQ:

`
HTTP Request Node (publish message):
Method: POST
URL: http://rabbitmq-server:15672/api/exchanges/{{ $json.exchange }}/publish
Headers:
Content-Type: application/json
Authorization: Basic * Body:
{
“properties”: {},
“routing_key”: “{{ $json.routingKey }}”,
“items”: [
{
“payload”: “{{ $json.messageBody }}”,
“payload_encoding”: “string”
}
]
}
`

The exchange name, routing key, and message body come from upstream workflow nodes. This makes the publish operation fully dynamic. Different workflows can route to different exchanges using the same node configuration.

For consuming messages, I use a polling pattern. A scheduled workflow checks the queue for pending messages every few seconds:

`
Schedule Trigger (every 5 seconds)
→ HTTP Request (GET /api/queues/{vhost}/{queueName}/get)
Headers: Authorization: Basic *
Body: { “count”: 50, “ackmode”: “ack_requeue_false”, “encoding”: “auto” }
→ Code Node (parse and process messages)
→ HTTP Request (publish to processing workflow via webhook)
`

The queue GET endpoint returns up to 50 messages at a time. Each message contains the payload, properties, and metadata. The Code node parses the payload and routes it to the appropriate processing logic. The n8n webhook node guide covers webhook-based message distribution.

Kafka Integration Patterns

Kafka handles higher throughput than RabbitMQ. Where RabbitMQ processes hundreds of messages per second, Kafka handles tens of thousands. The trade-off is complexity. Kafka requires more infrastructure and careful partition management.

I interact with Kafka through its REST proxy or direct protocol connections. The HTTP Request node works with Kafka’s schema registry and admin API:

`
HTTP Request Node (consume from Kafka topic):
Method: POST
URL: http://kafka-proxy:8082/subjects/{{ $json.topicName }}-value/versions
Headers:
Accept: application/vnd.schemaregistry.v1+json
Content-Type: application/vnd.schemaregistry.v1+json
Body:
{
“schema”: “{\”type\”:\”record\”,\”name\”:\”Event\”,\”fields\”:[{\”name\”:\”eventType\”,\”type\”:\”string\”},{\”name\”:\”data\”,\”type\”:\”string\”}]}”
}
`

For actual message consumption, I use a consumer group pattern. Multiple n8n workflows share the same consumer group, and Kafka distributes partitions among them. This enables horizontal scaling. Add more workflows, get more throughput.

The n8n queue mode workers guide explains how n8n’s own worker mode maps directly to Kafka consumer groups. Each worker processes a subset of partitions independently.

Event Schema Design

Message queues are only as good as the events they carry. I design schemas that are stable, versioned, and self-describing. Every event includes a type, timestamp, and source identifier.

Here’s my standard event schema:

`json
{
“event_type”: “user.created”,
“event_version”: “1.0”,
“timestamp”: “2024-03-15T10:30:00Z”,
“source”: “n8n-workflow-123”,
“correlation_id”: “abc-def-ghi”,
“data”: {
“user_id”: “USR-456”,
“email”: “user@example.com”,
“plan”: “pro”
}
}
`

The correlation_id links related events across workflows. When a user creates an account, that generates a user.created event. When the welcome email sends, it generates email.sent with the same correlation_id. Tracking correlations makes debugging straightforward.

I store event schemas in a Code node template that every publishing workflow references:

`javascript
function buildEvent(type, data) {
return {
event_type: type,
event_version: ‘1.0’,
timestamp: new Date().toISOString(),
source: $(‘Webhook’).item.json.workflowId || ‘manual’,
correlation_id: $(‘Webhook’).item.json.correlationId || crypto.randomUUID(),
data: data
};
}

return [{ json: buildEvent(‘{{ $json.eventType }}’, $input.item.json) }];
`

This function wraps any data object in a consistent event envelope. Downstream consumers parse the envelope to determine routing logic. The n8n expressions reference covers expression patterns for event building.

Dead Letter Queues and Error Handling

Messages fail. Invalid data arrives. Consumers crash. Dead letter queues capture failed messages so they don’t disappear. I configure separate queues for different failure types:

`
Main Queue (events)
→ Consumer Workflow
→ Success: Process normally
→ Parse Error: Dead Letter Queue – parse-errors
→ Validation Error: Dead Letter Queue – validation-errors
→ Timeout: Dead Letter Queue – timeouts
`

Each dead letter queue holds messages with error metadata attached. A weekly review workflow processes these queues, extracts the errors, and sends a report to the team. The n8n error workflow catch guide covers error handling patterns that apply to queue-based systems.

For Kafka, I use the error topic pattern. Failed deserialization goes to {topic}-errors. Failed processing goes to {topic}-dead-letter. These topics feed into alerting workflows that notify the team through the n8n Slack PagerDuty setup.

Connecting to Your Setup

The n8n Docker environment variables guide covers configuring RabbitMQ and Kafka connection strings as environment variables. This keeps connection details out of workflow definitions.

For production deployments, the n8n Docker deployment guide includes infrastructure recommendations for message queue integration.

Understanding n8n node configuration ensures your queue messages map correctly to workflow data structures.

Action Card: RabbitMQ Publisher Template

Use this HTTP Request configuration for any RabbitMQ publisher:

`
Method: POST
URL: http://RABBITMQ_HOST:15672/api/exchanges/{{ $json.exchange }}/publish
Headers:
Content-Type: application/json
Authorization: Basic {{ $json.rabbitmqCredentials }}
Body:
{
“properties”: {
“delivery_mode”: 2,
“content_type”: “application/json”
},
“routing_key”: “{{ $json.routingKey }}”,
“items”: [
{
“payload”: “{{ $json.eventPayload }}”,
“payload_encoding”: “string”
}
]
}
`

Set RABBITMQ_HOST, exchange, routingKey, and eventPayload as workflow variables. This template works with any RabbitMQ exchange and routing strategy.

References