n8n Queue Mode: Workers, Concurrency & Horizontal Scaling


⚙️ n8n Scaling Hub
Queue mode · Workers · Concurrency · Redis · PostgreSQL · Horizontal scaling
n8n Queue Mode: Workers, Concurrency & Horizontal Scaling
Redis message broker | PostgreSQL mandatory | Worker architecture | Concurrency limits | Task runners
This guide exclusively covers n8n queue mode — the architecture that decouples workflow triggering from execution, enabling horizontal scaling. You will learn how to set up Redis as a message broker, configure workers, tune concurrency limits, understand task runner isolation, and migrate from SQLite to PostgreSQL. All information follows official n8n scaling documentation.

1. What is n8n queue mode and why do you need it for production?

Queue mode separates the main n8n process (webhook receiver, editor) from workflow execution. It uses Redis as a message broker and PostgreSQL as a database. One or more worker containers pull jobs from the Redis queue and execute them independently. This allows unlimited horizontal scaling.

In default mode, every workflow execution runs inside the main n8n process. This works for testing and low volumes, but when you exceed ~50 executions per day or need high concurrency, the main process becomes a bottleneck. Queue mode solves this by introducing a Redis queue between the trigger layer and the execution layer.

📈 Key benefits of queue mode:
– Horizontal scaling: add workers without touching the main instance
– Higher concurrency: each worker runs multiple workflows in parallel
– Fault isolation: a worker crash does not affect the main editor or webhooks
– Dynamic load distribution: Redis distributes jobs to available workers

Without queue mode, you cannot scale beyond a single machine. Queue mode is the foundation for production n8n deployments handling thousands of executions per day. For a detailed comparison of scaling strategies, see the official queue mode overview.

2. What are the mandatory prerequisites for n8n queue mode?

Queue mode requires two external services: Redis ≥6.0 as the message broker and PostgreSQL as the database. SQLite is incompatible with queue mode. You must also set the same N8N_ENCRYPTION_KEY across all main and worker containers.

Redis handles job queues, dead-letter queues, and worker heartbeats. PostgreSQL replaces SQLite because it supports concurrent writes, transaction locking, and multi‑worker coordination. Attempting to enable queue mode with SQLite results in runtime errors.

🚨 Incompatibility warning: Do not set QUEUE_MODE_ENABLED=true while DB_TYPE=sqlite. The n8n main process will fail to start or will log “Queue mode requires PostgreSQL”. Always migrate to PostgreSQL first.

Minimum versions: Redis 6.0 (or later, including Redis 7.x), PostgreSQL 12 or higher. For production, use managed services (AWS ElastiCache, Google Cloud Memorystore, or a dedicated Redis container with persistence). The following Docker Compose snippet shows the required services:

services:
  redis:
    image: redis:7-alpine
    command: redis-server --appendonly yes
  postgres:
    image: postgres:15
    environment:
      POSTGRES_USER: n8n
      POSTGRES_PASSWORD: securepass
      POSTGRES_DB: n8n

Refer to the queue mode prerequisites for detailed setup.

3. How do you enable queue mode in a Docker Compose environment?

Set QUEUE_MODE_ENABLED=true on the main n8n container. Add environment variables for Redis (QUEUE_BULL_REDIS_HOST) and PostgreSQL (DB_TYPE=postgresdb). Then define one or more worker services with EXECUTIONS_PROCESS=worker. Workers must share the same encryption key.

Below is a minimal production Docker Compose file with one main instance and two workers:

version: '3.8'
services:
  n8n-main:
    image: n8nio/n8n:1.78.1
    environment:
      - QUEUE_MODE_ENABLED=true
      - QUEUE_BULL_REDIS_HOST=redis
      - DB_TYPE=postgresdb
      - DB_POSTGRESDB_HOST=postgres
      - DB_POSTGRESDB_DATABASE=n8n
      - DB_POSTGRESDB_USER=n8n
      - DB_POSTGRESDB_PASSWORD=securepass
      - N8N_ENCRYPTION_KEY=${N8N_ENCRYPTION_KEY}
      - N8N_HOST=n8n.yourdomain.com
      - N8N_PROTOCOL=https
    ports:
      - "5678:5678"
  n8n-worker-1:
    image: n8nio/n8n:1.78.1
    environment:
      - QUEUE_MODE_ENABLED=true
      - QUEUE_BULL_REDIS_HOST=redis
      - DB_TYPE=postgresdb
      - DB_POSTGRESDB_HOST=postgres
      - DB_POSTGRESDB_DATABASE=n8n
      - DB_POSTGRESDB_USER=n8n
      - DB_POSTGRESDB_PASSWORD=securepass
      - N8N_ENCRYPTION_KEY=${N8N_ENCRYPTION_KEY}
      - EXECUTIONS_PROCESS=worker
  n8n-worker-2:
    ... same as worker-1

The main instance exposes the editor and webhooks. Workers do not need published ports. Scale workers by adding more containers. Use a load balancer in front of the main instances if you need high availability for the editor itself.

4. How do you control concurrency per worker with N8N_CONCURRENCY_PRODUCTION_LIMIT?

N8N_CONCURRENCY_PRODUCTION_LIMIT defines how many workflows a single worker executes simultaneously. Default is -1 (unlimited). Set to 1 for strict sequential execution. Total cluster concurrency = number of workers × limit per worker. Monitor CPU and memory to find optimal value.

When a worker pulls a job from Redis, it starts executing it. If the worker has spare concurrency slots, it immediately pulls additional jobs. Without a limit, a worker may overload itself if workflows are CPU‑intensive. Start with a limit of 5 and adjust based on metrics.

📐 Concurrency calculation example:
3 workers each with N8N_CONCURRENCY_PRODUCTION_LIMIT=4 → max parallel executions = 12.
The main instance does not execute workflows in queue mode; only workers do.

Set this variable on the worker containers only (it is ignored on the main instance). For bursty traffic, a higher limit can reduce queue waiting time. For memory‑intensive workflows (e.g., large file processing), keep the limit low (e.g., 1 or 2). The environment variables reference provides further details.

5. How do you scale n8n workers horizontally in production?

Add more worker containers to the same Redis and PostgreSQL backend. Workers automatically register themselves and start consuming jobs. You can scale dynamically based on queue length. Use orchestration tools (Kubernetes, Docker Swarm) to auto‑scale workers using metrics like Redis queue depth.

Workers are stateless: they read the workflow definition from the database, execute it, and write results back. They do not store any persistent data locally. Therefore, you can add or remove workers at any time without disrupting ongoing executions (in‑flight jobs finish on the original worker).

To monitor scaling needs, observe the Redis queue length using redis-cli LLEN bull:queue:wait. If the queue grows consistently, add workers. Reduce workers when the queue stays empty. For Kubernetes, use the official Helm charts that support Horizontal Pod Autoscaling based on custom metrics.

⚠️ Note: The main n8n instance still processes webhooks and editor requests. If you need high availability for the editor, run multiple main instances behind a load balancer (all sharing the same Redis and PostgreSQL). However, only one main instance should handle webhook registrations; use a shared Redis for webhook routing.

6. What are n8n task runners and why are they important in queue mode?

Task runners are separate processes that execute Code node (JavaScript/Python) sandboxes. In queue mode, task runners are enabled by default. They isolate user code from the main worker process, preventing infinite loops, memory leaks, or crashes from affecting the entire worker.

Before task runners, Code node code ran inside the main worker thread. Malformed code could crash the worker. Task runners spawn a dedicated process for each Code node execution (or a pool of processes). The worker communicates with the task runner via gRPC. If a Code node crashes, only that task runner restarts; the worker continues processing other workflows.

Key task runner variables:

  • N8N_RUNNERS_ENABLED=true (default in queue mode).
  • N8N_RUNNERS_MAX_CONCURRENCY — max concurrent Code node executions per worker (default 5).
  • N8N_RUNNERS_TASK_BROKER_URI — internal URI for task broker.
🧩 Security note: Task runners increase isolation but also add slight overhead. For workflows without Code nodes, you can safely disable them with N8N_RUNNERS_ENABLED=false to save resources. In queue mode, the default is enabled.

More details are available in the task runners documentation.

7. How do you monitor queue mode performance and worker health?

Expose n8n Prometheus metrics (N8N_METRICS=true) to monitor queue length (n8n_queue_jobs_waiting), active workers, execution latency, and failure rates. Use Grafana dashboards. Additionally, monitor Redis memory usage and PostgreSQL connection pool.

The Prometheus endpoint includes metrics for queue mode:

  • n8n_queue_jobs_waiting — number of workflows waiting in Redis queue.
  • n8n_queue_jobs_active — workflows currently being executed by workers.
  • n8n_workers_count — number of connected workers.
  • n8n_workflow_execution_total and n8n_workflow_execution_failed_total — success/failure counts.

Example Prometheus alert rule: if n8n_queue_jobs_waiting > 100 for 5 minutes → scale up workers. Also monitor Redis eviction and memory fragmentation.

🔔 Proactive monitoring: Set up alerts for worker disconnection (e.g., absent(n8n_workers_count)). A worker that stops consuming jobs may indicate a crash or configuration error (e.g., mismatched encryption key).

For full metrics reference, see the metrics documentation.

8. What are common queue mode failures and how to fix them?

Typical issues: SQLite used instead of PostgreSQL (error “queue mode needs PostgreSQL”), Redis connection refused, mismatched encryption keys between main and workers (credentials cannot be decrypted), or worker concurrency set too high causing OOM kills. Check logs with docker logs and verify all environment variables.

Below is a troubleshooting table:

Error / Symptom Likely cause Solution
Queue mode enabled but SQLite detected DB_TYPE not set to postgresdb Set DB_TYPE=postgresdb and provide PostgreSQL credentials
Worker cannot decrypt credentials N8N_ENCRYPTION_KEY differs from main instance Set the exact same key on all containers
Redis connection refused QUEUE_BULL_REDIS_HOST missing or wrong port Verify Redis service name and port (default 6379)
Workers idle but queue not emptying Concurrency limit too low or workers not registered Check N8N_CONCURRENCY_PRODUCTION_LIMIT and worker logs
High memory usage on workers Concurrency too high for available RAM Reduce N8N_CONCURRENCY_PRODUCTION_LIMIT or add more workers

Always run docker logs <main_or_worker_container> to see detailed error messages. For persistent issues, refer to the n8n community forum or the GitHub issues.

⚠️ Disclaimer: This guide is for informational purposes. Queue mode changes your deployment architecture; always test in a staging environment. The author is not affiliated with n8n GmbH.


Leave a Reply

Your email address will not be published. Required fields are marked *