n8n Workflow Buffer, Cache & State Management Patterns

I manage state in n8n workflows because each execution is isolated. When a workflow runs, it gets fresh memory. When it finishes, everything disappears. This is great for security but terrible for workflows that need to remember things across executions. Here’s how I solve that problem.

Without state management, n8n workflows are stateless by design. You can’t store a counter between runs. You can’t cache API responses. You can’t track which items were already processed. I used to work around this by creating separate workflows for each state, which was a nightmare to maintain. The buffer and cache patterns I’ll show you are much cleaner.

Using the Database as State Storage

The most reliable state management technique in n8n is using a database. Whether you run PostgreSQL, MySQL, or SQLite, a dedicated state table gives you persistent storage that survives workflow executions.

Here’s my standard state table schema:

`sql
CREATE TABLE workflow_state (
id SERIAL PRIMARY KEY,
workflow_name VARCHAR(255) NOT NULL,
state_key VARCHAR(255) NOT NULL,
state_value TEXT,
updated_at TIMESTAMP DEFAULT NOW(),
UNIQUE(workflow_name, state_key)
);
`

The UNIQUE constraint on (workflow_name, state_key) ensures one value per key per workflow. Updates replace existing values rather than creating duplicates.

Reading and writing state in n8n uses the Database node:

`
Write state:
Operation: Insert / Update
Table: workflow_state
Columns: workflow_name, state_key, state_value
Value: {{ “my_workflow” }}, {{ “last_processed_id” }}, {{ $json.last_id }}

Read state:
Operation: Read All
Table: workflow_state
Filter: workflow_name = ‘my_workflow’ AND state_key = ‘last_processed_id’
`

This pattern is essential for pagination workflows. When you fetch page 1 of results, store the last ID. On the next run, read that ID and fetch page 2 starting from there. The workflow remembers where it left off without manual intervention.

For tracking processed items, maintain a list of IDs in the state table:

`javascript
// Code node: Update processed items list
const currentState = $input.all()[0].json?.processed_ids || [];
const newIds = $json.item_ids || [];
const merged = […new Set([…currentState, …newIds])];

return [{
json: {
processed_ids: merged,
total_processed: merged.length
}
}];
`

Store the merged list back to the database. Next execution reads the full list, adds new IDs, and deduplicates. This prevents reprocessing items you’ve already handled.

Implementing a Cache Layer

Caching reduces API calls and speeds up workflows. Instead of fetching the same data repeatedly, I cache results and serve them from memory or a fast store like Redis.

The simplest cache in n8n uses the database with a TTL (time-to-live) check:

`javascript
// Check if cached data is still valid
const cacheEntry = $input.all()[0].json;
if (!cacheEntry) return [{ json: { cache_hit: false } }];

const age = Date.now() – new Date(cacheEntry.updated_at).getTime();
const maxAge = 3600000; // 1 hour in milliseconds

if (age < maxAge) { return [{ json: { cache_hit: true, data: JSON.parse(cacheEntry.state_value) } }]; } else { return [{ json: { cache_hit: false } }]; } `

If the cache hit is true, use the cached data. If false, fetch fresh data from the API and store it in the cache table. This simple pattern eliminates redundant API calls for data that doesn’t change frequently.

For faster caching, I use Redis. The n8n Docker environment variables guide covers Redis configuration for n8n. Redis supports automatic expiration, so cached data expires without manual cleanup.

`
Redis cache pattern:
Key: cache:{workflow_name}:{state_key}
TTL: 3600 seconds (1 hour)
Value: JSON string of cached data

On workflow start:
1. Check Redis for key
2. If found, parse and use cached data
3. If not found, fetch from API, store in Redis, use fresh data
`

Redis is ideal for caching API responses, computed results, and lookup tables. The n8n node concurrency guide explains how Redis integrates with n8n’s queue mode for worker coordination.

Buffering Data Between Executions

Buffering means collecting data across multiple executions before processing it. Instead of processing each item immediately, I accumulate them in a buffer and process when the buffer reaches a threshold.

Here’s a buffering pattern using the database:

`
1. Workflow runs every 5 minutes
2. Each run appends new items to the buffer table
3. When buffer count reaches 100, process all items
4. Clear the buffer table
5. Repeat
`

`javascript
// Code node: Check buffer threshold and process
const buffer = $input.all()[0].json?.buffer_items || [];
const newItems = $json.new_data || [];
const combined = […buffer, …newItems];
const threshold = 100;

if (combined.length >= threshold) {
// Process and clear buffer
return [{
json: {
action: ‘process_and_clear’,
items: combined.slice(0, threshold),
remaining: combined.slice(threshold)
}
}];
} else {
// Keep buffering
return [{
json: {
action: ‘continue_buffering’,
buffer_count: combined.length,
items: combined
}
}];
}
`

This pattern is perfect for batch operations. Send 100 emails at once instead of 100 separate API calls. Insert 100 database rows in one query instead of 100 separate inserts. The buffer accumulates until it’s worth processing.

For the n8n queue mode workers, buffering works naturally. Each worker processes a batch of items from the queue. The queue itself acts as a buffer between workflow executions.

Managing Execution State Across Branches

n8n workflows often branch based on conditions. Each branch may modify state, and branches may converge later. Proper state management ensures data flows correctly through branches.

Use the Merge node to bring branches back together:

`
Branch A: Fetch user data -> modifies user_state
Branch B: Fetch order data -> modifies order_state
Merge: Combine user_state and order_state -> single unified record
`

The Merge node matches items by a key field. If both branches produce items with the same user_id, the Merge node combines them into a single output item. Fields from both branches are available in the merged result.

For state tracking across complex workflows, maintain a state object that flows through the entire execution:

`javascript
// Initialize state at workflow start
return [{
json: {
_state: {
started_at: new Date().toISOString(),
steps_completed: [],
errors: [],
user_id: $json.user_id,
source: ‘api_import’
}
}
}];

// Update state at each step
const state = $input.all()[0].json._state || {};
state.steps_completed.push(‘data_transform’);
return [{ json: { …$input.all()[0].json, _state: state } }];
`

The _state field travels with the data through every node. At the end of the workflow, you have a complete execution log. This is invaluable for debugging and auditing.

Best Practices for State Management

Use unique keys. Prefix state keys with workflow names to avoid collisions: workflow_name:key_name. This makes state tables sortable and searchable.

Clean up old state. Periodically delete state entries older than 30 days. The n8n backup and restore guide covers database maintenance best practices.

Monitor state size. Large state tables slow down queries. Keep state entries minimal — only what you need for the next execution.

Encrypt sensitive state. The n8n security guide covers encryption for credentials and sensitive data. Apply the same principles to state storage.

Action Card: State Management Starter

Quick setup for workflow state management:

`bash

1. Create state table in your database

CREATE TABLE workflow_state (
id SERIAL PRIMARY KEY,
wf_name VARCHAR(255),
state_key VARCHAR(255),
state_value TEXT,
updated_at TIMESTAMP DEFAULT NOW(),
UNIQUE(wf_name, state_key)
);

2. Add Database node to n8n workflow

– Write: INSERT INTO workflow_state VALUES ($wf_name, $key, $value) ON CONFLICT DO UPDATE

– Read: SELECT state_value FROM workflow_state WHERE wf_name = $wf_name AND state_key = $key

3. Use Code nodes to read/write state at key points

4. Schedule periodic cleanup workflow to delete old entries

`

This gives you persistent state across all workflow executions. Combine with caching for optimal performance.

References