Trace executions with OpenTelemetry
n8n can emit OpenTelemetry traces for workflow and node executions. Use these traces to monitor execution latency, debug failures, and track requests across services in your observability stack.
Watch an overview of OpenTelemetry tracing in n8n:
What you get
When you turn on tracing, n8n exports two kinds of spans for each execution:
workflow.execute: One span per workflow execution. It records the workflow ID, name, version, project, node count, execution mode, status, and any error type.node.execute: One span per node execution, nested inside its workflow span. It records the node ID, name, type, version, the number of input and output items, and the execution, workflow, and project it belongs to.
n8n exports a span when it ends. To see a run while it's still going, turn on start spans.
Each span includes resource attributes that identify the n8n instance:
service.name(defaultn8n)service.version(the n8n version)n8n.instance.idn8n.instance.role(for example,main,worker, orwebhook)
n8n also handles trace context propagation:
Inbound: If a webhook request includes a W3C
traceparentheader, n8n uses it as the parent for the workflow span. This links the n8n workflow trace to the upstream caller.Outbound: HTTP Request nodes (and other nodes that use the n8n HTTP helpers) can inject a
traceparentheader into outbound requests. Downstream services that support W3C trace context can therefore continue the trace.Sub-workflows: A sub-workflow's span uses the parent workflow's span as its parent.
Resumed workflows: When a workflow resumes after a wait, the new span links back to the previous span using a span link.
Enable tracing in the UI
Instead of setting environment variables, you can configure tracing from Settings > OpenTelemetry. n8n applies your changes without a restart, and reloads them across workers and webhook processors in queue mode.
To configure tracing:
Select Settings > OpenTelemetry.
Turn on Enable OpenTelemetry.
Under Collector connection, enter your OTLP endpoint and any other connection details.
Under Tracing, set your sampling and span options.
Select Save settings.
To check that n8n can reach your collector, select Send test trace under Verify configuration. n8n sends a single test span and reports whether the collector accepted it. You can run this before or after you save.
Each field maps to an environment variable, shown in the field's tooltip. For the full list, see OpenTelemetry environment variables.
Enable tracing with environment variables
Set the following environment variables on each n8n instance you want workflow tracing enabled (main, workers, and webhook processors):
Restart n8n. The instance starts exporting spans over OTLP HTTP using the Protobuf encoding. To export over gRPC instead, refer to Choose the OTLP protocol.
n8n appends /v1/traces to the endpoint by default. Point N8N_OTEL_EXPORTER_OTLP_ENDPOINT at the base URL of your collector, not the traces path. The endpoint must be an http:// or https:// URL. If you set another scheme, or no scheme, n8n logs a warning and uses the default endpoint.
If your collector needs authentication, set N8N_OTEL_EXPORTER_OTLP_HEADERS to a comma-separated list of key=value pairs:
For the full list of supported variables, refer to OpenTelemetry environment variables.
Choose the OTLP protocol
n8n can export traces over two OTLP transports:
http/protobuf(default): OTLP over HTTP with Protobuf encoding. Collectors listen for it on port 4318 by convention. It works through proxies, ingresses, and firewalls that don't support HTTP/2, and it's simpler to debug.grpc: OTLP over gRPC. Collectors listen for it on port 4317 by convention. HTTP/2 multiplexing and binary framing give it lower overhead per export. The difference matters most at high span volume. It needs HTTP/2 support end to end. Some proxies and load balancers require explicit configuration for HTTP/2.
Keep the default http/protobuf unless your collector only accepts gRPC, or you export a high trace volume through infrastructure that supports HTTP/2.
To select the protocol, set the Protocol field in Settings > OpenTelemetry, or set the environment variable:
The variable name and its values (http/protobuf and grpc) match the upstream OTEL_EXPORTER_OTLP_PROTOCOL specification. As with the other fields in Settings > OpenTelemetry, n8n disables the Protocol field when you set the environment variable.
TLS
The endpoint scheme controls TLS for both protocols. https:// turns TLS on and http:// turns it off. n8n doesn't accept a grpc:// scheme.
To trust a custom certificate authority, or to present a client certificate for mutual TLS (mTLS), use the upstream OpenTelemetry variables OTEL_EXPORTER_OTLP_CERTIFICATE, OTEL_EXPORTER_OTLP_CLIENT_KEY, and OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE. For the http/protobuf protocol, NODE_EXTRA_CA_CERTS also works. n8n has no setting of its own for certificates.
gRPC behavior
The grpc protocol differs from http/protobuf in these ways:
Include the port in the endpoint. A gRPC endpoint without an explicit port connects to port 443, the gRPC default, not 4317. Write
http://<your-collector-host>:4317.gRPC endpoints take no URL path. n8n ignores the Trace path setting (
N8N_OTEL_EXPORTER_OTLP_TRACING_PATH) and hides its row in the UI. n8n keeps the saved value and applies it again if you switch back tohttp/protobuf.Custom headers become gRPC metadata. n8n converts the keys to lowercase. It skips entries that gRPC rejects, including
-bin-suffixed keys with text values, and logs a warning instead of failing startup.The startup connectivity check waits for the gRPC channel to become ready. A ready channel proves that n8n can open a TCP connection, complete the TLS handshake for
https://, and establish an HTTP/2 connection. It doesn't prove that the endpoint serves OTLP. Use Send test trace in Settings > OpenTelemetry to confirm that the collector receives spans. The check doesn't block startup.
Sampling
By default, n8n exports every trace. To reduce volume in busy instances, set N8N_OTEL_TRACES_SAMPLE_RATE to a value between 0 and 1:
n8n uses a trace ID ratio sampler, so the same trace ID is either fully sampled or fully dropped across all spans in the trace.
Reduce span volume
Each node in a workflow produces its own span. For workflows with lots of nodes, this can produce more data than you need. To export only workflow-level spans, set:
To stop n8n from injecting traceparent headers into outbound HTTP requests, set:
Start spans
A long-running workflow or node is invisible in your backend until its span ends. With start spans on, n8n also sends a 0 ms child span the moment the workflow or node starts. The start span carries the same attributes as its parent at that moment, so you know right away what's running, and in which workflow and project.
workflow.execute.started
workflow.execute
Send workflow start spans
N8N_OTEL_TRACES_EMIT_WORKFLOW_START_SPAN
node.execute.started
node.execute
Send node start spans
N8N_OTEL_TRACES_EMIT_NODE_START_SPAN
Both are off by default. Turn them on under Settings > OpenTelemetry > Tracing, or set the environment variables to true. Node start spans also need Include node spans (N8N_OTEL_TRACES_INCLUDE_NODE_SPANS) on. In queue mode, set the variables on the main instance and on every worker.
To use the public API, call PUT /api/v1/settings/otel with emitWorkflowStartSpan and emitNodeStartSpan, using an API key with the otel:manage scope. This call replaces all OpenTelemetry settings. n8n rejects a body that leaves out a required field. If you leave out an optional field, exporterProtocol, emitWorkflowStartSpan, or emitNodeStartSpan, n8n resets it to its default.
Crashed executions
An execution is crashed when the n8n instance running it stopped without recording a result, for example after an out-of-memory kill or a container eviction. n8n marks the execution crashed later, once it notices the execution can't finish. Before n8n 2.42.0, these executions produced no span, so the error rate in your trace backend missed every platform failure. From n8n 2.42.0, a crashed execution produces one workflow.execute span.
Receive crashed spans
The instance that notices the crash emits the span, not the instance that ran the execution. In queue mode, that's the main instance, so enable OpenTelemetry on the main instance, not only on the workers. In regular mode, the single main instance runs and checks its own executions, so the normal setup covers it.
Read a crashed span
A crashed span carries the usual workflow span attributes plus:
n8n.execution.status
crashed
n8n.execution.error_type
WorkflowCrashedError
n8n.execution.crash.detector
Which check found the crash. See How n8n detects a crash.
n8n.execution.reconstructed
false when n8n ended the original span, true when it rebuilt one. See Tracked and rebuilt spans.
The span has status error and an exception event.
How n8n detects a crash
stall
Queue mode. The worker stopped renewing its job lock.
queue-recovery
Queue mode. A periodic check found a running execution with no job in the queue.
startup-recovery
n8n restarted and found an execution still marked as running.
start-failure
The execution couldn't start.
workflow-deactivation
Someone unpublished the workflow while it had executions in progress.
Tracked and rebuilt spans
The instance that starts an execution opens its workflow.execute span. If the same instance detects the crash, it ends that span in place with n8n.execution.reconstructed=false. The span keeps every attribute set at start. Node spans still open on that instance end with n8n.node.termination_reason=workflow_crashed.
If a different instance detects the crash, for example after the main instance restarted, no open span exists. n8n rebuilds a workflow.execute span in the same trace from the trace context stored on the execution, with n8n.execution.reconstructed=true. A rebuilt span carries the workflow ID, workflow name, workflow version ID, project ID, project and workflow custom span attributes, execution ID, execution mode, n8n.execution.is_retry, and n8n.execution.retry_of. It lacks:
n8n.workflow.node_count.Node spans of its own. Node spans exported before the crash stay in the trace. The node that was running when the instance died has no span.
A span link to the previous span, when the execution had resumed from a wait.
Its workflow custom attributes come from the workflow's current settings, not the version that ran.
Crashed execution caveats
Duration is detection lag, not run time. A crashed span ends when n8n detected the crash, not when the instance died. That's one to two minutes for
stall, the next periodic check forqueue-recovery, and the next restart forstartup-recovery. Exclude spans withn8n.execution.status=crashedfrom latency metrics. Count them in error-rate metrics only.Rebuilt spans show a missing parent. The stored trace context is the original span's ID, and OpenTelemetry can't reuse a span ID. The rebuilt span is a child of the span that never ended, so Jaeger and similar backends flag a missing parent. Expect the warning.
Stalled queue jobs changed status. Before n8n 2.42.0, a job whose worker died ended as
error. From n8n 2.42.0 it ends ascrashedwith detectorstall. Alerts that match onlyn8n.execution.status=errormiss these.
Agent tracing
n8n can also emit detailed spans for agent runs built with the Agents feature, using the same tracer as workflow tracing. This covers agent runs started from a workflow, from chat integrations, and from scheduled tasks.
Agent tracing rides along with the rest of the OTel module. With N8N_OTEL_ENABLED turned on, add:
Set it to false to keep workflow and node spans while dropping agent spans.
By default, agent tracing records prompts, tool arguments, responses, and tool results. To exclude sensitive input or output data:
For the full list of variables, refer to OpenTelemetry environment variables.
What you get
Each agent run produces one root span, named <agent name>.generate or <agent name>.stream depending on whether the run streams its response. Each tool call the agent makes produces a nested execute_tool <tool name> span.
These spans use the OpenTelemetry GenAI semantic conventions (gen_ai.* attributes), so they're readable in any OTLP backend, not just tools built for LangSmith.
Custom span attributes
You can add custom attributes to project, workflow, and node spans. n8n exports each custom attribute as an OpenTelemetry span attribute to your configured observability backend.
Don't include secrets, personal data, or other sensitive values in attribute values.
n8n supports the following custom attribute levels:
Project
Project settings
workflow.execute
n8n.project.custom.<key>
Workflow
Workflow settings
workflow.execute
n8n.workflow.custom.<key>
Node
Node Settings tab
node.execute
n8n.node.custom.<key>
Add project span attributes
To add project-level span attributes:
Open a project.
Select Project settings.
Under Custom Span Attributes, add one or more span attributes.
Select Save.
Use plain text for project attribute values.
Add workflow span attributes
To add workflow-level span attributes:
Open the workflow.
Open Workflow settings.
Under Custom Span Attributes, select Configure.
Add one or more span attributes.
Select Save.
Use plain text for workflow attribute values.
Add node span attributes
To add node-level span attributes:
Open the node and select the Settings tab.
Under Custom Span Attributes, select Add Attribute.
Enter a Key. Keys must be plain text.
Enter a Value. Values can be plain text or expressions, such as
={{ $json.environment }}.
Node attribute values must resolve to a string, number, or boolean.
Add attributes programmatically in a custom node
If you're building a custom node, you can attach custom key-value pairs from code. Call setMetadata from the node's execute method:
n8n prefixes each key with n8n.node.custom. on the exported span. Values must be strings, numbers, or boolean.
This API isn't available from the Code node. It's intended for node authors who want to enrich spans with domain-specific data.
If a node sets an attribute key here that's also configured as a custom node span attribute, the programmatic value takes precedence.
Try it out with Jaeger
You can send traces to a local Jaeger instance to see them in action.
Save the following as
docker-compose.yml:
Start Jaeger:
Start n8n with tracing turned on and pointed at Jaeger. Refer to starting n8n for setup details:
Run a workflow, then open the Jaeger UI at http://localhost:16686 - Select "n8n" as service" and click "Find traces" to see the OpenTelemetry traces emitted by n8n.
Span attributes
Workflow and node spans include the following n8n-specific attributes.
Workflow span (workflow.execute)
n8n.workflow.id
Workflow ID.
n8n.workflow.name
Workflow name.
n8n.workflow.version_id
Workflow version ID.
n8n.workflow.node_count
Number of nodes in the workflow. Absent on a rebuilt span for a crashed execution.
n8n.project.id
Project ID. Available from n8n 2.23.0.
n8n.project.name
Project name. Team projects only: a personal project's name contains its owner's name and email, so n8n leaves it out. Available from n8n 2.44.0.
n8n.execution.id
Execution ID.
n8n.execution.mode
Execution mode (for example, manual, webhook, trigger, retry).
n8n.execution.status
Final execution status, including crashed for a crashed execution.
n8n.execution.is_retry
true if the execution is a retry.
n8n.execution.retry_of
The original execution ID, when the execution is a retry.
n8n.execution.error_type
Error class name, set when the execution fails. WorkflowCrashedError on a crash.
n8n.execution.crash.detector
Which check found the crash: stall, queue-recovery, startup-recovery, start-failure, or workflow-deactivation. See How n8n detects a crash. Available from n8n 2.42.0.
n8n.execution.reconstructed
true when n8n rebuilt the span after a crash, false when it ended the original span. See Tracked and rebuilt spans. Available from n8n 2.42.0.
n8n.continuation.reason
Set on a span link when the workflow resumes after a wait.
n8n.project.custom.<key>
Custom attributes set through project-level custom span attributes.
n8n.workflow.custom.<key>
Custom attributes set through workflow-level custom span attributes.
Node span (node.execute)
n8n.node.id
Node ID.
n8n.node.name
Node name.
n8n.node.type
Node type (for example, n8n-nodes-base.httpRequest).
n8n.node.type_version
Node type version.
n8n.node.items.input
Number of input items the node received.
n8n.node.items.output
Number of output items the node produced.
n8n.node.termination_reason
Why a node span ended without a normal completion (for example, workflow_cancelled or workflow_crashed).
n8n.node.custom.<key>
Custom attributes set through node-level custom span attributes in the node settings or metadata.tracing in custom node code.
From n8n 2.44.0, node spans also carry n8n.execution.id, n8n.workflow.id, n8n.workflow.name, n8n.project.id, and n8n.project.name, with the same values as their workflow span. To find every span of one execution in Jaeger, search for the tag n8n.execution.id=<execution-id>.
When a node fails, n8n records an exception event on the span with the standard OpenTelemetry exception attributes (exception.type, exception.message, exception.stacktrace).
Agent run span (<agent name>.generate or <agent name>.stream)
gen_ai.operation.name
Always invoke_agent.
gen_ai.agent.name
Agent name.
gen_ai.request.model
Model ID, as <provider>/<model name>, when known.
gen_ai.conversation.id
Thread ID.
gen_ai.prompt
Serialized prompt, tool count, and tool catalog. Omitted when N8N_AGENTS_TRACING_RECORD_INPUTS is false.
agent_id
Agent ID.
project_id
Project ID.
thread_id
Thread ID.
source
Where the run started (for example, workflow, or a chat integration name).
user_id
User ID, when known.
model_id
Model ID, as <provider>/<model name>, when known.
execution_id
Execution ID, for workflow-triggered runs.
workflow_id
Workflow ID, for workflow-triggered runs.
node_id
Node ID, for workflow-triggered runs.
Tool call span (execute_tool <tool name>)
gen_ai.operation.name
Always execute_tool.
gen_ai.tool.name
Tool name.
gen_ai.tool.call.id
Tool call ID.
gen_ai.agent.name
Agent name.
gen_ai.tool.call.arguments
Tool call arguments. Omitted when N8N_AGENTS_TRACING_RECORD_INPUTS is false.
gen_ai.tool.call.result
Tool call result. Omitted when N8N_AGENTS_TRACING_RECORD_OUTPUTS is false.
Troubleshooting
No traces appear in your backend
If n8n can't reach the OTLP endpoint at startup, it logs an error:
Check that:
N8N_OTEL_ENABLEDis set totrue.N8N_OTEL_EXPORTER_OTLP_ENDPOINTpoints at the base URL of the collector (not the/v1/tracespath).The collector is reachable from the n8n container or host.
Any required
N8N_OTEL_EXPORTER_OTLP_HEADERS(such as authentication tokens) are set.
If you use the grpc protocol, also check that the endpoint includes an explicit port. Without one, n8n connects to port 443, not 4317.
n8n logs OpenTelemetry diagnostics at warn level by default. Set N8N_LOG_LEVEL=debug to see more detail.
Startup connectivity error with gRPC and a private CA
The startup connectivity check for the grpc protocol uses the default TLS trust store. It doesn't read OTEL_EXPORTER_OTLP_CERTIFICATE, OTEL_EXPORTER_OTLP_CLIENT_KEY, or OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE, while the exporter honors them. A collector behind a private certificate authority, or one that requires mTLS, can fail the check and log Failed to connect to OpenTelemetry OTLP endpoint during startup while exporting works. The check doesn't block startup. Use Send test trace in Settings > OpenTelemetry to confirm that the collector receives spans.
Custom span attributes are missing
Check that:
You have an Enterprise license.
You set
N8N_OTEL_ENABLEDtotrue.For node-level span attributes,
N8N_OTEL_TRACES_INCLUDE_NODE_SPANSisn't set tofalse.
A crashed execution has no span
Enable OpenTelemetry on the main instance. In queue mode, the main instance detects the crash and emits the span, so enabling it on the workers alone produces no crash spans. Set the same variables on every instance type so worker spans keep their parent context.
The node that was running when the instance died has no span. Node spans exported before the crash stay in the trace. See Crashed executions.
A rebuilt span shows a missing-parent warning
n8n rebuilds the span from the trace context stored on the execution. That context is the ID of the original span, which never ended, and OpenTelemetry can't reuse a span ID. That makes the rebuilt span a child of a span the backend never received. Expect the warning. The span carries n8n.execution.reconstructed=true. See Caveats.
Worker traces are missing parent context
In queue mode, workers read the parent trace context from the database. If you only set the OpenTelemetry environment variables on the main instance, worker spans won't link to the parent workflow trace. Set the same variables on every instance type.
No agent spans appear
Agent spans depend on the OTel module. Check that:
N8N_OTEL_ENABLEDis set totrue.N8N_AGENTS_TRACING_ENABLEDis set totrue.
With N8N_OTEL_ENABLED set to false, agent runs complete normally, but n8n emits no spans for them, even with N8N_AGENTS_TRACING_ENABLED set to true.
Related resources
Last updated
Was this helpful?