LLM observability: trace every step, prompts off by default

To make an LLM (large language model) feature observable, give each user request one trace, with a span for every retrieval, model call and tool call. Record the model, latency, errors, finish reason, token counts and whether the task succeeded, and keep prompts, outputs and secrets out by default. The OpenTelemetry semantic conventions for generative AI name most of these fields, though their status is still Development.

The standards below come from the OpenTelemetry and OWASP docs, last verified on 11 October 2026; none of it was tested for this post. This is part 21 of the series.

An LLM feature here means one inside your product, such as a support assistant or a search box that answers questions. Part 20 threat-modelled a feature like that, and Part 15 covered the signals to watch during a rollout.

One trace per request, with a span for every step

A trace is the record of one request as it moves through your system. A span is one timed step inside it, with a name, a start, an end and some attributes. OpenTelemetry’s context propagation keeps the spans of one request together: “Propagation is the mechanism that moves context between services and processes.”

A made-up support assistant shows the shape. A user asks where their order is. The app searches the help articles, calls the model, runs an order lookup tool the model asked for, and calls the model again to write the answer:

POST /support/answer              your own server span
├── retrieval help-articles       search the help articles
├── chat example-model            model asks for a tool
├── execute_tool lookup_order     your code runs the tool
└── chat example-model            model writes the answer

The span names follow the conventions. A model call is {gen_ai.operation.name} {gen_ai.request.model}, per the client inference page. A retrieval is {gen_ai.operation.name} {gen_ai.data_source.id}, and a tool run is execute_tool {gen_ai.tool.name}, per the GenAI spans page.

Tool spans are often yours to add

An instrumentation library for your model client sees the model calls. It usually cannot see the tool, because your own code runs it. The spans page says so: “Application developers are encouraged to follow this semantic convention for tools invoked by their own code and to manually instrument any tool calls that automatic instrumentations do not cover.”

Retries hide inside one model span

A model span covers every automatic retry of that call. The spans page says that “the corresponding span SHOULD cover the duration of the logical operation with all retries.” One slow chat span may hold several failed attempts and a success.

To see the attempts, instrument the HTTP client too. OpenTelemetry’s HTTP span conventions add http.request.resend_count “to each repeated span”. The GenAI attribute registry had no retry count on 11 October 2026, so a retry loop in your own code needs an attribute of your own.

The GenAI conventions are still in Development

The GenAI conventions README gives their status as Development, and so do the spans, metrics and events pages. Error and server attributes, such as error.type and server.address, come from the core conventions and are marked Stable on the client inference page.

The conventions have also moved. The old page on opentelemetry.io now says: “GenAI semantic conventions have moved to the OpenTelemetry GenAI semantic conventions repository.” That repository had no tagged release on 11 October 2026.

Names still change. The GenAI metrics page now says gen_ai.client.operation.duration “SHOULD NOT be reported for inference operations; use gen_ai.client.inference.duration instead.” A library built against an earlier version may still send the old name. Check what your instrumentation library actually sends, pin its version, and keep the names your dashboards use in one place.

What to record, what to opt into, what to keep out

The fields of an LLM trace fall into three groups. The first group is recorded by default. The second group is content, which the conventions mark Opt-In. The third group comes from the OWASP Logging Cheat Sheet and has no convention name, because it should not reach telemetry at all.

The convention rows come from the client inference page, the spans page and the attribute registry, read on the main branch on 11 October 2026.

The conversation row has a trap. For a request with no conversation id, the client inference page says “a new UUID, a trace identifier, or a hash of request content SHOULD NOT be used as a fallback value.”

Prompts, outputs and secrets stay out by default

The OpenTelemetry GenAI conventions start from the safe side. The spans page calls model instructions, user messages and model outputs “sensitive” and “often large in size”. Instrumentations “SHOULD NOT capture them by default, but SHOULD provide an option for users to opt in.”

The same page gives three patterns:

1. The default: don’t record instructions, inputs or outputs.
2. Record them on the spans. The page suggests this when “telemetry volume is manageable” and privacy rules are met, “for example, in pre-production environments”.
3. Store the content somewhere else and record a reference on the span. The page recommends this “in production environments where telemetry volume is a concern or sensitive data needs to be handled securely.”

The third pattern keeps prompts away from everyone who can read your traces. The page notes that “Using external storage enables separate access controls.”

Redact before the data leaves the process

The GenAI spans page gives content its own exit, where you can clean it. It describes an upload hook that “SHOULD be able to enrich and modify provided span, instructions, and message objects”. It also allows the upload to happen “in the telemetry processing pipeline (in-process or via a collector)”.

The page also says the hook runs “regardless of the span sampling decision”. Sampling fewer traces does not reduce the content you store.

For anything else in your logs, OWASP’s data to exclude list applies. It also suggests “deletion, scrambling or pseudonymization” of identifiers “where the individual’s identity is not required”.

Sample the boring traces, keep the failed ones

Sampling means keeping a share of traces instead of all of them. OpenTelemetry’s sampling page warns that with head sampling, decided at the start of a request, “you cannot ensure that all traces with an error within them are sampled”. Tail sampling decides after looking at all or most of the spans, so it can keep every trace with an error or high latency.

Keep latency and token counts as metrics as well. A metric is counted over every request, while a sampled trace keeps only some of them.

Decide retention and access before the first export

OWASP’s disposal rule sets the retention limit both ways. Logs “must not be destroyed before the duration of the required data retention period, and must not be kept beyond this time”.

Access needs the same care. OWASP’s protection section says: “The privileges to read log data should be restricted and reviewed periodically”. Traces from an LLM feature need the same rules as your logs. Stored prompts need the rules of your user data.

Put the business outcome on the same trace

A trace that ends without an error can still be a failure for the user. The answer may arrive quickly and still be wrong. So record whether the task succeeded, on the same trace as the model calls.

The GenAI registry had no attribute for task success or user feedback on 11 October 2026. OpenTelemetry’s naming guidance says a company-specific name should start with “your company’s reverse domain name”. It also advises against putting your own names under an existing OpenTelemetry namespace, such as gen_ai. Some example attributes for the support assistant, with made-up names:

com.example.support.outcome        resolved | handed_to_human | abandoned
com.example.support.feedback       thumbs_up | thumbs_down
com.example.support.orders_found   0, 1, 2 ...

Feedback often arrives after the trace has ended. Record it as an event that carries the trace id or gen_ai.response.id, so you can join the two later. The conventions use that pattern for automated quality checks: the gen_ai.evaluation.result event “SHOULD be parented to GenAI operation span being evaluated when possible or set gen_ai.response.id when span id is not available.”

With the outcome on the trace, you can compare resolved answers by gen_ai.response.model or by gen_ai.prompt.version. That comparison shows when a model or prompt change made things worse.

Two made-up failures, and the span that explains each

The two failures in this section are made up, to show how the recorded fields explain a user’s complaint.

A user says the answer stopped mid-sentence. The second chat span shows gen_ai.response.finish_reasons as ["length"], and its gen_ai.usage.output_tokens matches the gen_ai.request.max_tokens you set. The model ran out of room. The fix is a higher limit or a shorter answer format.

Another user says the assistant gave the wrong order status. The execute_tool lookup_order span has error.type set to timeout, and the chat span after it still finished with ["stop"]. The model wrote an answer without the tool result. That one is a bug in how your code handles a failed tool, and the trace points straight at it.

Two prompts for tracing an LLM feature

The two prompts below are for a coding agent working in the repository of your LLM feature. Neither relies on a tool-specific feature.

Prompt 1: audit what your LLM feature already records

An example prompt to find what your telemetry records today, before you add anything.

Task: audit the telemetry of <the LLM feature or service>.

Do not change any code. Read only what this task needs.

1. Find every place the code calls a model, runs a retrieval or runs a
   tool the model asked for. Cite the file and line.
2. For each one, list the spans, metrics, logs and attributes it records,
   and which library records them.
3. Check them against the OpenTelemetry GenAI semantic conventions:
   gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model,
   gen_ai.response.model, error.type, gen_ai.response.finish_reasons,
   gen_ai.usage.input_tokens, gen_ai.usage.output_tokens.
4. Flag anything that can carry prompts, model output, tool arguments,
   tool results, secrets, session ids or personal data into telemetry,
   including debug logs and error messages.
5. Check whether any setting turns on content capture, and in which
   environments.
6. Check whether one user request produces one connected trace.

Never print, expose or commit secret values. If you find one, name the
file and line only.
Do not commit, push, deploy, run destructive commands or make external
changes unless I explicitly authorise it.
If two reasonable attempts fail, stop and report the failure rather
than repeatedly guessing.

End with a table: signal, where it is recorded, risk, suggested change.

Check every flagged line yourself before deleting or masking anything.

Prompt 2: add tracing to one request path

An example prompt to instrument one request path, with a plan you approve first.

Task: add tracing to <the request path, for example POST /support/answer>.

First write a plan and wait for my approval. The plan lists:
- one trace per request, with child spans for each retrieval, model call
  and tool call, named as the OpenTelemetry GenAI conventions suggest
- the attributes for each span, using the convention names, and the
  library version that emits them
- an attribute under <our reverse domain prefix> for the task outcome
- where retries are counted

Rules:
- Do not record prompts, system instructions, model output, tool
  arguments, tool results or retrieval queries.
- Do not record secrets, session ids or personal data.
- Add a test that fails if any gen_ai.input.messages,
  gen_ai.output.messages or gen_ai.system_instructions attribute is set.

Never print, expose or commit secret values.
Do not commit, push, deploy, run destructive commands or make external
changes unless I explicitly authorise it.
If two reasonable attempts fail, stop and report the failure rather
than repeatedly guessing.

The test in the last rule keeps content capture from being switched on by accident later.

Quick answers

What is observability for an LLM application?

Observability for an LLM (large language model) application means each user request leaves one trace, with timed spans for retrieval, model calls and tool calls. The spans carry the model, latency, errors, finish reason and token counts, so you can explain a bad answer.

Should you log prompts and model responses in production?

Logging prompts and model responses is off by default in OpenTelemetry’s generative AI semantic conventions, which say instrumentations should not capture them unless you opt in. For production, they recommend storing that content externally, with separate access controls.

Are the OpenTelemetry GenAI semantic conventions stable?

The OpenTelemetry GenAI (generative AI) semantic conventions had the status Development on 11 October 2026, in their own repository with no tagged release yet. Attribute and metric names can still change, so check what your instrumentation library emits.

What is gen_ai.response.finish_reasons?

gen_ai.response.finish_reasons is an OpenTelemetry GenAI attribute that records why the model stopped generating, one value per generation. A value such as length usually means the answer hit the token limit, and error means the generation ended abnormally.

How do you track whether an LLM feature helps users?

Tracking whether an LLM feature helps users needs a business outcome on the same trace, such as whether the task was resolved and the user’s feedback. Name those attributes under your company’s reverse domain prefix, outside the gen_ai namespace.

More on AI coding is in the AI category.