What is OpenLLMetry? What it records, tested on 0.62.4

OpenLLMetry is a set of OpenTelemetry extensions that turn each LLM call into a span carrying the prompt, the completion, and the token counts.

After tracing a stubbed OpenAI call with version 0.62.4 and reading all 3 spans, here's what OpenLLMetry records and where the data goes.

What is OpenLLMetry? The 30-second answer

OpenLLMetry is Traceloop’s open-source extension of OpenTelemetry, released under the Apache 2.0 license. Dynatrace's knowledge base spells the name out as "Open Large Language Model Telemetry."

In practice, you add 2 lines of Python, the import and Traceloop.init(), and your calls to supported providers start producing spans that OpenTelemetry-compatible backends can read.

Traceloop's situation changed in March 2026, when it announced it's joining ServiceNow. The post states that OpenLLMetry "will remain open-source" and that Traceloop plans to keep contributing to the OpenTelemetry ecosystem.

Key features

The SDK. Install it with pip install traceloop-sdk. Beyond Python, Traceloop's docs also cover JavaScript/TypeScript and Go.

Two other languages come with caveats. The Ruby guide calls the Ruby SDK "still in beta" and says it needs manual logging of what you send to the model and what comes back. A Java repo exists, but Traceloop’s docs have no Java guide yet.

Decorators. @workflow, @task, @agent, and @tool mark the entry points you want grouped in a trace. Names are optional. For supported frameworks, Traceloop's annotations page adds that "OpenLLMetry will automatically detect the framework and annotate your traces."

What it captures. The openai.chat span carried the model name, the finish reason, and the full input and output messages. Token counts were 14 in, 11 out, and 25 total. Splunk's Lantern page also lists latency, error rates, and agent and tool activity.

Cost is the exception. None of the 3 spans carried a cost attribute, so you have to calculate it from the token counts in your backend. Langfuse, for example, multiplies model prices by token usage.

Coverage. The README lists 16 model providers, including Anthropic, Bedrock, Cohere, Gemini, Groq, and OpenAI. It also lists 7 vector databases, such as Chroma, Pinecone, Qdrant, and Weaviate.

The README also covers 10 frameworks, such as LangChain, LangGraph, LlamaIndex, and CrewAI, and it supports MCP, the Model Context Protocol.

How does OpenLLMetry work?

OpenLLMetry plugs LLM-aware instrumentation into the OpenTelemetry pipeline, so each LLM call becomes a span. Traceloop's docs say "Tracing is done in a non-intrusive way."

1. Initialize once. Traceloop.init() runs at startup, before your first request to a model.

2. Group with decorators. Decorated functions turn into workflow, task, agent, or tool spans.

3. Capture each call. An instrumented LLM call becomes a span carrying the model, tokens, prompt, and completion.

4. Export. Traceloop's introduction puts it this way: "You can choose to export the traces to Traceloop, or to your existing observability stack."

Here's the trace the run produced. The endpoint was a local stub that returned a canned reply and never called a model.

joke_workflow.workflow
└─ joke_creation.task
   └─ openai.chat (input_tokens = 14, output_tokens = 11)

The parent IDs matched at every level. openai.chat sat under the task, and the task sat under the workflow.

OpenLLMetry vs. OpenTelemetry: what's the difference?

The main difference between OpenLLMetry and OpenTelemetry is that OpenTelemetry covers application telemetry broadly, while OpenLLMetry adds the LLM attributes that default instrumentation does not capture.

New Relic's docs call OpenTelemetry "a powerful standard for collecting general application data (traces, metrics, logs)." Dynatrace notes that prompt input and the LLM's name, version, and token counts are "not captured by default."

Use OpenLLMetry for traces that include an LLM call, since OpenTelemetry’s default auto-instrumentation doesn’t capture prompts, model names, or token counts. OpenTelemetry’s own GenAI instrumentations cover some providers, with prompt capture off by default.

Services without LLM calls can stay on plain OpenTelemetry, because OpenLLMetry adds to the same setup.

Where does OpenLLMetry send its data?

OpenLLMetry can send traces to backends that accept OTLP, and Traceloop’s integrations page shows 30 destinations. Each vendor documents its own setup.

The same page includes Datadog, Grafana Tempo, Honeycomb, Dash0, Sentry, SigNoz, ServiceNow Cloud Observability, and 18 more beyond the 5 in the table. If you check the README instead, its 24 destinations leave out Langfuse, so use the integrations page.

What I liked and didn't like about OpenLLMetry

I tested OpenLLMetry 0.62.4 against a stubbed OpenAI call in a Python 3.13 sandbox, so these notes cover setup and span output, and I didn’t test production load.

Pros (what works)

• It took 2 lines of Python to get a first trace. Once the import and Traceloop.init() were in place, my OpenAI call produced an openai.chat span with token counts, and I changed nothing in the call. Adding @workflow and @task produced the full span tree.

• Content logging can be turned off. With TRACELOOP_TRACE_CONTENT=false, the message text disappeared from my spans while the model and the token counts remained.

• The backend list is long. Datadog, Honeycomb, and ServiceNow Cloud Observability are all on it.

Cons (what got in my way)

• Prompts are written to spans by default. My test prompt contained a fake Social Security number, and it appeared in plain text under gen_ai.input.messages.

• With content logging on, the decorators in my run logged return values too, and traceloop.entity.output contained the stub's canned reply.

• You have to calculate cost yourself, since none of the 3 spans carried a price.

• On Python 3.13 in my sandbox, the cuid dependency failed to install until I added setuptools and used --no-build-isolation. Then the import failed on a missing httpx. I did not test other Python versions.

Should you use OpenLLMetry? My take

Use OpenLLMetry when an LLM app already sends traces to an OpenTelemetry-compatible platform. I'd look at other tools when the main goal is a prompt-and-eval workspace.

Who OpenLLMetry fits

• If you’re already running Datadog, Grafana, Honeycomb, or Dynatrace and want LLM spans in the same trace as their services.

• Services in a supported language that call providers directly or through LangChain or LlamaIndex.

• Multi-step workflows where decorators can map workflow, task, agent, and tool spans.

• Companies that want prompt logging off in production and on in staging.

Who should skip OpenLLMetry?

• If you want a hosted workspace for prompts, datasets, and scoring from the start.

• Teams that write Java services, which the docs don't cover yet (see the FAQ).

• When you need to know whether answers were correct, since a trace records what happened, and grading needs a separate tool.

OpenLLMetry or Langfuse

OpenLLMetry and Langfuse can run together, according to both vendors' docs. Langfuse's OpenTelemetry page names OpenLLMetry among "Popular OpenTelemetry libraries," and Traceloop publishes its own Langfuse integration page.

Cekura's Langfuse vs. LangSmith comparison describes Langfuse as tracing everything your LLM does, with cost tracking, latency metrics, and dashboards you can configure. Cost tracking is the feature I found missing above. For how Langfuse compares with Helicone, see Cekura’s Helicone vs. Langfuse test.

Use the Langfuse SDK if Langfuse is the only backend you need, and OpenLLMetry if you want the same traces to reach other backends too.

How to get started with OpenLLMetry in 5 steps

These 5 steps take a Python app from no tracing to a first span, in the order I ran them.

1. Install the SDK. Run pip install traceloop-sdk.

2. Initialize it. Add Traceloop.init(app_name="joke_service", disable_batch=True) at the top of your entry file, where disable_batch=True is for local debugging.

3. Decorate your entry points. Wrap them with @workflow and @task, imported from traceloop.sdk.decorators.

4. Choose a destination. For Traceloop, run export TRACELOOP_API_KEY=your_api_key_here. For another destination, set TRACELOOP_BASE_URL and TRACELOOP_HEADERS, as in the New Relic example above.

5. Run it and open one span. Look for gen_ai.usage.input_tokens and the model name.

Here is the setup in one file.

from traceloop.sdk import Traceloop
from traceloop.sdk.decorators import workflow, task

Traceloop.init(app_name="joke_service", disable_batch=True)

@task(name="joke_creation")
def create_joke():
    ...  # your OpenAI call goes here

@workflow(name="joke_workflow")
def run():
    return create_joke()

run()

Tip. The Python guide suggests disabling batch sending locally "so you can see the traces immediately."

OpenLLMetry best practices from my test run

These practices come from my test run and Traceloop’s docs.

• Turn content logging off in production unless you need the text. Traceloop's privacy page states the default, "By default, OpenLLMetry logs prompts, completions, and embeddings to span attributes." It also describes overrides per workflow, task, agent, tool, and user.

• Check what your decorators record. In my run, with content logging on, a decorated function's return value was written to traceloop.entity.output, which means a function that returns a customer record would store that record in a span.

• Start by decorating entry points. The annotations docs say OpenLLMetry detects supported frameworks and annotates traces for you.

• Skip the deprecated async decorators. Traceloop marks @aworkflow and @atask deprecated in favor of the standard decorators, which handle sync and async functions.

Where tracing ends and agent quality testing starts

In my stubbed run, the openai.chat span reported finish reason stop, which only tells you the call ended. Checking whether the reply was right takes evaluation, which is separate from observability.

Cekura's guide to LLM monitoring separates monitoring, observability, and evaluation, and OpenLLMetry belongs to the observability layer.

For voice and chat agents, Cekura covers that grading step. Its observability docs describe three ways to import production calls: a Custom API, Auto-fetch from Vapi, ElevenLabs, Retell, and Bland AI, or a webhook.

Cekura covers that step with these features, among others:

• Simulations run synthetic conversations before launch.

• Red teaming sends jailbreak and persona-substitution probes against the agent.

• Observability scores the calls you import against the metrics you define, and trend alerts fire when a metric moves away from its baseline.

Cekura lists integrations with Vapi, Retell, ElevenLabs, LiveKit, and Pipecat, and it can pull production calls from Bland AI.

It’s SOC 2-, HIPAA-, and GDPR-compliant for transcript redaction, role-based access, and audit trails.

My verdict on OpenLLMetry

OpenLLMetry suits LLM apps that already export OpenTelemetry data. I tested the Python SDK only, and the 2-line setup and the span hierarchy worked as expected locally.

I didn't test any of the 30 backends on Traceloop's integrations page, which leaves the export step unverified on my side. Before production, decide whether prompt logging stays on and where you will calculate cost.

If you want a prompt-and-eval workspace, start with Langfuse. To check whether your agent answered correctly, add a tool that scores conversations.

Test agent quality beyond tracing

Once OpenLLMetry shows you what each call did, Cekura scores those calls against the metrics you define. Book a demo to see simulations and production scoring on your own agent.

Frequently asked questions

Is OpenLLMetry free?

Yes, OpenLLMetry is free and open source under the Apache 2.0 license, according to Dynatrace's knowledge base. Charges depend on the platform you choose, and OpenLLMetry itself adds no license fee.

Does OpenLLMetry support Java?

No, not in the docs yet. Traceloop’s docs have guides for Python, JavaScript/TypeScript, Go, and Ruby (beta), and none for Java.

The org's openllmetry-java repo had 2 commits and 3 stars when I checked on October 11, 2026.

What is the difference between OpenLLMetry and OpenTelemetry?

The main difference between OpenLLMetry and OpenTelemetry is that OpenLLMetry adds LLM data, such as model name, token counts, prompts, and completions, to OpenTelemetry's general traces, metrics, and logs.

It exports in the same format, so backends that accept OTLP can read it.

Is OpenLLMetry an alternative to Langfuse?

Partly, at the tracing layer. OpenLLMetry overlaps with the Langfuse SDK there, and it can also send its traces into Langfuse. According to both vendors' docs, OpenLLMetry produces the traces, and Langfuse receives them as OTLP over HTTP, so you can run both.

What is the best tool for testing AI agents traced with OpenLLMetry?

For voice and chat agents, I'd use Cekura, the product my team builds. It simulates full conversations before launch and scores live calls afterward.