LLM Observability Explained: Why Your AI Lies to You

The biggest problem with your AI app is that you have invited a pathological liar into your stack. Language models are built to be helpful and confident, and like a certain kind of developer, they are the last to admit something is wrong. When the backend is on fire, the model sweeps it under the rug and puts on a show, swearing everything is fine while returning a tidy 200 OK.

That gap broke monitoring. Before AI, the question was simple: is this service running? Now it is a harder one: is this answer right? Closing that gap is what LLM observability does, and this article walks through what it is, how the pieces fit, the part nobody has solved yet, and how to start for free.

What is LLM observability?

LLM observability is the practice of recording everything an AI application does so you can tell whether its answers are actually correct. That means capturing not just the final reply but the whole path the model took: the prompts you sent, the documents it pulled back, the tools it called, and the tokens it burned, all timestamped and saved. It is one of the fastest-moving corners of AI tooling right now.

The reason it had to become its own discipline is that AI broke the old model of correctness. A normal program is either up or down, and a bug throws a loud error. An AI app can be perfectly up, return a clean status code, and still hand you an answer that is confidently, completely wrong. You cannot see that from a uptime dashboard, so you need a different kind of visibility.

Why traditional monitoring went blind on AI

Traditional monitoring leans on three assumptions, and AI breaks all three. The tools we built for the last twenty years assume software is deterministic, that failures are loud, and that correctness is a yes-or-no. None of that survives contact with a language model, which is exactly why a green dashboard can sit on top of a broken product.

Run a normal function twice and you get the same result. Run a prompt twice and you can get two different answers, one solid and one nonsense, like dropping a marble into a Plinko board. When a normal service breaks it screams, with an exception and something red and obvious. An LLM fails politely, status code two hundred, tipping its fedora on the way out. And “did this work” used to be a boolean, but “was this answer any good” is a slider, and your old dashboard has no idea where the needle sits.

01 Same input, same output

Old world Run a function twice, you get the same result.

With AI Run a prompt twice, you can get two answers, one solid and one nonsense.

02 Failures scream

Old world A crash, an exception, something red and obvious.

With AI It fails politely. Status code 200, a wrong answer, no alarm anywhere.

03 Correctness is yes or no

Old world The request either worked or it did not.

With AI “Was this answer any good” is a slider, and the old dashboard cannot read it.

That is the difference between monitoring and observability, and for AI the two finally came apart. Monitoring watches the signals you knew to collect in advance: uptime, latency, error rate. Observability is about being able to ask new questions of a running system after the fact, including the one that matters most here, “why did it give that answer?”

What is a trace, and what are spans?

A trace is the full story of one request, from the moment it arrives to the answer that leaves, and spans are the individual steps inside it. Picture the trace as a flight recorder for a single request and each span as a line in the transcript. One span shows where the app searched the database, the next shows what came back, then the prompt it assembled, then the answer it gave.

When the final answer comes out wrong, you stop guessing and scroll back through the recording to find the exact step where it went sideways. A fluent, confident reply at the end tells you nothing about where the rot started. The trace does, because the bad step is sitting right there in the timeline with everything that led to it.

1 trace · one request, end to end

1. 01 +0 ms Request arrives “What were our Q3 refunds?”
2. 02 +40 ms Search the database Vector lookup over the docs
3. 03 +210 ms Documents come back Pulled the Q2 sheet, not Q3 root cause
4. 04 +230 ms Prompt assembled Question + the wrong document
5. 05 +1.9 s Model answers Confident, fluent, and wrong

The answer was wrong, but step 5 looked perfect. The rot started at step 3. Without the recording you would be guessing.

The reassuring part is that none of this is new technology. We have traced requests across messy distributed systems for years, and there is an open standard for it called OpenTelemetry, OTel for short. Its Generative AI conventions are still marked experimental as of mid-2026, but the direction is set: OTel is becoming the shared language for AI observability, which means you are not welded to one vendor’s box.

Why agent observability is harder than tracing one call

Tracing a single model call is the easy part, and it is mostly solved. One prompt goes in, one answer comes out, you log the pair with the token count, and you are basically done. The result is a flat little record with nothing to untangle.

An agent is a different animal, because it does not answer once and stop. It loops: it makes a plan, calls a tool, reads the result, decides it is not happy, re-plans, calls another tool, maybe hands the whole job to a sub-agent, and around it goes. One user question can spin off twenty or thirty steps, and your tidy flat record becomes a sprawling tree of spans nested inside spans, sometimes dozens deep.

One model call

prompt in answer out 1,240 tokens · 1.9 s

A flat record. Log the pair, you are basically done. Mostly solved.

One agent run

• invoke_agent the user question
• plan decide the next move
• call_tool: search reads the result
• re-plan not happy, tries again
• call_tool: fetch and again
• hand off → sub-agent …and around it goes
• + 20 to 30 more steps…

A sprawling tree, dozens deep. Where the nasty bugs live.

$47K One team's research agent looped for 11 days before the billing alert blinked. Every single step returned a cheerful 200, so nothing stepped in to stop it.

That tree is where the nasty bugs live. A single call cannot get stuck in a loop, but an agent absolutely can, and since every step returns a cheerful two hundred, nothing steps in to stop it. The money is not even the scary part. A wrong turn at step two does not announce itself, it poisons step nine, which poisons the final answer, which lands looking perfectly reasonable. Without the tree, you would never trace the rot back to where it started.

How do you evaluate an LLM’s output?

You grade it, with a second piece bolted on top of tracing called evaluation. A flight recorder is dead weight if no one reads the output, so you score the answers, and that scoring comes in two flavors. Offline evals run before you ship, against a fixed set of questions whose answers you already know, the same idea as unit tests except the thing under test is fuzzy language. Online evals run live, scoring a sample of real traffic as it flows through production.

The smart move is what you do with the failures. Every time your app blows it in the wild, you do not just patch it and move on. You grab that exact failure, label it, and drop it into your test set for good, so it cannot sneak past you twice. Braintrust describes exactly this loop, where production traces that reveal new edge cases get folded into your golden set.

Offline eval Before you ship

Run against a fixed set of questions you already know the answers to. Same idea as unit tests, except the thing under test is fuzzy language.

Online eval Live in production

Score a sample of real traffic as it flows through. No answer key, so the grading itself gets fuzzy (which is where the judge comes in).

The flywheel

App fails in the wild → Grab + label the exact failure → Add it to the test set for good → It can't sneak past twice

Eval-driven development: your traces turn into a growing pile of tests.

People call this eval-driven development, and it is the closest thing the field has to a discipline. The principle on evaldriven.org puts it bluntly: if your evals do not run on every change, they do not exist, so they belong in the pipeline right next to lint and build. Your traces stop being a rear-view mirror and turn into a growing pile of tests.

What is LLM-as-a-judge, and can you trust it?

LLM-as-a-judge means using a second model to grade the first one, and you can trust it about as far as you coach it. Online is where grading gets awkward: there is no answer key and no human to read every reply, yet you still need to score fuzzy language at scale. The popular answer is a little circular. You hand a second model the first one’s output, give it a rubric, and ask it to score the work.

It works better than it has any right to. The original MT-Bench study found a strong judge agreed with human reviewers over 80% of the time, which is roughly how often two humans agree with each other. But the same paper found the judge plays favorites. It leans toward whichever answer it sees first, it assumes longer means better, and best of all, it rates its own writing higher than anyone else’s.

~80%

A strong judge agreed with human reviewers about 80% of the time, roughly how often two humans agree with each other. It works better than it has any right to. Then the same study found the catch.

Position bias

It leans toward whichever answer it sees first. Swap the order and the verdict can flip.

Verbosity bias

It assumes longer means better, and rewards padding over a tighter, correct answer.

Self-preference

It rates writing from its own model family higher than anyone else’s.

Nobody is throwing it out, because it is the only thing that scales. They just coach it. Swap the order of the answers and average the scores. Use a judge from a different model family than the one on trial. Keep a human in the loop for the calls that matter. The lesson underneath is permanent: the moment you watch an AI with another AI, you have to keep half an eye on the watcher too.

The frontier nobody has nailed yet

Everything so far is the solved part. The unsolved part, the stuff that eats your weekends, is watching the agent’s decisions instead of its outputs. Think about how agents really go wrong, and notice that none of it shows up in the final span.

One agent drifts off the original task after a few turns, still sounding completely confident, just answering a subtly different question than the one you asked. Another retrieves ten documents and the one wrong document steers the whole decision. A third hands off to a sub-agent with half the context it needed, so the next step is valid on its own and wrong in the big picture. All three produce a clean-looking final answer.

Intent drift

It wanders off the original task after a few turns, still sounding completely confident, just answering a subtly different question than the one you asked.

The wrong chunk steers

It retrieves ten documents and the one wrong document quietly steers the whole decision. The final answer never mentions it.

Lossy handoff

It hands a sub-agent half the context it needed, so the next step is perfectly valid on its own and completely wrong in the big picture.

15–20% of long agent runs silently drift off-objective, by one practitioner's estimate. We can see what the agent did, but not reliably why it decided to.

That is the part nobody has standardized. There is no agreed schema for tracing an agent’s intentions and memory the way there is for a plain API call, so teams hand-roll it, bolting custom notes onto every step and hoping it holds together. We can see what the agent did, but not reliably why it decided to. Treat that 15 to 20% figure as one practitioner’s estimate, not a measured average, but the shape of the problem is real.

How do you start with LLM observability?

The single best piece of advice from people who have shipped this: add observability early, before production turns painful, not after. The teams who wait end up debugging with logs and vibes at two in the morning, reconstructing a failure they no longer have the data to see.

You do not need to buy a platform on day one. The cheapest start is a proxy: route your calls through it, change one line, and every request and its cost gets logged. One step up, an open-instrumentation library wires full traces into your app in a couple of lines, and because it speaks OTel, you can point it at almost any tool later. Add a single eval on top, keep your prompts in version control like the rest of your code, and you have the bones of a working setup.

1. 01
   Route through a proxy Change one line. Every request and its cost gets logged.
2. 02
   Add an open-instrumentation library A couple of lines wires full traces in. It speaks OTel, so you can point it anywhere later.
3. 03
   Bolt on a single eval One check that grades the output. Grow the set from real failures.
4. 04
   Keep prompts in git Version them like the rest of your code. Beats a dedicated prompt tool.

Add observability early, before production turns painful. The teams who wait end up debugging with logs and vibes at 2 a.m.

A word on the tools themselves. It is a crowded, noisy space, and it is consolidating fast, with big infrastructure players buying up the popular ones as we speak. In early 2026 alone, ClickHouse acquired Langfuse, Mintlify acquired Helicone, OpenAI moved to acquire Promptfoo, and Databricks bought Quotient.

The liar is still in your stack

The pathological liar has not left. It is still confident, still happy to swear everything is fine while the backend smolders. What has changed is that you now have the recordings. You can scroll the trace, find the span where it went wrong, grade the output, and fold that failure into a test so it never happens twice.

That is the whole point, and it is worth saying plainly. Monitoring told you the thing was running. Observability is how you find out whether it was telling the truth.

For the animated version, with the Plinko board, the fedora-tipping 200 OK, the runaway agent loop, and the biased judge all on screen, watch the breakdown on YouTube.