Online Evaluation & Drift Monitoring for LLM Systems (industry practice)

Type: article (synthesized industry best-practice; multiple practitioner sources) Tier: 3 (Engineering Guide) URL: https://langfuse.com/blog/2025-03-04-llm-evaluation-101-best-practices-and-challenges Accessed: 2026-06-09 Primary references:

• Langfuse — "LLM Evaluation 101: Best Practices and Challenges" — https://langfuse.com/blog/2025-03-04-llm-evaluation-101-best-practices-and-challenges
• Databricks — "Best Practices and Methods for LLM Evaluation" — https://www.databricks.com/blog/best-practices-and-methods-llm-evaluation
• VentureBeat — "Monitoring LLM behavior: Drift, retries, and refusal patterns" — https://venturebeat.com/infrastructure/monitoring-llm-behavior-drift-retries-and-refusal-patterns

The core distinction (carried from the concept map)

Observability records what happened; evaluation decides whether it was good. A trace tells you latency rose and tokens grew; it does not tell you the answers got worse. Production needs both: cheap telemetry on every request, plus sampled quality evaluation.

Offline vs. online evaluation — the loop

• Offline eval (Project 07): a frozen set, scored before shipping, to catch regressions.
• Online eval: runs continuously in production to surface drift and failure modes the

frozen set never anticipated. The two together are worth more than either alone — offline gates the release, online watches the deployment.

How online evaluation is run in practice

• Establish a baseline from initial deployment, then continuously compare live behavior

against it.

• Sample, don't grade everything. A background LLM-judge asynchronously samples ~5–10% of

traffic and grades it against the same rubric the offline eval used. Critically, the judge must not run synchronously on the request path — that doubles latency and cost. Quality scoring is a background job over sampled traces, not a gate on every response.

• Reuse offline asserts online. The pass/fail checks from the offline suite can score sampled

production traffic; a spike in malformed-output rate is the earliest warning of drift.

What a trace contains (the unit of monitoring)

A trace records everything the system did to produce a response: the user query, retrieved documents with relevance scores and source ids, tool calls with arguments and results, the assembled prompt, and the model's response — plus latency, tokens, cost, and finish reason. This is exactly what the OTel GenAI conventions standardize so tools agree on the field names.

Drift detection strategies

• Operational drift: error rate, latency, token/cost per route trending off the baseline.
• Behavioral drift: refusal-rate spikes, retry storms, malformed-output rate.
• Distributional drift: the input embedding distribution moving away from the golden set's

coverage — queries arriving that the system was never evaluated on — often before failure rates climb.

The monitoring principle that mirrors P09

Aggregate per route / per operation, not as one global mean. A healthy overall average can hide a single route that has silently gone to a 0% pass-rate or a 10× latency — the same trap Project 09 names for routing accuracy. Drift detection must be per-segment.

Why This Source Matters

It is the "decide whether it was good, continuously" half that the OTel telemetry ("record what happened") cannot provide alone. The learner builds a monitor that aggregates spans per route and flags the route that regressed against a baseline — the smallest real online-eval loop.

Key Claims

• Observability records what happened; evaluation decides whether it was good — production needs both: cheap telemetry on every request plus sampled quality evaluation.
• Online evaluation runs a background LLM judge over ~5–10% sampled traffic against the same offline rubric; it must not run synchronously on the request path (doubles latency and cost).
• Drift is operational (error/latency/cost), behavioral (refusal spikes, retry storms, malformed output), and distributional (input-embedding shift) — and must be aggregated per route, never as one global mean that hides a regressed segment.

Relevant To

• Elective 04 — Observability & Ops (the online-evaluation / drift-monitor half).
• Related: opentelemetry-genai-semconv.md (the telemetry vocabulary), mt-bench.md (scoring with numbers over a live window), Project 07 (offline eval), Project 09 (per-route aggregation).

Known issues / cautions

• A synchronous production judge is an anti-pattern (latency/cost); sample asynchronously.
• Drift detection on a global mean hides per-route failure; segment it.
• An alert threshold too tight = alert fatigue; too loose = missed regressions (the same

precision/recall dial as Electives 02/03).