Observatory

In progress

The live, in-browser demo for Observatory is being built. The full technical teardown is on the right →

The diagnostic layer that tells you what degraded last Tuesday — and the gate that blocks the regression.

Impact

p50 / p95

tracked, not avg

$/req

per request

gate

on every PR

An observability layer bolted onto the RAG system: every step traced, quality metrics tracked over time, and a CI regression gate that blocks a deploy when faithfulness or latency slips.

The problem

A RAG system that works on Monday can quietly rot by Friday. Averages hide it. Without per-step traces and quality tracked over time, “it feels worse” is unprovable — and nothing stops the next bad change from shipping.

The approach

Instrument every step so any request is reconstructable, track the metrics that matter (p50/p95, $/request, citation coverage, failure rate) over time, and wire the golden set into CI as a hard gate.

Architecture

RAG request

live traffic

PROCESS

Instrumented steps

retrieve · rerank · generate

STORE

Langfuse traces

spans + token counts

STORE

Metrics store

p50/p95 · $/req · coverage

OUT

Diagnostic dashboard

what degraded, when

cross-cutting

GATE

CI regression gate

golden set blocks deploy

·Every step of every request is traced — any failure is fully reconstructable.
·Quality is tracked over time, not averaged away.
·The same golden set runs in CI and blocks a deploy the moment a metric drops.

How it was built

Phase 1

Tracing

✓Instrument every RAG step with Langfuse
✓Per request: chunks, rerank order, prompt, response, tokens

Phase 2

Quality over time

✓Latency p50/p95
✓Cost per request
✓Citation coverage + failure rate
✓A dashboard that explains what degraded

Phase 3

Regression gating

✓Golden set runs in CI
✓Build fails below threshold
✓Prompts/config versioned

Stack

LangfuseOpenTelemetryPostgresGitHub ActionsGrafana

Source

GitHub ↗Live dashboard

Want this on your team?

hello@miske.dev ← back to work