Agent Evaluation

This section covers the Agent Output Quality Evaluation framework (Story 94)---a deterministic, fixture-driven way to score what each agent produces, used as a CI release gate.

Purpose

LLM-based agents are non-deterministic, but the shape of their output is not. Scarlet emits SPO triples, Ayana filters items, Eliza grounds claims, Reagan writes review sections. The evaluation framework scores these structured outputs against fixed rubrics so we can detect regressions before they ship.

The framework is:

Deterministic -- no LLM calls during scoring. Same fixtures in, same scores out.
LLM-free -- rubrics inspect emitted structure (triples, labels, claims, sections), not prose quality.
Fixture-driven -- every agent ships golden (should-pass) and negative (should-fail) cases.
A release gate -- CI fails the release build if any agent's golden average drops below its threshold.

Rubrics

Each agent has one primary rubric. A rubric inspects the agent's emitted output and returns a score in [0.0, 1.0].

Agent	Rubric	What It Measures	Threshold
Scarlet	`SourceCoverage`	Fraction of emitted `{"triples":[…]}` carrying a non-empty `source`	0.70
Ayana	`FilterAccuracy`	`(TP + TN) / total` over labelled items	0.80
Eliza	`GroundingScore`	Fraction of claims grounded in the supplied SPO context	0.70
Reagan	`ReviewCompleteness`	Fraction of required review sections present	0.80

Reagan's required sections:

Recommendation
Conviction
Key Positives
Key Risks
Reviewer Notes

Scoring

A rubric scores a single case. An agent's score aggregates those case scores.

Per-agent score = average over GOLDEN cases only (should_pass = true).
Negative cases (should_pass = false) must score below the threshold---this confirms the rubric actually discriminates.
Coverage -- at least 10 cases per agent (8 golden + 4 negative is the working baseline).
Release gate -- release_gate_check fails if any agent's golden average falls below its threshold.

agent_score(agent) = mean(score(case) for case in golden_cases(agent))

release_gate_check:
    for agent in agents:
        assert agent_score(agent) >= threshold(agent)
    # negative cases independently asserted < threshold

Metrics

After an evaluation run, emit_agent_eval_metrics emits a structured tracing event:

Target: qadra.eval
Event: agent_eval_complete
Fields: eval_agent, eval_score, eval_rubric

These flow through the standard pipeline---OTLP -> OTel Collector -> Loki -> the Eval Quality Grafana dashboard (deploy/otel/dashboards/eval-quality.json)---so per-agent scores can be charted against their thresholds over time. No new exporter or /metrics endpoint is required.

Code Locations

File	Purpose
`crates/qadra-traits/src/eval.rs`	Rubric and case types, scoring trait
`crates/qadra-epistemic-core/src/eval/rubric.rs`	Rubric implementations (SourceCoverage, FilterAccuracy, GroundingScore, ReviewCompleteness)
`crates/qadra-epistemic-core/src/eval/harness.rs`	Aggregation, `release_gate_check`, `emit_agent_eval_metrics`
`crates/qadra-epistemic-core/src/eval/fixtures.rs`	Golden + negative cases per agent
`tests/eval_harness.rs`	Integration tests (gate check, threshold enforcement, metric emission)

Constraints

Rubrics must stay aligned to the live SPO persona output formats. If a persona changes the structure it emits (triple shape, section headings, label schema), the corresponding rubric and fixtures must be updated in lockstep---otherwise the gate either passes blindly or false-fails.
Scoring stays deterministic and LLM-free: a rubric may only inspect emitted structure, never call a model.
Every agent keeps both golden and negative fixtures so the gate verifies discrimination, not just a high average.

Qadra Documentation