Observability & Audit Tracing

Qadra runs two complementary observability systems. They answer different questions and are stored in different backends:

Both reference the same trace_id, so a log line in Loki and its full decision tree in MongoDB can be correlated. Logs are for operational troubleshooting; audit traces are for compliance, debugging, and post-hoc analysis of agent behaviour.

The Grafana dashboards (Qadra Overview, Container Monitoring, KG-SMILE, Eval Quality) and alert rules are documented in Infrastructure. This page covers the observability model — how data is produced, correlated, and retained.

System 1: Structured Logs & Metrics (OpenTelemetry)

Pipeline

Every service emits over OTLP to a central collector, which fans out to log and metric stores:

Services (Rust Core, Gateway, Agent)
    |
    v  OTLP (gRPC :4317 / HTTP :4318)
OTel Collector
    |
    +---> Loki (logs)
    +---> Prometheus (metrics, :8889)
    |          |
    |          v
    +------> Grafana (visualization, :3200)

The collector receives, batches, and routes telemetry. No service talks to Loki or Prometheus directly — everything flows through the collector so the export targets can change without touching service code.

OTel Collector Config

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
  # Loki accepts OTLP natively via /otlp (the legacy `loki` exporter was removed from contrib)
  otlp_http/loki:
    endpoint: http://loki:3100/otlp

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp_http/loki]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]

Structured Log Format

All services emit structured JSON to stdout. The collector ingests it; nobody parses free-form text.

{
  "timestamp": "2026-01-20T12:00:00.000Z",
  "level": "info",
  "message": "Task completed",
  "service": "qadra-core",
  "trace_id": "abc123",
  "span_id": "def456",
  "tenant_id": "00000000-0000-0000-0000-000000000001",
  "fields": {
    "task_id": "...",
    "duration_ms": 152
  }
}

Required Fields

Every log entry MUST carry these for correlation and tenant scoping:

Correlation

Every request gets a trace_id propagated across service boundaries, so a single user action can be reassembled from logs emitted by Gateway, Core, and Agent:

Log Levels

KG-SMILE Gate-Run Metrics

KG-SMILE attribution sessions are persisted to PostgreSQL (attribution_sessions, triple_attributions) and emitted as structured tracing::info events so they can be dashboarded without a dedicated metrics endpoint. The events use the target kg_smile.gate_run and carry fields gate, result, coverage, connectivity (plus fidelity, faithfulness, stability once the verification gate is wired).

These flow through the same OTLP -> Collector -> Loki pipeline as everything else — no new exporter, no /metrics scrape. The KG-SMILE Grafana dashboard computes pass/fail rates and average gate scores via LogQL over these log lines:

# Competence-gate pass/fail rate
sum by (result) (
  rate({container="qadra-core"} |= "kg_smile.gate_run" | json | gate="competence" [5m])
)

# Average coverage over time
avg_over_time({container="qadra-core"} |= "kg_smile.gate_run" | json | unwrap coverage [5m])

The agent output-quality eval signal follows the same pattern: qadra.eval tracing events feed the Eval Quality dashboard with per-agent scores versus threshold. Both signals are log-derived metrics — emit a structured event, query it with LogQL.

System 2: Audit Decision Traces (MongoDB)

Where logs are flat, audit traces are hierarchical. Every request produces a single trace document in MongoDB capturing the full decision tree — which agent ran, which atomics it spawned, which requirements resolved, which SPO/LLM calls fired, and how they nested.

MongoDB Connection

MONGODB_URL=mongodb://qadra:qadra_traces_dev@localhost:27017/qadra_traces?authSource=admin

Trace Structure

Trace (root document)
├── trace_id (UUIDv7 — time-sortable)
├── tenant_id
├── request_id
├── method, path
├── input (sanitized request body)
├── output (response body)
├── duration_us
├── status (success | error | timeout | rate_limited)
├── decisions[] (hierarchical tree)
│   └── agent_invoke
│       ├── input, output, metadata
│       └── children[]
│           └── atomic_spawn
│               └── children[]
│                   └── requirement_resolve
│                       └── children[]
│                           ├── capability_match → spo_query
│                           ├── embedding_generate
│                           └── memory_search
└── metrics (aggregated)
    ├── total_tokens_in, total_tokens_out
    ├── llm_call_count, spo_query_count
    └── cache_hits / misses

Decision Types

Each node in the tree is typed by the layer that produced it:

Recording Decisions

Decisions are opened and closed explicitly; nested decisions automatically parent to the open one:

#![allow(unused)]
fn main() {
// Start a decision
let decision_id = trace_ctx.start_decision(DecisionType::AgentInvoke);

// Update with metadata
trace_ctx.update_decision(decision_id, |d| {
    d.input = json!({ "agent_name": &agent_name });
    d.metadata.agent_name = Some(agent_name.clone());
});

// Nested decisions automatically parent to the open one
let child_id = trace_ctx.start_decision(DecisionType::AtomicSpawn);
// ... do work ...
trace_ctx.end_decision(child_id);

trace_ctx.end_decision(decision_id);
}

RAII Guards

For automatic cleanup, use DecisionGuard — the decision ends when the guard drops, even on early return or error:

#![allow(unused)]
fn main() {
let _guard = trace_ctx.decision_guard(DecisionType::LlmCall);
// Decision automatically ends when guard drops
}

Fire-and-Forget Persistence

Trace persistence must never block the request. Finalization hands the trace to a detached tokio::spawn, so a slow or failing MongoDB write has no effect on response latency:

#![allow(unused)]
fn main() {
impl AuditTrace {
    pub fn finalize_and_persist(self, output, status, error) {
        self.ctx.finalize(output, status, error);
        if let Some(repo) = self.repo {
            if let Some(trace) = self.ctx.take_trace() {
                tokio::spawn(async move { repo.insert(trace).await });
            }
        }
    }
}
}

Per-Request Metrics Aggregation

Each trace carries pre-aggregated counters, so you can answer "how many tokens / LLM calls / SPO queries did this request cost?" without walking the tree:

{
  "metrics": {
    "total_tokens_in": 1250,
    "total_tokens_out": 340,
    "llm_call_count": 2,
    "spo_query_count": 5,
    "agent_invoke_count": 1,
    "tool_call_count": 0,
    "cache_hits": 2,
    "cache_misses": 1
  }
}

Querying Traces

# Latest trace
docker exec qadra-mongodb mongosh \
  "mongodb://qadra:qadra_traces_dev@localhost:27017/qadra_traces?authSource=admin" \
  --quiet --eval "db.traces.find().sort({timestamp: -1}).limit(1).toArray()"

// Traces with errors
db.traces.find({ status: "error" })

// Traces containing a specific decision type
db.traces.find({ "decisions.decision_type": "llm_call" })

// Slow traces (>1s)
db.traces.find({ duration_us: { $gt: 1000000 } })

Graceful Disabled Mode

When MongoDB is unavailable, tracing degrades to a no-op. Requests continue to work; traces are simply not recorded:

#![allow(unused)]
fn main() {
let ctx = TraceContext::disabled();  // No-op implementation
}

Audit Traces vs Logs

The two systems overlap intentionally but serve distinct purposes:

Both reference the same trace_id for correlation — start from a log line, jump to the full decision tree.

Retention

Audit-trace retention is driven by compliance requirements, not disk economics:

Enforce the default via a MongoDB TTL index:

db.traces.createIndex(
  { "timestamp": 1 },
  { expireAfterSeconds: 7776000 }  // 90 days
)

Prometheus retains metrics for 30 days (5GB max); Loki log retention is configured for short-lived operational troubleshooting. The authoritative, long-lived record of why a decision was made lives in MongoDB.

Environment Variables

OTEL_ENABLED=true
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
OTEL_SERVICE_NAME=qadra-core
RUST_LOG=info                                  # Log level
MONGODB_URL=mongodb://qadra:...@mongodb:27017/qadra_traces?authSource=admin
GRAFANA_PASSWORD=your_grafana_password