Infrastructure

This section covers how to run Qadra---the Docker services, messaging, configuration, and operational concerns.

Docker Stack

Qadra runs as a Docker Compose stack. All services are containerized for consistency across development, staging, and production.

Core Services

Service Responsibilities:

• nats: The message bus. All inter-service communication flows through NATS request-reply. JetStream enabled for durable streams. Monitoring at :8222.
• postgres: The source of truth. All authoritative state, embeddings (pgvector HNSW), and SPO triples.
• mongodb: Append-only audit logs. Decision trees, request traces, NATS audit trail (nats_audit collection).
• qadra-core: The truth layer. Stateless Rust application that handles all NATS subjects (qadra.*). Eight handler groups: epistemic, workload, auth, email, audit, observer, admin, flow.
• qadra-agent: The intelligence layer. Python service that handles LLM-driven orchestration, workload execution, agent queries, and document verification.
• qadra-gateway: The HTTP surface. Express server that translates HTTP requests into NATS messages. Handles JWT creation/validation, CORS, file uploads to MinIO, and image processing via Imaginary. Never touches PostgreSQL directly.
• qadra-ui: The legacy frontend. Open WebUI fork with Qadra branding.
• kong: The single entry point for external traffic. JWT validation, rate limiting, routing, SSL termination.
• minio: S3-compatible file storage. Stores avatars, tenant logos, uploaded documents. Gateway uploads directly; metadata recorded in PostgreSQL via NATS.
• imaginary: Image processing microservice. Resizes avatar uploads to 256x256 WebP. Falls back gracefully if unavailable.

Observability Stack

Why a full observability stack?

AI workloads are opaque by nature. When a task takes 30 seconds, you need to know: Is it waiting for an LLM? Database query? NATS timeout? The observability stack answers these questions:

• Traces (OTel): See the full request path, including LLM calls and database queries
• Metrics (Prometheus): Track throughput, latency percentiles, error rates
• Logs (Loki): Search across all services for debugging
• Dashboards (Grafana): Visualize everything in one place
• Container Metrics (cAdvisor): CPU, memory, network, disk per container

Starting the Stack

# Clone and enter directory
cd qadra

# Copy environment template
cp .env.example .env

# Edit environment variables
vim .env

# Start all services
docker compose up -d

# View logs for a specific service
docker compose logs -f qadra-core
docker compose logs -f qadra-gateway

# Stop services
docker compose down

Environment Variables

Required Variables

# Database
DATABASE_URL=postgres://qadra:password@postgres:5432/qadra
POSTGRES_PASSWORD=your_secure_password

# Authentication
JWT_SECRET=your_jwt_secret_min_32_chars
JWT_REFRESH_SECRET=your_refresh_secret_min_32_chars

# LLM Provider
OPENROUTER_API_KEY=sk-or-v1-...

Optional Variables

# MongoDB (for audit traces)
MONGO_PASSWORD=qadra_traces_dev

# Email (SMTP2GO)
SMTP2GO_API_KEY=api-...
SMTP2GO_SENDER=noreply@qadra.dev
EMAIL_VERIFICATION_URL=http://localhost:5173/verify-email

# File Storage (MinIO)
MINIO_ROOT_USER=qadra
MINIO_ROOT_PASSWORD=qadra_minio_dev
MINIO_ENDPOINT=http://localhost:9000
MINIO_BUCKET=qadra-files

# Image Processing
IMAGINARY_URL=http://localhost:9002

# LLM (additional)
CEREBRAS_API_KEY=csk-...

# Python Agent
EXA_API_KEY=...                     # Exa neural search for autonomous research
QADRA_LLM_MODEL=google/gemini-3-flash-preview

# Observability
OTEL_ENABLED=true
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
GRAFANA_PASSWORD=your_grafana_password

# Server
PORT=3000
RUST_LOG=info

Resource Limits

All containers have memory limits to prevent runaway processes from affecting other services.

What happens when limits are hit?

• Soft limit (reservation): Guaranteed minimum; service will not be evicted below this
• Hard limit: If exceeded, container is killed and restarted
• Postgres shared_buffers: Set to 25% of memory limit (512MB)

Network Configuration

All services run on the qadra-network Docker network:

networks:
  default:
    name: qadra-network

Internal service discovery uses container names:

• postgres:5432
• mongodb:27017
• nats:4222
• minio:9000
• imaginary:9000 (internal port; host-mapped to 9002)
• loki:3100
• otel-collector:4317

NATS Message Bus

NATS is the backbone of Qadra's inter-service communication. All three application containers (Core, Agent, Gateway) connect to NATS.

Architecture

React SPA (:5173)
    |
    v  HTTP
Node.js Gateway (:4000)
    |
    v  NATS request-reply
Rust Core (:3000 internal)  <-->  Python Agent (:8080 internal)
    |                                    |
    v                                    v
PostgreSQL / MongoDB            LLM Providers (OpenRouter)

Subject Hierarchy

Monitoring

NATS monitoring is available at http://localhost:8222:

• /healthz -- health check
• /connz -- active connections
• /subsz -- subscription details
• /jsz -- JetStream status

Kong API Gateway

Kong is the single entry point for all external traffic. It sits in front of the Gateway and UI.

Why Kong?

• JWT validation: Verify tokens before requests reach the application
• Rate limiting: Protect against abuse without application logic
• Routing: Direct traffic to API vs. UI based on path
• SSL termination: Handle HTTPS at the edge
• Load balancing: Distribute requests across multiple instances

Kong runs in DB-less mode with declarative configuration (no database needed):

Configuration (kong/kong.yml)

_format_version: "3.0"

services:
  - name: qadra-gateway
    url: http://qadra-gateway:4000
    routes:
      - name: api-route
        paths:
          - /api
        strip_path: true

  - name: qadra-ui
    url: http://qadra-ui:8080
    routes:
      - name: ui-route
        paths:
          - /

plugins:
  - name: jwt
    config:
      secret_is_base64: false
      claims_to_verify:
        - exp

  - name: rate-limiting
    config:
      minute: 100
      policy: local

Header Injection

Kong validates JWTs and injects headers downstream:

X-User-Email: user@example.com
X-Tenant-Id: uuid
Authorization: Bearer <validated-token>

Observability

Observability is not optional for AI systems. When an agent takes 45 seconds to produce a report, you need to understand where time is spent.

OpenTelemetry Pipeline

Services (Rust Core, Gateway, Agent)
    |
    v  OTLP (gRPC :4317 / HTTP :4318)
OTel Collector
    |
    +---> Loki (logs)
    +---> Prometheus (metrics, :8889)
              |
              v
          Grafana (visualization, :3200)

Configuration files:

deploy/otel/
├── otel-collector-config.yaml  # OTLP receivers, processors, exporters
├── prometheus.yml              # Scrape targets (otel-collector, cadvisor)
├── grafana-datasources.yml     # Loki + Prometheus datasource provisioning
├── grafana-alerting.yml        # Alert rules, contact points, notification policies
└── dashboards/
    ├── qadra-overview.json     # OTel pipeline metrics, logs, collector health
    ├── container-monitoring.json  # cAdvisor container CPU/memory/network/disk
    ├── kg-smile.json           # KG-SMILE gate pass/fail + coverage/connectivity, Story 104
    └── eval-quality.json       # Agent output-quality eval scores per agent vs threshold, Story 94

Accessing Grafana

URL: http://localhost:3200
Username: admin
Password: (from GRAFANA_PASSWORD, default: qadra-grafana)

Pre-configured Dashboards:

• Qadra Overview (/d/qadra-overview) -- Request rates, error rates, OTel pipeline health
• Container Monitoring (/d/qadra-containers) -- CPU, memory, network, disk per container
• KG-SMILE -- KG-SMILE gate pass/fail + coverage/connectivity, fed by kg_smile.gate_run tracing events (Story 104)
• Eval Quality -- Agent output-quality eval scores per agent vs threshold (Story 94), fed by qadra.eval tracing events

Alert Rules

8 provisioned alerts in Grafana:

Database Migrations

Migrations are stored in migrations/ and run via sqlx. There are currently 30 migration files (001 through 030).

# Install sqlx-cli if needed
cargo install sqlx-cli

# Run migrations
DATABASE_URL=postgres://qadra:password@localhost:5432/qadra sqlx migrate run

# Create new migration
sqlx migrate add <name>

# Revert last migration
sqlx migrate revert

# Generate offline query data for CI
cargo sqlx prepare

Key Migration Groups:

Health Checks

Gateway Health

# Liveness
curl http://localhost:4000/health

# NATS connectivity
curl http://localhost:4000/health/ready

Container Health

docker compose ps

All containers should show healthy status. Each service has a configured healthcheck in docker-compose.yml.

Backup and Recovery

PostgreSQL Backup

# Backup
docker exec qadra-postgres pg_dump -U qadra qadra > backup.sql

# Restore
docker exec -i qadra-postgres psql -U qadra qadra < backup.sql

MongoDB Backup

# Backup
docker exec qadra-mongodb mongodump --out /backup

# Restore
docker exec qadra-mongodb mongorestore /backup

MinIO Backup

MinIO data is stored in the minio-data Docker volume. For backup, use the mc (MinIO Client) tool or mount the volume externally.

Scaling

Horizontal Scaling

Qadra Core and Gateway are stateless and can be horizontally scaled:

services:
  qadra-core:
    deploy:
      replicas: 3

NATS distributes messages across subscribers automatically.

Database Scaling

For production:

• Use managed PostgreSQL (RDS, Cloud SQL)
• Enable read replicas for query distribution
• Use connection pooling (PgBouncer)