Observability, CI/CD, DR & Auto-scaling

11. Observability (Metrics, Logs, Tracing)

Metrics (Prometheus + Grafana)

Stackhouse exposes /v1/stats → Prometheus scrapes every 15s.

Key metrics: stackhouse_requests_total{method,path,status}, stackhouse_request_duration_seconds (histogram), stackhouse_active_connections, stackhouse_db_query_duration_seconds, stackhouse_cache_hit_rate, stackhouse_rate_limited_requests_total, stackhouse_auth_failures_total, container_{cpu,memory}_usage (from cAdvisor).

Alerting rules (Alertmanager → PagerDuty):

• p99 latency >500ms for 5 minutes
• Error rate >5% for 2 minutes
• DB connection pool utilization >80%
• Pod restart count >3 in 10 minutes
• Disk usage >80%

Logs (Structured JSON → Log Aggregator)

Format: {"ts":"2026-03-24T10:00:00Z","level":"info","method":"POST","path":"/v1/push/users","status":201,"latency_ms":12,"user_id":1234}

Log levels: ERROR → WARN → INFO → DEBUG (configurable). Separate log files:

• stackhouse.log (all application logs)
• error.log (ERROR level only)
• query.log (slow queries >100ms)

Shipping: Fluentd/Vector → Elasticsearch/Loki/Datadog.

Tracing (OpenTelemetry → Jaeger/Tempo)

Every request gets an X-Request-ID header. Traces: HTTP request → DB query → Cache → Response. Useful for finding bottlenecks and debugging slow requests.

Provider Observability Options

12. CI/CD Pipeline

Rollback trigger: error rate >10% OR p99 latency >2s after deploy → kubectl rollout undo deployment/stackhouse

Docker Build — Multi-stage for Minimal Image

# Build stage (cargo build)
FROM rust:1.82 AS builder
WORKDIR /app
COPY Cargo.toml Cargo.lock ./
RUN cargo fetch
COPY src/ ./src/
RUN cargo build --release --bin stackhouse

# Runtime stage (minimal image)
FROM debian:bookworm-slim
RUN apt-get update && apt-get install -y ca-certificates && rm -rf /var/lib/apt/lists/*
COPY --from=builder /app/target/release/stackhouse /usr/local/bin/stackhouse
EXPOSE 3000
USER 1000  # Non-root
ENTRYPOINT ["stackhouse"]

# Final image size: ~50MB vs 2GB+ for full Rust image

13. Disaster Recovery & Backups

Backup Strategy

RPO = Recovery Point Objective (max data loss acceptable). RTO = Recovery Time Objective (max downtime acceptable).

Target: RPO <5 minutes, RTO <30 minutes.

Backup types:

1. PostgreSQL WAL Archiving (continuous, scaling target) — WAL segments shipped to object storage; would enable PITR to any second within the retention window; not wired into Stackhouse's own code today (external Postgres tooling would provide this)
2. Application-level logical backups (stackhouse_backups table + /v1/admin/backups/*, real, implemented today) — BackupManager runs SELECT * FROM "<table>" per table and writes the result to a backup file; restore and delete are exposed via POST /v1/admin/backups/:id/restore and DELETE /v1/admin/backups/:id. This is a logical snapshot mechanism, not WAL-based — no continuous point-in-time granularity between snapshots.
3. PitrService (stackhouse/src/storage/backups/pitr.rs, real WAL-replay logic, but not wired to any HTTP route) — implements real point-in-time restore by replaying WAL from a logical replication slot to a target timestamp. The logic exists and works when called directly, but there is no /v1/admin/* (or any other) endpoint that invokes it today, so it isn't reachable in a running server.
4. Object Storage Versioning — all file uploads versioned; 30-day version retention; cross-region replication for critical buckets (scaling target, not verified against current object-storage code)

Recovery scenarios:

14. Auto-scaling Strategy

Horizontal Pod Autoscaler Logic

Metrics collected every 15 seconds: CPU utilization per pod, memory utilization per pod, requests per second (custom metric via Prometheus adapter), WebSocket connection count.

Pod count limits:

Scale simulation at 100k users:

* PgBouncer limit

Cluster Autoscaler (Node-level)

Node pool config: min nodes 3 (HA, spread across 3 zones) · max nodes 10 (cost ceiling) · node type e2-standard-4 (4vCPU, 16GB) on GCP