Observability

How the Qubital backend is observed in production: logs, metrics, and the agents that collect and ship them.

Overview

Backend (Sevalla)
  ├── stdout (JSON)          → Sevalla internal log viewer
  ├── OTLP HTTP              → Grafana Alloy (internal Sevalla URL)
  │                              └── Loki (Grafana Cloud)
  └── Betterstack handler    → Betterstack (direct)

Backend /metrics endpoint   → Grafana Alloy scrapes
                                   └── Grafana Cloud (Prometheus / Mimir)

Grafana Alloy runs as a co-located Sevalla service and communicates with the backend over internal Sevalla networking — not over the public internet.

---

Logging

Logging is initialised in internal/app/slog-setup.go using Go's standard slog library with a fanout pattern (samber/slog-multi) that writes to multiple sinks simultaneously.

Handlers by environment

Development (APP_ENV=development)

• Single tint handler: human-readable coloured output to stdout with file/line source and time-only timestamps.

Staging / Production (APP_ENV=staging or prod)

Three handlers, all active simultaneously:

Handler	Sink	Activation
JSON stdout	Sevalla internal log viewer	Always active
OTLP HTTP (otlploghttp)	Grafana Alloy → Loki (Grafana Cloud)	Only if OTEL_EXPORTER_OTLP_ENDPOINT is set
Betterstack (slogbetterstack)	Betterstack	Only if BETTER_STACK_TOKEN is set

OTLP setup

The backend uses go.opentelemetry.io/contrib/bridges/otelslog to bridge slog into the OpenTelemetry log pipeline. Logs are batched and exported via OTLP HTTP. The instrumentation scope name is github.com/StartingQuoTechDivision/qubital-backend — used as a namespace for filtering logs in Grafana dashboards.

The OTEL_EXPORTER_OTLP_ENDPOINT variable accepts both http:// (insecure, used for internal Sevalla URLs) and https:// endpoints.

Relevant env vars

Variable	Purpose
APP_ENV	Controls which handlers are active (development, staging, prod)
OTEL_EXPORTER_OTLP_ENDPOINT	OTLP HTTP endpoint (Alloy internal URL). Omit to disable OTel export.
BETTER_STACK_TOKEN	Betterstack source token. Omit to disable.
BETTER_STACK_ENDPOINT	Betterstack ingest URL. Defaults to https://in.logs.betterstack.com.

---

Metrics

The backend exposes a Prometheus-compatible /metrics endpoint (via pkg/metrics/, wrapped by prometheus/client_golang). Grafana Alloy scrapes this endpoint on a regular interval and remote-writes the metrics to Grafana Cloud (Prometheus / Mimir).

Mimir query client

internal/platform/grafana/mimir/ is a read client — it queries Grafana Cloud Mimir via PromQL (/api/v1/query_range). It is used by backend features (e.g. analytics) to read metrics back from Mimir at runtime.

Authentication: HTTP Basic Auth with instanceID and apiToken from the GrafanaEnv config group.

Env var group	Field	Purpose
GrafanaEnv	MimirURL	Grafana Cloud Mimir base URL
GrafanaEnv	InstanceID	Grafana Cloud instance ID (Basic Auth username)
GrafanaEnv	APIToken	Grafana Cloud API token (Basic Auth password)