Observability

Every service in Golden Reports emits structured logs, Prometheus metrics, and OpenTelemetry traces. This page is the catalog: what's emitted, where it's emitted from, and how to wire it into the rest of your monitoring stack.

What's emitted

The platform follows the modern Kubernetes-native convention: logs go to stdout, metrics are scraped over HTTP, traces are pushed over OTLP. There's no proprietary collector to deploy.

Logs

All services log structured JSON. The key fields you'll see across every service:

• timestamp - ISO 8601.
• level - Debug, Information, Warning, Error, Critical.
• message - human-readable.
• traceId and spanId - OpenTelemetry trace context.
• tenantId - when the request has tenant context.
• correlationId / plan_id - for query-engine operations.
• service.name - the service emitting the log (api, query-engine, etc.).

Logs are emitted at Information level by default. Increase to Debug per-service through the Logging:LogLevel:* environment variables exposed in each component's env block.

Collecting logs

Any Kubernetes log collector that reads stdout works:

• Fluent Bit - DaemonSet that tails container logs and ships to your sink.
• Vector - same shape.
• Loki + Promtail - for Grafana-native logging.
• CloudWatch / Stackdriver / Datadog - managed equivalents.

A useful filter: service.name="api" AND level >= "Warning" is a good starting point for a "what's not right" stream.

Metrics

Each service exposes Prometheus metrics on /metrics on its container port. The chart's Service already exposes the port; you only need to point your Prometheus instance at it.

Suggested scrape config

- job_name: golden-reports
  kubernetes_sd_configs:
    - role: pod
  relabel_configs:
    - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
      regex: golden-reports
      action: keep
    - source_labels: [__meta_kubernetes_pod_container_port_name]
      regex: http
      action: keep

Or, if you run the Prometheus Operator, add a ServiceMonitor selecting the chart's Services by label.

Key metrics by service

Platform API

• http_server_request_duration_seconds_bucket - request latency histogram, labelled by route and status.
• Runtime counters (GC, thread pool, exceptions) - useful for capacity tuning.
• Handler-level timing - internal-handler duration histogram, useful when narrowing down a slow request.

Query Engine

• engine_executions_total - counter, labelled by execution status.
• engine_execution_duration_seconds - histogram covering fetch and in-engine processing phases.
• engine_rows_returned_total - counter.
• engine_active_executions - current gauge.
• engine_errors_total - counter, labelled by error category.

Data Provider

• connector_invocations_total - counter for connector operations.
• connector_invocation_duration_seconds - histogram.
• connector_load_errors_total - counter.

Dashboards worth building

A minimum useful dashboard set:

• API latency - request duration p50/p95/p99 by route.
• Engine throughput - executions per second, error rate, p95 latency.
• Active reports - engine_active_executions gauge.
• Error breakdown - engine_errors_total and failed connector_invocations_total stacked by category.

Tracing

OpenTelemetry tracing is enabled by default. To send spans to a collector, set:

OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production

on each Deployment via the chart's env blocks.

What's instrumented

• HTTP server (Platform API): every inbound request.
• HTTP client (Platform API): every outbound call.
• Inter-service calls (API → engine, engine → data-provider): every call.
• Platform-database queries (parameter values redacted).
• Internal handler timing: every command/query handler.
• Custom spans: report rendering, plan building, query execution phases.

The plan_id correlation

The query engine emits a plan_id (UUID) on every execution. This propagates as the correlation ID from the API into the engine and onwards:

GraphQL request (API)
└── span: ExecuteReport
    ├── span: BuildPlan
    └── span: QueryEngine.Execute    plan_id=<uuid>
        └── span: Connector.Fetch    plan_id=<uuid>

Searching your trace store for a plan_id returns the full execution path across services. The API also includes the plan_id in the export response, so you can search backwards from a customer-reported "this report failed" all the way to the connector's outbound call.

Audit log

A separate concern from operational telemetry. Every privileged action - workspace member change, data-access policy edit, share grant, plugin install, billing change - writes an audit record to the platform database.

The Manager exposes a queryable view at Manager → Audit with filters by:

• Actor (user).
• Action category (membership changes, policy edits, share grants, plugin installs, billing changes).
• Target (record affected).
• Time range.

The audit log is append-only at the platform level - it cannot be deleted or modified through the API. For compliance regimes that require external archival, export the audit log periodically through the API and ship it to your SIEM.

Recommended alerts

A starter alert set:

Tune the thresholds to your own traffic and SLOs.

Health endpoints

In addition to /metrics, each backend component exposes:

The Manager doesn't have dedicated health endpoints - readiness defaults to a 200 on /.

What's not (yet) provided

• Built-in dashboards. The platform doesn't ship Grafana dashboards. Build your own from the metrics above; we may publish reference dashboards in a future release.
• Built-in alerting. No bundled Alertmanager rules. The alert list above is a starting point.
• Per-tenant metric labels. Most metrics are intentionally low-cardinality. Trace context is where tenant-specific detail lives.

Next

• Configure the deployment: Helm chart reference.
• Understand the runtime topology: System overview.
• Read the audit model: Sharing & permissions.