The query engine | Golden Reports Docs

The Query Engine is the service that fetches data for reports. It sits between the Platform API (which decides what data is needed) and the Data Provider (which holds the connectors that actually talk to your sources).

You usually don't need to think about the engine - it just runs reports. This page is for the people who do think about it: operators tuning throughput, debugging slow reports, or capacity-planning. For programmatic API details, see the Developer Portal; this page covers the model and the operational knobs.

What the engine does

Every time a report runs, the API hands the engine a query plan. The plan describes:

Which data source to read from (and the credentials to use, encrypted in transit).
Which fields to fetch.
Filter, sort, and pagination hints the connector can use to be efficient.
Post-operations the engine guarantees regardless of what the connector did.
Limits - max rows, max execution time, max output bytes.

The engine validates the plan, asks the data provider to run the fetch, applies any post-processing the connector couldn't push down, enforces the limits, and streams results back.

Push-down vs. post-processing

This is the most important concept to understand if you're trying to improve performance.

A plan carries two things that look related but are distinct:

Hints are opportunities. The plan says "ideally, filter total > 100, sort by date descending, and only return 50 rows." A capable connector translates these into a more efficient native query (a WHERE clause for SQL; a query-string parameter for REST). A simple connector ignores them.
Post-operations are requirements. The engine applies every requirement after the fetch, regardless of what the connector did. They run in a well-defined order so results match SQL semantics.

The split is why the same data context can answer queries over a Postgres view and over a CSV file: hints make the database fast, post-ops keep the CSV correct. Connectors are free to optimize without ever risking incorrect results.

Where this matters in practice

Slow reports against PostgreSQL are usually database-side: missing indexes, large unfiltered scans. The engine's metrics for that source will show fast in-engine post-processing and slow fetch - fix it at the database.
Slow reports against REST or CSV are usually because the connector couldn't push down. The engine fetches everything and then filters in memory. Solutions: tighten the data context's hint surface, or shape the source itself.
High engine CPU is usually post-processing on large result sets. Either reduce the row count or add push-down support to the connector.

Streaming

Result rows are streamed back to the API in batches (typically 1,000 rows per batch). This means:

The platform never loads a 100,000-row result into a single response payload.
The API can start rendering the report before the last batch arrives.
The engine and API both apply backpressure - if downstream consumers are slow, the engine pauses upstream fetches.

Operators don't tune batch size directly per query, but it's configurable per deployment.

Limits the engine enforces

Every query plan carries an explicit limits block:

Limit	What it does	Default
Max rows	Truncates the result if a post-op result exceeds this. Returns a row-limit-exceeded error.	100,000
Max execution time	Cancels the fetch if total time exceeds this. Returns a timeout error.	60 seconds
Max output bytes	Stops streaming if the cumulative byte count exceeds this. Returns an output-size error.	128 MB

These are platform-wide defaults set per deployment; the per-report values are bounded by them. Tighten them in shared multi-tenant deployments - a single misbehaving report can otherwise tie up an engine replica for the full timeout window.

Tenancy

Every plan carries a tenant identifier. The engine enforces it is set and consistent, and two tenants hitting the same physical database hit it through independent connections with independent credentials.

For capacity planning, treat each tenant's executions as independent rather than amortized.

Errors you'll see

Errors come back as structured codes. The most common in operational alerts:

Code	Meaning	Where to look
`INVALID_PLAN`	The plan failed validation.	Likely a recent data-context change.
`CONNECTOR_NOT_FOUND`	No connector loaded for the data source's type.	Check the data provider's loaded connector list.
`CONNECTOR_VERSION_MISMATCH`	Loaded connector is incompatible with the plan.	Roll the connector to a compatible version.
`CONNECTOR_EXECUTION_FAILED`	Connector returned an error.	Check the connector's logs; usually a source-side issue.
`DATA_SOURCE_UNREACHABLE`	Network or auth to the source failed.	Check source connectivity, credentials.
`EXECUTION_TIMEOUT`	Plan ran longer than the timeout.	Tune the query or the timeout.
`ROW_LIMIT_EXCEEDED`	Post-op result exceeded `max_rows`.	Add filters to reduce row count.
`OUTPUT_SIZE_EXCEEDED`	Cumulative stream exceeded `max_output_bytes`.	Reduce columns or add filters.

Error messages returned upstream are sanitized for safe display. Full diagnostic detail is in the engine logs.

Scaling

The engine is stateless and scales horizontally. A useful operator mental model:

Throughput scales with replica count.
Per-query latency is connector-bound (the actual fetch) plus post-processing (CPU). Adding replicas helps the first only if your source can take more concurrent connections.
Memory is dominated by the largest in-flight result set per replica. Tight max_rows and max_output_bytes cap this directly.
Concurrency limit per replica is configurable. The default suits a balanced workload; raise it if your reports are mostly waiting on the network, lower it if they're CPU-bound in post-processing.

The engine is the most likely component to hit CPU first under load. Watch the executions metric (below) and the autoscaler's CPU target.

Metrics that matter

The engine exposes Prometheus metrics. The ones to dashboard:

Metric	What to watch for
`engine_executions_total{status}`	Throughput and error rate.
`engine_execution_duration_seconds{phase}`	Where time goes - `fetch` vs `post-processing`.
`engine_active_executions`	Concurrency saturation.
`engine_errors_total{error_code}`	Specific failure categories.
`engine_rows_returned_total`	Aggregate row volume.

A balanced engine spends most of its time in fetch (the connector is waiting on the source). When post-processing dominates, you're doing work in the engine that ideally lives in the source - investigate push-down.

See Observability for the full metric and tracing catalog.

See what's on the other side of the connector boundary: Connectors.
Deploy the engine alongside everything else: Deployment overview.
Wire up monitoring: Observability.