Parquet and DuckDB as the Agent Query Layer

A human analyst checks a revenue dashboard once a morning, forms one question, and runs one query to answer it. A fleet of AI agents working the same metric will issue dozens of speculative probes in parallel, re-investigating the number from different angles within the same minute, and that difference in behavior is why the query infrastructure built for human analysts cannot safely serve agents. Parquet-materialized datasets queried through embedded DuckDB can serve that fleet safely.

How agent query behavior differs from human query behavior

A human analyst forms a question, writes a query, and reads a result back. An agent does not close the loop after one query. Research on agent query engines frames this as a fundamental break in the assumptions behind existing data systems: the shift from human operators to AI agents means orders of magnitude more probes hit the query layer, most of them redundant or speculative, and the layer itself has to become what that research calls "satisficing-aware," built to tolerate a client that is still figuring out what it wants to ask.

Each individual query an agent issues is cheap to run. A fleet of agents working concurrently, each iterating through its own schema-discovery-and-retry loop, produces a load profile that nobody sized for, because the permission model and the capacity plan behind a production OLTP database were built around a roster of human users, not a swarm of clients that multiply their own query count by design. When a fleet of agents re-investigates the same revenue metric from different angles at once, each issuing speculative probes in parallel, the aggregate load climbs and the lack of schema awareness becomes acute. That is the condition platforms like Dreambase are built to address: by pre-materializing governed datasets, agents no longer need to discover schema or reconstruct business logic from scratch, which turns what would otherwise be a pile of redundant speculative queries against production into a single, fast lookup against a stable analytical interface. The failure mode is not a database that crashes outright: it is a database that slows down enough to affect the application running the actual business, or one that stays fast but hands an agent read access to tables nobody designed for analytical use.

Why production Postgres cannot safely absorb analytical agent traffic

Handing an agent database credentials to a live Postgres instance is the fastest way to get it working and the slowest kind of mistake to notice, because the risk compounds quietly with every agent added to the fleet. Raw SQL access over a production credential exposes every table that credential can reach. That includes tables nobody built as an analytical interface, tables with internal naming conventions no agent was expected to parse, and tables holding fields that were never meant to leave the application layer.

The deeper pathology is that business logic lives in conversations, not in any shared, inspectable place. Dreambase addresses that pathology by materializing analytical datasets as governed Parquet files queryable through embedded DuckDB, so agents work from pre-calculated metrics and modeled data without ever inheriting production credentials or touching live Postgres.

There's also a compliance dimension that doesn't go away just because the agent is "just reading." In regulated contexts, every query against PHI or financial records carries audit and retention obligations, so an agent with a live production credential inherits those obligations but has no way to discharge them on its own. The underlying problem, agents querying production for analytical work, is not something any default setting resolves, because it is an architectural choice about where analytical traffic lives, not a configuration flag.

What Parquet materialization does for the agent query problem

Materializing refined datasets as Parquet files draws the boundary between operational truth and analytical truth, and that boundary is what makes it safe for agents to query at any frequency they want. Parquet is a compact, columnar format built for the kind of full-column scans, aggregations, and filters that analytical workloads issue, as distinct from the row-level point lookups a transactional database like Postgres is optimized to serve. An agent asking "what was net revenue by week for the last two quarters" is issuing exactly the kind of scan Parquet handles well and a row-oriented OLTP engine handles poorly under concurrent load.

The decision to materialize is primarily about where business logic lives and who touches production, not about file size or compression ratio. In the data factory architecture, source systems keep responsibility for operational truth, and the materialized Parquet dataset becomes the reusable analytical interface that sits in front of it. That richness matters directly for agent safety: a dataset carrying permissions and scope as first-class fields means an agent querying it can never accidentally modify production data, can never see tables outside that dataset's declared scope, and can repeat the same query a thousand times without affecting anyone else using the system.

Why DuckDB is the right engine for Parquet datasets

Parquet handles storage and DuckDB handles execution, and the pairing is not incidental. DuckDB is an embedded analytical SQL engine, capable of joins, aggregations, filters, window functions, and time-series transformations, the exact operations agents tend to issue, without needing to push every request back to the source database.

The word "embedded" carries the weight of the argument. DuckDB runs in-process, so an agent can query a Parquet dataset without a network round trip to a separate query service, and each agent can carry its own instance without fighting other agents over shared state. That matters specifically for agent workloads because agents run concurrently and unpredictably, and a shared query service becomes a bottleneck or a contention point exactly where a fleet of agents needs the opposite. A DuckDB ecosystem newsletter describes a concrete version of this pattern, where each agent runs its own embedded DuckDB instance for direct, peer-to-peer analytical data exchange, built around "immutable analytical slices" identified by catalog, dataset, snapshot, semantic contract, and slice digest. That architecture decentralizes computation and sidesteps the concurrency problems that a shared analytical server would otherwise introduce.

The engine is also getting faster in ways that matter specifically for this use case. DuckDB's upcoming v2.0 release introduces asynchronous I/O for Parquet and CSV files, making better use of remote storage bandwidth, and that matters for agent workloads because Parquet files often live in object storage. For teams whose production data lives in Postgres, as it does for every Supabase team, the fit runs deeper still: Hydra's pg_duckdb extension accelerates analytics queries on Postgres by more than 600x, which points to DuckDB as the natural analytical companion for Postgres stacks specifically, not just a general-purpose engine that happens to work.

How pre-calculated metrics make the Parquet layer useful

A Parquet file full of raw rows is safer than a production credential, but an agent querying it still has to reconstruct KPI logic inside every prompt. Pre-calculating metrics and storing them as governed objects in the materialized layer closes that gap.

The data factory model handles this: the factory itself pre-calculates and stores metrics, so no agent has to rebuild the same formula inside every task. An agent reading that object is reading a settled answer, not assembling one from scratch.

The startup KPIs where this discipline earns its keep fall into a few familiar buckets: growth metrics like signups and MQL-to-SQL conversion, retention metrics like rolling-period retention, churn, and NPS, revenue metrics like MRR, ARR, expansion MRR, and net revenue retention, and efficiency metrics like CAC payback, burn multiple, and LTV:CAC. The semantic layer is the technical home for that agreement: metrics get declared in version-controlled form, alongside the schema mappings, relationships, dimensions, measures, and joins that define them, so an agent reads a settled definition. This is the same principle behind giving a board and a founder the same number on the same dashboard: both are reading from the identical pre-calculated, governed object, so there is no room for the number to quietly diverge depending on who asked and how.

MCP as the delivery mechanism that makes governed Parquet queryable by any agent

Parquet is the storage layer and DuckDB is the execution layer, but something still has to carry queries from an arbitrary agent to that layer, or every integration has to be built by hand. The Model Context Protocol fills that role. The data factory architecture described earlier terminates exactly there, at "dashboards, reports, APIs, and agents over MCP."

MCP itself got materially easier to run at scale this year. The July 28, 2026 specification revision converted MCP from a bidirectional stateful protocol into a request/response stateless one, so a remote MCP server that once needed sticky sessions and a shared session store can now sit behind a plain round-robin load balancer. That shift lands in an ecosystem where MCP SDKs were downloaded more than 97 million times a month by mid-2026, which says something about how quickly agent-to-data connectivity is standardizing on this protocol.

None of that solves the harder problem, though. If an agent connects over MCP to a pile of ungoverned tables, it will still return confidently wrong answers, just faster and with less friction than before. Treating MCP as a substitute for a clean semantic layer is one of the more common mistakes in 2026 agent architectures. The governance that actually makes MCP safe sits upstream of the protocol: lock the MCP server to a curated semantic layer with reviewed metrics and dimensions, require human review on the first version of any new dashboard, and log every prompt-to-SQL pair so a sampled slice can be scored for faithfulness against the underlying data, with alerts on drift. MCP is the transport. The governed Parquet-and-DuckDB layer underneath it is what makes the transport worth using.

What this architecture looks like for a Supabase team

None of this requires a Supabase team to tear out its stack and stand up a separate warehouse. The Parquet, DuckDB, and MCP combination extends Postgres as the analytical home for a business rather than replacing it, which matters for teams that have already built their operational system around Supabase and have no appetite for a parallel infrastructure project.

The right practice is to keep OLTP and OLAP concerns separated at the storage layer from the start. Supabase's Analytics Buckets, built on Apache Iceberg, already let teams offload large datasets out of Postgres into queryable analytical storage without standing up a separate ETL pipeline. A full flow in this pattern runs from Supabase as the system of operational truth, through a scoped, read-only connection, into schema and relationship discovery, then cross-source joins and cleaning that bring in billing data from Stripe and event data from PostHog, then business definitions and metric logic, then materialized Parquet datasets, then profiling, quality checks, and anomaly detection, then a semantic layer, and finally out to dashboards, reports, APIs, and agents over MCP. The rule that holds the whole flow together is simple to state: agents query the materialized Parquet layer, never the live Supabase database, keeping the production database out of the query path for analytical work.

Dreambase implements this pattern as a managed data factory built for Supabase teams specifically. Agents query modeled Parquet data on DuckDB, a single MCP server exposes the governed layer to any agent workflow a team wants to run, and the production Supabase database never sees an analytical query. You get it from $29 a month if you are a solo founder or small team, higher-tier plans fit startups and growing teams, and every paid plan comes with a 14-day free trial.

The tradeoffs and open questions this architecture does not resolve

The Parquet and DuckDB architecture is the right query layer for governed, analytical agent access, but it is not a free lunch, and three tradeoffs deserve a plain statement.

Materialization introduces latency by design. A dataset refreshed at 6 AM does not reflect a transaction that happened at 7 AM, and for most analytical agent queries that lag is acceptable, but a team still has to decide deliberately which metrics can tolerate it and which cannot. A fraud check probably cannot.

The refinement pipeline behind the governed layer, schema discovery, cross-source joins, metric logic, profiling, and the semantic layer itself, is not zero-effort to build or maintain. The metric definitions that make the semantic layer valuable are only as good as the discipline applied to keeping them current: when a business changes how it counts an "active user," every downstream agent, dashboard, and report has to update from that single governed definition, which makes the model powerful when maintained and a liability the moment an update gets missed.

You should also know about a genuinely different architecture, one built around per-cell, async-native SQL execution that evaluates expensive LLM calls only when downstream operators actually demand them. Work on JS-native query engines for agent workloads, under the name Squirreling, reports running LLM-shaped async UDFs a large multiple faster than DuckDB-WASM on filter-bounded queries. That is a different tradeoff, one that favors lower-latency, per-turn execution over the governed-layer model this piece has described, and it is worth tracking even where the governance argument still favors pre-materialization for most teams. Every architecture carries some risk. The Parquet and DuckDB model trades some freshness flexibility for safety and consistency, and for any team where agents and humans need to agree on the same numbers, that is the trade worth making.