Estuary adds targeted backfills and scalable discovery to PostgreSQL CDC connector

Estuary has released updates to its open-source PostgreSQL change data capture connector that improve operability for large databases and recovery scenarios. The connector now allows operators to apply SQL expressions as Additional Backfill Filters to individual table bindings, limiting re-backfills to specific row subsets such as date ranges or identifier windows without affecting ongoing WAL-based CDC. For environments with extensive schemas, new discovery controls enable inclusion or exclusion of specific schemas, glob-based table pattern matching, and optional discovery of unpublished tables, while metadata discovery is batched into smaller groups for reliability. Operators can also tune how frequently the connector rediscovers schema changes, with a default 15-minute interval that trades off detection latency against catalog-query overhead.

These changes address practical pain points in running CDC pipelines on large PostgreSQL deployments, where full-table backfills and unconstrained discovery can strain production systems. The connector continues to use PostgreSQL logical replication to read inserts, updates, and deletes from the write-ahead log, supports PostgreSQL 10.0 and later, and works across both self-hosted and managed PostgreSQL services.

Backfill and discovery architecture changes

• Targeted Backfill Filter: A new Additional Backfill Filter accepts a SQL expression per table binding that is applied directly to backfill queries, allowing recovery or re-backfill operations to read only a subset of rows such as those matching an indexed timestamp or identifier range.
• Backfill-CDC isolation: The SQL filter operates exclusively on backfill queries; ongoing change data capture from the PostgreSQL WAL proceeds independently and is not filtered, ensuring no event loss for new changes.
• Schema-scoped discovery: Discovery filters now support including or excluding specific PostgreSQL schemas, glob-based table patterns, and optionally surfacing tables not yet added to the PostgreSQL publication, reducing noise in multi-tenant or large-schema environments.
• Batched metadata discovery: Table metadata is processed in smaller groups during discovery to prevent failures and resource exhaustion when scanning databases with very high table counts.
• Configurable rediscovery interval: The frequency of automatic rediscovery for schema changes and new tables is now operator-configurable, defaulting to 15 minutes, allowing adjustment based on tolerance for catalog-query load versus schema-change detection latency.

Why it matters

These improvements make Estuary's open-source PostgreSQL connector more viable for high-scale production CDC pipelines, reducing operational risk when backfilling terabyte-class tables or managing databases with thousands of tables. For Linux-based data infrastructure and cloud-native deployments that rely on PostgreSQL as a primary data store, finer-grained backfill control and discovery scalability directly translate to lower recovery times and reduced replication slot pressure.