Change Data Capture (CDC) represents a fundamental pattern in data pipeline architecture, enabling systems to identify and propagate modifications to source data. The two primary approaches—native CDC and snapshot-based CDC—differ significantly in their mechanisms, requirements, and operational characteristics. Understanding these distinctions is essential for designing efficient data integration solutions ¹⁾

Change Data Capture serves the critical function of detecting modifications (inserts, updates, and deletes) within source databases and propagating these changes to downstream systems. Rather than reprocessing entire datasets, CDC enables incremental data movement, reducing computational overhead and latency ²⁾

Organizations typically encounter two distinct patterns when implementing CDC:

Native CDC utilizes change logs or transaction logs directly emitted by source systems. These systems—including PostgreSQL with logical decoding, Oracle with LogMiner, and MySQL with binlog—provide structured, authoritative records of modifications as they occur.

Snapshot-Based CDC infers changes by comparing successive data snapshots across time intervals. When source systems lack native change feed capabilities, snapshot comparison offers an alternative mechanism for change detection.

Native CDC implementations leverage database transaction logs or change streams that source systems maintain for their own recovery and replication purposes. These logs represent the single source of truth for data modifications within the system ³⁾

Key characteristics of native CDC include:

* Log-Based Architecture: Source databases maintain binary logs (MySQL), write-ahead logs (PostgreSQL), or transaction logs (Oracle) that record every modification with precise ordering and timing information * Change Stream Emission: Specialized connectors or database features expose these logs as structured change streams, often including metadata such as transaction IDs, timestamps, and operation types * Low Processing Overhead: Native CDC avoids full table scans, consuming only the incremental changes since the last checkpoint * Ordering Guarantees: Native logs preserve strict ordering of operations within transactions, enabling accurate downstream state reconstruction * Latency Minimization: Changes propagate with minimal delay, often within seconds of source modification

Source systems supporting native CDC include PostgreSQL (logical decoding via replication slots), MySQL (binlog consumption), MongoDB (change streams), and cloud databases like AWS DMS and Azure Data Factory. Streaming platforms such as Apache Kafka frequently serve as intermediaries, offering change stream consumption through connectors like Debezium ⁴⁾

Snapshot-based CDC addresses scenarios where source systems lack native change feed capabilities. This approach periodically exports complete dataset snapshots, then compares successive snapshots to identify modifications ⁵⁾

Operational characteristics of snapshot-based CDC:

* Full Table Comparisons: Requires scanning entire tables or datasets to identify rows that have changed between snapshots, necessitating significant computational resources for large datasets * Change Inference: Modifications are inferred through comparison logic—missing rows indicate deletes, new rows indicate inserts, and differing column values indicate updates * Latency Considerations: Change detection latency depends on snapshot frequency; hourly snapshots cannot detect sub-hourly modifications * Resource Intensity: Full table scans impose computational burdens on source systems and consume network bandwidth * Applicability: Useful for APIs, files, data warehouses, and legacy systems that cannot emit native change logs

Snapshot-based CDC often applies to semi-structured data sources, API-based integrations, and systems where query capabilities exist but change logging does not. CSV files, JSON datasets, and REST APIs frequently depend on snapshot comparisons for change detection.

Modern data integration platforms increasingly treat both patterns as first-class CDC approaches rather than hierarchical alternatives. Databricks' AutoCDC framework, for instance, implements automatic change detection for snapshot-based scenarios, reducing manual engineering overhead while maintaining compatibility with native CDC streams ⁶⁾

This unified approach enables:

* Pattern-Agnostic Processing: Downstream consumers receive consistent change event schemas regardless of underlying CDC mechanism * Automatic Mechanism Selection: Systems automatically select native CDC when available, falling back to snapshot comparison for incompatible sources * Reduced Pipeline Complexity: Eliminates the need for separate code paths and transformation logic depending on change detection mechanism * Cost Optimization: Minimizes unnecessary full-table scans by leveraging native capabilities when possible

Selecting between native and snapshot-based CDC requires evaluating multiple technical dimensions:

Completeness: Native CDC preserves exact operation order and intermediate states, while snapshot-based CDC cannot detect intermediate updates that revert to original values within snapshot intervals.

Scalability: Native CDC scales linearly with change volume, while snapshot-based CDC scales with total dataset size, creating performance challenges for large tables with sparse modifications.

Cost: Native CDC requires minimal additional processing, while snapshot-based CDC incurs full-table scan costs proportional to dataset size and snapshot frequency.

Latency: Native CDC detects changes within seconds, while snapshot-based CDC introduces latency equal to snapshot intervals plus comparison duration.

Coverage: Native CDC depends on source system capabilities, while snapshot-based CDC applies universally to any queryable or exportable data source.

• Change Data Capture (CDC)
• AutoCDC vs Hand-Coded CDC
• AutoCDC
• Genie Code
• Deduplication