Designing Event-Time Based Partition Finalization for Stream Processing

The Challenge of Partition Finalization in Streaming

In traditional batch processing, data completeness (or "finalization") for a time-partitioned dataset is straightforward: once a batch job finishes writing all data for a specific time window, that partition is considered final. However, in stream-based ingestion systems, especially when dealing with change data capture (CDC) and potentially late-arriving events, a partition can continuously receive updates long after its wall-clock time has passed. This makes it challenging for downstream consumers, such as batch analytics jobs that run only once per hour, to know when a partition is truly complete and stable enough to process without missing data. The article highlights how Pinterest's migration to a stream-based ingestion framework necessitated a new, unified mechanism to infer this finalization state.

Pinterest's Event-Time Based Solution Architecture

Pinterest introduced an event-time based partition finalization mechanism directly into their streaming pipeline. This solution is integrated with the Flink job responsible for writing CDC events to Iceberg tables. The core idea is to measure the progress of event time within the stream and publish a finalization watermark directly into the Iceberg table's metadata. This allows downstream consumers to query the metadata and wait for the watermark to advance past their target partition before initiating processing.

Key Components and Logic:

• Event Time Extraction: Configurable extractors normalize the event timestamp from each record.
• Fact Collection (Event Time Statistics): Within each Flink checkpoint window, minimum, maximum, and percentile event times are collected using compact t-digest sketches to represent the distribution.
• Fact Storage (Iceberg Snapshot Summary): These statistics are stored in the Iceberg snapshot summary, leveraging existing per-commit metadata. T-digests ensure a small, fixed footprint and are mergeable across parallel subtasks.
• Partition Finalization Watermark: After each commit, an algorithm processes recent commit statistics to produce a monotonically increasing watermark. This watermark signifies the point up to which partitions are considered complete.
• Opinion Storage (Iceberg Table Property): The finalization watermark, a single table-level value, is stored as an Iceberg table property, making it easily discoverable by consumers via metadata lookups.