Deduplicator behavior in batch and streaming meters
The Deduplicator behaves differently in streaming meters and batch meters.
| UI option or combination | Streaming mode | Batch mode | Behavior | Examples/Notes |
|---|---|---|---|---|
| Processing Time + Rolling Duration | Supported | Supported | Records are remembered for a moving duration based on arrival time. | Ignore duplicate events for 24h after arrival. |
| Processing Time + Fixed Calendar Window | Supported | Supported | Deduplication resets at fixed calendar boundaries (hour/day/week/month). | Allow one record per customer per day. |
| Event Time + Rolling Duration | Not supported | Not supported | This combination is unsupported. | Validation rejects this configuration. |
| Event Time + Fixed Calendar Window | Supported | Supported | Uses timestamp inside payload instead of arrival time. | Best for replay/backfill imports. |
| Deduplication across multiple runs | Supported | Not supported | Streaming persists state across executions; batch clears state after completion. | Streaming blocks replays across days. Batch only within the current run. |
| Late events handling | Supported | Supported | Late-arriving records are assigned to the correct event-time window. | Yesterday's delayed event still goes into yesterday's window. |
| Replay protection | Strong support | Limited support | Streaming continuously protects against replays. Batch protection ends with the job. | Reprocessed files are ignored in streaming but not across batch reruns. |
| Event Time state retention | Retained for the configured state lifetime | Retained only within the current batch run | Event Time determines window assignment, but deduplication state is not kept forever. | By default, persistent state is kept for 60 days. In batch mode, deduplication state is cleared when the batch run ends. |
Streaming meters
In streaming meters:
-
The pipeline runs continuously.
-
Deduplication state persists over time.
-
Time windows and retention settings are meaningful across events.
-
A duplicate arriving days later can still be detected.
Use streaming meters when you need replay protection across days or weeks.
Batch meters
In batch meters:
-
The job processes a finite set of records.
-
Deduplication applies only within the current batch run.
-
Deduplication state is cleared when the batch run ends.
Use batch meters when deduplication is required only within the current import or batch file.
Rolling Duration in batch meters
Rolling Duration is supported in batch meters, but the retention settings do not have practical effect after processing completes. During the batch run, records are still compared against other records in the same batch.
Fixed Calendar Window in batch meters
Calendar windows still matter in batch meters. Records can still be partitioned by hour, day, week, or month within the same batch run.
For example, two identical records on Monday are duplicates, while the same record on Tuesday is treated as new again, even within the same batch.
Event Time in batch meters
Event Time with Fixed Calendar Window also works in batch meters. This is useful for historical imports, backfills, replay processing, and multi-day data files.
Event Time determines which calendar window is used for duplicate checking, but it does not imply unlimited historical lookup. In batch mode, deduplication state still ends with the batch run. In persistent mode, retained state is still bounded and not infinite.