Streaming and batch meters
Streaming and batch meters allow you to process either continuously arriving events or a finite set of input data in Zuora Mediation.
Every meter in Zuora Mediation is classified as either a streaming meter or a batch meter. The meter type is determined automatically based on the configured source and how that source delivers data.
- Streaming meters process data continuously as new events arrive.
- Batch meters process a finite set of data and complete automatically.
Understanding the difference helps you choose the right source, interpret meter behavior, and understand how your meter runs during execution. Use a streaming meter when data arrives continuously and should be processed as soon as it is available. Use a batch meter when you process scheduled files, database extracts, historical data, or other finite data sets.
Streaming meters
A streaming meter continuously processes data from a source that is constantly receiving new events. Instead of processing a fixed data set and stopping, the meter remains active and processes events as they arrive.
Common streaming sources include the following:
| Source. | Example use case. |
|---|---|
| Streaming API. | Your application sends usage events continuously through the Mediation Streaming API. |
| Kafka. | Process high-volume application, IoT, or telecom events in near real time. |
| Amazon S3 (incremental mode). | Continuously monitor an Amazon S3 folder and automatically process newly uploaded files. |
For example, a SaaS platform can send usage events to the Streaming API throughout the day as they occur. A telecommunications provider can also publish call detail records to Kafka so that the streaming meter processes them as new messages arrive.
Streaming meters have the following characteristics:
- They continuously process new data.
- They are designed for real-time or near real-time processing.
- They run until they are explicitly stopped.
- They can be paused and later resumed.
- They are suitable for high-volume event streams.
A streaming meter typically starts, enters a running state, continues processing incoming events, and remains active until it is stopped.
Batch meters
A batch meter processes a finite set of input data. After all available records have been processed, the meter automatically completes.
Common batch sources include the following:
| Source. | Example use case. |
|---|---|
| Upload File. | Upload a CSV, JSON, Parquet, or other supported file for one-time processing. |
| Amazon S3 (batch mode). | Process the files currently available in an Amazon S3 folder, then complete. |
| Snowflake. | Read data from a Snowflake table or query and process the results. |
| Event Store. | Read previously ingested events for replay, validation, consolidation, or scheduled processing. |
For example, a customer can export the previous day’s usage records to Amazon S3 each night and then start a batch meter to process the files and complete. Another example is a monthly billing process in which usage is collected in Event Store throughout the month and then processed in one batch run at month-end.
Batch meters have the following characteristics:
- They process a finite data set.
- They automatically complete after all data has been processed.
- Each execution is independent.
- They are best suited for scheduled or periodic processing.
A batch meter typically starts, processes the available input data, and then completes automatically after input processing and downstream output processing finish.
Key differences between streaming and batch meters
| Feature. | Streaming meter | Batch meter |
|---|---|---|
| Data source | Continuously receives new events | Finite set of records |
| Processing | Continuous | One-time execution |
| Typical sources | Streaming API, Kafka, Incremental Amazon S3 | Upload File, Batch Amazon S3, Snowflake, Event Store |
| Execution | Runs continuously | Runs until all input is processed |
| Natural completion | Does not complete automatically | Completes automatically |
| Typical status | Running | Processing Completed |
| Resume behavior | Can be resumed after being paused | Each start creates a new execution |
Amazon S3 support for both modes
Amazon S3 can be configured as either a streaming source or a batch source.
In batch mode, the meter processes the files currently available in the configured folder and then completes. In incremental mode, the meter continuously monitors the configured folder for newly uploaded files and processes them automatically when they appear.
Snowflake and Event Store
Although Snowflake and Event Store can store large volumes of events, they are typically used for batch processing.
Common scenarios include the following:
- Processing daily or monthly usage data stored in Snowflake.
- Reprocessing historical events from Event Store.
- Validating or consolidating usage before billing.
- Running scheduled aggregation jobs.