Meter Audit counts and Event Store counts
When a meter reads data from a source such as Amazon S3 and writes it to an Event Store, the counts shown in Meter Audit might not always match the number of records stored in the Event Store.
This behavior is expected in certain situations and does not necessarily indicate data loss, duplicate records, or billing issues. Understanding the purpose of each metric helps determine which count to use for monitoring and which count to use for data validation.
Meter Audit counts
The Meter Audit page shows how many records were processed by the meter during execution. Use Meter Audit counts to monitor meter activity and troubleshoot processing behavior.
Examples of information shown in Meter Audit include:
- Records processed by the meter
- Operator input and output counts
- Processing errors
- Meter execution activity
Event Store counts
The Event Store contains records that have been successfully stored and are available for querying, reporting, and downstream processing. If you need to validate how many records were ultimately persisted, use the Event Store count as the source of truth.
Why the counts can be different
During execution, records can pass through multiple processing stages before they are fully stored in the Event Store.
As a result, Meter Audit can sometimes count records that:
- Were processed during meter execution but not yet stored
- Were processed again as part of an automatic recovery process
- Were processed in one meter run but stored during a later run
- Were still being finalized when the meter was stopped and resumed
Because Meter Audit metrics represent processing activity and Event Store counts represent stored records, the two values might not always match.
Which count to use
Use Meter Audit counts for:
- Monitoring meter activity
- Understanding throughput and processing behavior
- Troubleshooting meter execution
- Investigating processing errors
Use Event Store counts for:
- Data reconciliation
- Reporting
- Billing validation
- Verifying persisted records
- Confirming final ingestion volumes
If you need to know how many records were successfully stored, use the Event Store count.
Impact
Most customers are not impacted because billing, reporting, and downstream processing typically use the data stored in the Event Store.
However, the difference between Meter Audit counts and Event Store counts can be noticeable if you:
- Compare Meter Audit counts directly against source file counts
- Validate ingestion success by using Meter Audit metrics alone
- Reconcile data volumes on a per-run basis
- Stop and restart meters during large ingestion jobs
- Investigate processing activity after interruptions or recoveries
In these situations, Meter Audit counts might not exactly match the number of records ultimately stored in the Event Store.
In many cases, the Event Store contains the correct final data even when Meter Audit counts do not match exactly. The difference is typically related to how processing activity is reported compared with how successfully stored records are counted.
If you suspect data loss, compare the expected data volume with the records available in the Event Store and contact Zuora Global Support for assistance.
Example 1: Meter recovers after an interruption
A meter reads 10 million records from Amazon S3 and encounters an interruption while processing.
The meter automatically recovers and resumes processing.
During recovery, some records can be processed again to ensure data consistency.
As a result:
- Meter Audit count: 12 million records processed
- Event Store count: 10 million records stored
In this example, the Event Store contains the correct final data, while the Meter Audit page reflects the additional processing activity that occurred during recovery.
Example 2: Meter is stopped and later resumed
A large ingestion job is stopped before completion and resumed later.
Some records can be processed during the first run but only become visible in the Event Store after the meter resumes.
For example:
| Run | Meter Audit count | Event Store records added |
|---|---|---|
| Run 1 | 3 million | 2 million |
| Run 2 | 4 million | 5 million |
Although the counts for individual runs do not align, the final Event Store total is correct after all runs complete.
Example 3: Validate data for billing
A customer loads 15 million usage records into an Event Store that is later used for billing.
The Meter Audit page shows 16.5 million processed records, while the Event Store contains 15 million records.
For billing validation, reconciliation, and reporting, use the Event Store count of 15 million records because it represents the records that were successfully stored and are available for downstream processing.
Example 4: Monitor meter performance
A customer wants to understand whether a meter is actively processing data.
In this case, the Meter Audit page is the appropriate tool because it shows processing activity, throughput, and operational behavior while the meter is running.
The Event Store count can temporarily lag behind processing activity, which is expected.