The Kafka Source operator subscribes to a Kafka topic, reads messages, converts them into Mediation events, and sends them through downstream operators such as transformers, enrichments, aggregators, and targets. Kafka sources are commonly used for real-time usage ingestion, event-driven integrations, streaming analytics, data lake and warehouse integrations, and usage-based billing workflows.

Meter volume limits

Production follows the same limits as Central Sandbox.

• Message validation uses the tenant-configurable payloadMaxLength setting, whose default value is approximately 5 MB.

• Kafka consumer defaults:

  • fetch.min.bytes = 131072

  • fetch.max.wait.ms = 500

  • max.poll.records = 1000

Configuration

Configure the following fields for the Kafka Source operator:

Name - Enter a user-defined name for the Kafka source.

Example:

Kafka-Source

Event Definition - Select the Event Definition that describes the structure of incoming Kafka messages. The Event Definition determines the available fields in the pipeline, data types, and validation rules.

Example Event Definition:

accountNumber: String
subscriptionNumber: String
quantity: Number
eventDate: DateTime

Connection - Select the Kafka connection that contains connectivity and authentication details for your Kafka cluster.

Example:

stage-kafka

Topic - Specify the Kafka topic from which the source consumes messages.

Example:

usage-events

Data Format - Select how the Kafka messages are interpreted:

• JSON reads messages as JSON documents.
• AVRO reads messages as Avro records.
• RAW_STRING reads messages as plain text.

Offset Reset Strategy - Select where the source starts reading when no previously processed position exists:

• Latest starts reading newly arriving messages. For example, if a topic already contains 10,000 messages, a new source configured with Latest starts with the next message that arrives after the source starts. Use this option for real-time processing and production streaming workloads.
• Earliest starts reading from the oldest available message. For example, if a topic contains 10,000 existing messages, a source configured with Earliest processes those messages. Use this option for historical imports, initial onboarding, and backfill scenarios.

Examples

Example incoming JSON message:

{
  "accountNumber": "A100",
  "subscriptionNumber": "S100",
  "quantity": 500,
  "eventDate": "2026-06-01T12:00:00Z"
}

Example Kafka Source configuration:

Connection: stage-kafka
Topic: usage-events
Data Format: JSON
Offset Reset Strategy: Latest

The message is converted into a Mediation event and made available to downstream operators.

Best practices

• Use Latest for continuous production ingestion.
• Use Earliest for historical imports and backfills.
• Ensure that the selected Event Definition matches the incoming message structure.
• Use AVRO when your Kafka ecosystem relies on schema-managed data.