Kafka to Apache Iceberg With Kafka Connect
It’s Monday morning and support wants to know if payment failures spiked after the weekend release. The checkout service has been sending events to Kafka all night. The dashboard last refreshed at midnight. Someone pulls raw messages by hand, answers the first question, and gets asked to do it again an hour later.
With Kafka Connect, you can send those same events to Apache Iceberg and give the analyst a table to query throughout the morning. I’d start with one topic feeding one table. Once that first batch lands, you can see how long it takes to get a row into the hands of the person who needs it.
Table of Contents
- From a Kafka record to a queryable row
- Start with one topic and one table
- See what happens when a worker restarts
- Decide what a row should represent
- Give the pipeline a freshness target
- Keep the table easy to query
From a Kafka record to a queryable row
The Iceberg sink reads records from a Kafka topic. A converter turns each record’s value into something the sink can map to columns. For a checkout event, those columns might hold an event ID, an order ID, the event type and the time it happened. The sink’s tasks write data files to the warehouse, often on object storage. A coordinator commits batches of those files to the Iceberg table. After the commit, a query can read the new snapshot, a consistent view of the table at that point.
The catalog is how the sink and your query engine find the same table. It resolves a name like analytics.checkout_events to Iceberg metadata, which tracks the files that belong to each snapshot. If a worker logs that it wrote files and your query still comes back empty, check whether a snapshot has committed. It’s a quick way to narrow down where the delay is happening.
Start with one topic and one table
Install the Iceberg sink plugin on each Kafka Connect worker and restart the workers so they can load it. Give the connector access to the source topic, the Kafka resources it uses to coordinate commits, the Iceberg catalog and the warehouse storage. The example uses an existing checkout_events topic. If your cluster requires manual topic creation, create checkout_iceberg_control with one partition before starting the sink.
For the table, assume Spark has a catalog alias named lake pointing to the same Iceberg REST catalog as the connector. Create the analytics namespace first. I’d define the columns up front for this test so I can check exactly how IDs and timestamps land. Daily partitioning fits the checkout reports, which tend to filter by date.
CREATE TABLE lake.analytics.checkout_events (
event_id STRING,
order_id STRING,
event_type STRING,
event_time TIMESTAMP
)
USING iceberg
PARTITIONED BY (day(event_time));
Send the following JSON to Kafka Connect’s POST /connectors endpoint. Replace the catalog URI and warehouse name with your own values, and add the catalog and storage authentication your environment requires. The settings assume string keys and plain JSON values. The timestamp transform parses a value like 2026-11-16T09:10:00Z so you can filter and group by time in SQL.
{
"name": "checkout-iceberg",
"config": {
"connector.class": "org.apache.iceberg.connect.IcebergSinkConnector",
"topics": "checkout_events",
"tasks.max": "1",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"value.converter.schemas.enable": "false",
"transforms": "eventTime",
"transforms.eventTime.type": "org.apache.kafka.connect.transforms.TimestampConverter$Value",
"transforms.eventTime.field": "event_time",
"transforms.eventTime.target.type": "Timestamp",
"transforms.eventTime.format": "yyyy-MM-dd'T'HH:mm:ssX",
"iceberg.tables": "analytics.checkout_events",
"iceberg.catalog.type": "rest",
"iceberg.catalog.uri": "https://catalog.example.com",
"iceberg.catalog.warehouse": "checkout-warehouse",
"iceberg.control.topic": "checkout_iceberg_control",
"iceberg.control.commit.interval-ms": "60000"
}
}
Send a payment_failed event with ID evt-101, order ID ord-42 and the sample timestamp. Use the order ID as the Kafka key. After a commit, query the table for that event ID and compare the timestamp in UTC. A time zone mix-up is much easier to spot while you have one row to check.
Try a real producer message next. If your order IDs look like 00042, check that the leading zeros survive before anyone joins on that column. Send an event with a missing timestamp, too. A row with a null timestamp fits this schema and can slip past an ordinary date filter. That’s worth catching while you’re still deciding which fields the producer must send.
See what happens when a worker restarts
The Iceberg sink is designed for exactly-once delivery. To see what that means for your setup, send a batch with known event IDs and restart a worker while records are still flowing. Keep the connector’s name, offsets and control topic in place so it resumes from saved progress. Once it catches up, group the rows by event ID and check that each expected ID appears once. A simple total can hide a missing row if another row appears twice.
Kafka tracks records by offset within each partition. If the checkout service publishes the same event_id twice as two separate Kafka records, both can land in an append-only table. Keep a stable event ID in the payload so you can spot that kind of duplicate and decide how to count it. Two genuine payment attempts on the same order should have distinct event IDs.
Say your analyst spots a reporting gap and asks for last week’s events. I’d replay the relevant records into a staging table, compare event IDs with the live table, and backfill the missing rows. Check the topic’s time and size retention settings first. They determine how much history Kafka can still supply.
Decide what a row should represent
Order ord-42 could fail at 9:10 and go through on the customer’s second attempt at 9:13. Keep both events if the table tracks checkout history. They let the analyst count failed attempts and see how many orders eventually went through. A support agent checking whether ord-42 is paid needs the current state of that order.
If your Kafka topic carries change data capture, or CDC, it may describe inserts, updates and deletes from a database. Keep the source key and sequence with each change. For a current-state table, a merge step such as Spark’s MERGE INTO with the Iceberg extensions can apply the latest change for each order. Apply changes only when their sequence is newer than the last one you processed for that key. Keep that sequence after a delete, too. Try an insert, an update and a delete, then send the old update again. Check that the order stays gone.
Give the pipeline a freshness target
Ask the people reading the table how long they can wait for a new event. A two-minute target might make sense for a checkout incident dashboard. The example sets a one-minute commit interval. The Iceberg sink defaults to five minutes. A record can also spend time waiting behind a backlog or waiting for the query engine to pick up the latest snapshot. Measure the full trip before promising anyone a number.
Send a test event and time how long it takes to appear in the same engine your analysts use. Watch consumer lag alongside that measurement. Rising lag tells you the sink is falling behind. If you see lag drop and the test event still hasn’t shown up, check the connector tasks and the table’s recent snapshots. In Spark, you can inspect lake.analytics.checkout_events.snapshots to see when the last commit went through. A known test event also gives you something to check when real traffic goes quiet.
In real traffic, an event that happened yesterday can arrive and get committed this morning. Its event timestamp still puts it in yesterday’s reporting window. If people export the dashboard every evening, tell them how long late arrivals can keep changing those daily numbers. That gives them a fair basis for comparing two exports of the same day.
Keep the table easy to query
A low-volume stream that commits every minute can leave a surprising number of small files. The query engine still has to open them, even if a report scans just one day. Look at file counts and sizes for the dates people query most. In Spark, Iceberg’s rewrite_data_files procedure can compact small files into larger ones. Set a snapshot expiration policy based on how far back the team needs to query or roll back.
Give schema changes the same attention you’d give a checkout API change. The Iceberg sink can add missing fields when you enable schema evolution. If an event used to report amount_cents and a new producer wants to send decimal dollars, give the new amount its own field. Check a sample of old and new events before changing the sales report. That saves someone from having to explain why revenue jumped a hundredfold overnight.
Put the time of the latest Iceberg commit included in the report next to the payment failure count. On a busy Monday, that timestamp helps support tell the difference between a quiet checkout and a dashboard that’s fallen behind. Then they can get back to figuring out what happened after the release.
