Forget everything you know about ingestion engines

We're Doron Porat and Liran Yogev, and we work at Eon. Eon is building the autonomous data foundation for the AI era. It was founded by the team behind AWS Disaster Recovery and Migration services, and it backs up cloud data for enterprises: exabytes across hundreds of thousands of databases, buckets and disks.

Operating at that scale teaches you how enterprise data actually behaves. So Eon now also ingests, processes and optimizes data for AI agents and analytics, and that is where Eon Yukon, the engine in this post, comes in.

We've been getting data into lakes for eleven years, and every stop had a catch:

• 2015, Hadoop jobs: always a batch behind.
• 2017, Secor (Kafka topics to object storage): just files, no table, no updates.
• 2019, a CDC lake (Debezium, Kafka, Spark, Hudi): far too many knobs, and brittle.
• 2022, Flink and Spark into Delta Lake and Iceberg: compaction and sorting were on us, costly and complex.
• 2025, Flink + Apache Paimon at petabytes a day: every checkpoint waited for the slowest task.

This post is the written version of our Current 2026 talk, Forget everything you know about ingestion engines. It covers why streaming into Iceberg breaks at massive scale, what Flink and Paimon cost us on one real pipeline, and how Yukon works instead. Yukon is a commercial product, so weigh our numbers accordingly; the benchmark behind our head-to-head comparison is open at eon-io/ingest-bench.

Iceberg was built for batch. Streams are not.

Iceberg wants few, big, atomic changes; a stream makes many small ones, forever. Three design choices make that collision expensive:

1. Data is visible only at a commit. Readers see snapshots, never rows, so a table is never fresher than its commit interval.
2. Every commit adds metadata. One commit writes data files, a manifest, a manifest list and a new metadata.json that lists every snapshot. At one commit a minute that is 1,440 a day, and every query plan and every commit slows down until someone expires snapshots and rewrites manifests.
3. Upkeep fights ingest. Compaction and sorting rewrite data files and commit to the same table as ingest. Commits are optimistic, so a rewrite that takes minutes keeps racing an ingest commit that lands every minute.

So streaming into Iceberg is five jobs that never finish: consume, write Parquet, commit, compact and maintain.

One customer, one table

All of this is fine at a gigabyte a day. It falls apart at the scale of our running example: a customer who lands Avro files in a GCS bucket and wants one Iceberg table out of them.

The queries need four things from that table:

• Partitioned by tenant ID and hour: every query filters one tenant and a time range.
• Sorted inside each partition: so file and row-group statistics can skip data.
• Fresh within minutes of a file landing in the bucket.
• Exactly once: no loss and no duplicates through restarts and redeliveries.

One giant, ten thousand small

There is no average tenant. About 10,000 tenants share the stream, and in its biggest hour, one of them carried about a quarter of all rows: 868 GiB and 6.33 billion rows into one partition, as 7,526 files with a median size of 120 MiB.

Those files were well sized, but unsorted. A BigQuery COUNT for that tenant and hour, filtering two more columns, scanned all 868 GiB because no file's min/max could rule anything out. The giant needs sorting, not compaction.

The small tenants are the opposite. Partitioning already narrows their queries, but a commit every minute closes a file for every tenant that received rows: up to 600,000 files an hour. They need compaction, not sorting, and both jobs run on the same table while the next commit lands.

The price of staying readable

Keeping that table queryable means rewriting the same bytes again and again. A back-of-the-envelope model, built on the measured hour:

• The giant: sort it every 15 minutes and each sort rewrites the partition so far: 217 GiB, then 434, 651 and 868, and all 868 again when late files arrive. With the 868 GiB ingest wrote in the first place, that is 3.8 TiB written to keep 868 GiB, 4.5×.
• A small tenant writing 1 MB a minute, bin-packed every 5 minutes: Iceberg's bin-pack picks files under 75% of the 512 MiB target, so every pass rewrites everything merged so far. With the original writes, that is 450 MB written to keep 60 MB, 7.5×.

At 1.4 PB a day, writing every byte 4.5 to 7.5 times is 6 to 10 PB written every day. And every rewrite pass is a commit, racing the ingest commit that lands every minute.

The textbook answer: Flink and Paimon, in production

The standard way to stream this into Iceberg is Flink, and for a stream like this, Flink with Apache Paimon. It's a good answer, and we didn't just try it: we ran it in production on this customer's stream, at 600 billion events a day.

On paper it covers everything:

• Flink is the standard streaming engine. Files commit with the checkpoint, so you get exactly-once. It scales out with parallelism, and every checkpoint is a commit, so data stays fresh.
• Paimon was built for streaming into a lake. It started in the Flink community as Flink Table Store (FLIP-188), stores data as an LSM tree so small writes land fast, and can compact inside the streaming job. It aims for data queryable within a minute and handles schema changes mid-stream.
• Iceberg readers still work. With metadata.iceberg.storage set, Paimon writes Iceberg metadata after each snapshot, so every engine reads the same files as an Iceberg table.

The pipeline: GCS sends a Pub/Sub notification for every new file; a Flink job reads the Avro, writes Parquet and commits on every checkpoint into an append-only Paimon table; Paimon publishes the Iceberg metadata.

It worked. It was also expensive:

Every workaround we tried moved a number, but the pipeline stayed expensive, lagged at peaks and needed babysitting.

Why streaming into Iceberg with Flink isn't really streaming

None of what follows is a Flink bug. It's what the checkpoint model forces on you at this scale, and it shows up in four places, in the order the data meets them.

1. Operator skew

We keyed each incoming file by tenant, so all of a tenant's files went to one task. That's good for files, one per tenant per checkpoint, until the biggest tenant's task falls behind and backpressure slows the whole job.

The usual fix is to split big tenants over N tasks. We kept a static map in the pipeline config ({"whale-org-1": 32}) and keyed those tenants as {org}#{hash(object key) mod N}. The load spread, but now N tasks each wrote a smaller file for that tenant at every checkpoint. N was set by hand, changing it meant a redeploy, and the busy tenants changed through the day.

2. The checkpoint barrier

In Flink, data commits only when a checkpoint completes. At the barrier every task stops taking rows, turns what it buffered into Parquet and uploads it. Most tasks finish in seconds; the task holding the giant tenant takes far longer, and the checkpoint waits for it.

Then the commit runs: a Paimon snapshot plus Iceberg metadata, which on the biggest partition sometimes took more than five minutes. Only then is anything visible, from every task at once. That wait, not the data, set our freshness: 2 to 4 minutes. It isn't streaming; it's a batch per checkpoint.

Every fix pushed it further toward batch:

# Flink settings we ended up with
execution.checkpointing.interval: 60s
execution.checkpointing.min-pause: 60s   # at 0, a slow checkpoint ran straight into the next and tasks ran out of memory
execution.checkpointing.timeout: 30min   # raised from 10min: Paimon commits on the giant partition took 5+ minutes
# Paimon table options
# 'write-only' = 'true'                  -- compaction moved to a separate Spark job
# 'write-buffer-for-append' = 'true'     -- rows buffer and are encoded to Parquet at the barrier

The upload burst at each barrier had its own fix. With 64 upload threads, a task manager held about 5 GiB of upload buffers and ran out of memory on a catch-up checkpoint, so we dropped to 16 and accepted slower uploads. Unaligned checkpoints weren't an option, Paimon 1.4 rejects them at submit.

3. Rigid scale and recovery

Flink's unit of progress is the checkpoint. The Kubernetes operator can rescale a job, but a rescale is a restart: decide from x minutes of metrics, wait for a checkpoint, stop every task, restore and catch up. Nothing is redone, but it's slow, so you rescale rarely and size for the burst. We targeted 40% utilization.

Spot is worse. Our task managers ran one per spot node, and one keyBy linked every task, so a reclaimed node restarted the whole job from the last checkpoint, often minutes old. Every file notification pulled since then was redelivered, because notifications are acked only when a checkpoint completes. Then the job waited one to three minutes for a new node. Spot made each task manager cheap and each loss expensive.

4. Upkeep after the fact

Paimon can compact inside the streaming job, and that's where we started. But inline compaction runs next to the writers and takes their throughput: every merge slowed ingest, and lag piled up.

So the ingest jobs went write-only, and compaction moved to a second fleet of Spark jobs running sys.compact and a chained, single-threaded sys.compact_manifest. Files arrive up to 12 hours late, so every run re-scans the last 12 hours. We also had to raise compaction.file-num-limit from 100k to 2 million for terabyte-scale tables.

Each workaround made sense on its own. Together they described a batch system in streaming clothes.

The principles we built Yukon on

Tuning wasn't enough, because the problems are structural: one global checkpoint, work hashed onto fixed tasks, state to restore, and upkeep bolted on afterwards. So we built a new engine from scratch, with one principle per problem, plus two we learned along the way.

Under the hood, Yukon is written in Rust and works on Arrow batches from decode to Parquet, with no per-record objects. It ships as one image with seven binaries and runs on Kubernetes in your own cloud.

How Yukon works

Yukon splits the engine in two: roles that move bytes and roles that move metadata. Writers read each file once and write Parquet, and compactors bin-pack and sort; neither ever talks to the catalog. One committer per table is the only process that commits to it, and it never reads a row of data. The cleaner is the only process that deletes.

Only small messages cross between the halves: file pointers with column statistics, compaction tasks and their results, and files to remove. Each hand-off is a message on a queue or a small object in the bucket, so no pod holds state that has to be restored.

Writers: decode once, never wait

A writer fetches a source file (or a spooled batch of Kafka messages), decodes it into Arrow once, splits the rows by table partition and appends each partition to an open Parquet file. A file closes at 128 MiB, at its age limit (60 s by default) or after 15 s with no new rows. Then the committer gets a pointer to it, with its column statistics.

For Kafka, separate reader pods drain the brokers into object storage, so records are durably archived before they are decoded. Writers then take the objects like any other source file.

Three things make writing skewless:

• A busy partition opens more files side by side. Each lane is an independent encoder with its own open file, and a hot partition opens another lane when its current one keeps refusing writes. It never slows the other partitions.
• Small partitions are limited by memory, not threads. Memory budgets are shares of the pod's cgroup limit, admitted by bytes, so an open file reserves only what it needs and many stay open at once. The writer reads the cgroup every 250 ms and flushes its largest open files as it nears the limit.
• Backpressure goes back to the bucket. A full writer simply stops asking for work, so the backlog waits in object storage rather than in memory. KEDA adds writers on CPU and on the age of the oldest waiting file.

The result: no partition slows the others, and CPU, memory and IO stay busy instead of idling behind the slowest task.

Commits without a global checkpoint

When a file closes, its writer publishes a pointer (path, row count, column statistics) to a queue. The committer drains that queue continuously, records the open window in a write-ahead log in the bucket, and commits one append per interval. Nothing locks: no barrier stops the writers, and a commit never waits for a writer to finish.

So freshness comes down to two timers you set: how long a writer keeps a file open, and the commit interval.

Exactly-once comes from the same hand-offs. Every snapshot carries its window's transaction ID, so after a crash the committer checks which windows landed instead of replaying them. A source is acknowledged only once every file it fed is published, and if a source is read twice, the duplicate is dropped at commit.

The trade-off: writers close files on their own cadence, so freshness has a floor of roll age plus commit interval.

Compaction and sorting, all the time

Compaction is heavy compute but light metadata: a rewrite is just a list of files removed and files added. So Yukon runs it on three separate clocks:

• Commit keeps a fixed cadence. New rows go in first as an append, so they never wait for a rewrite; finished rewrites follow as a metadata-only commit that swaps file pointers. One committer makes both commits, so upkeep never races ingest.
• Planning runs all the time inside the committer, from the file index its commits already keep in memory. It never reads a Parquet footer, and a full pass over the manifests every 15 minutes catches idle partitions.
• Compute runs on an elastic fleet of stateless compactors on spot nodes. They rewrite files, send pointers back and never touch the catalog.

Here is what that looks like in one production table's snapshot history. Appends add rows in files of about 25 MiB; seconds later, a compaction commit changes no rows, removes files and writes ones of 80 to 100 MiB.

The planner is cost-based. It prices every candidate rewrite by the object-store requests it would save a query: a bin-pack for a full scan, a sort for a single-key point query. It runs the most valuable work first, and a rewrite that doesn't pay yet waits and grows into a bigger one.

The rewrites themselves come in two steps:

1. Pack small files, LSM style. Files under half the 512 MiB target merge once eight pile up, and a merged file that is still small merges again a level up. No sorting.
2. Sort in groups. Once large files add up to a group, 10 GiB by default, they are sorted together into one group, and a sorted group is never rewritten again. The sort order is lexicographic, Z-order or Hilbert.

The trade-off: groups can overlap in key range, so a query may open a few more files than after a global sort. In exchange, no byte is ever re-sorted.

Sorting decides what every query pays, so we wrote our own external merge sort. It counts keys first, and the few keys that hold most rows each get their own file, already sorted. The rest sorts a short byte key per row, with strings as dictionary codes, and spills compressed runs that one k-way merge streams straight into the final Parquet files. On one 155 GiB partition in an internal study, DuckDB took 1,223 s and our best variant took 73.5 s.

Scaling: nothing to restore

Nothing lives in a pod; the state is in the bucket. Scaling out means KEDA starts a pod and it asks for work, while every other writer keeps writing. Scaling in or rolling out is the reverse: a pod stops asking for work, finishes what it holds and hands back the rest.

Losing a spot node is the same story with about 15 seconds of warning. On a real reclaim, a writer handed back 1,032 files in 282 ms and exited in under 3 seconds, and other pods picked them up. With no warning at all, its lease lapses and its files go to the others.

Work is handed out by a coordinator built on nothing but the bucket:

• Leadership: one pod leads by holding a 30-second lease object, written with the object store's compare-and-swap.
• Work: a writer leases a batch of files for a few minutes and acks them once written. A lease is a reservation, not a lock: nobody waits on it.
• Durability: every request is appended to a log in the bucket before it is answered. Object stores take about one update a second per object, so one write a second carries every request of that second, up to 8,192 of them.
• Failover: a new leader loads the latest state snapshot and replays the log on top, to exactly the same state, and a returning old leader is fenced out.

No ZooKeeper, no database, no extra service to run. A lost leader costs seconds, never correctness.

In-stream: open the data once

Decoding is the most expensive thing a writer does, so Yukon does it once and does everything else while the batch is still in memory. Anything that needs a global view is computed as a small partial result and folded later, off the writer's path.

Shape the data on the way in. Filters drop rows before you pay to store them, transforms derive, cast and rename columns with scalar SQL expressions, and UDFs are your own Rust compiled to WebAssembly. All of it runs vectorized over the Arrow batch that's already decoded. A sketch of the pipeline config:

[transform]
input_filter = "event != 'heartbeat'"

[[transform.columns]]
name = "utm_source"
expr = "LOWER(REGEXP_EXTRACT(url, 'utm_source=([^&]+)'))"
type = "string"

Know your data as it lands. In the same pass that encodes each file, Yukon counts per-column statistics such as null rate, min and max. It also builds sketches (Theta for distinct counts, KLL for quantiles, Misra-Gries for top values) and a weighted sample that keeps rare rows and thins common ones. All of them are small and mergeable, and the committer folds them at each commit.

Upserts without rewriting files. Writers never look anything up. Next to each Parquet file, a writer ships a tiny map of the keys it wrote and where they landed. The key index lives in object storage as Parquet files that carry their key range in the footer, so a merge skips files that can't overlap without opening them.

Every commit, the committer runs one loop that fetches and updates the index shards the new keys touch, in parallel. The newest row for each key wins, and every older copy becomes a bit in an Iceberg v3 deletion vector, committed in the same snapshot as the new data. No data file is rewritten.

The index shards itself: when a shard gets too heavy, the committer splits it, and since keys map to fixed virtual shards, nothing is rehashed. Compaction just tells the committer how the index changed for the files it rewrote. An update that lands mid-compaction carries the newer version and wins: no conflict, no retry. This needs Iceberg v3 tables, and readers that apply v3 deletion vectors.

Aggregates, the same way. You declare group-bys per key, such as events in the last 7 days or spend in the last 30. Writers fold each row into a partial result as they decode it, aggregator pods merge the partials and publish only the keys that changed, and each aggregate lands as its own Iceberg table. We also enable serving the same state per key in under a millisecond, over a Redis protocol, HTTP or Arrow Flight.

Built for agents to run, and to use

Agents will operate data systems and consume their data, so we built for them from the start:

• Our agents watch it. They watch every pipeline and triage alerts from metrics, logs, configs and change history, then post the likely cause.
• Your agents drive it. The yukon CLI ships with its own skill, so any coding agent can create, change and check streams. A change to a protected stream waits for a person's approval.
• Live context for every table. From the statistics, the pipeline's changes, your notes and the sample, agents keep a living description of what each table means and what just changed, served over MCP. The metrics they choose to track feed back into what the writers sample.

$ yukon skill install          # teach your coding agent the CLI
$ yukon top                    # every stream: status, health, rows/s, time since last commit
$ yukon top --output json      # the same view, for an agent

Results: an open benchmark, and the same stream in production

We couldn't find a benchmark that measures streaming ingestion the way a lakehouse actually runs, so we built one and published it: eon-io/ingest-bench. It finds the smallest fleet that keeps up, and measures four things together:

• Exactly once: every row committed once, which an independent scorer checks by row ID.
• Freshness: lag p95 at most 90 s and max at most 120 s, at the full offered rate.
• File sizes: the small files each engine leaves behind, because they slow every read.
• Compute: requested vCPU and dollars per TB, with compaction included.

Producers replay about a million 200-column rows a second into Kafka for an hour, on skewed or uniform keys. The engine under test writes an Iceberg table through a REST catalog, and the scorer reads back every row ID.

On the skewed track, at 1,376 MB/s for an hour:

Ingest only, Yukon needed 6.8× fewer vCPU than Flink. With compaction included, it was 4.4× cheaper per TB than Spark and left the fewest files.

The caveats. Flink is fresher at p95 when it only ingests (63 s against our 74). Its maintenance run fails on skew because range distribution calibrates at startup; on uniform keys it passes at 422 vCPU. Dollars per TB cover engine compute only, on the same Graviton machines with one price rule, and every fleet is the smallest we've found so far, not a proven minimum. Kafka Connect's Iceberg sink, in our configuration, didn't keep up on the skewed track. If you find a smaller configuration that passes, send it in and we'll update the comparison.

The same stream, in production

The customer from the start of this post now runs on Yukon, with the same stream into the same table:

The freshness numbers are for the oldest file in each commit, measured over 72 hours in October 2026. Each writer core turns about 56,000 rows a second of 200-column Avro into Parquet.

What we'd tell a Flink user

If your stream is modest or evenly spread, Flink with Paimon or Iceberg, Kafka Connect's Iceberg sink, or a managed service like Confluent Tableflow will probably serve you well. We ran Flink for years. The checkpoint model starts to hurt when three things meet: heavy skew, one shared table, and a freshness target measured in minutes.

If you want to check our numbers, run the benchmark or read how we built it and the full results. More on the engine is at eon.io/yukon.

Doron Porat and Liran Yogev, Eon