AI is expanding what organizations want from their event data. Clicks, transactions, and sensor readings can support model training, feature pipelines, and agents that need current business context. As teams bring more streams into their lakehouse, the cost of keeping that data usable can limit how far they scale.
Table maintenance is part of ingestion. Compaction, sorting, and clustering keep a streaming table efficient to query. With general-purpose engines, that maintenance usually means separate jobs or services to configure, monitor, and pay for. The cost of a stream includes all of it.
We built Eon Yukon to reduce the compute needed to keep streaming tables fresh and organized. It is a managed service that handles ingestion and table maintenance together, so your team has no separate jobs to run or tune. It writes standard Iceberg tables to your own storage, where any compatible engine can query them directly. In the benchmarks below, it used 4-7x less ingestion compute than Spark and Flink, and its cost advantage grew further once compaction was included.
Built for continuous ingestion and maintenance
Eon Yukon's ingestion framework separates writing, committing, and table maintenance. Writers produce files in object storage, a committer publishes them to Iceberg, and maintenance services manage file size and data layout.
Keep every core busy. Eon Yukon is built in Rust, with no JVM or garbage collector competing for CPU. Because a separate committer publishes their files, writers never pause for a commit or wait on a slower peer, unlike micro-batches, where one slow task leaves the rest idle. Every core keeps doing useful work, which is a large part of why a smaller fleet keeps up.
Share hot keys across writers. A single customer, device, or event type can account for a large share of incoming data. Eon Yukon assigns work to writers by bytes, so a hot key spreads across the writers already running.
Maintain file size and data layout together. Eon Yukon integrates compaction, sorting, and clustering into the ingestion framework. Compaction consolidates small files. Sorting and clustering organize the data for efficient queries. These services run alongside ingestion and scale independently of the writers.
Measuring ingestion and maintenance together
We found no benchmark that measured streaming ingestion the way a lakehouse runs it, keeping a table fresh at high volume while accounting for small files and compaction. These goals pull against each other. Committing often keeps a table fresh but leaves many small files that slow reads, and compacting them costs compute of its own. We built ingest-bench to measure them together: freshness at volume, the file sizes each engine leaves behind, and the compute to compact them.
Every engine receives the same pre-generated Kafka input at a fixed rate. The ingestion benchmark checks the table itself for missing or duplicate rows and measures how far it trails the stream. Uniform and skewed inputs test how key distribution changes the required fleet size. The compaction comparison adds maintenance compute.
The target is the smallest tested fleet that meets the workload's requirements. The public harness includes Spark and Flink configurations and results, with an interface for adding other Kafka-to-Iceberg engines.
Benchmarks
For event ingestion, we compared Eon Yukon with Apache Spark and Apache Flink at 1 million rows per second of 200-column events (1,375 bytes per row) from Kafka into Iceberg, about 86 billion rows a day.
To qualify, a configuration must land every row exactly once and keep p95 table lag below 90 seconds and maximum lag below 120 seconds for a full hour. Table lag measures how far the queryable table trails the stream.
Ingestion
For ingestion alone, Eon Yukon kept up with a fraction of the provisioned compute used by Spark and Flink.
Skew without a larger ingestion fleet. The skewed run keeps the same total input but sends 15% of rows to one key. Eon Yukon ran the same fleet for both.
Ingestion and compaction cost
Frequent commits make new rows visible sooner, but can leave small files behind. More files add planning and storage-request overhead, so keeping data fresh also creates maintenance work. Ingestion efficiency needs to be assessed alongside the compute required to consolidate those files.
On both streams, we compare Eon Yukon's built-in compaction with separate compaction jobs in Spark and Flink. All three use the same rules for which files to merge and the same 512 MiB target file size. Eon Yukon compacts with less compute and at lower cost.
Counting ingestion and compaction together shows the compute required to publish new rows and maintain the target file size. The combined cost remains lower for Eon Yukon than for Spark and Flink.
Upserts, too
Eon Yukon also supports upserts and deduplication. We tested upserts at 49,000 order changes per second (1 KB per row), where the table must hold only the newest version of each order. The savings carry over: Eon Yukon met the same freshness requirement with far less compute and memory.
How we measured
The exact configurations, raw results, and code to rerun Spark and Flink are in the ingest-bench repository.
All engines ran on AWS Graviton. The tested versions were Spark 4.1, Flink 2.2, and Iceberg 1.12, with Spark and Flink tuned for the workloads. Resource figures describe provisioned capacity, not a complete cloud bill. We also tried Kafka Connect with the Iceberg sink. On 288 vCPU, the configuration we tested committed only about three quarters of the stream, with four minutes between commits, so it has no published result. The attempt and its limits are in the repository.
Eon Yukon is commercial, so Eon ran its tests, under the same checks as every other engine. The repository includes Eon Yukon's results. If you find a smaller configuration that passes, submit a result and we'll update the comparison.
