Eon Yukon vs. Dataproc
Why teams run ingest and table upkeep on Eon instead of a Spark cluster on Google's Managed Service for Apache Spark (formerly Dataproc)
Ingest without a cluster
Eon lands Kafka and GCS data as clean Iceberg tables in one pass, without a Structured Streaming job, merge query job or compaction schedule to run.
Tables BigQuery reads
Eon writes Apache Iceberg to Cloud Storage and registers it in BigLake (now Google Cloud Lakehouse), so BigQuery and every other Iceberg engine read the same tables in place.
Pay per query, nothing when idle
Existing Spark batch jobs can connect over Spark Connect and run serverless, billed per query, so interactive Spark no longer needs a standing cluster. RDD-based and custom-JVM jobs, and GPU training, stay on Spark.
Eon vs. Dataproc at a glance
Data freshness
Query-ready data within seconds.
As fresh as the Spark Structured Streaming job you build. Iceberg advises triggers of at least one minute.
Data preparation & maintenance
Event-level data lands ready to query, with optional updates, deduplication and aggregates. Table maintenance is built in.
Upserts, dedup, compaction and snapshot cleanup on BigLake tables are Spark jobs you write and schedule.
Pipeline complexity
Eon Stream: one managed pipeline produces the tables you choose in one pass, with no separate processing or maintenance jobs.
A streaming job, merge query logic and a maintenance schedule per table, on a cluster or serverless batch.
Context for AI agents
Eon Context: built-in table descriptions, statistics and change history help agents understand and query the data.
Gemini-generated descriptions in Knowledge Catalog, run on demand. No per-commit statistics, value-matched joins or change log.
Open formats & interoperability
Iceberg tables, readable by compatible engines.
Spark reads and writes Iceberg.
Querying & compute management
Eon Query: serverless SQL and Spark Connect, billed per query with no idle compute cost.
Clusters that start in 90 seconds or less on average, or serverless batches billed per second with a 1-minute minimum.
Real-time application serving
Eon Realtime: current values, looked up by key in under 1 ms through Redis-compatible and HTTP APIs.
No serving layer. Results land in a sink, and apps read from a separate cache.
Real-time aggregations
Summaries update alongside event-level data, written to tables and served live, with no state for your team to manage.
Stateful windows in Spark or Flink jobs you write and operate.
Operational overhead
Fully managed by Eon, with no tuning required. Dedicated deployment in your region or BYOC.
Google runs the clusters, and your team runs the jobs, their tuning and on-call.
Compute efficiency
At 1M events/s: 35 vCPU for ingest and 63 with compaction, 4× Spark's compute efficiency.
135 vCPU for Spark Structured Streaming, which Dataproc runs (about 7,400 rows/s per vCPU), and as much again to compact it.
How Eon Yukon compares with Dataproc
Ingest and table upkeep without a cluster
Landing Kafka streams or files arriving in Cloud Storage in lake tables usually takes a Structured Streaming job, a scheduled job to merge or rebuild tables, and compaction and snapshot cleanup on top.
Dataproc runs those Spark jobs well, on clusters or serverless batches. The jobs themselves are still yours: merge query logic, small-file cleanup, snapshot expiry and the schedule that ties them together.
Eon Stream does it in one pass: deduplication, upserts and deletes on compound keys, transforms, sorting and compaction, committed together to Iceberg in Cloud Storage, queryable within seconds.
BigQuery reads the same tables
Many Dataproc teams analyze in BigQuery and don't want a second copy of the data.
With Dataproc, Spark writes Iceberg tables and BigQuery reads them through BigLake, as long as the jobs keep them fresh and compacted.
Eon Stream writes Apache Iceberg to Cloud Storage and registers it in BigLake on commit. BigQuery stays your BI engine on the same tables, with no copies and no load jobs, and Spark, Trino and Databricks read them too.
Spark jobs, unchanged and per query
Spark ETL and interactive Spark run on standing clusters or serverless batches sized in executors.
Dataproc clusters start, scale and shut down in 90 seconds or less on average, and serverless batches bill per second with a 1-minute minimum, starting at 2 executors. Interactive sessions bill at the premium rate.
Eon Query bills per query, with nothing to pay when idle, and existing PySpark and SQL jobs can connect over Spark Connect, serverless. Eon Query answers counts, ranges and distinct values from statistics written at ingest. RDD-based and custom-JVM jobs and GPU training stay on Spark.
Context agents read before they query
Agents need to know what a table means, how it joins and what changed before they can answer from it.
Spark writes standard Iceberg statistics. Descriptions and joins live wherever someone writes them down.
Eon Context writes a description of every table with each commit: grain, column meaning, distributions, joins found by matching values, and recent changes. Agents read it over MCP, so one call replaces the exploratory queries.
How Dataproc pipeline costs compound
Streaming ingest
Included.
A long-running Structured Streaming job.
Upserts and dedup
Included, applied as data lands.
Scheduled merge query or rebuild jobs.
Compaction and snapshot cleanup
Included, continuous.
Spark maintenance jobs on BigLake tables.
Batch ETL and interactive Spark
Per query, nothing to pay when idle. Spark jobs can connect over Spark Connect.
Clusters or serverless batches, with interactive sessions at the premium rate.
Where the difference shows up most:
High-volume streams
compute grows with event rate. Eon uses 4× to 7× less compute than Spark and Flink to ingest and compact the same stream.
More tables
every new table brings its own streaming job, merge query and maintenance schedule on Dataproc. With Eon, cost follows data volume.
Interactive and agent queries
standing clusters and premium-rate sessions cost money between questions. Eon bills only for the queries it runs.
See how Eon works in your environment
Bring a stream you process on Dataproc today and we will show you that data landing in Eon Yukon tables, with freshness and compute compared side by side.