Apache Iceberg on Amazon S3, Made AI-Ready: How Eon Builds the Semantic Layer AI Agents Need

Apache Iceberg on Amazon S3 gives an AI agent typed columns, snapshots, and engine-agnostic reads. It does not tell the agent what a column means, which tables describe the same business entity, how two sources join when they share no key, or which of several near-identical tables is current. Eon builds each of those layers automatically as data lands in open Iceberg on S3, then serves the result to agents over the Model Context Protocol (MCP) and Agent-to-Agent (A2A) — the open interfaces AI agents call.

What sits between an Iceberg lake on S3 and an AI agent

A lakehouse decision on AWS usually ends at the table format: land the data as Iceberg on Amazon S3, register it in the AWS Glue Data Catalog, point Amazon Athena or Amazon Redshift at it. Iceberg gives you ACID commits, schema and partition evolution, snapshots, and the ability to point any Iceberg-compatible engine at the same files on S3. None of that is semantics. A catalog adds structural metadata — names, types, lineage — but structure is not meaning. An AI agent needs semantic definitions of what a column actually holds: verbal descriptions, not data types and column names.

Three gaps remain once the Iceberg tables exist on S3:

• Meaning. Column names and types do not say what a column holds. AI agents need verbal descriptions of the data, beyond its schema.
• Joinability. Business questions span sources, and the relevant columns rarely share a key or a name. A production domain is not a CRM customer_name.
• Ambiguity. Real lakes carry duplicate and near-duplicate tables with drifting definitions; an agent that "selects one that matches and moves on" produces inconsistent or wrong answers.

The conventional fix is a hand-built semantic layer (dbt Semantic Layer, Cube, AtScale) curated per use case, alongside manually declared joins. Those layers are maintained by hand and must be re-curated as sources and use cases evolve — which is the gap Eon closes by generating the layer from the capture path. Join discovery in particular is a studied problem: systems like JOSIE frame it as overlap set-similarity search over column values, and Freyja ranks candidate joins by shared values rather than names. In both, joins are found in the data, not the metadata. Eon's contribution is to run these layers from the capture path and assemble them into a single queryable structure it calls the context graph: a graph whose nodes are columns, tables, and the business entities they cluster into, and whose edges are the value-based join relations and natural-language descriptions derived on ingest. Every layer above the substrate reads from and writes back to this one structure.

Reference architecture: Iceberg on Amazon S3, end to end

The path is deliberately one-directional and read-only into production. Eon reads sources through a single read-only AWS IAM role, writes incremental immutable backups into the Eon Vault on Amazon S3, and registers the result as open Apache Iceberg tables. From there every consumer — analytics engines and AI agents alike — reads the same Iceberg tables with zero ETL and no restore.

The AWS services and their roles in the architecture:

• Sources — Amazon RDS / Amazon Aurora, Amazon Keyspaces, Amazon DynamoDB. Production databases captured read-only via IAM; no agents deployed in production.
• Streaming — Amazon Kinesis Data Streams, Amazon MSK. A first-class path for continuously arriving, high-throughput data into the same Iceberg tables.
• Storage — Amazon S3, with S3 Object Lock and AWS KMS. Immutable, air-gapped Iceberg data files and snapshot storage — the Eon Vault.
• Catalog and governance — AWS Glue Data Catalog, AWS Lake Formation. Iceberg table registration plus fine-grained row- and column-level access control.
• Analytics — Amazon Athena, Amazon EMR, Amazon Redshift (Spectrum). Ad-hoc, batch/Spark, and warehouse query engines reading Iceberg directly — zero ETL.
• AI / ML consumers — Amazon SageMaker, Amazon Bedrock. SageMaker reads Iceberg directly for training data and features; AWS-hosted agents (for example, ones built on Amazon Bedrock) call Eon's MCP tools like any other MCP client — there is no Bedrock-specific integration.

Because the surface is standard Iceberg on S3, the same tables stay readable by non-AWS engines (Databricks, Snowflake, Trino) through their existing Iceberg integrations. AWS is the reference implementation, not a lock-in: the open format is what consumers depend on.

System model: agentless capture into open Iceberg on S3

Eon maps a multi-cloud estate from a single read-only AWS IAM role: no appliance, no compute in the data path, no gateways, no network access into production systems. Data teams define a contextual rule ("all production relational databases with financial information") and Eon ingests every matching source; freshness and retention are configuration, not pipeline engineering. Agentless capture across the estate is one path in, not the only one: Eon also ingests streaming sources through Amazon Kinesis Data Streams and Amazon MSK as a first-class path, and can pull from backups and archives into the same lake. Whichever path it takes, the data lands in the same open Iceberg substrate on Amazon S3 and flows through the same context layers.

Ingested data lands directly in open Apache Iceberg, the open table-format standard for large analytic tables, with files stored on Amazon S3 under S3 Object Lock for immutability and encrypted with AWS KMS. The design is standard Iceberg on the outside, Eon's patented storage engine underneath: an open format any engine can read, over a proprietary storage implementation built for warehouse-grade economics without lock-in. Tables register in the AWS Glue Data Catalog and are governed by AWS Lake Formation, so the context layers publish into the catalog your engines and BI tools already use, and Amazon Athena, Amazon EMR, and Amazon Redshift read them through their native Iceberg support — zero ETL, no restore.

Why the Iceberg format matters for AI/ML reads on AWS:

• Schema evolution is metadata-only. Iceberg tracks columns by unique ID, so add/drop/rename/widen never rewrites data files on S3 or resurrects old data. Ingested production schemas drift constantly; this absorbs the drift without breaking downstream embeddings or training inputs.
• Hidden partitioning yields planned scans. Iceberg derives partition values from a column transform and prunes non-matching files on S3, so a RAG or training job run from Athena or EMR gets a bounded, predictable scan instead of full-table churn — and consumers don't hand-write partition filters.
• Partition evolution is also metadata-only. Old data keeps its old spec; both layouts coexist via split planning, so layout can be tuned to a workload without rewriting data on S3.
• Snapshots and time travel. Point-in-time reads (SELECT ... AS OF) give reproducible training sets and audit, without a separate audit store.

Everything above is the substrate. The remaining layers are produced on top of it, automatically, as data is captured.

Self-describing data: classification at ingest

On ingestion Eon runs an AI agent that generates a contextual natural-language description for every column in every table, then a clustering step groups tables into logical entities that represent business context. Both the descriptions and the entity clusters land in the context graph, which becomes the queryable catalog feeding context to LLMs through AI agents.

Metadata cannot do this part. A type and a name (acct_id BIGINT) tell an agent nothing about whether the column is a billing account, a cloud account, or a login. A verbal description grounds the column so retrieval matches intent instead of string-matching headers. At the scale automatic ingest creates — thousands of databases pulled by one rule — manual cataloging cannot keep pace, which is why the description step runs inline with capture rather than as a later annotation pass.

The classification path is already productized in a narrower form: Eon auto-classifies data classes (PII, financial, PHI) at the resource level, with an API to override or revert. That maps cleanly onto AWS Lake Formation governance — the generated classifications inform fine-grained access policies, and the override path means the layer is reviewable, not infallible.

Join discovery: relations from value overlap

Join discovery is the layer hardest to derive automatically. Customer adoption metrics live in a production Amazon RDS database; the customer's name and account details live in a CRM. Answering "which accounts are expanding usage" requires joining them, but the columns that connect the two share neither a name nor, often, a key. Agents that fall back on semantic similarity fail here, because a production domain is simply not the CRM customer_name; name-and-metadata matching has weak success rates on exactly the joins that matter.

Eon attacks this with two complementary signals, and it derives them from the backup on S3 itself — with no running database to query.

Declared keys, taken as truth. Eon reads the foreign keys the source database already knows about and enters them into the relation map at full confidence. Declared keys are facts, not guesses.

Relationships inferred from the data. The harder and more valuable case is the relationship no one ever declared: the cross-database join, the denormalized copy, the constraint that was never written down. Here Eon looks at the values rather than the names. While capture runs, Eon takes a lightweight statistical fingerprint of each column without scanning the full dataset. Later, outside the capture path so nothing slows down, it compares those fingerprints to find columns whose values genuinely overlap across otherwise unconnected sources. Two columns that hold the same underlying values are flagged as joinable and confidence-scored, even when their names, types, and keys all differ.

Filtering comes first, so the joins Eon proposes are the ones that mean something: low-variety categorical fields, flags, and timestamps that technically "match" everything are excluded, and Eon compares only type-compatible columns. Declared keys are never filtered away. Finally, Eon clusters the surviving relationships into entities, so when many columns across many tables carry the same identifier they collapse into a single group and the system reasons about join paths, not just isolated pairs. The whole map — declared and inferred — feeds the context graph automatically.

Eon documents one worked example: the anchor that linked production adoption metrics to CRM customer records was the cloud-account-ID, correlated against the cloud marketplace identifier listed in the CRM exports. Once that link is established by value overlap, rows from the two systems join even though no one ever declared a foreign key.

The value-overlap half of this is the approach the data-lake join-discovery literature converged on: joinable columns are the ones whose value sets overlap, a signal invisible to names and types. The same literature documents the weakness: pure overlap over-fires when unrelated columns happen to share a value range, like sequential IDs. Eon's advantage is that value overlap never stands alone: declared keys anchor the map where they exist, noise columns are filtered out before anything is scored, and surviving candidates are reconciled against the context graph's descriptions, entity clusters, and freshness before they reach an agent. Overlap proposes a join; the rest of the system decides whether it's real.

Ambiguity, similarity, and freshness

Duplicate and near-duplicate tables are normal: the same column names mean different things across databases, and legacy tables hold stale data. An agent cannot tell them apart, and its default is to pick one and move on — so the same question asked twice can return two answers, and a user usually never learns a source was chosen from several.

Eon uses the context graph and the joinability map to find tables that contain similar data, and it monitors freshness — naturally captured as Iceberg snapshots on Amazon S3 — as an indicator it passes to the agent. When similarity crosses a threshold, the agent is handed the differences, including how current each candidate is, to surface to the user or look up in the organization's knowledge bases, rather than guess. The reconciliation reuses the same two structures the earlier layers already build, so no new index is required.

Serving the context to agents on AWS: retrieval and acceleration

Eon serves the context graph to agents three ways, because a context graph is only useful if an agent can hit it at query time.

• Semantic search and prompt-to-table embeddings. Even when a matching view or table exists, the agent has to find it. When matching isn't straightforward, Eon embeds the prompt and the destination table/view in a vector store and retrieves over them; RAG works well for this routing problem, and the retrieved tables are standard Iceberg on S3.
• Auto-managed SQL views. On ingest Eon generates SQL views to capture complex queries, and when an agent iterates to build a new query, the resulting SQL is saved as a reusable view so similar questions reuse it instead of re-deriving it. Views stay virtual when they're cheap to resolve and are materialized when complexity or runtime warrants, so the agent pays the multi-step derivation once and subsequent reads — from Amazon Athena or Amazon Redshift — hit precomputed results.
• Native MCP/A2A serving. Eon exposes the retrieval surface directly from the substrate over MCP (Model Context Protocol) and A2A (Agent-to-Agent): smart Iceberg layouts, cached scans, and vectorized compute in place, with planned scans for RAG and no separate serving tier. In shipped form, the Eon AI Agent discovers relevant tables across environments, runs cross-resource and cross-snapshot analysis with automatic join detection, generates SQL, and is callable from Claude Code, Codex, and Gemini Enterprise through the Eon MCP server and A2A. Because the interface is standard MCP, any MCP-compatible AI application — including AWS-hosted agents such as those built on Amazon Bedrock — can call the same tools; there is nothing Bedrock-specific about it.

Personalization: a hierarchical knowledge graph

"Active customer" means one thing to a CS engineer and another to finance. Eon stores these subjective definitions as knowledge served to LLMs through agents, in a hierarchical knowledge graph — company, then team, then person — layered on top of the context graph so a query resolves against the requester's definitions. These definitions are configuration-driven, not a self-correcting loop, so the same prompt returns the answer the requester's role means.

Governance and trade-offs

Running on AWS, the generated layers inherit AWS-native controls, and the honest limitations carry over:

• Governance is AWS-native. Iceberg data files sit on Amazon S3 under S3 Object Lock (immutable, air-gapped) and AWS KMS encryption; access is enforced through AWS Lake Formation fine-grained policies. Eon's auto-classification (PII, financial, PHI) feeds those policies, with an override API.
• Joinability mixes fact and inference. Declared foreign keys enter as ground truth; inferred relationships are probabilistic and can over-fire when unrelated columns share a value range. Filtering, plus reconciliation against context and freshness, mitigates this but doesn't eliminate false positives.
• Generated layers need review. Column descriptions, entity clusters, and saved views are model-derived. The classification override path signals that human-in-the-loop is expected; descriptions and views warrant the same review.
• The layers are continuous compute. Describing and profiling every column across thousands of databases, then monitoring freshness, is ongoing work, not a one-time index build.

How the layers compose

Each layer reuses the one beneath it — from capture, up through classification, the join map, reconciliation, serving, and the knowledge graph. The result is one self-describing substrate, and what an engineering team plugs into is open Iceberg tables on Amazon S3 and an MCP/A2A endpoint: no separate semantic-layer build, no separate serving tier.

If you own an Iceberg or lakehouse decision on AWS and are working out what your tables on S3 actually need before agents can use them, that architecture is the right thing to walk through end to end. We're happy to do that against your own estate: a technical session or a scoped pilot on your AWS sources — RDS, Keyspaces, DynamoDB, or S3 — landing in open Iceberg, queryable from Athena, EMR, and Redshift, and served to your AI agents over open MCP.

FAQ

What does it mean for an Iceberg lake on AWS to be "AI-ready"?

An AI-ready lake adds the layers an agent needs above the table format: natural-language descriptions of every column, clustered business entities, a cross-source join map, and ambiguity resolution. Apache Iceberg on Amazon S3 — registered in the AWS Glue Data Catalog — gives an agent typed columns and engine-agnostic reads, but not the meaning, relationships, or currency signals it needs to answer reliably. Eon derives those layers automatically from the capture path.

How does Eon store and catalog data on AWS?

Eon writes captured data as open Apache Iceberg files on Amazon S3, immutable under S3 Object Lock and encrypted with AWS KMS, and registers the tables in the AWS Glue Data Catalog with governance through AWS Lake Formation. Amazon Athena, Amazon EMR, and Amazon Redshift then read the same tables directly — zero ETL, no restore.

How does Eon detect joins across sources that share no key?

Eon combines two signals. It reads the foreign keys the source database already declares, and, for the relationships no one declared, it analyzes column values rather than names. While capture runs, Eon takes a lightweight statistical fingerprint of each column, then flags pairs whose values overlap across otherwise unconnected sources, filtering out low-variety and mistyped columns first. In Eon's worked example, a cloud-account-ID correlated against the cloud marketplace identifier in CRM exports anchored a join between production adoption metrics and CRM customer records, with no declared foreign key.

Does building an AI-ready Iceberg lake on AWS create vendor lock-in?

No. Eon writes standard, open Apache Iceberg on Amazon S3, so AWS engines (Athena, EMR, Redshift) and non-AWS engines (Databricks, Snowflake, Trino) read the same tables through their existing Iceberg integrations. A patented storage engine sits beneath the open format for warehouse-grade economics, but the surface consumers depend on is standard Iceberg.

How do AI agents query Eon's context graph on AWS?

Eon exposes its retrieval surface natively over MCP and A2A directly from the substrate, with no separate serving tier. The Eon AI Agent discovers relevant tables, runs cross-resource and cross-snapshot analysis with automatic join detection, and generates SQL, and is callable from Claude Code, Codex, and Gemini Enterprise through the Eon MCP server. Because the surface is standard MCP, any MCP-compatible AI application — including AWS-hosted agents such as those built on Amazon Bedrock — can call the same tools; nothing is Bedrock-specific.