.jpg)
Abstract
Enterprise data agents are usually tested on synthetic data written by language models, where schemas are tidy, distributions are uniform and systems rarely disagree. We introduce a benchmark built on Era, which generates complete simulated companies whose data distributions are drawn from aggregated metadata of the exabytes of enterprise data Eon backs up. Each company is consistent across cloud storage and SaaS systems, and every question has an exact answer planted by the generator, so grading is done in code with no judge model.
On everyday analyst questions, the recently released NVIDIA Nemotron 3.5 Super VL open model, self-hosted on a single node with eight NVIDIA H100 GPUs, answered 88.9% correctly at $0.067 per correct answer, level with GPT-6 Astra (88.9%) at 27× lower cost. It never abstained on a question that had an answer, which every API model tested did between three and five times. The benchmark also isolated a specific failure: on questions with no answer in the data, the model sometimes turned “nothing found” into a value.
Using Era as a post-training environment, we trained a rank-32 LoRA-SFT adapter on 324 of the model's own grader-correct episodes from ten other simulated companies, with no human labels, judge model or teacher model, in 29 hours for $609 across four training iterations. On a company the model had never seen, malformed abstentions fell from 3 to 0, correct abstentions rose from 65 to 74 of 90, and cost per correct answer fell to $0.059.
Introduction
An agent that answers questions about a company's data has to find the right records across several systems, page through results, join and aggregate them, and recognise when the requested fact does not exist. Testing this well needs data that behaves like a real enterprise estate and questions with answers that can be checked exactly. Synthetic data written by an LLM tends to fail the first requirement, and grading with a judge model weakens the second.
This report has two aims. The first is to measure enterprise data-agent capability, calibrated abstention and cost per correct answer on an environment with exact ground truth. The second is to test whether the same environment can generate targeted training data to improve a specific behavior, and what that costs. We use NVIDIA Nemotron 3.5 Super as the subject of both, because its open weights let us fine-tune, merge and serve the model ourselves.
Era: simulated companies with exact ground truth
Era generates complete simulated companies: a cloud estate (cloud provider accounts, databases, storage) and the SaaS systems that run on it, including Salesforce, HubSpot, Zendesk, Jira, Gong and Slack. Each company is one business, with the same customers, deals and employees in every system on one timeline, exposed through vendor-compatible APIs and MCP servers.
Era's data distributions are drawn from aggregated metadata sketches of the enterprise data Eon backs up across cloud databases, object storage and SaaS applications, never from customer records. A generated company therefore carries the skew, pagination and cross-system inconsistency of a real estate. All Era data is simulated.
The generator picks each answer first and then plants the records that produce it. Every question has one exact key, computed from the same records the tools serve, and the key is never exposed to the model. A question is unanswerable when the fact was never placed in any system, which makes abstention exactly gradable as well.
Experiment design
What the experiment measures
- Answering through real interfaces. Questions can only be answered by calling the company's tools, and require retrieval across systems, complete pagination, joins, grouping, aggregation and correct arithmetic.
- Calibrated abstention. The benchmark separates correct abstentions, confident answers to unanswerable questions, malformed abstentions, and false abstentions on answerable questions.
- Cost per correct answer. Every model is reported with its price per correct answer alongside accuracy, with self-hosted open models at both Spot and on-demand GPU prices.
- Generalisation. Training and evaluation use disjoint generated companies, so a gain on the held-out company reflects a learned behaviour rather than memorised records.
Task suites
- Everyday questions: 33 factual questions about one simulated fintech, answerable only through S3, Salesforce, HubSpot, Zendesk, Jira, Gong and Slack. Answers are counts, sums, medians and "which account had the most" lookups.
- Unanswerable questions: 30 questions whose answer exists in no connected system, in three forms: an outside-company fact (an analyst-report placement), an absent system (on-call pages with no paging system connected), and an empty field (a credit rating field present but null).
Harness and grading
- The model is given three tools, list_systems, describe_tools and call_tool, and up to 165 tool calls per run.
- It submits answer, abstain and note. A correct abstention is answer: null, abstain: true.
- Strict scoring accepts only an exact match with the key, with no partial credit. An in substance score is reported alongside for answers with the right content that fail the strict check.
- Each model answers every question several times, the score is the share of correct runs.
- Every model in this report ran on the same company build, tools, budget and grader.
Experimental setup
Nemotron 3.5 Super is NVIDIA's open hybrid Mamba-2 and mixture-of-experts model, with 120B total parameters and 12B active parametersa 1M context and a built-in reasoning mode. We ran it from the NVIDIA NIM container on one self-hosted AWS 8×H100 node behind an OpenAI-compatible API. Closed models ran through their providers' APIs and are priced at list price. Nemotron is priced at Spot GPU rates, with on-demand figures reported beside them.
Results
Everyday questions
Nemotron 3.5 Super was level with GPT-6 Astra on strict grading, 3 points below GPT-6.1-Sol, 11 points above DeepSeek V3.2, 16 points above GPT-5.6-Sol and 24 points above self-hosted Qwen3-Coder-Next. It never refused a question that had an answer; every API model did so between one and five times. Per 1,000 correct answers at Spot prices it cost $67, against $1,796 for GPT-6 Astra.
Abstention
In most of its 25 failures Nemotron searched thoroughly and then converted "nothing found" into an answer. An empty field became "MRR is 0", or the note said no information exists while the answer field held the string "None", which the grader correctly reads as a value. In one held-out episode the agent is asked for a company's current monthly recurring revenue. It calls HubSpot and S3, finds the MRR field null and no billing document, and submits {"answer": "0", "abstain": false}. Every tool call is reasonable; the error is the final step.
Even on these runs, Nemotron cost $0.255 per run at Spot prices, about 10× less than GPT-5.6 Sol. This is the failure the post-training in the next section targets: after tuning, correct abstentions rose to 74 of 90 and malformed abstentions fell to zero.
Post-training on Era
Method
Because Era's grader is code, a model's own correct attempts can serve as supervised examples. Nemotron 3.5 Super generated rollouts on answerable and unanswerable questions across ten Era companies, and only the episodes the grader marked correct were kept. From them we built 324 training examples: 142 final answers, 61 steps that re-read a truncated tool result (a narrower query, or the same page with a smaller limit), and 121 abstentions. There were no human labels, no judge model and no API teacher.
We trained a rank-32 LoRA adapter on the attention, Mamba in_proj and shared-expert projections, with the base weights frozen and loss on the supervised turn only. The adapter was merged into the NGC checkpoint and served with the vLLM bundled in the NIM image, using the same flags as the base model. The tuned model was evaluated on an eleventh company that appeared in no training data, with the same questions, harness and grader.
Cost
All post-training compute, across four training iterations, came to $609 over 29 hours on one AWS Spot 8×H100 node. An earlier, independent LoRA-SFT run on a different training set also removed every malformed abstention and raised correct abstentions, from 65 to 69 of 90.
Results on the held-out company
After tuning, every abstention used the expected shape and no run overflowed the context window. On the outside-company question about an analyst-report quadrant, the base model abstained but wrote "None" in the answer field; the tuned model submitted answer: null with a note listing every system it searched. On the absent-system question about on-call pages, the base model answered "0"; the tuned model abstained and noted that no paging system is connected. We are excited about the implications and the potential this has for post-training models.
Discussion
Three factors kept the cost of this loop low. The deterministic grader turned the model's own correct attempts into labelled training data, so nothing had to be annotated or judged. Nemotron's mixture-of-experts design kept serving cost low on long, tool-heavy episodes averaging 1.96M tokens on the abstention runs, and its 262K context held a full episode. Lastly, the whole run fit on a single Spot node, where GPUs cost about a fifth of on-demand.
The result that matters most for enterprise use is that the same environment served as the benchmark, as the filter that selected training data, and as the held-out test. For an open model, finding a failure, post-training against it and re-measuring on an unseen company ran on one node. The cost result was strong enough that we plan to use Nemotron 3.5 Super internally beyond benchmarking.
This is especially compelling for us: in earlier experiments, fine-tuning other open-weight models on the same data did not produce comparable gains.
Limitations
- This is LoRA, not full fine-tuning, and the training targeted a narrow behaviour. Everyday accuracy moved by two runs, within run-to-run noise; we did not attempt to raise it.
- Nemotron ran on self-hosted Spot capacity while API models ran on provider platforms with different reliability guarantees. At on-demand GPU prices, Nemotron's cost per correct answer (about $0.36 base, $0.31 tuned) is close to GPT-6.1-Sol's $0.378 and below GPT-5.6-Sol's $0.440 and DeepSeek V3.2's $0.557.
- The tuned model still mostly answers when a field exists but is empty: 2 of 6 such runs abstained correctly after tuning, against 0 of 6 before. Format failures went from 0 to 2 of 90.
- Results come from one held-out company in one industry.
Open questions and future work
- Empty fields versus real zeros. Whether the empty-field case should be its own suite, and whether training contrast pairs (an empty field next to a real zero) close the remaining gap.
- Reinforcement learning on the grader. The deterministic grader is also a reward. Comparing grader-filtered SFT with RL on the same companies would show how much of the remaining headroom each method recovers.
- Budgets and reasoning mode. Reporting accuracy at several tool-call budgets and with reasoning mode on and off, to keep comparisons between open and closed models fair.
