Understanding Reliability Bottlenecks in LLM Agents for HPC I/O Telemetry Analysis

The question: can an agent answer a facility's I/O questions reliably?

HPC centers record a lot about how jobs use storage. Darshan profiles each job's I/O, the scheduler logs who ran what on how many nodes, and operations staff log outages and maintenance. Turning those records into an answer is still manual work. A question such as "why was my job's I/O three times slower than last week?" means joining profiler records across runs, checking whether the job landed on a different file system or allocation, and knowing which Darshan counters actually measure time spent in I/O.

A tool-using LLM agent looks like a good fit for this work: it can write the SQL, run the Python, and hand back a table. The paper asks the question a facility would ask before trusting one. Does the agent get the answer right reliably, across repeated runs, and which design choices change that? A plausible answer produced once is not enough.

To answer it, the authors built CIODI-Bench (Complex I/O Data Insights Benchmark, pronounced "coyote-bench"), 49 expert-curated tasks over public production telemetry from the Polaris supercomputer at Argonne. They then held the data, the tools, and the scoring fixed and varied three things: the context the agent is given, how the work is split among LLM roles (the orchestration), and the model.

The main result is that no single design choice wins everywhere. Which choice matters depends on the kind of task. Curated context decides extraction tasks. A reviewer that checks the work decides analysis tasks whose answer is one of a few labels. On open-ended questions and on the hardest tasks, neither helps much. What limits those tasks is the agent's own judgment about how to approach the problem, and a single clarifying sentence from an expert helps far more than extra agent machinery.

What the benchmark asks agents to do

The data. The tasks run against one PostgreSQL database that links three public sources by job ID:

The Darshan logs are parsed into per-module counter and summary tables for STDIO, POSIX, and MPI-IO, plus a table for traced jobs that did no instrumented I/O. That is nine tables in all.

The tasks. Seven tasks reproduce case-study analyses from an earlier study of the Polaris logs. The rest came from a survey of I/O experts and national-lab facility staff, who were asked what they would realistically ask an agent with access to this data. Six tasks are deliberately unanswerable. For example: "For jobs using Lustre file systems in 2024, what fraction had good striping settings in place?" Polaris's Darshan collection has no Lustre module data, so the right response is to decline.

The tasks fall into four types, which differ mainly in how open the answer is:

Difficulty is set by a countable proxy: the number of distinct steps (queries and post-processing operations) in the reference solution. Easy tasks are nearly directly observable in the data. Medium tasks require mapping common analysis concepts onto the schema. Hard tasks require domain reasoning over intermediate results.

The scoring. Every task has one reference CSV. The grader normalizes the agent's CSV (types, common formatting differences) and scores each column according to what it holds, so a run earns partial credit. The task score is the mean of the column scores, between 0 and 1. Columns holding short free-text explanations are scored by an LLM judge that compares one reference answer with one candidate row and gives 0, 0.5, or 1.

Repeated runs. Each configuration solves each task three times. The paper reports two numbers per configuration:

• average-of-3: the mean score of the three runs. This is what a user should expect from one run, and it is the paper's main reliability measure.
• best-of-3: the best of the three runs. It shows what the configuration can do when sampling goes its way.

A configuration whose best-of-3 is far above its average-of-3 sometimes solves a task and sometimes does not. That gap is the instability the benchmark is designed to expose.

The agents being compared

All configurations get the same three tools: a read-only SQL tool, which shows the agent up to 10 rows and saves the full result as a CSV in the agent's sandbox; a persistent Python/Bash kernel; and a submit tool, which ends the run and sends the CSV to the grader. They differ along three axes.

Context. By default, the agent's system prompt documents the schema: table and column meanings, and conventions that are easy to misread, such as whether a table is per job or per file. It also carries a short list of domain conventions that pilot runs showed agents getting wrong. Agents treated partial logs as failures. They misread rank=-1, which marks data shared across all ranks. They computed throughput as bytes over wall time instead of over actual I/O time. And they misunderstood how MPI-IO activity also appears in the POSIX records beneath it. The "online" variants, marked (O), get none of this and must discover the schema with exploratory queries.

Orchestration. The paper arranges the designs as a ladder of increasing structure. Step through it here:

Direct-Solver

1. The solver reads the task, its required output format, and the curated schema and domain notes.
2. It queries the database. It sees a 10-row preview, and the full result is saved as a CSV in its sandbox.
3. It post-processes that CSV in a persistent Python kernel, and can go back and forth between querying and computing.
4. It submits one CSV. The run ends and the grader scores the file column by column.

Plan-and-Execute

1. The planner turns the task into an ordered list of steps: all at once (Plan-Execute), or one step at a time based on results so far (the iterative variant, Plan-Execute (I)).
2. It hands the next step to a separate executor.
3. The executor carries out the step with the SQL and Python tools.
4. It reports a detailed summary back. If a step failed, the planner revises the plan before continuing.
5. When the plan is done, the result is submitted for grading.

Plan-Execute-Review (process-aware)

1. Planner and executor produce a candidate CSV, exactly as in Plan-and-Execute.
2. A third role, the reviewer, receives the task and output format, the planner's steps with the executor's summaries, and the candidate CSV.
3. It checks the key decisions independently, running cross-checks and alternative approaches that either confirm the result or call it into question.
4. The reviewer submits the final CSV.

Plan-Execute-Review (PB), process-blind

1. Planner and executor produce a candidate CSV, as before.
2. The reviewer receives only the task and the candidate CSV. It does not see how the answer was produced.
3. Without the trace it cannot target the weak decisions, so it tends to rebuild the same framing or solve the task again from scratch.
4. The reviewer submits the final CSV.

The paper also runs DS-STAR, a recent open-source data-analysis agent with seven specialized roles (Analyzer, Planner, Coder, Debugger, Router, Verifier, Finalizer). It is adapted as lightly as possible: it gets the same SQL and Python tools, a one-shot solvability check before execution (it has no way to decline a task otherwise), and the same cap of 20 refinement rounds as the original work.

Model. Unless noted, every role uses OpenAI's GPT-5, with high reasoning effort for the solver and planners and the default (medium) for executors. The (M) variants swap in the cheaper GPT-5-mini for the whole Direct agent, for only the planner (M-P), or for only the executor (M-E).

All 11 configurations at a glance

Configuration	Context	Roles	Model
Direct	curated	one solver	GPT-5
Direct (O)	discovered by the agent	one solver	GPT-5
Direct (M)	curated	one solver	GPT-5-mini
Plan-Execute	curated	planner (full plan up front) + executor	GPT-5
Plan-Execute (I)	curated	planner (one step at a time) + executor	GPT-5
Plan-Execute (M-E)	curated	planner + executor	GPT-5 planner, GPT-5-mini executor
Plan-Execute (M-P)	curated	planner + executor	GPT-5-mini planner, GPT-5 executor
Plan-Execute-Review	curated	planner + executor + reviewer who sees the trace	GPT-5
Plan-Execute-Review (PB)	curated	planner + executor + reviewer who sees only the answer	GPT-5
DS-Star	curated (replaces DS-STAR's Analyzer)	DS-STAR's multi-agent loop	GPT-5
DS-Star (O)	discovered by DS-STAR's Analyzer	DS-STAR's multi-agent loop	GPT-5

Result 1: the bottleneck moves with the kind of task

Each panel below plots one task type. The x-axis is average-of-3, what to expect from one run. The y-axis is best-of-3, the best of three tries. Better configurations sit toward the upper right. The dashed diagonal is where all three runs scored the same.

Data extraction: context is what matters

Extraction tasks have one correct table. Here the deciding factor is whether the agent understands the schema. Direct with the curated context averages 93.5%; the same agent discovering the schema on its own, Direct (O), averages 73.0%, a 20.5-point gap. The online agents' errors are upstream ones, such as misreading what a column means or joining on the wrong identifier. Everything after that step can be carried out correctly and still produce a wrong table.

DS-STAR's design makes this point sharper. Its Analyzer role summarizes the tables one at a time in a fresh context before planning begins. DS-Star (O) still reaches only 75.1%, close to Direct (O). Spending more agent effort on understanding the schema does not replace a vetted description of it. Once the context is curated, even the configurations that use GPT-5-mini stay in the 86.6–90.1% range. In this regime the model matters less than what the agent is told.

Data analysis: review helps when the answer is one of a few labels

Analysis tasks still require interpretation, but the final answer is often one label from a short list, such as shared_access_is_better versus independent_access_is_better. A reviewer can check that kind of answer: it can rerun the few alternative choices that would flip the label and see whether the label holds.

The numbers follow this. Plan-Execute raises the average-of-3 from 51.5% (Direct) to 56.9%, and Plan-Execute-Review reaches 64.7%, 13.2 points above Direct. The reviewer must see the process, though. The process-blind reviewer reaches only 53.2%. Seeing the trace tells the reviewer which assumptions were made and which checks were skipped, so it can test something new instead of repeating the original line of reasoning.

Insight generation: everything drops, and extra structure stops paying

Insight tasks ask the agent to decide which signals matter and to explain a finding. The best configuration is plain Direct, at 45.7%. Plan-Execute-Review, the winner on analysis, scores 43.7%. Decomposition and review help when they can anchor on a check that tells answers apart. On an open-ended question, a reviewer adds another layer of open-ended reasoning on top of the first.

Averaged over all configurations, the score falls from 87.4% on extraction to 38.5% on insight generation. The paper's failure analysis names the most common failure in both analysis and insight tasks tunnel vision. The agent commits early to one framing (one metric, one table, one hypothesis) and finishes the task inside it. One early wrong choice of grouping level, time window, or definition of "better" still yields a complete, well-formatted table that looks plausible and is wrong.

Unsolvable: the model making the decision matters

GPT-5 configurations reliably recognize when the data cannot answer a question. The exceptions are the configurations that make that call with a smaller model. Direct (M) and Plan-Execute (M-P), which plan with GPT-5-mini, fall to 72.2% and 66.7%. Plan-Execute (M-E), which uses GPT-5-mini only to execute, still scores 100%. The paper reads this as model capability mattering most at decision points: whether to answer, which plan to follow, and which hypothesis to test. Insight tasks point the same way, though less cleanly. Every GPT-5-mini variant scores below its GPT-5 counterpart there (Direct (M) 40.3% against Direct's 45.7%), and the two split-model variants are the lowest of all, the one with a mini planner at 32.8% and the one with a mini executor at 31.7%.

Summary. As tasks move from constrained extraction to semi-constrained analysis and then to open-ended interpretation and abstention, the main limit moves from semantic grounding (understanding the schema), to verification (whether the answer can be checked cheaply), to upstream judgment (choosing the right approach in the first place). The paper condenses this into a failure taxonomy:

Result 2: hard tasks, and what one sentence from an expert does

Sorting the same results by difficulty shows how strongly these limits bind. Averaged over all configurations, the score falls from 87.4% on easy tasks to 22.6% on hard ones.

The hard tasks show why. Take one that asks the agent to find applications using MPI-IO's read-modify-write optimization (data sieving). An expert starts from a cross-layer fact about Darshan: MPI-IO is implemented on top of POSIX, so MPI-IO activity on a file should come with POSIX activity on the same file. From that follows a concrete signature. Look for files with nonzero MPI-IO write volume and POSIX reads and writes on the same file. Even when told the general fact that MPI-IO behavior shows up in the POSIX layer, agents do not turn it into this test. They ignore the POSIX layer and look only at MPI-IO counters, or they pick a different cross-layer rule that does not work.

So the bottleneck on hard tasks is not carrying out a known plan. It is coming up with the right plan. Newer models do not remove it. With the Direct agent, GPT-5.4 reaches 34.5% and GLM-5 31.7% on hard tasks, against 26.9% for GPT-5 as plotted, and GPT-5.4 mini drops to 18.4%. These values are read from the paper's figure. None of them is reliable enough to hand the work to an agent unattended.

One clarification per task. As a diagnostic, the authors looked at the main failure on each hard task and added a single sentence of clarification that addressed it without giving away the answer. For the data-sieving task, the added instruction was along the lines of "use the cross-layer POSIX read+write signature on the same file as the detection rule." That one sentence turns a near-impossible task into routine processing, much like an extraction task.

Post-hoc clarification depends on hindsight: someone had to see the failure first. As a first step toward doing this live, the authors gave the agent a decision-logger tool and asked it to record every decision the prompt did not settle, treating each logged decision as a question an expert would answer. This is preliminary evidence that much of the gain is possible without a post-hoc rerun, but the agent logs many decisions that did not need an expert. Measured per logged decision, the gain is 11.44% on hard tasks and 0.40% on easy ones. Expert attention is worth far more on the hard tasks, and the open problem is routing only those decisions to the expert.

Result 3: what the extra reliability costs

The paper measures cost as output tokens, the text the agents generate. Input tokens depend heavily on how verbose the tools are and on prompt caching, so output tokens are the fairer comparison. At OpenAI's March 2026 prices, 20k output tokens cost about $0.20 with GPT-5 ($10 per million) or $0.04 with GPT-5-mini ($2 per million).

The configurations fall into clear tiers:

• About 18–30k tokens per task: Direct, Direct (M), and the Plan-and-Execute variants. Most of the generated text goes into one end-to-end solve.
• About 33k: Direct (O). Exploring the schema is added on top of a direct solve.
• About 43–53k: the two reviewer configurations, 2.3× and 2.8× Direct. The process-blind reviewer costs more because without the trace it tends to solve the task again.
• About 100–150k: DS-STAR, 5.4× Direct with curated context and 8.1× when its Analyzer also explores the schema. Its router can throw away steps it judges wrong, so work gets done, abandoned, and done again.

Over all 49 tasks, Plan-Execute-Review is the most reliable configuration (67.7% against Direct's 64.0%), for 2.3× the output. DS-STAR spends more than any other configuration and scores below Direct. Curated context is the cheap intervention. It raised reliability on extraction and analysis, and Direct (O) generated more tokens than Direct because it had to rediscover the schema.

What this means for an HPC facility

The authors do not argue for replacing dashboards. Recurring, well-understood analyses belong in dashboards, curated queries, and templates, which are validated once and reused. Agents are being measured on the long tail: new, multi-source questions whose joins, choices, and checks are not known in advance. For that long tail, the results suggest an order of investment:

1. Write down the schema semantics and local conventions first. It is the largest single gain on well-specified extraction work, and it lowers generation cost instead of raising it. Letting the agent discover conventions on its own is fragile, and more exploration effort does not fix it.
2. Add a lightweight, process-aware reviewer only where the answer can be checked, meaning tasks with a small set of possible outcomes that a counter-check can confirm or overturn. Give the reviewer the trace; a reviewer that sees only the answer adds cost for little gain.
3. Spend on a stronger model at the decision points, such as planning and deciding whether a question is answerable, before adding more roles.
4. Keep an expert in the loop for hard, open-ended analysis. One targeted clarification moved the hard tasks from about a quarter to 80.7%. The open question is how to ask only when it matters. The paper suggests triaging each unsettled decision by running the possible branches, and escalating only when they lead to different answers.

Once a novel analysis has been validated and becomes routine, it should move out of the agent and into a curated query or dashboard.

Limits of the study

One site. All tasks are about Polaris, so the numbers are controlled single-site evidence, not estimates for other facilities. The paper argues the pattern should carry over because it follows from task structure: undocumented local conventions will be hard to discover anywhere, checkable answers will benefit from targeted review anywhere, and open-ended analysis will be limited by framing and judgment anywhere. That is an argument, not a measurement at a second site.

Three runs per task. Three runs estimate a typical run but not the tail of the outcome distribution, so the paper emphasizes patterns across task types over rankings between nearly tied configurations. To check that three runs are enough, the authors reran Direct nine times on 12 tasks, three per type:

The two estimates differ by at most 0.04 in any task type.

Cost covers generation only. Output tokens are a fair proxy for generation work, but they leave out input tokens and wall-clock time.

Reproducing it. The artifact (benchmark, database, agent implementations, and a launch notebook for Chameleon Cloud) is public. After 60–90 minutes of setup, its functional check runs one task under each of the 11 configurations, plus one hard task with and without its clarification, in about two hours and for 5–10 USD of API calls. Rerunning the full study, three runs of all 49 tasks under every configuration, would take about 12 days serially and about 800 USD in output tokens.

@inproceedings{egersdoerfer2026reliability,
  author    = {Chris Egersdoerfer and Orcun Yildiz and Kevin Harms and
               Robert Latham and Robert B. Ross and Dong Dai},
  title     = {Understanding Reliability Bottlenecks in {LLM} Agents for
               {HPC} {I/O} Telemetry Analysis},
  booktitle = {Proceedings of the International Conference for High
               Performance Computing, Networking, Storage and Analysis, {SC}
               2026, Chicago, IL, USA, November 15-20, 2026},
  publisher = {{ACM}},
  year      = {2026},
  note      = {To appear}
}