← All posts

PDSW'26 Invited Talk: Agents in Data Infrastructure

TL;DR

  • On well-defined telemetry questions, agents already do well; on hard ones, stronger models and heavier orchestration barely help, while one targeted sentence from an expert can turn a near-impossible task into routine extraction.
  • The core failure is that agents do not know what context they lack and do not ask. Running the alternative choices and asking an expert only when the results differ is one way to make that loop efficient.
  • For optimization, wider actions demand stronger verifiers, real-application trials are slow and noisy, and LLM optimizers tend to lock onto their first guess, making a trustworthy, affordable reward signal the central open problem. Our community's knowledge decides how well these agents work.

On November 16 I gave the last invited talk at PDSW'26, the Parallel Data Systems Workshop held with SC'26. PDSW is a small community of people who build and study parallel storage, HPC I/O, and data systems. It is also the community I grew up in, so I used the twenty minutes to say where I think our field fits now that LLM agents are arriving in the software we run every day.

This post is the written version of that argument.

Agents will become part of the data-infrastructure stack. How well they work will depend less on model reasoning than on what our community knows and has not yet written down: what telemetry means, which reward signal we can trust and afford, and when to bring in a human expert.

The loop most of you already know

I organized the talk around the standard picture from reinforcement learning: an agent observes the state of an environment, takes an action, and receives a reward. Data infrastructure fits this picture almost too well.

1.00

  • The agent is an LLM agent that reads telemetry, reasons about what is going on, and changes the system.
  • The environment is the data infrastructure: storage and data services, plus the workloads running on them.
  • The state is telemetry: metrics, logs, traces, and server counters.
  • The actions are knobs, configurations, policies, and eventually code.
  • The reward comes from running a workload and comparing it with a baseline on performance, cost, or reliability.

The loop itself is not new. But challenges are still there: the state arrives without its meaning, and the reward is noisy and expensive. For example, a Darshan log (the lightweight per-job I/O summary collected at many HPC facilities) does not tell an agent what a strange combination of counters implies. In addition, a tuning trial is not a cheap simulator step; it is a real job on a real file system, and running the same job twice can give very different numbers. Almost everything we learned over the last two years follows from those two facts.

The easy half is already here

Start with the good news.

Telemetry analysis is turning from a pattern-recognition task into an agentic one. For more than a decade our community built the collection and diagnostic layer, from Darshan, UMAMI, DXT Explorer, and Drishti to ML-based tools such as AIIO, and that line continues with WisIO (ICS'25) and DataCrumbs (ProTools@SC'26). Since 2023, many groups have started putting LLMs and agents on top of that layer. On our side, that progression runs from an early HPDC'23 poster on file-system log anomalies to ION (HotStorage'24), IOAgent (IPDPS'25), and CIODI-Bench (SC'26), alongside recent systems such as EPIC, SMARTTalk, and KORAL in HPC storage and D-Bot and RCACopilot in database and cloud operations. Slide 5 of the deck has the full timeline as a reading list.

For well-defined questions (look something up, join two tables, aggregate a column), frontier models are already strong. Ask "What fraction of monthly node-hours comes from jobs that have Darshan traces?" and a current model, drawing on what it learned about Darshan in training and on its ability to write SQL and Python, already gets most of the way there. In CIODI-Bench, the benchmark Chris Egersdoerfer presented in the SC'26 main program, a single GPT-5 agent discovering the schema on its own averaged 73.0% across three runs on data-extraction tasks; giving it a curated schema and a short list of domain conventions raised that to 93.5%. Once someone writes down what the columns mean, extraction is mostly a solved problem.

Hard questions are hard to understand

The surprise came from questions that are open-ended or ambiguous, or that depend on unstated relationships across layers of the storage stack.

Across our experiments, the usual remedies did little: switching to stronger models, asking the agent to repeat and recheck, giving it history or extra tools, or building heavier multi-agent pipelines. In CIODI-Bench, the best configuration on open-ended insight questions was a plain single agent at 45.7%; adding planning and review, which helped most on structured analysis tasks, scored 43.7%.

What did help was one sentence from an expert. The clearest example asks which jobs use MPI-IO data sieving. Data sieving is an MPI-IO optimization for scattered writes: rather than issuing many small writes, the library reads a large contiguous block, modifies it in memory, and writes the whole block back. Darshan never records "this job used data sieving." An expert infers it across layers. MPI-IO is implemented on top of POSIX, so a file with MPI-IO writes and both POSIX reads and POSIX writes is the signature of read–modify–write.

The agents did not find this test. Even when told, in general terms, that MPI-IO behavior shows up in the POSIX layer, they kept looking only at MPI-IO counters. When we added one targeted clarification per hard task, for this one roughly "use the cross-layer POSIX read+write signature on the same file," accuracy on the hard tasks jumped from about a quarter to 80.7%. A nearly impossible task became routine data extraction.

So the bottleneck on hard questions is not executing a plan. It is coming up with the right plan, and that depends on knowledge the model does not have and the telemetry does not carry.

The agent does not know what it is missing, and it does not ask

This is the point I most wanted the PDSW audience to take home.

Missing context is normal. Every new student or technician who joins a facility lacks the small, annoying details of that specific system. The difference is that people ask and agents don't.

The harder part is that we often have the missing context. We just don't know which piece the model is missing. It turns into a cat-and-mouse game where we try to guess what the agent will need and supply it in advance. Usually we cannot, so every new situation first has to fail once.

For instance. We asked an agent to label machine-check error events as Minor, Major, or Critical, and pointed it to the Intel manual. Intel's machine-check status registers have several flag bits, including UC (uncorrected), PCC (processor context corrupt), VAL (valid), and OVER (overflow). The agent scanned the manual and anchored on the most visible bit: OVER was set on 1.67 million of 1.93 million rows. It then wrote its own rule promoting those events to Major. The code ran cleanly, passed every check, and produced plausible output. Read from the Intel manual alone, that rule is defensible; for our site, it is wrong. The site's convention uses only UC, PCC, and VAL to grade severity, while OVER only says that the error-record bank overflowed. Nothing in the run looked like a failure, and nothing prompted the agent to ask.

A related failure is tunnel vision: once the agent commits to an interpretation, it defends it. We asked an agent which features explained the high performance of the best MPI-IO jobs. It noticed that the file names in the MPI-IO records on /lus/eagle (the Lustre parallel file system) did not match those in the POSIX records on /local/scratch, concluded "ignore POSIX," and dropped the POSIX data. Yet the POSIX activity on /local/scratch lined up in time with the MPI-IO activity, and the observed MPI-IO throughput matched node-local storage rather than Lustre. Within the same time window, these jobs were using /local/scratch for local caching and staging, so the high performance reflected node-local storage speed. That relationship shows up in timing and throughput, not in file names. Once the agent decided the two layers were unrelated, it never looked.

Any solutions?

We tried several obvious fixes. They are worth listing because they are the first things anyone tries.

  • Prompting the agent to be curious and ask more questions. It either asked about everything or about nothing. Often it simply did not know whether a given decision deserved a question.
  • An optional "ask the expert" tool. It did not work well.
  • Online clarification, where the agent logs every decision the prompt did not settle and an expert answers. It helped on hard tasks, but the CIODI-Bench paper describes "substantial unnecessary escalations."
  • A reviewer that sees only the final answer. On data analysis tasks it reached 53.2%, against 51.5% with no review at all.

Two things did work.

The first is a reviewer that can see the process. When the reviewer reads the agent's trace, meaning which assumptions it made and which checks it skipped, data-analysis accuracy rose from 51.5% to 64.7%. The useful thing to review is the work, not the conclusion. This is the same lesson behind my earlier post on checking agent claims against the work record.

The second is our attempt at deciding when an expert should be involved. It comes from ongoing, unpublished work in my group on telemetry preparation, so treat it as early. The idea has three parts.

  1. Do not hard-code a workflow, but divide the work into stages. Stages keep the result stable when the model guesses badly on a given day, and they give the agent fixed points to check its own understanding.
  2. When the agent faces an uncertain decision, it does not ask right away. It forks: it runs each plausible choice. For example, when deduplicating log events, it may be unsure whether a field belongs in the event identity, so it tries using the field, not using it, and using it partially.
  3. Deduplication is deterministic code, so each branch can actually be executed on the same saved input and the results compared on row counts and label distributions. If the branches agree, the decision does not matter and the expert is not disturbed. If they disagree, the agent asks, and it brings the executed evidence.

1.00

In the example above, two of the three options remove almost nothing (0 and 2 rows out of 307,931). The third treats rows with the same error code, timestamp, and failed component as one hardware error, keeping only 128,710 rows (removing 58.2%). A difference that large is worth an expert's time. In our runs so far this needed at most four questions per run, and each question was about a decision that actually changed the output.

Good answers to hard questions still cost too much

There is an important distinction on model strength. When a site convention or cross-layer rule is missing, as with the OVER bit or data sieving, extra reasoning tokens cannot invent a convention nobody wrote down. When the telemetry fields are already understandable and the difficulty comes from searching across many tables and hypotheses, frontier models at their highest reasoning setting can do quite well. In one setting, we gave an agent a short time window of telemetry from an HPC system and asked it to find a slowdown and its root cause. Frontier models handled this well, as long as you ignore latency and cost.

Running that capability 24/7 over a whole cluster, with near-real-time feedback, does not make economic sense today. We do not have a clear answer to this yet. (On the slide all costs were estimates, because we use subscriptions; otherwise, as I said in the talk, we would go bankrupt. That is a real concern, not only a joke; see Will AI bankrupt my lab?.) Even here, cost points back to the same lesson: the cheapest way to make an agent reliable is not burning more reasoning tokens on every window, but giving it the right domain knowledge at the right time.

The other half of the loop: a reward we can trust and afford

The second half of the talk moved from reading the state to acting on the system: autotuning.

Storage tuning has moved from search and reinforcement learning (H5Tuner, CAPES) to LLM agents. An LLM can read the file-system manual, the source code, and the application's I/O trace, and propose a sensible configuration after very few trials. Our STELLAR system (SC'25) retrieves parameter rules from the Lustre manual, analyzes the application's Darshan trace, proposes configurations, reruns the application, and reaches near-expert performance within five attempts. We have since applied the same approach to DAOS, an object-based HPC storage system. More recent work in the broader systems community goes further and lets agents synthesize code directly, including AlphaEvolve, ADRS, and PolicySmith.

At its core, though, agent-driven optimization is still trial and error. Understanding the code and the manual makes each proposal far better than blind search, yet every proposal still has to be tested on a real system with a real workload before the agent can make the next move. Once you look at optimization through that trial-and-error loop, three difficulties reinforce one another.

First, the wider the action space, the harder it is to verify the reward. Tuning knobs, as STELLAR does, is conservative and safe because the system design stays fixed, even though knobs cap the upside and the spaces are large: Lustre 2.12.5 exposes at least 159 tunable user parameters, and Ceph Nautilus comes with 1,536 parameters, though not all of them are meant to be tuned. Letting an agent rewrite code raises the ceiling, yet generated policies can game the score, need strict verifiers, and often transfer unevenly: in PolicySmith, a caching heuristic synthesized on one trace beats hand-written baselines on anywhere from 14% to 64% of the remaining traces in the same dataset. Sitting between knobs and code is an option that, as far as I know, no agent uses yet: choosing a composition of components. Frameworks such as Mochi already break HPC data services into composable building blocks, where an agent could pick the combination that fits a workload. The practical path is to climb gradually from knobs to composition to code, building verifiers first and having agents read code to set knobs before they rewrite it.

Second, evaluating the reward itself is caught between fidelity, runtime, and noise. Tuning against synthetic I/O and metadata benchmarks such as IOR, IO500, or mdtest is cheap, yet a mismatch between benchmark metrics and application behavior has been observed (Liem et al., SSDBM'26), and good parameter settings depend on the application, platform, problem size, and concurrency (Behzad et al.). Tuning on the full application is faithful, and far too slow for many iterations. Noise makes the bill worse: on the Polaris supercomputer at Argonne, among 20,956 pairs of jobs with nearly identical Darshan summaries, 758 pairs still differ in I/O performance by more than 5×, which means each trial has to be repeated. The middle ground is an I/O kernel: extract a small program that preserves the application's I/O behavior, check it against a real run's profile, tune on the kernel, and confirm on the application. The open question I care most about here is how much trace is enough to rebuild a faithful kernel. A Darshan summary is too coarse; a per-call trace from Darshan DXT or Recorder is too heavy. We have no results on that threshold yet.

Third, expensive rewards punish LLM optimizers in particular because of tunnel vision. In the standard loop (propose, run, reflect, propose again), we have observed that a good first trial gets refined while a bad first trial stays stuck. Success is largely decided on the first attempt. That is the same early lock-in we saw in telemetry analysis, now showing up in optimization, and it matches controlled findings elsewhere that LLMs "do not robustly engage in exploration without substantial interventions" (Krishnamurthy et al., NeurIPS'24). To be clear, this is an observation from our own runs; no storage-tuning paper has measured it systematically yet. When every trial costs multiple cluster runs, getting stuck near a bad initial guess is fatal, which is why ideas like population and novelty search (AlphaEvolve, ShinkaEvolve), candidate archives, LLM-guided Bayesian optimization (GPTuner), and RL-style exploration schedules (CAPES) deserve attention in storage tuning.

Put those pieces together and the bottleneck on the action side of the loop is clear: for optimization, the open problem is a reward signal we can trust and afford.

What this means for our community

The question is no longer whether agents will enter the data-infrastructure stack. They will. The question is who makes them work well, and with what.

Right now, frontier models already handle well-defined queries and routine code generation reliably. My view, as a judgment rather than a measurement, is that this is why the bottleneck has shifted so visibly onto the things surrounding the model: what telemetry actually means across layers, which reward signals are faithful and affordable, and how to bring a human expert in at the moments that change the answer. Those are problems our community has worked on for decades, and model vendors cannot solve them from outside. Our domain knowledge has not lost value because of LLMs; it has become the deciding factor.

Concretely, here is where I think effort will pay off over the next few years:

  • Ship semantics with the data. When a facility publishes telemetry, it should also publish a machine-readable description of what the fields mean and how they relate across layers, and expert clarifications should be reusable across runs and across sites. My guess is that curated semantics could become facility-level infrastructure, much as Darshan made I/O characterization routine.
  • Keep humans in the loop, and make that loop efficient. The goal is not full autonomy. It is to interrupt an expert only when the answer changes the outcome. Fork-and-compare is one way to do it; there will be better ones.
  • Build verifiers before agents rewrite code.
  • Explore component composition. Composable services such as Mochi make this possible, and nobody has an agent choosing compositions yet.
  • Build cheap, faithful I/O kernels, and find out how much trace they actually need.

I ended the talk by saying I was not there to sell one system; I wanted to find out who wants to work on which piece. That still holds. If one of these problems is close to yours, I would like to hear from you.

Further reading: the CIODI-Bench paper report has the full telemetry-analysis results; the STELLAR paper report covers LLM-driven Lustre tuning. For a broader view of where human effort still matters in agent-driven research, see Auto Research Is Real. It's Just Not What People Think.