Observe, act, verify: how we think about AI agents for documents

An AI agent that works in documents needs three kinds of tools. It needs ways to observe a file without reading all of it, ways to act on exactly the part it means to change, and ways to verify that the result is what was asked for. Leave out any one of the three and the agent either guesses, damages the file, or stops at "done" without knowing. This post explains why documents make each of these hard, and the principles we design around. It stays at the level of ideas; the details of our own agent are for another day.

1. 1

   Observe

   Look at the structure first, then read only the part the task needs, as it is stored.

2. 2

   Act

   Make one targeted edit that names exactly what it changes and stops if the file is not as expected.

3. 3

   Verify

   Read it back, recompute, render or compile, and compare the result with the request.

A failed check is a new observation, and the loop starts again. When the checks pass, the change goes to you to keep or discard.

Why documents are a hard place for agents

Code is the setting most agents were built for, and code is friendly to them. It is plain text, it is split into many small files, and it comes with its own checks: a compiler, a type checker, a test suite. An agent can read a file, change a line and run the tests.

Office documents and long papers are different in four ways:

• What you see is not what is stored. A Word paragraph that reads as one sentence can be stored as a dozen pieces, each with its own formatting. A spreadsheet cell that shows 12.5% stores 0.125 and a number format, and may hold a formula that produced it. A slide's layout is a set of coordinates and inherited settings, not a picture.
• They are big, and most of each file is irrelevant to the task. A thesis can run to hundreds of pages. A workbook can hold tens of thousands of rows. The part that matters to a request is usually small.
• Formatting is meaning. In code, whitespace rarely matters. In a document, a heading style, a number format or the position of a text box is part of what the author made. An edit that gets the words right and quietly drops the formatting has still broken the file.
• There is no test suite. Nothing tells the agent that a slide now has overlapping text, or that a total no longer adds up. Those checks have to be built.

So the question we keep asking is: what would an agent need in order to work in a document the way a careful person does?

Observe: let the agent look before it reads

A careful person opening a 200-page report doesn't read it start to finish before fixing one table. They look at the outline, jump to the section, read the table and a little of the text around it. An agent should work the same way, for two reasons. Its working memory is limited, so a whole document crowds out the request itself. And the more irrelevant text it holds, the easier it is to edit the wrong place.

Good observation tools share a few traits:

• Start with structure. An outline of headings, a list of sheets and their used ranges, a list of slides with their titles. From there the agent can decide where to look.
• Sample, then zoom in. A few rows of a table, the first lines of each section, or a summary of a slide's shapes, followed by a full read of only the part that matters.
• Search by meaning and by position. Find the paragraph that mentions the budget; read cells B2 to F40; show slide 7.
• Show what is stored, not only what is shown. For a spreadsheet, the agent needs both the formula and its value. For a paragraph, it needs the text and the style it carries. For a slide, the text and where each box sits. An agent that sees only rendered text will "fix" a number by typing over the formula that made it.
• Report what was left out. If a read is cut short, the agent should be told, so it doesn't treat a partial view as the whole file.

The aim is a view of the document that is small enough to think with and faithful enough to act on.

Act: edits that say exactly what they change

Once the agent knows what to change, it needs a way to change only that. The common approach is to write a script that opens the whole file with a general-purpose library and saves it again. That works for many jobs, and we describe its trade-offs in why our agent edits Office files with native operations. At a higher level, these are the properties we want every edit to have:

• Targeted. An edit names its object in the document's own terms: this paragraph, this cell range, this shape on this slide. Everything it doesn't name stays as it was.
• Explicit about intent. "Set the number format of B4 to a percentage" is a better edit than "rewrite this sheet". A specific edit is easier to check, easier to explain to the person reviewing it, and harder to get subtly wrong.
• Checked before it writes. If the edit expects certain text at a certain place and finds something else, it should stop and say so. A file that changed since the agent last looked is a reason to look again, not to write anyway.
• Respectful of what is there. New text should take on the formatting around it. A new row should match the rows next to it. The author's styles come first.
• Reversible. Every edit should be easy to undo exactly, so that the person in charge can say no. We wrote about how we do that in how Discard puts a file back byte for byte.

An edit that fails loudly is better than one that succeeds quietly at the wrong thing. If the agent asks for something the edit can't do, the agent should hear about it right away, so it can try another way or tell the person.

Verify: give the agent a way to check its own work

This is the step most often missing. After an edit, a model will often report success because the edit call returned without an error. But "the call succeeded" and "the document is now right" are different claims. Grounding means checking the second one against the file itself.

What verification looks like depends on the format:

• Read it back. After changing a paragraph, read the paragraph again from the saved file. After setting a formula, read the cell's formula and its computed value.
• Recompute. For spreadsheets, check that totals still add up and that formulas still point at the ranges they should.
• Render and look. Layout problems, like text running off a slide or two boxes overlapping, only show up when the file is drawn. Rendering the page or slide and looking at the picture catches what reading the XML cannot.
• Compile. For LaTeX, the compiler is the test suite. A clean build, with no undefined references, is the check. Our guide to common LaTeX errors lists what that check usually catches.
• Compare with the request. Look at the change itself, before and after, and ask whether it does what was asked and nothing else.

Each check produces a new observation, and the loop starts again. If the check finds a problem, the agent fixes it. If it passes, the agent can say what it changed and how it knows.

The last verifier is you

No automated check knows what you meant as well as you do. That's why every edit the Nebula agent makes waits for your decision: you can open Inspect changes to see each change with its Before and After, then choose Keep edit or Discard edit. The agent's own checks are there to make sure that what reaches you is worth reviewing. Your review decides what stays. The user side of this is in keep or discard: how to check every AI edit.

What we are still working on

None of this is solved. Some of the questions we spend our time on:

• How small can a view of a document be while staying faithful to it?
• Which checks are worth their cost after every edit, and which only at the end?
• How should an agent show its evidence, so that reviewing its work takes less effort than doing the work yourself?

We'll write more as we learn. If you want to see the loop in practice, the use-case guide shows three real runs on Word, Excel and PowerPoint files.

Related

• Why our agent edits Office files with native operations
• How Discard puts a file back byte for byte
• What happens to your files when you use the agent