Letting an agent edit the canvas

When an agent produces visual work, the format of its output decides what you can do next. A generated image can be looked at or regenerated. A paragraph of instructions leaves you to do the work and to guess what was meant. Neither gives you what you would want from a colleague working in a design tool, which is a layout you can click on, adjust and carry on with.

In March 2026 I built a canvas prototype for the agent platform I work on around the alternative, in which the agent and the person edit the same document. Its interaction model was inspired by Paper. An agent wrote the first version under my direction, with me as co-author, and I carried it on from there. The prototype was later retired.

One document

Everything else follows from the data model. The canvas kept a map of typed nodes, five types in all, among them artboards, containers, text and images. Every node had an ID, a name, a parent, children, a position, a size, styles and visible and locked flags, and each type added the fields it needed: a background for an artboard, flex direction, gap, padding and alignment for a container, content, font and colour for text, a source and object fit for an image. A single Zustand store held those nodes alongside the selection, the active tool and the camera, which could pan and zoom between 0.15x and 4x. It also held a small amount of state about the agent itself: whether it was active, its status, which artboards it was working on and the prompt history. The toolbar, layer panel, properties panel and prompt bar all sat on top of that store.

Nothing in the model records where a node came from. A container the agent made and a container you made have the same shape and change through the same store operations, so there is no separate thing called generated output that has to be imported, converted or redrawn before you can touch it.

What the agent sees

Each canvas had its own agent session, and the agent worked from a snapshot rather than from the whole document. The snapshot listed each artboard’s ID, name, geometry, background and child count, the total node count, the selected IDs with their subtrees down to a depth of two, the current tool, the artboards being worked on and the last six prompts.

The IDs give the agent exact names to use. The selection does something more useful, because it removes most of the ambiguity from a prompt. If you select a card and type “make this wider”, the word “this” now has a referent the model can act on, along with two levels of structure beneath it.

What the agent writes

The agent’s replies stayed ordinary messages, with its changes carried in fenced canvas_ops blocks. The parser looked for every such block in a reply, parsed the JSON, accepted either an object with an ops array or a bare array, kept only the entries that were objects and ignored anything malformed, so a broken block didn’t take the rest of the reply with it. There were seven kinds of operation, including create_artboard, parse_html, update_styles and set_text. This is a constructed example rather than a captured reply, with the field names simplified:

{ "ops": [
  { "op": "create_artboard", "ref": "pricing", "name": "Pricing" },
  { "op": "parse_html", "parent": "pricing",
    "html": "<section style='display:flex;gap:24px'><h1>Plans</h1></section>" }
] }

And on a later turn, once the snapshot has shown the real IDs:

[
  { "op": "set_text", "id": "node_42", "text": "Simple pricing" },
  { "op": "update_styles", "id": "node_17", "styles": { "gap": "32" } }
]

The awkward part of letting a model write operations is references. In the first block the model has to name an artboard that doesn’t exist yet and whose ID the client will assign. The system prompt therefore set two rules: a new artboard could be given a temporary alias through id or ref, or reached through the token LATEST_CREATED_ARTBOARD, and anything that already existed had to be addressed by its exact ID. The client-side executor kept an alias map as it worked through the operations. create_artboard made the real ID and recorded the aliases against it, and later operations resolved through the map and then checked that the node actually existed in the store. The executor also tracked what was last created and last touched, so when an operation needed a default artboard, it tried the last created, then the last touched, then the selected or first artboard. It recognised four spellings of these placeholders, which nobody would design into an API for human programmers and which makes sense when the caller is a language model.

HTML as notation

Creation used HTML, which is a compact notation for layout and one the browser can already parse. parse_html wrapped the fragment in a root element, ran it through DOMParser and walked the result. Text tags became text nodes, frame tags became layout nodes, img and svg became image nodes, and script, style, meta and link were ignored. A replace mode cleared the target’s children first. Every node was added with selection turned off, so a stream of new nodes couldn’t take your selection away from whatever you were doing.

The result of that conversion was a tree of typed nodes the user could edit. Inline styles passed through with only their keys normalised, so this was a document conversion step, not a general-purpose HTML sanitiser.

Updates took the same route as any other edit. update_styles and set_text resolved their IDs, dropped null values and called the store’s updateNode, where one sanitiser normalised style keys, dropped empty values and coerced numeric strings, so the "32" in the example above became a number. This is the main benefit of the structured approach. The document has one write path with one set of rules, so an agent’s edit and a person’s edit end up in the same format.

If I were doing this again for another kind of document, a spreadsheet or a slide deck or a schema, I would make the same decisions in the same order: a typed document that the user already edits, a bounded view with real IDs and the user’s selection, operations that go through the same write path as the user’s own changes, and a way for the model to refer to something it has only just created.

Chris Chabot, November 2026