Dimitar Nedev

Working with AI agents

This page is a way into my public notes on instructing coding agents and chat assistants, and on checking what they produce. On cases picked because they had failed without my standing instructions in claude.ai, 16% of runs failed with them, against 37% without. Further down are a team benchmark of AI agents on Snowflake work and ways to try the notes.

The setup

The instructions around my coding agent and for data work sit in layers, with one kind of rule in each. The tools are Claude Code and Snowflake CoCo (formerly Cortex Code).

Always loadedOn demandFor one delegated runEnforced

Around my coding agent

The setup around my coding agent: what is always loaded, what is enforced, and what the main session loads on a trigger, delegates, or reads on demand, with links to the notesAlways loadeduser-level CLAUDE.mdworkspace AGENTS.mdrepository AGENTS.mdagent memory indexoutput style (main session)Every session, at startEnforced, not requestedhooksguardsnoticessession continuitysettingspermissionsmodelsidle compactionHooks block, warn, re-injectmain sessionLoaded on a triggerskillsWhen the trigger named in theinstructions matchesDelegatedguarded agentsscoutcriticdocs-readerworkflowsEach gets a self-contained packetRead on demandprivate notes vaultworkstream notesVault: index, folder README, one note
Grouped by how each piece reaches a session. A rule that must hold every time goes in a test, a hook or a setting, because prose only requests it. A conditional procedure goes in a skill, and its trigger stays in the always-loaded instructions, because the model might not notice the condition on its own.As it stood on November 10, 2026. The scaffold overview has each piece and why it exists, and each box links to its part.

For data work

The seven instruction layers for data work, with links to the notesL0EnforcementAnything that must hold every timeAlways: permissions, hooks, CI, server-side scopeL1General coreEvidence, candor, calibrationAlways, in both toolsL2Execution contractConnection discipline, execution rulesAlways, in the CoCo CLI and desktop appL3Data layerRules for every data turnAlways, in the CoCo CLI and desktop appL4Repository filesReal identifiers, commands, gatesInside that repositoryL5Task-type modulesRules for one task typeOn demand, as skillsL6Per-call promptOne delegated runOnly for that run
A new rule goes to the first layer that fits. That's L0 if it must hold every time, L1 if it's true for any agent turn, L4 if it names a real identifier, command or gate, L6 if it's true for one run, L5 if it's true for one task type, and L3 if it's true for every data turn. Only L0 enforces anything. A rule that must always hold needs a control there, and until it has one it's a request.The design is in decision 0002, and each box links to its sample or to the decision.

The profile

Most of this work is one fairly large set of standing instructions, which I call my profile. It sits in the account-level instructions field in claude.ai, so it goes out with every chat and Cowork task on my account, and a sanitized copy is in the notes.

When the same failure shows up across unrelated conversations, the fix goes in the profile. A failure tied to one kind of work goes into that project's instructions. The aim is that every later conversation starts a little closer to how I want to work. For coding agents the same idea runs across more layers, including a private notes vault, and The setup above shows how they fit together. The loop below is how I decide which changes earn their place, and the charts are what I've measured so far.

The loop

When a conversation goes wrong, I take it through these steps. Each box links to the part of the notes that covers it, and a worked example follows one real failure through every step.

The loop, in seven steps, with links to the notesA conversation goes wrongAudit it in a fresh sessionClassify each failureagainst the taxonomyChange anything?yesFix it in one layer:account or projectAdd a regression checkwhere a stable one existsRe-run after a modelor harness changenot yetWatch itemfails again
A finding becomes a fix in one layer (an account-level rule or a project-level prompt) or a watch item. A re-run that fails again goes back to classifying.

What I've measured

These are my own runs, so n=1.

Profile versions and checks, by model era

Profile versions and checks, by model eraOpus 4.6: 4 profile versions. Opus 4.7: 9 profile versions. Opus 4.8: 9 profile versions; checks: Jun: first checks, 4 of 4 pass. Opus 5: 4 profile versions; checks: Jul: 7 valid passes, no fails. Opus 5.5: 1 profile version; checks: Sep: 15 of 15 with and without; Oct: the new suite.Profile versions writtenChecks runOpus 4.64 versionsOpus 4.79 versionsOpus 4.89 versionsJun: first checks, 4 of 4 passOpus 54 versionsJul: 7 valid passes, no failsOpus 5.51 versionSep: 15 of 15 with and withoutOct: the new suite
Of 27 versions, 22 were written before Opus 5, mostly before the checks could tell a difference. In September the checks passed 15 of 15 runs with the profile and without it, so the October suite uses cases that had failed without it.Versions are counted by the date they were written.

Runs that failed the checks

Share of runs that failed the checks, where both judges agreedWithout the profile (platform prompt only): 37% failed, 39 of 105 runs. Profile minus one rule: 22% failed, 25 of 113 runs. Full profile: 16% failed, 18 of 114 runs.Without the profile (platform prompt only)37% 39 of 105Profile minus one rule22% 25 of 113Full profile16% 18 of 114
On 23 cases picked because they had failed without the profile, 16% of runs failed with it, against 37% without. The price was answers about 75% longer and runs about 70% slower.545 runs, graded by two model judges. Counts use the runs they agreed on, and "minus one rule" drops the first rule each case lists.

My labels against the judges'

My labels against the judges', 20 runs, five per boxWith the profile, judges said PASS: Agreed on 3; 2 undecided. With the profile, judges said FAIL: Disagreed on 4; 1 undecided. Without the profile, judges said PASS: Disagreed on 4; 1 undecided. Without the profile, judges said FAIL: Agreed on 2, disagreed on 2; 1 undecided. Agreed on 5 of the 15 runs I could decide.Judges said PASSJudges said FAILWith the profileAgreed on 32 undecidedDisagreed on 41 undecidedWithout the profileDisagreed on 41 undecidedAgreed on 2,disagreed on 21 undecidedI passedI failedI couldn't decideI disagreed with the judges
I agreed with the judges on 5 of the 15 runs I could decide, about half once weighted back to all the runs they agreed on. The checks grade whether one rule was followed, and I graded the whole answer.I labeled 20 runs without seeing the judges' verdicts.

Data work

For data work I use Snowflake CoCo next to Claude Code, in the terminal, in Snowsight and over MCP from my other coding agents. In my use, CoCo is good at the Snowflake side. It knows where to find the metadata, and its bundled skills cover a lot of everyday work.

We also benchmarked AI agents on two kinds of Snowflake work. That's team work, separate from my notes, and these are the results I posted on LinkedIn on November 5, 2026.

Building on the platform

Share right, by model, in the same setupOpus 5.5: 77%. Opus 5: 71%. Opus 4.7: 40%. Sonnet 5.5: 79%. Haiku 5.5: 56%.Opus 5.577%Opus 571%Opus 4.740%Sonnet 5.579%Haiku 5.556%
Coding agents in CoCo and Claude Code answered 24 questions, mostly from real work, graded blind by an AI judge. Model version was the one clear difference. In the same setup, Claude Opus 5.5 got 77% right and Opus 4.7 40% (the orange bars), the only gap in the chart that held up statistically. Sonnet 5.5 did as well as Opus 5.5 at about a third of the cost, and effort and an evidence-labeling pack made no measurable difference.All answers came from one account in one day, graded by Claude models.

Answering business questions

Median answer time, by modelOpus 5.5: 17 seconds. Sonnet 5.5 and 4.6: 9 seconds. Haiku 5.5: 14 seconds.Opus 5.517 sSonnet 5.5 and 4.69 sHaiku 5.514 s
One Cortex Agent over a semantic view answered 12 questions, 4 tries each, on five models and on auto, where Snowflake picks the model. Every one got all 48 answers right, so the model changed speed and cost. The Sonnet models were fastest, and Haiku 5.5 cost about 1/40 of Opus 5.5 per answer.The questions were clear, over a clean semantic view of synthetic data.
  • My own test pointed the same way. In a small test of a data agent, a stronger model removed almost all of the recurring judgment errors. Higher effort on the weaker model didn't, and neither did an evidence-labeling pack or an earlier version of my data layer. I keep the layer for what a transcript shows, like writes and data handling.
  • A tool allowlist alone isn't a read-only boundary. With the SQL tool allowed, a headless CoCo run wrote without a prompt. A restricted session scope refused a table create. Access that still holds when an agent acts for a person is platform work, and it's on the main page.

The CoCo notes have the details, with the data layer and the execution contract as samples.

Try it

  • Hand it to your agent. The hand-off prompt asks the agent or assistant you use to pick out what's useful for you. It asks for proposals, not changes.
  • Start a task. The task brief is a message you paste as you start a task, alongside your existing instructions. There's nothing to install.
  • Audit a conversation. The audit prompt goes into a fresh chat with the transcript attached, on a surface that transcript is allowed to reach. It's for the occasional bad conversation, not every one.
  • Borrow a rule. The samples are tuned to my own voice and defaults, so take single rules rather than whole files. The harness index lists them by tool.

The notes' Start here has more ways in, in four groups.

Figures from the notes are as of the 2026-11-11 release, and the version counts from the 2026-10-15 release.