How a CSM Used an AI Agent to Audit 27 Credit Proposals

The standard worry about AI at work is that it’s confidently wrong. A language model writes a wrong answer in the same calm tone as a right one, and it doesn’t flag its own mistakes. Most of the people I work with are already sold on AI, so I rarely hear the worry out loud. It’s still a fair worry, and the place it hurts most in customer success is arithmetic, because the numbers end up in front of customers.

When my credit sizing numbers came out wrong across 27 proposals, though, the AI wasn’t the cause. The math came from an internally built tool. The AI agent is how I found the errors, measured them across my whole book and fixed them.

I run most of my work as a Strategic CSM at Netlify through Claude Code, an AI agent that works on files on my laptop. I wrote about the setup in the previous post. This one is about that audit and the checking habits around it.

The audit

Netlify moved enterprise customers to a credit-based model, where usage of each resource draws down a pool of credits. For renewals, I size each customer’s pool from 6 to 12 months of their real usage, plus some headroom.

The sizing formula came from our in-house customer agent, the internal tool the team uses to look up a customer’s usage, adoption and invoices. My own sizing pages used the same formula, and it had two errors, both of which pushed the numbers up:

It double counted one kind of request. Those requests were already included in another column of the usage export, and the formula added them on top.
It counted a type of usage that isn’t billable at all.

Separately, the sizing pages had a rounding problem. It rounded the monthly figure up to a “nice” number from a short ladder of round values, then multiplied by 12. A monthly figure that landed just over one step got pushed to the next step, and multiplying by 12 carried that whole jump into the annual pool. On one account, the pool came out roughly 50% above its basis, headroom included. Across my book, rounding alone inflated the totals by about 11.5%.

I found it myself. Before the fix, one customer had already received a proposal built on the old formula. It wasn’t one of the accounts the errors hit hardest.

None of these were exotic. Each was the kind of mistake a person makes in a spreadsheet. The difference is that a tool makes it the same way on every account it touches, every time, and nobody notices unless they go looking.

The fix

I went looking with Claude Code. I had the agent re-audit all 27 sizing pages. Each one was regenerated with the corrected formula, then reconciled against an independent recomputation from that account’s own usage export, done in a separate pass. The worst deviation between the two was 0.0000%. All 27 were done on the same day. What I found went back to the team that owns the customer agent, and the tool got fixed and revamped.

The rounding fix is one line of logic:

Before: monthly figure -> round up to a nice number -> x 12
After:  monthly figure x 12 -> round the annual pool up -> monthly = pool / 12

The pools still come out as clean numbers, and the overshoot from rounding is now a fraction of what it was.

On my side, the fix went into writing, in two places. The corrected formula went into my own sizing workflow (a skill, in Claude Code’s terms) and the script that builds each sizing page, so it gets used. The rule went into the brief, the standing instructions file the agent reads at the start of every session, so it doesn’t come back:

- Pool rounding: round the ANNUAL pool, never the monthly baseline.
  Fixed 2026-09-04.

The brief also says why. The old rounding is described there, with the error it caused, so a future session that “simplifies” the math back runs into the explanation first.

Know which source is ground truth

The next question was which data is actually true.

The sizing runs on usage data from our data warehouse, so it’s easy to treat that as the source of truth. It isn’t. Billing invoices are. When I had the agent compare the two across my book, they agreed within a few percent in most months. In some older months, though, the warehouse held only about half of the usage that was actually billed, across every kind of usage at once. Any sizing built on those months would come out low.

So the script that builds a sizing page now compares the usage data with the invoices and prints a warning line on the page when a month looks short. I don’t have to remember which months are bad. The page tells me.

Recompute before anything goes out

The other standing rule is about checking. Every document with numbers in it gets its headline figures recomputed from the source file in a separate pass before it’s handed to me, and any mismatch gets reported. From my brief, with the specific figures taken out:

- Before handing over any document with numbers, recompute the headline
  figures from the source file in a separate pass and report any mismatch.
  I have caught errors by hand; I would rather you catch them.

The errors I’d caught by hand included a bandwidth figure off by a factor of four and a fee added on top of another fee.

For credit proposals, the recompute is one item on a checklist that runs every time, with results reported back to me. A condensed version:

1. Recompute every headline number from the inputs file; fix any mismatch.
2. No em dashes in the document.
3. Nothing that looks like an internal account id, if customer-facing.
4. No prose that counts a request twice or calls non-billable usage billable.
5. The Download PDF button and print styles are present.

Item 4 exists because of the audit. Fixing the formula wasn’t enough, since the old mistake could still show up in the explanation around the numbers. The checklist scans the prose too.

Ask before building

The cheapest check happens before any work starts. If I haven’t said who a document is for, what format it’s in, how long it should be, or what basis to use, the agent asks in a short numbered list and waits. I answer in seconds. It stops the agent from guessing on the things that matter most, like whether a document is for the customer or for me.

Corrections become dated rules

Every correction goes into the brief with a date. A recent one, from a multi-year proposal:

Steps never compound year over year (set 2026-10-13).

That rule came from a proposal built the wrong way. One proposal got fixed by hand, and the rule fixes every proposal after it.

Since September, the brief has turned into the most accurate record I have of how pricing and billing actually work for my accounts. Each fact carries the date I verified it and what I verified it against.

The human holds the irreversible actions

The agent drafts. It builds pages, writes proposals, and deploys previews where I can look at things. It asks before any production deploy, and it never sends anything to a customer on its own. Every email, Slack message and proposal goes out from me, after I’ve read it.

Where it goes wrong for me these days is mostly writing: tone, length, phrasing that sounds like a machine. I vet everything as much as I can before it leaves. It has a dedicated writing pass that strips out the usual AI tells, plus a growing list of style rules, like no em dashes and a short list of stock headings it may never use.

Speed and checking

An internal tool made the same arithmetic mistake on every account it sized. Checking that by hand would have meant redoing 27 proposals one at a time. With the agent, I audited all of them in a day against an independent recomputation, instead of spot-checking a few and hoping. Writing the fix down as a dated rule means a new session starts out knowing about it.

Speed is the easy part. Verification is the part that takes work.