An agent without a tool contract is a user with an undocumented admin panel. It will look productive in a demo and expensive in production.
I treat tools the same way I treat a public API: typed inputs, explicit side effects, auth that cannot be talked out of, and a log you can read without replaying a chat.
This sits on top of the harness split I use at AvenFort — Python loop, .NET tenant API, Azure runtime. The harness is allowed to call. It is not allowed to invent a new way to mutate tenant state.
Tool contracts
Write the tool as if a second team will implement the agent.
- Name and intent. One verb, one object.
create_quoteis a contract.handle_requestis a wish. - Input schema. Required fields, enums, and size limits. The harness rejects anything else before the API sees it.
- Idempotency. If the model retries, the API must not create a second record. Put the key in the contract, not in a comment.
- Auth. The tool runs as a principal, not as “the agent.” Tenant scope comes from the session, not from a field the model can set.
- Error taxonomy. Validation, conflict, not found, and upstream timeout are different. If everything is a string, evaluations cannot tell a bad prompt from a down dependency.
If you cannot write those five lines, the tool is not ready. Prompting harder will not finish the design.
Evaluations
I do not start with a leaderboard. I start with fixtures.
A fixture is a frozen world: tenant, user, and a transcript that should produce one tool call or a refusal. The harness runs the loop against a fake API that records calls. Pass means the calls match the contract. Fail means the agent skipped a field, double-applied a side effect, or called a tool it was not offered.
Keep three buckets:
- Happy path. The obvious job, once.
- Refusal. Missing permission, missing field, or a request that should stay in chat.
- Retry. The API returned a timeout; the second call must not duplicate work.
You can add model-quality scoring later. If these three buckets are empty, you are scoring poetry.
Observability
Log the loop as a trace, not as a novel.
I want, per turn: agent id, tool name, schema version, latency, outcome, and a correlation id that matches the tenant API request. I do not want the full prompt dumped into the same stream as payment logs.
When something is wrong, the question is “which contract failed,” not “what did it say.” If you cannot answer that from traces, the harness is still coupled to the conversation.
What this is not
This is not a claim that evals replace review. A new tool still needs a human who understands the domain object. The contract is how that review scales across time zones: the PR shows schema, tests, and traces, not a screen recording of ChatGPT.
Next
Once tools are boring, modernization work looks familiar again. The next notes are about cutting over a live product while the rest of the team keeps shipping — agents or not.