Agent quality
How the Ingot agent gets a coding task right with a cheap model, and the head-to-head tasks that check it on every change.
Beta The agent behavior on this page is on for everyone; there is nothing to turn on. The head-to-head tasks are part of Ingot's own test suite: every pull request runs them with a scripted model, and a real-model run is one click away for maintainers.


Small edits instead of whole files
An agent changes an existing file with a REPLACE block: the exact lines to find and what to put there instead, one pair per change. It only sends the lines that change, so a long test file is never retyped, never cut off halfway and never loses a test.
<<<REPLACE index.js
<<<<<<< SEARCH
return parts.join('-');
=======
return parts.join('-').slice(0, maxLength);
>>>>>>> REPLACE
>>>END
- The search text must match the file once. Ingot also accepts a match that only differs in trailing spaces or indentation, and writes the replacement with the file's own tabs or spaces.
- When a search text is not found, or matches more than one place, that file is left unchanged and the agent is told which change failed, so it reads the file again and sends a corrected block. Other files in the same reply are still saved.
- New and short files are still written whole.
It runs your tests when you ask
When your message asks for tests or lint to pass ("make sure npm test passes", "fix the lint errors"), the agent runs the project's checks after its last edit and fixes what fails before it stops. Its final reply only says the work is done when that run was green; otherwise it says what still fails.
No files you did not ask for
A task leaves only the files it was asked to change in git status:
- Artifacts (plans, walkthroughs, screenshots) live in
.ingot/artifacts/, which ignores itself. - Shared context is only written in projects that already have
.ingot/context/. Elsewhere the learning stays in the chat note with Share. - The last test run is remembered in
.ingot/frozen.jsononly when the project already uses Frozen Zones. - The codebase index in
.ingot/index/ignores itself too.
Commits only when you ask
The agent creates branches and commits only when your message asks for it ("fix it and commit", "ship it"). "Fix this bug" or "add this feature" leaves your changes uncommitted for you to review.
Head-to-head tasks
Three tasks run in the real Ingot window against a copy of the open-source slugify library, each scored out of 10 by checking the files on disk, npm test and what the agent said:
| Task | What the agent is asked | What scores |
|---|---|---|
| Code question | How slugifyWithCounter avoids returning the same slug twice, walking through three inputs, without changing files | no files changed, file and line citations, the right three results in order, the mechanism |
| Bug fix | slugify('Hello World!') returns hello-world-; find the root cause, fix it, add a regression test, make npm test pass | lint, tests, hidden behavior checks, a regression test, only index.js and test.js changed, no stray files, an honest final reply |
| Feature | Add a maxLength option with types, readme and tests, and make npm test pass | lint, tests, hidden behavior checks, index.d.ts, readme, new tests, no stray files, an honest final reply |
A final reply is honest when it either reports what still fails or claims success only after a test run that was green after the agent's last edit.
Latest scores
The default cheap model (Claude Haiku 5.5 through OpenRouter), November 4, 2026:
| Task | Score | Time | Cost |
|---|---|---|---|
| Code question | 10/10 | 21 s | $0.003 |
| Bug fix | 10/10 | 38 s | $0.011 |
| Feature | 10/10 | 143 s | $0.026 |
Before these changes the feature task scored 9/10 in 480 seconds for $0.107: the agent retyped whole files, one reply was cut off, and it left artifacts and a shared learning in the repository.
Run them yourself
In a checkout of the Ingot repository, after npm run build:browser:
npm run h2h # real model when OPENROUTER_API_KEY is set, the scripted model otherwise
INGOT_H2H_REAL=0 npm run h2h # always the scripted model (free, offline)
INGOT_H2H_ONLY=feature npm run h2h # one task
INGOT_H2H_CAP_USD=0.4 npm run h2h # this run's hard cost cap
| Variable | Default | What it does |
|---|---|---|
OPENROUTER_API_KEY | empty | Runs the real model. Without it the scripted model answers. |
INGOT_H2H_CAP_USD | 0.60 | Hard cost cap for one run, at most 0.95. A local proxy adds up the cost OpenRouter reports for every request and refuses further requests once the run reaches the cap. |
INGOT_H2H_ONLY | all three | Comma-separated tasks: question, bug, feature. |
INGOT_H2H_REAL | 1 | 0 forces the scripted model even when a key is set. |
INGOT_H2H_MODEL | openrouter-haiku | The Ingot model id to run. |
The run also stops before it starts when the key has used 85% of its OpenRouter limit. Scores, the evidence behind them, and every request and reply go to .ingot-tmp/h2h/; the key is sent in a request header and never written. Screenshots of each finished chat in light, dark and High Contrast go to shots/.
In CI, the Head-to-head tasks (mocked model) job runs all three on every pull request with the scripted model and needs a 10/10 on each. The Head-to-head (real model) workflow runs when you start it from the Actions tab or add the h2h-real label to a pull request; with the OPENROUTER_API_KEY secret it needs 9/10 or better on each task.