Beta Coming soon. This feature is still being built, so names and details may change before it ships.

Agent quality

How the Ingot agent gets a coding task right with a cheap model, and the head-to-head tasks that check it on every change.

Beta The agent behavior on this page is on for everyone; there is nothing to turn on. The head-to-head tasks are part of Ingot's own test suite: every pull request runs them with a scripted model, and a real-model run is one click away for maintainers.

The Ingot agent chat after the head-to-head feature task: maxLength added to slugify with REPLACE edits, a green npm test check, and a final reply that says npm test passed and names one edge case it left unfixedThe Ingot agent chat after the head-to-head feature task: maxLength added to slugify with REPLACE edits, a green npm test check, and a final reply that says npm test passed and names one edge case it left unfixed
The Ingot agent chat after the head-to-head feature task: maxLength added to slugify with REPLACE edits, a green npm test check, and a final reply that says npm test passed and names one edge case it left unfixed

Small edits instead of whole files

An agent changes an existing file with a REPLACE block: the exact lines to find and what to put there instead, one pair per change. It only sends the lines that change, so a long test file is never retyped, never cut off halfway and never loses a test.

<<<REPLACE index.js
<<<<<<< SEARCH
	return parts.join('-');
=======
	return parts.join('-').slice(0, maxLength);
>>>>>>> REPLACE
>>>END
  • The search text must match the file once. Ingot also accepts a match that only differs in trailing spaces or indentation, and writes the replacement with the file's own tabs or spaces.
  • When a search text is not found, or matches more than one place, that file is left unchanged and the agent is told which change failed, so it reads the file again and sends a corrected block. Other files in the same reply are still saved.
  • New and short files are still written whole.

It runs your tests when you ask

When your message asks for tests or lint to pass ("make sure npm test passes", "fix the lint errors"), the agent runs the project's checks after its last edit and fixes what fails before it stops. Its final reply only says the work is done when that run was green; otherwise it says what still fails.

No files you did not ask for

A task leaves only the files it was asked to change in git status:

  • Artifacts (plans, walkthroughs, screenshots) live in .ingot/artifacts/, which ignores itself.
  • Shared context is only written in projects that already have .ingot/context/. Elsewhere the learning stays in the chat note with Share.
  • The last test run is remembered in .ingot/frozen.json only when the project already uses Frozen Zones.
  • The codebase index in .ingot/index/ ignores itself too.

Commits only when you ask

The agent creates branches and commits only when your message asks for it ("fix it and commit", "ship it"). "Fix this bug" or "add this feature" leaves your changes uncommitted for you to review.

Head-to-head tasks

Three tasks run in the real Ingot window against a copy of the open-source slugify library, each scored out of 10 by checking the files on disk, npm test and what the agent said:

TaskWhat the agent is askedWhat scores
Code questionHow slugifyWithCounter avoids returning the same slug twice, walking through three inputs, without changing filesno files changed, file and line citations, the right three results in order, the mechanism
Bug fixslugify('Hello World!') returns hello-world-; find the root cause, fix it, add a regression test, make npm test passlint, tests, hidden behavior checks, a regression test, only index.js and test.js changed, no stray files, an honest final reply
FeatureAdd a maxLength option with types, readme and tests, and make npm test passlint, tests, hidden behavior checks, index.d.ts, readme, new tests, no stray files, an honest final reply

A final reply is honest when it either reports what still fails or claims success only after a test run that was green after the agent's last edit.

Latest scores

The default cheap model (Claude Haiku 5.5 through OpenRouter), November 4, 2026:

TaskScoreTimeCost
Code question10/1021 s$0.003
Bug fix10/1038 s$0.011
Feature10/10143 s$0.026

Before these changes the feature task scored 9/10 in 480 seconds for $0.107: the agent retyped whole files, one reply was cut off, and it left artifacts and a shared learning in the repository.

Run them yourself

In a checkout of the Ingot repository, after npm run build:browser:

npm run h2h                         # real model when OPENROUTER_API_KEY is set, the scripted model otherwise
INGOT_H2H_REAL=0 npm run h2h        # always the scripted model (free, offline)
INGOT_H2H_ONLY=feature npm run h2h  # one task
INGOT_H2H_CAP_USD=0.4 npm run h2h   # this run's hard cost cap
VariableDefaultWhat it does
OPENROUTER_API_KEYemptyRuns the real model. Without it the scripted model answers.
INGOT_H2H_CAP_USD0.60Hard cost cap for one run, at most 0.95. A local proxy adds up the cost OpenRouter reports for every request and refuses further requests once the run reaches the cap.
INGOT_H2H_ONLYall threeComma-separated tasks: question, bug, feature.
INGOT_H2H_REAL10 forces the scripted model even when a key is set.
INGOT_H2H_MODELopenrouter-haikuThe Ingot model id to run.

The run also stops before it starts when the key has used 85% of its OpenRouter limit. Scores, the evidence behind them, and every request and reply go to .ingot-tmp/h2h/; the key is sent in a request header and never written. Screenshots of each finished chat in light, dark and High Contrast go to shots/.

In CI, the Head-to-head tasks (mocked model) job runs all three on every pull request with the scripted model and needs a 10/10 on each. The Head-to-head (real model) workflow runs when you start it from the Actions tab or add the h2h-real label to a pull request; with the OPENROUTER_API_KEY secret it needs 9/10 or better on each task.