Rule tests
Rule tests check that your rules actually change what the agent does. Each test sends a prompt with your rules to a cheap model and checks the reply. Ingot reports the pass rate, and a hard cost cap stops a run before it can spend more than you allow.
Write a test
Put tests in .ingot/tests/*.yaml:
tests:
- name: Types go in a types file
prompt: Add a CartItem interface for the cart. Where does it go?
files: [src/cart/cart.ts] # optional: picks the glob rules, like an open file would
runs: 3 # optional: repeat to measure flakiness (at most 10)
assert:
- contains: cart.types.ts
- judge: The interface goes in a .types.ts file and is imported with import type
mock: Put it in src/cart/cart.types.ts and import it with import type.
The model gets your always-on rules plus the rules whose globs match files. Use rules: [name, ...] to send only some rules.
| Assertion | Passes when the reply… |
|---|---|
contains / not_contains | has (or doesn't have) the text, ignoring case |
matches / not_matches | matches (or doesn't match) a regular expression |
max_words | has at most that many words |
judge | meets the criterion, as graded by the same model (PASS or FAIL) |
mock is the reply of the free mock model, so the test can run in CI without a key or any cost.
Run tests in the IDE
Open the Rules & MCP page, pick the Rules tab and click Test, or run Test Rules… from Find Action.
- Pick a model. Mock (free, offline) costs nothing; the others are cheap OpenRouter models, with
google/gemini-2.5-flash-liteas the default. - Set the Cost cap. The default is $0.05 and it can't go above $1.
- Click Run Tests.
The dialog shows the pass rate, the cost of the run against the cap, and every test with its failed assertions and the model's reply. The last run is saved in .ingot/.state/rules-test-report.json, and the next time you open the dialog it shows that run.
Real models need an OpenRouter key. Add it on the Models page; it stays in your OS keychain and only the backend uses it.
The cost cap
Before each call Ingot works out the most it could cost (the prompt plus the full reply limit) at the model's price. If that would take the run over the cap, the call is not made and the test counts as skipped. A run never spends more than its cap, even when the model answers at length.
From the terminal and in CI
npx @ingotide/cli rules test # default model, $0.05 cap
npx @ingotide/cli rules test --max-cost 0.2 # raise the cap (at most $1)
npx @ingotide/cli rules test --mock # free mock model (CI)
| Option | Meaning |
|---|---|
--model <id> | OpenRouter model id, or mock |
--mock | the free mock model; INGOT_RULES_TEST_MOCK=1 does the same |
--max-cost <usd> | hard cap for the whole run |
--max-tokens <n> | reply limit per call (default 400) |
--min-pass-rate <0-1> | exit 1 below this pass rate (default 1) |
--json | the report as JSON |
The CLI reads the key from OPENROUTER_API_KEY, or from ai.openrouterApiKey in the project's gitignored ingot.config.json. It exits 0 when every test passes, 1 when a test fails, and 3 when the cost cap skipped a test.
In CI, run rules test --mock so pull requests never call a paid model. The workflow from ingot export --init-ci (Export rules) already does.