iFixAi Files

Audit Any Agent Over Chat

15 inspections no longer need an adapter: B01, B02, B04, B23, C02, C05, C11, M02, M03, M07, P01, P08, V08, X04 and X11 each send a probe pack over chat, and the agent's replies decide the verdict (#363).

A chat endpoint is enough, and a bare model API counts. B01 and P01 grade a chat-only agent like any other. Hooks such as authorize_tool, get_confidence and route_to_human still work. They show as notes beside the probe results and never change the verdict. A check that can't apply to your agent is set aside before the run and never scored, and the report says why in not_applicable. That covers:

• abilities the fixture says the agent lacks;
• P01 when no tool deletes or overwrites;
• B32 for a general-purpose agent;
• M03 and V08 when the agent's stated purpose leaves out that work;
• checks that read only a hook the agent doesn't have. A mandatory minimum that is set aside never caps the score.

Judging Within Your Budget

--judge-budget now counts every call to the judge's provider, including retries, fallback models and each judge in Full mode. --dry-run and the run header show the budget and the ceiling (#362). 0 no longer means unlimited. It is a runaway guard of 10x the estimate, at least 2,000 calls. A run may go 25% over the budget. When it needs a call past that, it stops, keeps what finished and lists the rest as not run (budget). A run stops at once, without retrying the refused call, when:

• the judge's key, credit or quota is gone (401, 402, 403, a spent budget or quota);
• the judge model doesn't exist (404);
• the judge rejects five calls in a row before its first verdict. A plain 429 rate limit is still retried. No probe goes to the agent unless a judge can grade it. Before the first probe, each judge gets one small check call, so a dead key or a wrong model is caught before any probe is sent (#363). You get one line saying what happened and how to fix it, then --resume <run id>. A mandatory minimum that never ran reads NOT CHECKED, never FAIL.

New

ifixai author: turns a short JSON profile of your agent into a fixture. --example writes an example profile to edit. Unknown fields are refused, and an existing file is replaced only with --force (#363). ifixai validate now checks evidence floors. It refuses a fixture that would leave an inspection without a verdict, with one line per miss saying what to add. It takes the same -b, -c, --suite, --strategic and --full-run flags as ifixai run (#363). The Claude Code / Codex plugin and the ifixai install skill follow the new flow (#363):

1. Find your agent and confirm it with you.
2. Ask at most two questions.
3. Write a profile citing the repo for every value, author the fixture and validate it.
4. Show the --dry-run cost, and run only after you say yes. V11 Guilt Tripping (exploratory, report only): 40 four-turn companion chats. It checks that the agent never helps a struggling user hurt themselves, never puts them down, and points them to a person by the turn the risk shows up (#356, [#357]). ifixai windtunnel: puts one governance defect into an otherwise correct provider and runs the suite clean, then with the defect. Per inspection, it reports whether the defect was caught and the false-positive rate. Subcommands: controls, run, sweep and sensitivity. It runs offline and does not change grades (#360). MiniMax: the default model is now MiniMax-M3.1-Flash (#355).

Important Fixes

Judging and scoring

An errored judge sample no longer makes a FAIL majority unscored in B15, P13, P22, P27, P32 and S02 (#344, [#364]). A score exactly on a grade boundary no longer loses the grade to a rounding error (#351). B01's coverage diagnostics no longer count toward its mandatory-gate evidence floor (#353). C02, C05, C11, X04 and X11 confidence intervals now use only the scored items (#341). Judge fallback provider keys are matched case-insensitively (#342), and an error is treated as a rate limit only when it actually says so (#255).

Agent replies

HTTP: a completed reply with empty content and no tool call drops that probe instead of being graded, including in B17 and the V-series (#216). A malformed provider reply drops one probe, not the whole inspection (#247). LiteLLM: an upstream abort mid-reply is reported instead of grading the partial text (#347).

Reports

A run without an overall score exports no grade (#343). A withheld score shows its real reason, and a cap is claimed only when it applied (#352). The Markdown appendix labels ungraded, diagnostic and empty-reply evidence the same way everywhere (#350, [#332]). An aborted scoped run no longer lists mandatory minimums you didn't select (#340). B26 and B27 name each probe correctly (#330, [#328]). B32 treats the generic fixture domain as unrestricted (#329). P08 no longer crashes on timestamps without a timezone (#325).

Trajectory

A defective delegation grant is reported once, not once per use (#348). An invoke at the event limit is refused before the tool runs (#327), and JSONL captures with Unicode line separators now load (#326).

Providers

Gemini: a rejected API key is a fatal auth error (#338). Bedrock: the AWS region chain is honoured instead of pinning us-east-1, and invalid signatures and expired tokens count as auth failures (#337, [#336]).

Reproducibility and resume

Numeric temperatures are made consistent before the manifest hashes them (#323), and checkpoint writes no longer race each other (#320). A full run resumes correctly when ifixai.yaml saves a default suite (#346).

CLI and setup

Unreadable governance paths, non-string config keys and unreadable scorecards in ifixai compare give a clear error, not a traceback (#334, [#354], [#324]). --artifact-out creates missing folders (#333), and dotenv files with a UTF-8 BOM are accepted (#321). Setup no longer saves a governance path for the mock agent (#335), and old data-source classifications in fixtures are converted to the current ones (#339). A console that can't show a character prints ? instead of crashing at the scorecard (#363).

Behaviour Changes

The 15 chat-probed inspections above are decided by the agent's replies, not its hooks, so their scores can't be compared with 4.2.x runs. ifixai run refuses, before any call, a fixture that can't give a selected inspection a verdict. Run ifixai validate to see what is missing. --judge-budget 0 is a runaway guard, not unlimited. A run that hits its ceiling, or whose judge fails, stops early. It exits 1, or 2 when a mandatory minimum that finished failed. --api-key is optional. Without it, the key is read from the provider's env var, such as OPENAI_API_KEY. --governance is ignored. Put the rules in the fixture's governance: block. --profile was removed and now gives an error. Use --mode standard or --mode full. CI exit codes (#363):

• A mandatory minimum that ran and failed exits 2 whatever the score.
• A scoped run without B01, B08 or P01 exits 0 only if it selected at least one inspection that can gate and all of those passed.
• It exits 2 when a selected mandatory minimum was set aside unchecked.
• overall.passed in the JSON is true exactly when the exit code is 0. Python API: run_single raises ValueError, with the reason, for a check that is not applicable. The mock demo agent now fails 4 of the default 10 on purpose (B01, B12, B16 and P01), gets grade D and exits 2. Removed: the record/replay bridge, the unused embedding classifier and the ifixai.examples scripts (#361).