ASSERT Files

What's Changed

Sandboxed action mediation (target.sandbox)

Run a real configured agent against a real tool surface during an eval — with zero real side effects — while preserving the tool-call and network evidence the judge needs.

• New target.sandbox config entry ([docs/config/schema.md](https://github.com/responsibleai/ASSERT/blob/main/docs/config/schema.md#L191)). Points at either a stock Docker container that ASSERT starts and removes per test case, or an already-running endpoint.
• Two-layer YAML that separates safety from fidelity: policy.yaml decides whether each tool call is passed / mocked / blocked; mocks.yaml decides what a mocked call returns. The mock layer can never weaken enforcement — pinned by test.
• Per-use-case mocks match on tool arguments, not just tool name. Stateful mock backends so a later read reflects an earlier mocked write, isolated per test case.
• Deny-by-default audited egress, host-side model credentials only, automatic container cleanup per case.
• Judge sees the full action + network trace as first-class evidence.
• Lives under assert_ai/integrations/sandbox/; no new dependencies (stdlib + PyYAML). See [examples/sandbox_action_mediation](https://github.com/responsibleai/ASSERT/tree/main/examples/sandbox_action_mediation).

2. Permissible / impermissible metric split and the run-assert-eval skill

run-assert-eval skill. Natural-language front-doors for Copilot, Claude Code, and Cursor. Users type "evaluate my agent for budget-violation risk" and the skill drives the whole pipeline — Clarity risk discovery via MCP → per-risk atomic eval config → smoke-run of 3 real test cases before the full inference sweep → policy-violation and overrefusal rates with trace-cited failure examples. Ships as .claude/skills/run-assert-eval/, .github/prompts/run-assert-eval.prompt.md, and .cursor/rules/assert.mdc.

Metric split. Behaviors now carry a canonical permissible flag ([assert_ai/core/io.py](https://github.com/responsibleai/ASSERT/blob/main/assert_ai/core/io.py)), and the headline surface splits accordingly:

• not_permissible_policy_violation_rate — one vote per conversation over non-permissible nodes only. No more overrefusal contamination in the policy-violation number.
• permissible_overrefusal_rate — computed on the ok-filtered permissible rows, driven by a dedicated overrefusal judge dimension ([assert_ai/core/judge.py](https://github.com/responsibleai/ASSERT/blob/main/assert_ai/core/judge.py)).
• CLI, viewer, and export_suite_results.py all use the same node-split definition. Legacy per-row metrics are preserved for stale taxonomies with no permissible flag.

3. GitHub Actions CI gate

Wire ASSERT into pull requests and fail on evidence of new safety regressions. New docs entry point at [docs/ci/README.md](https://github.com/responsibleai/ASSERT/blob/main/docs/ci/README.md), backed by the external action at [responsibleai/assert-ai-action](https://github.com/responsibleai/assert-ai-action).

• One-command onboarding: npx skills add responsibleai/assert-ai-action --skill wire-assert-ci --yes.
• Generated workflows call responsibleai/assert-ai-action@v1 and reference the user's own model credentials as CI secrets — no shared endpoint.
• Backed by the science regression gate ([#263](https://github.com/responsibleai/ASSERT/pull/263)): 6 canonical efficacy metrics, per-seed paired tests, McNemar's exact one-sided test, Holm-Bonferroni correction. The action is the customer-facing surface; the gate is the statistical backbone.

New Contributors

• @ded-furby made their first contribution in https://github.com/responsibleai/ASSERT/pull/214
• @changliu2 with @Copilot made their first contribution in https://github.com/responsibleai/ASSERT/pull/192
• @ango10 made their first contribution in https://github.com/responsibleai/ASSERT/pull/254
• @ShilJain made their first contribution in https://github.com/responsibleai/ASSERT/pull/250

Full Changelog: https://github.com/responsibleai/ASSERT/compare/v0.1.0...v0.2.0