promptfoo Files

0.125.0 (2026-10-23)

⚠ BREAKING CHANGES

• deps: make jks-js an optional peer dependency (#11739)
• providers: make Claude Agent and Codex SDKs opt-in (#11134)

Features

• add provider-only environment overrides (#11889) (2eed6e1)
• assertions: add rouge-l and rouge-s assertion types (#10256) (6842fc3)
• assertions: filter trace spans by attributes (#10648) (faf08b4)
• assertions: optional Unicode normalization for equals and contains (#10555) (92ae1ec)
• eval: add safe mode to disable inline JavaScript execution (#11382) (80a27b6)
• eval: disclose cache-hit rows in the eval summary (#11329) (3635a03)
• eval: report stability across repeated tests (#11416) (3f7bfa7)
• examples: add local Hermeneutic wording assertion (#10663) (636d59e)
• providers: add Daybreak access program config (#11737) (8f873ce)
• providers: add portable OpenAI config schemas (#11746) (3d6cdf2)
• providers: add templated request metadata for a2a provider (#10242) (f86c893)
• providers: support current Gemini media models (#11581) (f511091)
• redteam: add arabic-presentation-forms static encoding strategy (#11404) (ce7adab)

Bug Fixes

• accounting: ignore malformed optional token usage (#10112) (4f3b359)
• app: simplify navigation and refresh the local UI (#11740) (bbe73b5)
• assertions: honor the not- prefix for tool-call validity assertions (#11488) (f5b1d39)
• assertions: isolate cached answer relevance samples (#11628) (a98f031)
• assertions: match actual required XML elements (#11669) (6c3ffc5)
• assertions: preserve custom reason in not-javascript GradingResult inversion (#10880) (62cbfee)
• assertions: reject non-finite similarity calculations (#11627) (895bbe4)
• bound telemetry request lifetime (#11536) (04ab6fb)
• cache: honor disabled caching across package copies (#10793) (1b256fe)
• ci: refresh code scan action before OIDC expires (#11717) (7fd37b7)
• ci: secure code-scan action releases (#10108) (b5d6b20)
• cli: expose MCP errors and remove simulated tests (#11676) (0044799)
• code-scan: explain empty scan results instead of bare "No files to scan" (#11543) (0864873)
• config: remove the default configuration cache (#11166) (90c4a7a)
• config: scope runtime environment defaults (#11091) (fb50526)
• correct provider billing, configuration, and JUnit output (#11060) (c09adb3)
• csv: parse similar:cosine/dot/euclidean assertion types from CSV (#11365) (eb07d64)
• dataset: gracefully extract personas from varied JSON responses during dataset synthesis (#11293) (#11298) (2310a94)
• deps: align Storybook and update Mermaid/KaTeX (#11914) (03844fb)
• deps: hold IBM SDK before redaction regression (cbebb0e)
• deps: make jks-js an optional peer dependency (#11739) (1c28848)
• deps: preserve Claude permissions on SDK upgrade (#11696) (80f0a11)
• deps: raise webpack-dev-server Express minimum to 4.22.3 (#11865) (77c78d7)
• deps: synchronize Anthropic SDK upgrades (#11483) (be8b169)
• deps: synchronize Claude Agent SDK upgrade (#11553) (b4caec3)
• deps: update anthropic packages (#11860) (480893f)
• deps: update anthropic packages (#11871) (fa8886a)
• deps: update dependency @hono/node-server to v2.1.2 (#11535) (7afa715)
• deps: update dependency @hono/node-server to v2.1.3 (#11551) (6177a8b)
• deps: update dependency @hono/node-server to v2.1.4 (#11862) (0ed9ffd)
• deps: update dependency @hono/node-server to v2.1.5 (#11951) (5657b66)
• deps: update dependency @modelcontextprotocol/sdk to ^1.32.1 (#11778) (09ac4d4)
• deps: update dependency js-yaml to v5.4.3 (#11789) (eac66f6)
• deps: update DOMPurify to v3.4.17 (#11941) (ea1df7f)
• deps: update ip-address to 10.7.3 (#11691) (06ecb0d)
• deps: update ip-address to 10.7.4 (#11911) (6eb4f17)
• deps: update optional Hono to 4.13.12 (#11690) (ef0a459)
• deps: update optional OpenCode SDK to 1.18.34 (#11687) (69a4f4a)
• deps: update protobufjs to 8.8.1 (#11908) (424f09c)
• deps: update Zod to 4.6.6 (#11912) (15bcb0d)
• deps: update Zod to v4.6.7 (#11942) (da1d402)
• docker: detect pip-installed ModelAudit in the image (#11677) (16c7151)
• docker: refresh Alpine packages in CI builds (#11694) (a9ec196)
• eval: allow empty dynamic variable values (#10551) (244b7a2)
• eval: count cached graders in repeat stability (#11584) (2a461ff)
• eval: honor cache setting from a config passed with -c (#11327) (579043a)
• eval: insert nested var values literally when they contain $ patterns (#11366) (af6fa65)
• eval: keep shared providers alive across evaluations (#11020) (7d11ea2)
• eval: preserve row metrics and harden sanitization (#10889) (76dd354)
• eval: propagate grading cancellation through embedding providers (#11017) (f29d43e)
• examples: require patched Pydantic AI minimum (#11484) (2a54f11)
• generation: handle unexpected response shapes in dataset and assertion synthesis (#11302) (4b6a417)
• isolate evaluation state and schema loading (#11626) (2e347de)
• matchers: fail closed when llm-rubric grader response omits its verdict (#10611) (37cfe71)
• preserve ordered JSON keys and verify meta history (#11476) (0bef225)
• preserve per-provider prompt filters through loading and replay (#11443) (ddf052a)
• preserve prompt metadata and provider controls (#11260) (14178fb)
• prompts: accept JSON and JSONL prompt files with a UTF-8 BOM (#11473) (8014282)
• providers: align Azure Responses service tier (#10149) (fcfd526)
• providers: align MCP initialization and cleanup (#10676) (94356d0)
• providers: clean up custom Codex version probes (#11588) (0b6a9d2)
• providers: constrain MCP retries and repair OAuth discovery (#10967) (1da4637)
• providers: decode base64 OpenAI embeddings (#11383) (e4e3388)
• providers: harden malformed completion responses (#9872) (60da518)
• providers: honor scoped environment overrides (#11094) (763b437)
• providers: honor scoped Envoy gateway URLs (#10859) (e4347d3)
• providers: preflight Codex SDK compatibility (#9937) (c8f2ff2)
• providers: preserve active WebSocket streams and retry stalls (#10305) (f7ddd45)
• providers: preserve cancellation, quota accounting, and evaluation deadlines (#10860) (00d6ce4)
• providers: preserve prompts in raw HTTP request URLs (#11438) (1f304f9)
• providers: preserve remote provider errors (#11867) (af8f09a)
• providers: reject invalid OpenAI Agents retry policies (#11679) (7885ca6)
• providers: retain MCP paths across evaluations (#11569) (e4a1b17)
• providers: retain supported scoped environment settings (#10744) (75c4ac6)
• providers: return a clean error on empty choices in localai and OpenAI completions (#10437) (81cd1c1)
• providers: share cache bypass and replay metadata (#10687) (e7eb4a7)
• providers: update Google embeddings and catalog (#11583) (7c11c1e)
• python: fail requests when workers crash or hang (#10929) (ad5eaec)
• redteam: correct generated provider targets (#10857) (20e221a)
• redteam: do not invent an SSRF severity when the grader failed (#11300) (33be2e3)
• redteam: honor explicit redteamProvider in custom strategy (#10978) (2603c54)
• redteam: keep $ sequences literal when datamarking an injection (#11381) (667ebe3)
• redteam: normalize generation error callbacks (#11591) (ff83c4f)
• redteam: preserve attack prompts in jailbreak templates (#11437) (39226cc)
• redteam: preserve grader errors in iterative attacks (#11301) (1d1477e)
• redteam: preserve setup plugin options and workload counts (#11218) (6ae0024)
• redteam: use FIPS-compatible config fingerprints (#11070) (ebb4948)
• redteam: validate web page tracking identifiers (#11522) (46102b7)
• remove stale MCP evaluation summary cache (#11190) (58ca5a6)
• scheduler: require 429 to be a standalone token (#11369) (40ff7a5)
• util: ignore braces inside strings when extracting JSON objects (#10268) (cf63b47)
• webui: redirect stale dashboard URLs to eval (#11477) (a6922a1)

Performance Improvements

• providers: make Claude Agent and Codex SDKs opt-in (#11134) (d46ee50)