promptfoo Files

0.123.0 (2026-09-10)

⚠ BREAKING CHANGES

• providers: default GPT-5.6+ to Responses (#10803)

Features

• assertions: grade native audio with llm-rubric (#10814) (39c4fb5)
• providers: add Claude Fable and Mythos 5.1 (#10601) (ac6ba5b)
• providers: add GPT-6 Astra support (#10625) (cd4c68b)
• providers: add grok-4.6, grok-imagine-image-2.0, and grok-imagine-video-1.5 (#10587) (48a71cd)
• providers: default GPT-5.6+ to Responses (#10803) (b051efd)
• providers: expose MCP tool calls in response metadata (#10516) (891d870)
• providers: support Gemini 3.8 Flash (#10616) (b89a25d)
• providers: support GPT Image 2.5 generation (#10812) (71346b0)
• providers: support GPT Transcribe file uploads (#10813) (c85b395)
• providers: support Muse Spark 1.3 (#10617) (8e0de03)

Bug Fixes

• assertions: parse multiline tool calls and clarify guides (#10789) (5f0324f)
• assertions: recognize OpenAI Responses function_call items in tool-call-f1 (#10665) (028282c)
• claude-agent-sdk: recognize documented model selectors (#10741) (c7e32f6)
• claude-agent: preserve per-server env for stdio MCP servers (#10557) (6fd4c41)
• cli: remove retired GitHub example downloads (#10876) (bf09397)
• cloudflare: use the versioned Mistral gateway route (#10734) (ea6d6d3)
• config: avoid stack traces for missing optional configs (#10523) (3d67680)
• db: preserve writes after lock failures and during shutdown (#9009) (130389e)
• deps: preserve browser production dependencies (#10755) (0b4ccbe)
• deps: update anthropic packages (#10571) (93f963d)
• deps: update anthropic packages (#10679) (152c423)
• deps: update dependency @inquirer/core to v12 (#10554) (90fa399)
• deps: update dependency ai to ^6.0.264 (#10585) (da073db)
• deps: update dependency js-yaml to v5.4.0 (#10629) (58fe7cc)
• deps: update dependency zod to v4.5.4 (#10788) (03b0564)
• deps: update Hono to v4.13.5 (#10815) (6a7568d)
• deps: update js-yaml to 5.4.1 (#10680) (bd18712)
• deps: update OpenCode SDK to v1.18.21 (#10565) (93b9f8c)
• deps: update webpack to v5.110.3 (#10863) (fbfc5fb)
• eval: keep the CLI alive while watch mode is watching (#10562) (89d4138)
• eval: watch mode ignored scalar and generator tests references (#10169) (71499e3)
• examples: honor framework model selection (#10781) (34cfb68)
• examples: use selected CrewAI credentials (#10799) (e2cabfa)
• go-provider: support importable packages (#10540) (10deb31)
• google: route stable Omni through Interactions (#10737) (9ef3c96)
• guard empty drift stats and clarify provider options (#10645) (bfcd067)
• harden eval and provider edge cases (#10576) (1adb6e5)
• http: clear error context when signature fails due to missing private key (#10586) (7abf349)
• http: support Windows drive letter and file:// source paths in multipart (#10566) (22468f4)
• huggingface: normalize classification and embedding responses (#10735) (d31716d)
• mcp: use valid provider model selectors (#10806) (8bcf17d)
• n8n: omit request bodies for HEAD webhooks (#10730) (7f897cf)
• openai: preserve model routing and historical compatibility (#10810) (460e221)
• openclaw: sign the server challenge timestamp (#10739) (517e181)
• providers: address AI code quality findings (#10552) (9cd1924)
• providers: align billing rates and zero-token usage (#10743) (0a2852c)
• providers: align Hyperbolic audio requests and metadata (#10786) (e680c7f)
• providers: align Vercel generation with AI SDK 6 (#10728) (6514ebc)
• providers: allow zero-argument MCP commands (#10647) (1d32ab1)
• providers: correct Bedrock embeddings and knowledge base requests (#10751) (3f0787b)
• providers: correct Databricks serving endpoint requests (#10747) (b0b3874)
• providers: correct stale Bedrock pricing, regional rates, and Nova cache billing (#10593) (dad89cd)
• providers: defer FAL SDK loading until inference (#10792) (3afb563)
• providers: drop Claude model ids that resolve nowhere (#10635) (281f8d4)
• providers: finish the GitHub Models retirement (#10849) (b67f9e0)
• providers: forward Azure request settings (#10729) (685b354)
• providers: forward moderation cancellation (#10804) (5a4288e)
• providers: handle Vercel stream cancellation (#10805) (753c4e4)
• providers: isolate FAL SDK client configuration (#10749) (e3bbad6)
• providers: isolate SageMaker completion cache (#10731) (8e0a58f)
• providers: keep SageMaker request and cache settings consistent (#10808) (f81947d)
• providers: normalize typed Responses messages (#10600) (a966cc5)
• providers: omit unset Bedrock generation settings (#10774) (189c458)
• providers: preserve Bedrock Reel IDs and clarify serving routes (#10807) (2bc1354)
• providers: preserve local model parameters (#10732) (c8e606c)
• providers: preserve MiniMax and Moonshot request overrides (#10745) (01f4d9c)
• providers: preserve OpenRouter BYOK and cached costs (#10801) (790e768)
• providers: preserve Perplexity usage and search data (#10809) (c105c2c)
• providers: preserve Replicate creation sharing on cancellation (#10773) (884f2b0)
• providers: preserve selected credential sources (#10811) (1ca6fe2)
• providers: preserve service connection settings (#10742) (c8aa59d)
• providers: preserve WatsonX caller cancellation (#10861) (3113c6f)
• providers: prevent duplicate Realtime prompts and correct transcription metrics (#10868) (7e4e706)
• providers: price auto-quality xai image edits (#10613) (8e06f14)
• providers: raise equal Claude thinking token budgets (#10498) (ae3615a)
• providers: recognize Daybreak reasoning aliases (#10723) (b9ce5cc)
• providers: recover gateway model discovery (#10778) (ff69bd3)
• providers: remove retired OpenAI models (#10636) (3f4280b)
• providers: retain failed security scan costs (#10521) (04e36f3)
• providers: retire GitHub Models inference (#10746) (cf7adb7)
• providers: route Grok 4.6 inference profiles on Bedrock (#10589) (48bad74)
• providers: scope Hyperbolic legacy audio fields (#10796) (819a3d9)
• providers: scope WatsonX pricing metadata (#10802) (9cc4a36)
• providers: select TrueFoundry tasks explicitly (#10790) (83b12b9)
• providers: separate Azure chat and embedding defaults (#10784) (79454fa)
• providers: separate connection and model settings (#10681) (35f3434)
• providers: snapshot SageMaker request cache keys (#10798) (3b40040)
• providers: stop forwarding portkey provider config as HTTP headers (#10536) (3f94284)
• providers: support Abliteration Large V2 reasoning (#10615) (51b8f12)
• providers: treat Claude Sonnet 5 as thinking-on-by-default (#10631) (5bf2ad1)
• providers: unbreak Grok on Azure and multimodal Responses prompts (#10592) (d7ce7ec)
• providers: update OpenAI models and defaults (#10865) (607c820)
• providers: use OpenRouter reported costs (#10777) (ef82931)
• redteam: standardize target response attribution (#10862) (01f9e87)
• replicate: stop local polling when evaluations abort (#10740) (2cbfcd8)
• security: match env credential names case-insensitively, and stop redacting vendor names (#10638) (e5767f2)
• security: redact credential-named variables in persisted env maps (#10628) (811158e)
• slack: preserve scoped auth and returned DM conversations (#10738) (c0aa003)