garak Files

What's Changed

New features

• Feature: technique and intent annotation and initial IntentProbe iteration by garak-maintainers in https://github.com/NVIDIA/garak/pull/1984

"Power is in tearing attacks to pieces and putting them together again in shapes of your own choosing."

This initial feature represents the first step in enabling users to provided their own context and requirements for target expectations and enable identification of attack vectors that show the edges of the underlying systems safeguards and runtime posture from new perspectives. This feature includes the initial groundwork for bringing user context into account during evaluation of a target. In its first iteration, it provides a new facet of information while keeping the existing probes primarily unchanged. The trait and intent concepts are explored, with policy introduced only as a reference definition. Further iteration and community feedback will guide how these new concepts are consumed, and influence how the broader Context Aware Scanning feature evolves.

• arch: support & data for trait/intent, and for policy datatype by @leondz in https://github.com/NVIDIA/garak/pull/1421
• cas feature: intent service by @leondz in https://github.com/NVIDIA/garak/pull/1481
• cas: technique & intent minimal pilot by @leondz in https://github.com/NVIDIA/garak/pull/1598
• cas: add technique tags to existing probes by @leondz in https://github.com/NVIDIA/garak/pull/1691
• avoid stored probename side-effects in evaluation by @jmartin-tech in https://github.com/NVIDIA/garak/pull/1788
• cas: intents for integrated probes by @leondz in https://github.com/NVIDIA/garak/pull/1693
• Fix IntentProbe crash when intent has no detector mapping by @patriciapampanelli in https://github.com/NVIDIA/garak/pull/1798
• cas: Add eval-level intents and digest technique/intent matrix by @patriciapampanelli in https://github.com/NVIDIA/garak/pull/1807
• Unified run.spec selection grammar by @patriciapampanelli in https://github.com/NVIDIA/garak/pull/1831
• feat(cas): expand intent->detector mapping coverage by @patriciapampanelli in https://github.com/NVIDIA/garak/pull/1861
• probes.base: balance IntentProbe pruning across intents by @patriciapampanelli in https://github.com/NVIDIA/garak/pull/1838
• Feature/intent run spec selector by @patriciapampanelli in https://github.com/NVIDIA/garak/pull/1866
• report: enrich technique_intent_matrix with names and descriptions by @patriciapampanelli in https://github.com/NVIDIA/garak/pull/1890
• fix: drop name fallback for stub-less intents by @patriciapampanelli in https://github.com/NVIDIA/garak/pull/1876
• cas: donotanswer intents by @leondz in https://github.com/NVIDIA/garak/pull/1716
• docs: cas user dev guide for technique and intent by @patriciapampanelli in https://github.com/NVIDIA/garak/pull/1874
• Refactor/cas config to run by @patriciapampanelli in https://github.com/NVIDIA/garak/pull/1886
• add probe summary entries and augment digest based on them by @jmartin-tech in https://github.com/NVIDIA/garak/pull/1932
• Feat: report technique and intent tabs lists by @otavionvidia in https://github.com/NVIDIA/garak/pull/1879
• double escape for 'dot' literal match in regex by @jmartin-tech in https://github.com/NVIDIA/garak/pull/1989

New plugins

• generator: add native Anthropic generator by @NishchayMahor in https://github.com/NVIDIA/garak/pull/1809
• probe: add simple adaptive attacks probe by @neerazz in https://github.com/NVIDIA/garak/pull/1742

Breaking changes

This release introduces breaking changes:

• introduces a new unified spec syntax to enable activation and suppression of probes, buffs and newly added intents
• cli plugin selection options (backwards compatible with existing options now marked as deprecated)
• configuration file formats:
• probe_spec and buff_spec move into the run definition as part of a unified spec
• report.jsonl output structure changes:
• new entry types "probe_summary"
• expanded "digest" format
• removed "digest" values for older *_spec configuration no longer consumed

Improved plugins

• fix: catch OpenAI AuthenticationError before multiprocessing pickle by @DevamShah in https://github.com/NVIDIA/garak/pull/1859
• Enhance error handling for Mistral API responses by @jka236 in https://github.com/NVIDIA/garak/pull/1769
• feat: add suppressed_params support to BedrockGenerator by @u7k4rs6 in https://github.com/NVIDIA/garak/pull/1842
• fix: repair dead blank-prompt guard in NVCF 400 handling by @anxkhn in https://github.com/NVIDIA/garak/pull/1901
• Fix: continuation probe not pruning triggers with prompts by @Osamaali313 in https://github.com/NVIDIA/garak/pull/1976
• Guard nvcf 500 error parsing against non-JSON bodies by @chuenchen309 in https://github.com/NVIDIA/garak/pull/1945
• escape regex metachars in StringDetector word matching by @bereketDeneke in https://github.com/NVIDIA/garak/pull/1880
• Fix: improve extra_params support in NeMoGuardrailsServer by @chuenchen309 in https://github.com/NVIDIA/garak/pull/1947
• feat(detectors): add Unicode normalisation to StringDetector by @PhinehasNarh in https://github.com/NVIDIA/garak/pull/1884
• Handle non-list LangServe /invoke output in langchain_serve generator by @WatchTree-19 in https://github.com/NVIDIA/garak/pull/1916
• web_injection: harden markdown exfil detectors against domain/extension/param evasion by @WatchTree-19 in https://github.com/NVIDIA/garak/pull/1928
• fix(rest): clear error when response_json_field matches a non-text response value by @xr843 in https://github.com/NVIDIA/garak/pull/1892

Documentation

See "Feature: technique and intent" references

Tuning & fixes

• test: reduce test-suite runtime by @patriciapampanelli in https://github.com/NVIDIA/garak/pull/1854
• test(detectors): mock data to manage avoid HF Hub 5xx instead of failing by @JakeBx in https://github.com/NVIDIA/garak/pull/1833
• test: add unit tests for glitch, snowball, and ansiescape probe modules by @nuthalapativarun in https://github.com/NVIDIA/garak/pull/1755
• fix(probes): download wordnet lexicon when it is missing from the database by @adityasingh2400 in https://github.com/NVIDIA/garak/pull/1820
• fix calibration pathlib paths by @ron1po in https://github.com/NVIDIA/garak/pull/1847
• Resolve symlinked install paths in data_path escape protections by @ABeltramo in https://github.com/NVIDIA/garak/pull/1860
• test(detectors): add unit tests for snowball detector module by @nuthalapativarun in https://github.com/NVIDIA/garak/pull/1824
• test(probes): un-shadow leakreplay probe structure test by @anxkhn in https://github.com/NVIDIA/garak/pull/1914
• tests: cover a completed parallel_attempts run by @EphraiemSarabamoun in https://github.com/NVIDIA/garak/pull/1816
• fix(analyze): guard analyze_log against zero total_evaluated by @anxkhn in https://github.com/NVIDIA/garak/pull/1941
• fix: reverse_translation_outputs reassembled in wrong order via list.pop() by @chuenchen309 in https://github.com/NVIDIA/garak/pull/1959
• test(config): assert aiohttp User-Agent test actually verifies the request by @anxkhn in https://github.com/NVIDIA/garak/pull/1933
• fix(probes): correct inverted target_lang_name_en guard message by @anxkhn in https://github.com/NVIDIA/garak/pull/1934
• fix(cli): report file path in plugin option-file errors by @anxkhn in https://github.com/NVIDIA/garak/pull/1951
• Fix: image support in OpenAICompatible._conversation_to_list by @chuenchen309 in https://github.com/NVIDIA/garak/pull/1944
• UX: Show probe name in detector progress bar by @feiiiiii5 in https://github.com/NVIDIA/garak/pull/1942
• fix: close prior httpx client before reload to stop FD leak in long fitd runs by @mvanhorn in https://github.com/NVIDIA/garak/pull/1787
• task: update npm and python deps by @jmartin-tech in https://github.com/NVIDIA/garak/pull/2007
• lock nltk due to incompatible path restrictions by @jmartin-tech in https://github.com/NVIDIA/garak/pull/2017
• fix: default config_root to _config in InjectAtbash by @anugram in https://github.com/NVIDIA/garak/pull/2006
• fix(detectors): honour config_root in goodside.RileyIsnt by @manunicholasjacob in https://github.com/NVIDIA/garak/pull/2014

New Contributors

• @NishchayMahor made their first contribution in https://github.com/NVIDIA/garak/pull/1809
• @nuthalapativarun made their first contribution in https://github.com/NVIDIA/garak/pull/1755
• @ron1po made their first contribution in https://github.com/NVIDIA/garak/pull/1847
• @DevamShah made their first contribution in https://github.com/NVIDIA/garak/pull/1859
• @ABeltramo made their first contribution in https://github.com/NVIDIA/garak/pull/1860
• @jka236 made their first contribution in https://github.com/NVIDIA/garak/pull/1769
• @neerazz made their first contribution in https://github.com/NVIDIA/garak/pull/1742
• @anxkhn made their first contribution in https://github.com/NVIDIA/garak/pull/1901
• @u7k4rs6 made their first contribution in https://github.com/NVIDIA/garak/pull/1842
• @EphraiemSarabamoun made their first contribution in https://github.com/NVIDIA/garak/pull/1816
• @chuenchen309 made their first contribution in https://github.com/NVIDIA/garak/pull/1959
• @Osamaali313 made their first contribution in https://github.com/NVIDIA/garak/pull/1976
• @bereketDeneke made their first contribution in https://github.com/NVIDIA/garak/pull/1880
• @PhinehasNarh made their first contribution in https://github.com/NVIDIA/garak/pull/1884
• @WatchTree-19 made their first contribution in https://github.com/NVIDIA/garak/pull/1916
• @feiiiiii5 made their first contribution in https://github.com/NVIDIA/garak/pull/1942
• @mvanhorn made their first contribution in https://github.com/NVIDIA/garak/pull/1787
• @xr843 made their first contribution in https://github.com/NVIDIA/garak/pull/1892
• @anugram made their first contribution in https://github.com/NVIDIA/garak/pull/2006
• @manunicholasjacob made their first contribution in https://github.com/NVIDIA/garak/pull/2014

Full Changelog: https://github.com/NVIDIA/garak/compare/v0.15.1...v0.16.0