Search Results for "test scenario"

Rogue is an open-source evaluation and red-team framework designed to test the reliability, safety, and policy compliance of AI agents. The platform automatically interacts with an AI agent by generating dynamic scenarios and multi-turn conversations that simulate real-world interactions. Instead of relying solely on static test scripts, Rogue uses an agent-as-a-judge architecture where one agent probes another agent to detect failures or unexpected behaviors.

Molecule project is designed to aid in the development and testing of Ansible roles. Molecule provides support for testing with multiple instances, operating systems and distributions, virtualization providers, test frameworks and testing scenarios. Molecule encourages an approach that results in consistently developed roles that are well-written, easily understood and maintained. Molecule supports only the latest two major versions of Ansible (N/N-1), meaning that if the latest version is 2.9.x, we will also test our code with 2.8.x. Depending on the driver chosen, you may need to install additional OS packages. ...

Meta Agents Research Environments (ARE) is a simulation and benchmarking platform. It is designed to evaluate AI agents in dynamic, evolving, multi-step tasks. Unlike static benchmarks, ARE supports environments where agents must adapt to changes over time and reason over sequences of actions. It interacts with applications and faces uncertainty. The included Gaia2 benchmark offers 800 scenarios across multiple “universes”. It can test reasoning, memory, tool use, and adaptability....

Ansible for DevOps is a collection of Ansible playbooks, roles, and infrastructure-as-code examples that accompany the book Ansible for DevOps by Jeff Geerling. Rather than being theoretical, the examples span real-world infrastructure setups: multi-server orchestration, LAMP stacks, Docker deployments, Kubernetes cluster spins, rolling updates, and security hardening. You can clone the repo and play with actual scenarios using Vagrant, VirtualBox, or cloud hosts, making it ideal for both...

Inspect Petri is an open-source alignment auditing agent that lets researchers rapidly test concrete safety hypotheses against target models using realistic, multi-turn scenarios. Instead of building bespoke evals, Inspect Petri automatically generates audit environments from seed “special instructions,” orchestrates an auditor model to probe a target model, and simulates tool use and rollbacks to surface risky behaviors. Each interaction transcript is then scored by a judge model using a...

A Distributed Experimenting and Testing harness in Python. Automatically deploy, execute, and report your distributed test cases in easy steps. Write arbitrarily scenario scripts using 8 keywords. Fine grained distributed sync with built-in API.