Download
AgentBench is an open-source benchmark designed to evaluate the capabilities of large language models when used as autonomous agents. Unlike traditional language model benchmarks that focus on static text tasks, AgentBench measures how models perform in interactive environments that require planning, reasoning, and decision-making. The benchmark includes multiple environments that simulate realistic scenarios such as web interaction, database querying, and problem solving tasks. These environments require agents to interpret instructions, take actions, and adapt their strategies based on feedback from the environment. AgentBench also includes an evaluation framework that measures success rates, rewards, and task completion performance across different agent implementations. By testing models across diverse scenarios, the benchmark highlights strengths and weaknesses in reasoning, long-term planning, and tool usage.
Features
- Benchmark framework for evaluating large language models acting as autonomous agents
- Multiple interactive environments simulating real-world tasks and decision processes
- Evaluation metrics including success rate, rewards, and task completion accuracy
- Support for testing models with multi-step reasoning and tool use
- Architecture consisting of task servers, agent servers, and evaluation clients
- Research platform for comparing agent performance across diverse scenarios
Project Samples
