AgentBench
A Comprehensive Benchmark to Evaluate LLMs as Agents (ICLR'24)
...The benchmark includes multiple environments that simulate realistic scenarios such as web interaction, database querying, and problem solving tasks. These environments require agents to interpret instructions, take actions, and adapt their strategies based on feedback from the environment. AgentBench also includes an evaluation framework that measures success rates, rewards, and task completion performance across different agent implementations. By testing models across diverse scenarios, the benchmark highlights strengths and weaknesses in reasoning, long-term planning, and tool usage.