Search Results for "test cases"
Sort By:
Showing 29 open source projects for "test cases"
View related business solutions-
Ship Agents FasterGemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.
-
Error to trace to log to deploy. One click. No SSH.AppSignal links every error to the trace, the trace to the log, the log to the deploy that shipped it.
-
1
Opik
Debug, evaluate, and monitor your LLMapps, RAG systems, and agentic AI
Confidently evaluate, test, and monitor LLM applications. Opik is an open-source platform for evaluating, testing, and monitoring LLM applications. Built by Comet. Record, sort, search, and understand each step your LLM app takes to generate a response. Manually annotate, view, and compare LLM responses in a user-friendly table. Log traces during development and in production. Run experiments with different prompts and evaluate against a test set. Choose and run pre-configured evaluation... -
2
promptfoo
Evaluate and compare LLM outputs, catch regressions, improve prompts
Ensure high-quality LLM outputs with automatic evals. Use a representative sample of user inputs to reduce subjectivity when tuning prompts. Use built-in metrics, LLM-graded evals, or define your own custom metrics. Compare prompts and model outputs side-by-side, or integrate the library into your existing test/CI workflow. Use OpenAI, Anthropic, and open-source models like Llama and Vicuna, or integrate custom API providers for any LLM API. -
3
gpt-prompt-engineer
Experimental prompt optimization toolkit built around notebooks
gpt-prompt-engineer is an experimental prompt optimization toolkit built around notebooks and LLM-assisted evaluation. It lets users describe a task, provide test cases, and generate many candidate prompts automatically. The system then tests those prompts against the examples and ranks their performance through an ELO-style scoring process. The repository includes versions for general prompt generation, classification tasks, Claude-based workflows, and model-to-model prompt conversion. It is meant to make prompt engineering more systematic than manual trial and error. ... -
4
AICGSecEval
A.S.E (AICGSecEval) is a repository-level AI-generated code security
...The project was developed to address concerns that AI-assisted programming tools may produce insecure code containing vulnerabilities such as injection flaws or unsafe logic. The framework constructs evaluation tasks based on real-world software repositories and known vulnerability cases derived from CVE records. By simulating realistic development scenarios, the benchmark assesses how well AI code generation systems handle security-sensitive programming tasks. AICGSecEval combines static and dynamic evaluation techniques to analyze generated code for vulnerabilities and functional correctness. The framework includes datasets, test cases, and evaluation metrics that measure how AI programming tools perform across multiple programming languages and vulnerability categories. -
Save Up to 91% on Cloud Compute With Spot VMsRun batch jobs at 60-91% off with Spot VMs. Long-running workloads get automatic discounts with sustained use.
-
5
Micro Agent
AI CLI agent that writes code by iterating until tests pass
Micro Agent is a command-line tool designed to generate and refine code using a test-driven approach powered by large language models. Instead of producing one-shot code outputs, it creates or uses test cases and repeatedly iterates on the generated code until those tests pass successfully. This workflow emphasizes reliability by using structured feedback from failing tests to guide improvements, reducing the need for manual debugging and iteration. -
6
ASSERT
Requirement-driven evaluation harness for AI agents and LLM
...It turns natural-language specifications, policies, product requirements, and launch criteria into structured tests that can be reviewed, executed, scored, and improved. The pipeline derives behavior categories, generates single-turn and multi-turn test cases, runs them against a target system, and uses an LLM judge to score conversations against the stated policies. It can evaluate hosted models, custom agents, multi-agent systems, REST clients, and frameworks such as LangGraph, CrewAI, AutoGen, DSPy, LlamaIndex, and OpenAI Agents SDK. ASSERT is designed to close the gap between what a system is supposed to do and what evaluation actually measures. ... -
7
PySpur
Visual tool for building, testing, and deploying AI agent workflows
PySpur is a visual development environment designed to help AI engineers build, test, and iterate on agent-based workflows more efficiently. It provides a structured playground where users can define test cases, construct agents either through Python code or a graphical interface, and continuously refine their behavior. It addresses common challenges in AI agent development such as prompt tuning difficulties and lack of visibility into workflow execution. -
8
Rogue
AI Agent Evaluator & Red Team Platform
Rogue is an open-source evaluation and red-team framework designed to test the reliability, safety, and policy compliance of AI agents. The platform automatically interacts with an AI agent by generating dynamic scenarios and multi-turn conversations that simulate real-world interactions. Instead of relying solely on static test scripts, Rogue uses an agent-as-a-judge architecture where one agent probes another agent to detect failures or unexpected behaviors. The system allows developers to... -
9
aisuite
Simple, unified interface to multiple Generative AI providers
...Using an interface similar to OpenAI's, aisuite makes it easy to interact with the most popular LLMs and compare the results. It is a thin wrapper around Python client libraries and allows creators to seamlessly swap out and test responses from different LLM providers without changing their code. Today, the library is primarily focused on chat completions. We will expand it to cover more use cases in the near future. Currently supported providers are - OpenAI, Anthropic, Azure, Google, AWS, Groq, Mistral, HuggingFace and Ollama. To maximize stability, aisuite uses either the HTTP endpoint or the SDK for making calls to the provider. -
Our Free Plans just got better! | Auth0You asked, we delivered! Auth0 is excited to expand our Free and Paid plans to include more options so you can focus on building, deploying, and scaling applications without having to worry about your security. Auth0 now, thank yourself later.
-
10
LabClaw
Operating Layer for LabOS (Stanford-Princeton AI Co-Scientists)
...LabClaw is designed to integrate with various large language models and external tools, enabling flexible configurations that adapt to different use cases such as automation, research, and product prototyping. It also includes logging and observability features that help developers track agent decisions and debug behavior during execution. -
11
Sandbox Agent
Run Coding Agents in Sandboxes
...Developers can use Sandbox Agent to simulate real-world workflows, debug agent decisions, and evaluate outcomes in a contained setting before deploying to production. It also supports extensibility, allowing integration with custom tools, APIs, and workflows tailored to specific use cases. -
12
Aden Hive
Outcome driven agent development framework that evolves
Hive is an open-source agent development framework that helps developers build autonomous, reliable, self-improving AI agents by letting them describe goals in ordinary natural language instead of hand-coding detailed workflows. Rather than manually defining execution graphs, Hive’s coding agent generates the agent graph, connection code, and test cases based on your high-level objectives, enabling outcome-driven agent creation that fits real business processes. Once deployed, agents can capture failure data, evolve automatically to meet their success criteria, and redeploy without constant manual intervention, delivering continual improvement over time. The framework also includes human-in-the-loop nodes, credential management, cost and budget controls, and real-time observability so teams can monitor execution and intervene as needed. ... -
13
FlowLens MCP
Open-source MCP server that gives your coding agent
FlowLens MCP Server is an open-source tool designed to give AI-powered coding agents (like Claude Code, Cursor, GitHub Copilot / Codex, and others) full, replayable browser context to dramatically improve debugging, bug reporting, and regression testing for web applications. It works together with a companion browser extension: when a user reproduces a bug or a complicated UI interaction, the extension captures a rich session log, including screen/video recording, network traffic, console... -
14
Microsoft Agent Skills
Skills, MCP servers, Custom Agents, Agents.md for SDKs
Microsoft Agent Skills is an actively maintained repository of skills, custom agents, templates, and MCP configuration files designed to extend AI coding assistants with deep knowledge about Azure SDKs and Microsoft AI Foundry services. The project bundles over a hundred domain-specific skills that teach AI agents how to perform tasks like Azure resource provisioning, SDK usage patterns, infrastructure setup, and common DevOps workflows, bridging the gap between agent reasoning and... -
15
Cake
Distributed LLM and StableDiffusion inference
Cake is a compact, powerful toolkit that combines a flexible TCP/UDP proxy, port forwarding system, and connection manager designed for both development and penetration testing scenarios. It enables users to create complex networking flows where traffic can be proxied, relayed, and manipulated between endpoints — useful for debugging networked applications, inspecting protocols, or tunneling traffic through different hops. The tool is designed to work with multiple protocols and supports... -
16
Telegram.Bot
.NET Client for Telegram Bot API
...The guides here can even be useful to bot developers using other languages/platforms as it shows best practices in developing Telegram chatbots with examples. This project is fully tested using Unit tests and Systems Integration tests before each release. In fact, our test cases are self-documenting and serve as examples for Bot API methods. Once you learn the basics of Telegram chatbots, you will be able to easily understand the code in examples and use it in your own bot program. -
17
Arthur Bench
Bench is a tool for evaluating LLMs for production use cases
Bench is a tool for evaluating LLMs for production use cases. Whether you are comparing different LLMs, considering different prompts, or testing generation hyperparameters like temperature and # tokens, Bench provides one touch point for all your LLM performance evaluation. -
18
OpenAI Quickstart Python
Python example app from the OpenAI API quickstart tutorial
...It provides practical, beginner-friendly examples to help developers quickly learn how to send requests, handle responses, and build basic applications using the OpenAI Python SDK. The examples folder includes small, self-contained projects showcasing common use cases like chat completions, tool usage, and interactive interfaces. Each example is designed to be easily runnable with minimal setup—requiring only Python, a virtual environment, and an API key. The repository also includes environment setup guides and example scripts, such as a simple Flask web app for chat interactions, allowing developers to test OpenAI API integrations locally. ... -
19
ZeroStep
Supercharge your Playwright tests with AI
ZeroStep is a tool that enhances Playwright tests with AI capabilities, aiming to improve the efficiency and effectiveness of end-to-end testing. -
20
Aviary
Ray Aviary - evaluate multiple LLMs easily
Aviary is an LLM serving solution that makes it easy to deploy and manage a variety of open source LLMs. Providing an extensive suite of pre-configured open source LLMs, with defaults that work out of the box. Supporting Transformer models hosted on Hugging Face Hub or present on local disk. Aviary has native support for autoscaling and multi-node deployments thanks to Ray and Ray Serve. Aviary can scale to zero and create new model replicas (each composed of multiple GPU workers) in... -
21
HttpRunner
Testing framework that began with API and performance testing
HttpRunner is an open-source testing framework that began with API and performance testing and has evolved into a general, extensible test platform. The current major version is implemented in Go, with the legacy Python edition split to a separate repository; this shift emphasizes a single, fast, cross-platform runtime for modern pipelines. It provides declarative test cases, data-driven parametrization, and plugin mechanisms so teams can compose reusable steps and validations at scale. ... -
22
TensorFlow.js models
Pretrained models for TensorFlow.js
This repository hosts a set of pre-trained models that have been ported to TensorFlow.js. The models are hosted on NPM and unpkg so they can be used in any project out of the box. They can be used directly or used in a transfer learning setting with TensorFlow.js. To find out about APIs for models, look at the README in each of the respective directories. In general, we try to hide tensors so the API can be used by non-machine learning experts. New models should have a test NPM script. You... -
23
Coursera Machine Learning
Coursera Machine Learning By Prof. Andrew Ng
CourseraMachineLearning is a personal collection of resources, notes, and programming exercises from Andrew Ng’s popular Machine Learning course on Coursera. It consolidates lecture references, programming tutorials, test cases, and supporting materials into one repository for easier review and practice. The project highlights fundamental machine learning concepts such as hypothesis functions, cost functions, gradient descent, bias-variance tradeoffs, and regression models. It also organizes week-by-week course schedules with links to exercises, lecture notes, and additional resources. ... -
24
Bottery
A conversational agent prototyping platform
...It encourages authors to model intents, dialog turns, and transitions in a compact, declarative form that’s easy to visualize and simulate. A built-in simulator lets you “play” the conversation, test edge cases, and tweak flows without wiring up back-end services first. Because behavior is defined in a DSL, teams can version the conversation design alongside code and keep product, research, and engineering aligned. The repo includes examples of slots, branching, and side effects, showing how to express common patterns like confirmations, fallbacks, and small talk. ... -
25
MOEA Framework
A Free and Open Source Java Framework for Multiobjective Optimization
The MOEA Framework is a free and open source Java library for developing and experimenting with multiobjective evolutionary algorithms (MOEAs) and other general-purpose multiobjective optimization algorithms. The MOEA Framework supports genetic algorithms, differential evolution, particle swarm optimization, genetic programming, grammatical evolution, and more. A number of algorithms are provided out-of-the-box, including NSGA-II, NSGA-III, ε-MOEA, GDE3 and MOEA/D. In addition, the MOEA...
