Search Results for "safety"
Sort By:
Showing 36 open source projects for "safety"
View related business solutions-
$300 Free Credits for Your Google Cloud ProjectsLaunch your next project with $300 in free Google Cloud credits—no strings attached. Test, build, and deploy without risk. Use your credits across the entire Google Cloud platform to find what works best for your needs. After your credits are used, continue with always-free tier services. Only pay when you're ready to scale. Sign up in minutes and start exploring.
-
Ship Agents FasterGemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.
-
1
PKU Beaver
Constrained Value Alignment via Safe Reinforcement Learning
PKU Beaver is an open-source research project focused on improving the safety alignment of large language models through reinforcement learning from human feedback under explicit safety constraints. The framework introduces techniques that separate helpfulness and harmlessness signals during training, allowing models to optimize for useful responses while minimizing harmful behavior. To support this process, the project provides datasets containing human-labeled examples that encode both performance preferences and safety constraints across multiple dimensions. ... -
2
Purple Llama
Set of tools to assess and improve LLM security
Purple Llama is an umbrella safety initiative that aggregates tools, benchmarks, and mitigations to help developers build responsibly with open generative AI. Its scope spans input and output safeguards, cybersecurity-focused evaluations, and reference shields that can be inserted at inference time. The project evolves as a hub for safety research artifacts like Llama Guard and Code Shield, along with dataset specs and how-to guides for integrating checks into applications. ... -
3
FuzzyAI Fuzzer
A powerful tool for automated LLM fuzzing
...FuzzyAI provides testing tools, datasets, and evaluation workflows that help researchers measure how well models resist harmful instructions or attempts to bypass safety mechanisms. -
4
In-The-Wild Jailbreak Prompts on LLMs
A dataset consists of 15,140 ChatGPT prompts from Reddit
In-The-Wild Jailbreak Prompts on LLMs is an open-source research repository that provides datasets and analytical tools for studying jailbreak prompts used to bypass safety restrictions in large language models. The project is part of a research effort to understand how users attempt to circumvent alignment and safety mechanisms built into modern AI systems. The repository includes a large collection of prompts gathered from real-world platforms such as Reddit, Discord, prompt-sharing communities, and other public sources. ... -
Stop Storing Third-Party Tokens in Your DatabaseRolling your own OAuth token storage can be a security liability. Token Vault securely stores access and refresh tokens from federated providers and handles exchange and renewal automatically. Connected accounts, refresh exchange, and privileged worker flows included.
-
5
Claude Code Tools
Practical productivity tools for Claude Code, Codex-CLI
...Some components enable Claude Code to interact with terminal multiplexers such as tmux so that it can run programs, debug applications, and interact with scripts that require user input. The toolkit also provides safety mechanisms that prevent potentially dangerous shell commands from being executed automatically by AI agents. -
6
ZAPI
ZAPI by Adopt AI is an open-source Python library
ZAPI is a developer-centric API framework that streamlines building, testing, and deploying APIs with strong type safety and minimal boilerplate, helping teams deliver backend services faster with fewer errors. It emphasizes a declarative router and schema model that uses types to define request and response formats, providing clear contracts for frontend and backend teams while automatically generating documentation. Zapi abstracts many repetitive tasks such as validation, authentication flows, and error handling so developers can focus on business logic instead of infrastructure plumbing. ... -
7
AI Agents Masterclass
Follow along with my AI Agents Masterclass videos
...The project includes structured lessons, code examples, and practical exercises that cover foundational concepts like prompt engineering, chaining agents, tool usage, plan execution, evaluation, and safety considerations. It breaks down how autonomous agents interact with external systems, handle iterative reasoning, and integrate with third-party services or APIs to perform real tasks — for example, web search, browsing, scheduling, or coding assistance. Students of the masterclass can follow written modules or Jupyter notebooks that illustrate concepts step by step and progressively build more capable agents. ... -
8
OpenHands
Open-source autonomous AI software engineer
...So we're building all our agents in the open on GitHub, under the MIT license. Our agents can do anything a human developer can: they write code, run commands, and use the web. We're partnering with AI safety experts like Invariant Labs to balance innovation with security. -
9
Pearl
A Production-ready Reinforcement Learning AI Agent Library
Pearl is a production-ready reinforcement learning and contextual bandit agent library built for real-world sequential decision making. It is organized around modular components—policy learners, replay buffers, exploration strategies, safety modules, and history summarizers—that snap together to form reliable agents with clear boundaries and strong defaults. The library implements classic and modern algorithms across two regimes: contextual bandits (e.g., LinUCB, LinTS, SquareCB, neural bandits) and fully sequential RL (e.g., DQN, PPO-style policy optimization), with attention to practical concerns like nonstationarity and dynamic action spaces. ... -
Secure File Transfer for Windows with Cerberus by RedwoodCerberus supports unlimited users and connections on a single IP, with built-in encryption, 2FA, and a browser-based web client — all deployable in under 15 minutes with a 25-day free trial.
-
10
Claw Code
AI agent harness for AI coding agents
...It emphasizes harness engineering—how agents are structured, how they interact with tools, and how they maintain context during execution. The system is being actively expanded, with a Rust-based runtime in development to improve performance and memory safety. Overall, Claw Code serves as a research-driven platform for advancing agent-based software development systems. -
11
Heretic
Fully automatic censorship removal for language models
Heretic is an open-source Python tool that automatically removes the built-in censorship or “safety alignment” from transformer-based language models so they respond to a broader range of prompts with fewer refusals. It works by applying directional ablation techniques and a parameter optimization strategy to adjust internal model behaviors without expensive post-training or altering the core capabilities. Designed for researchers and advanced users, Heretic makes it possible to study and experiment with uncensored model responses in a reproducible, automated way. ... -
12
Inspect Petri
An alignment auditing agent capable of exploring alignment hypothesis
Inspect Petri is an open-source alignment auditing agent that lets researchers rapidly test concrete safety hypotheses against target models using realistic, multi-turn scenarios. Instead of building bespoke evals, Inspect Petri automatically generates audit environments from seed “special instructions,” orchestrates an auditor model to probe a target model, and simulates tool use and rollbacks to surface risky behaviors. Each interaction transcript is then scored by a judge model using a consistent rubric so results are comparable across runs and models. ... -
13
Supabase MCP Server
Query MCP enables end-to-end management of Supabase via chat interface
An open-source MCP server that enables comprehensive management of Supabase projects through natural language interactions, providing capabilities such as SQL execution, schema management, and API integration. -
14
highway-env
A minimalist environment for decision-making in autonomous driving
HighwayEnv is an OpenAI Gym-compatible environment focused on autonomous driving scenarios. It provides flexible simulations for testing decision-making algorithms in highway, intersection, and merging traffic situations. -
15
openpilot
Open source driver assistance system
openpilot is an open-source driver assistance system designed to improve upon the existing driver assistance of most modern cars today. openpilot gives you Tesla Autopilot-like functionality with functions like Adaptive Cruise Control (ACC), Automated Lane Centering (ALC), Forward Collision Warning (FCW) and Lane Departure Warning (LDW). All these with just a push of a button. openpilot also includes a camera-based driver monitoring feature that alerts distracted and asleep drivers while it... -
16
A.I.G
Full-stack AI Red Teaming platform
AI-Infra-Guard is a powerful open-source security platform from Tencent’s Zhuque Lab designed to assess the safety and resilience of AI infrastructures, codebases, and components through automated scanning and evaluation tools. It brings together AI infrastructure vulnerability scanning, MCP server risk analysis, and jailbreak evaluation into a unified workflow so that enterprises and individuals can identify critical security issues without relying on external services. -
17
VibeVoice
Open-source multi-speaker long-form text-to-speech model
...The model integrates a Qwen2.5-based large language model with a diffusion head to produce realistic acoustic details and capture conversational context. Training involved curriculum learning with increasing sequence lengths up to 65K tokens, allowing VibeVoice to handle very long dialogues effectively. Safety mechanisms include an audible disclaimer and imperceptible watermarking in all generated audio to mitigate misuse risks. -
18
DeepEval is a simple-to-use, open-source LLM evaluation framework, for evaluating and testing large-language model systems. It is similar to Pytest but specialized for unit testing LLM outputs. DeepEval incorporates the latest research to evaluate LLM outputs based on metrics such as G-Eval, hallucination, answer relevancy, RAGAS, etc., which uses LLMs and various other NLP models that run locally on your machine for evaluation. Whether your application is implemented via RAG or fine-tuning,...
-
19
OpenAI Agents SDK
A lightweight, powerful framework for multi-agent workflows
The OpenAI Agents Python SDK is a powerful yet lightweight framework for developing multi-agent workflows. This framework enables developers to create and manage agents that can coordinate tasks autonomously, using a set of instructions, tools, guardrails, and handoffs. The SDK allows users to configure workflows in which agents can pass control to other agents as necessary, ensuring dynamic task management. It also includes a built-in tracing system for tracking, debugging, and optimizing... -
20
Pedalboard
A Python library for audio
pedalboard is a Python library for working with audio: reading, writing, rendering, adding effects, and more. It supports the most popular audio file formats and a number of common audio effects out of the box and also allows the use of VST3® and Audio Unit formats for loading third-party software instruments and effects. pedalboard was built by Spotify’s Audio Intelligence Lab to enable using studio-quality audio effects from within Python and TensorFlow. Internally at Spotify, pedalboard... -
21
Future AGI
Open-source platform for evaluating, observing, and improving LLM
...It supports both cloud and self-hosted deployment models, making it useful for teams with different privacy, infrastructure, and compliance needs. Future AGI is especially relevant for agent-heavy products where reliability, regression testing, and safety checks matter before and after release. Its main value is turning AI agent development into a measurable engineering process instead of an informal cycle of prompting, guessing, and manual review. -
22
Semantic Router
Superfast AI decision making and processing of multi-modal data
Semantic Router is a superfast decision-making layer for your LLMs and agents. Rather than waiting for slow, unreliable LLM generations to make tool-use or safety decisions, we use the magic of semantic vector space — routing our requests using semantic meaning. Combining LLMs with deterministic rules means we can be confident that our AI systems behave as intended. Cramming agent tools into the limited context window is expensive, slow, and fundamentally limited. Semantic Router enables lightning-fast and cheap tool usage that can scale to many thousands of tools. ... -
23
Guardrails
Framework for validating and controlling LLM outputs in AI apps
...Guardrails also supports generating structured data from language models, allowing developers to enforce schemas or type constraints on responses. A companion ecosystem known as a hub provides reusable validators that can be combined into input and output guards to address different reliability and safety concerns. -
24
TypeChat
Library for building type-safe natural language interfaces with LLMs
...Traditional natural language interfaces often relied on complex decision trees to interpret user intent and gather required inputs. With the rise of large language models, developers can interpret user requests more easily, but they still face challenges related to output reliability, safety, and structured responses. TypeChat addresses these challenges by replacing traditional prompt engineering with a concept called schema engineering. Instead of writing complex prompts, developers define types that represent the intents supported by their applications. It then uses those type definitions to construct prompts for language models and translate user input into structured data that follows the defined schema. -
25
Rogue
AI Agent Evaluator & Red Team Platform
Rogue is an open-source evaluation and red-team framework designed to test the reliability, safety, and policy compliance of AI agents. The platform automatically interacts with an AI agent by generating dynamic scenarios and multi-turn conversations that simulate real-world interactions. Instead of relying solely on static test scripts, Rogue uses an agent-as-a-judge architecture where one agent probes another agent to detect failures or unexpected behaviors.
