Search Results for "safety"
Sort By:
Showing 42 open source projects for "safety"
View related business solutions-
$300 Free Credits to Build on Google CloudPut your $300 in credit toward real workloads, then keep building with free monthly usage for 20+ products. No commitment and no charge until you upgrade.
-
Ship Agents FasterGemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.
-
1
Purple Llama
Set of tools to assess and improve LLM security
Purple Llama is an umbrella safety initiative that aggregates tools, benchmarks, and mitigations to help developers build responsibly with open generative AI. Its scope spans input and output safeguards, cybersecurity-focused evaluations, and reference shields that can be inserted at inference time. The project evolves as a hub for safety research artifacts like Llama Guard and Code Shield, along with dataset specs and how-to guides for integrating checks into applications. ... -
2
FuzzyAI Fuzzer
A powerful tool for automated LLM fuzzing
...FuzzyAI provides testing tools, datasets, and evaluation workflows that help researchers measure how well models resist harmful instructions or attempts to bypass safety mechanisms. -
3
In-The-Wild Jailbreak Prompts on LLMs
A dataset consists of 15,140 ChatGPT prompts from Reddit
In-The-Wild Jailbreak Prompts on LLMs is an open-source research repository that provides datasets and analytical tools for studying jailbreak prompts used to bypass safety restrictions in large language models. The project is part of a research effort to understand how users attempt to circumvent alignment and safety mechanisms built into modern AI systems. The repository includes a large collection of prompts gathered from real-world platforms such as Reddit, Discord, prompt-sharing communities, and other public sources. ... -
4
Heretic
Fully automatic censorship removal for language models
Heretic is an open-source Python tool that automatically removes the built-in censorship or “safety alignment” from transformer-based language models so they respond to a broader range of prompts with fewer refusals. It works by applying directional ablation techniques and a parameter optimization strategy to adjust internal model behaviors without expensive post-training or altering the core capabilities. Designed for researchers and advanced users, Heretic makes it possible to study and experiment with uncensored model responses in a reproducible, automated way. ... -
Custom VMs From 1 to 96 vCPUs With 99.95% UptimeLive migration and automatic failover keep workloads online through maintenance. One free e2-micro VM every month.
-
5
Codex-5.5-codex-instruct-5.5
Prompt-injection utility aimed at modifying how Codex CLI behaves
...Its documentation also points users toward a newer visual project with desktop UI, provider switching, session management, and packaged installers. Because the stated purpose is to bypass safety restrictions, it should be treated as unsafe prompt-bypass research material rather than a normal developer productivity tool. -
6
OpenHands
Open-source autonomous AI software engineer
...So we're building all our agents in the open on GitHub, under the MIT license. Our agents can do anything a human developer can: they write code, run commands, and use the web. We're partnering with AI safety experts like Invariant Labs to balance innovation with security. -
7
Claude Code Tools
Practical productivity tools for Claude Code, Codex-CLI
...Some components enable Claude Code to interact with terminal multiplexers such as tmux so that it can run programs, debug applications, and interact with scripts that require user input. The toolkit also provides safety mechanisms that prevent potentially dangerous shell commands from being executed automatically by AI agents. -
8
Inspect Petri
An alignment auditing agent capable of exploring alignment hypothesis
Inspect Petri is an open-source alignment auditing agent that lets researchers rapidly test concrete safety hypotheses against target models using realistic, multi-turn scenarios. Instead of building bespoke evals, Inspect Petri automatically generates audit environments from seed “special instructions,” orchestrates an auditor model to probe a target model, and simulates tool use and rollbacks to surface risky behaviors. Each interaction transcript is then scored by a judge model using a consistent rubric so results are comparable across runs and models. ... -
9
Supabase MCP Server
Query MCP enables end-to-end management of Supabase via chat interface
An open-source MCP server that enables comprehensive management of Supabase projects through natural language interactions, providing capabilities such as SQL execution, schema management, and API integration. -
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
10
ZAPI
ZAPI by Adopt AI is an open-source Python library
ZAPI is a developer-centric API framework that streamlines building, testing, and deploying APIs with strong type safety and minimal boilerplate, helping teams deliver backend services faster with fewer errors. It emphasizes a declarative router and schema model that uses types to define request and response formats, providing clear contracts for frontend and backend teams while automatically generating documentation. Zapi abstracts many repetitive tasks such as validation, authentication flows, and error handling so developers can focus on business logic instead of infrastructure plumbing. ... -
11
AI Agents Masterclass
Follow along with my AI Agents Masterclass videos
...The project includes structured lessons, code examples, and practical exercises that cover foundational concepts like prompt engineering, chaining agents, tool usage, plan execution, evaluation, and safety considerations. It breaks down how autonomous agents interact with external systems, handle iterative reasoning, and integrate with third-party services or APIs to perform real tasks — for example, web search, browsing, scheduling, or coding assistance. Students of the masterclass can follow written modules or Jupyter notebooks that illustrate concepts step by step and progressively build more capable agents. ... -
12
Pearl
A Production-ready Reinforcement Learning AI Agent Library
Pearl is a production-ready reinforcement learning and contextual bandit agent library built for real-world sequential decision making. It is organized around modular components—policy learners, replay buffers, exploration strategies, safety modules, and history summarizers—that snap together to form reliable agents with clear boundaries and strong defaults. The library implements classic and modern algorithms across two regimes: contextual bandits (e.g., LinUCB, LinTS, SquareCB, neural bandits) and fully sequential RL (e.g., DQN, PPO-style policy optimization), with attention to practical concerns like nonstationarity and dynamic action spaces. ... -
13
Future AGI
Open-source platform for evaluating, observing, and improving LLM
...It supports both cloud and self-hosted deployment models, making it useful for teams with different privacy, infrastructure, and compliance needs. Future AGI is especially relevant for agent-heavy products where reliability, regression testing, and safety checks matter before and after release. Its main value is turning AI agent development into a measurable engineering process instead of an informal cycle of prompting, guessing, and manual review. -
14
VulnClaw
Based on AI Agent + MCP toolchain + penetration Skill orchestration
...Its newer architecture uses a goal-driven solving engine instead of a fixed-round loop, helping the agent stop when the goal is reached, the search space is exhausted, or a safety budget is met. It also includes evidence checks designed to reduce hallucinated conclusions by requiring real tool output before accepting key findings. VulnClaw is intended for authorized testing, CTFs, security education, and controlled red-team environments. -
15
Claw Code
AI agent harness for AI coding agents
...It emphasizes harness engineering—how agents are structured, how they interact with tools, and how they maintain context during execution. The system is being actively expanded, with a Rust-based runtime in development to improve performance and memory safety. Overall, Claw Code serves as a research-driven platform for advancing agent-based software development systems. -
16
openpilot
Open source driver assistance system
openpilot is an open-source driver assistance system designed to improve upon the existing driver assistance of most modern cars today. openpilot gives you Tesla Autopilot-like functionality with functions like Adaptive Cruise Control (ACC), Automated Lane Centering (ALC), Forward Collision Warning (FCW) and Lane Departure Warning (LDW). All these with just a push of a button. openpilot also includes a camera-based driver monitoring feature that alerts distracted and asleep drivers while it... -
17
highway-env
A minimalist environment for decision-making in autonomous driving
HighwayEnv is an OpenAI Gym-compatible environment focused on autonomous driving scenarios. It provides flexible simulations for testing decision-making algorithms in highway, intersection, and merging traffic situations. -
18
Semantic Router
Superfast AI decision making and processing of multi-modal data
Semantic Router is a superfast decision-making layer for your LLMs and agents. Rather than waiting for slow, unreliable LLM generations to make tool-use or safety decisions, we use the magic of semantic vector space — routing our requests using semantic meaning. Combining LLMs with deterministic rules means we can be confident that our AI systems behave as intended. Cramming agent tools into the limited context window is expensive, slow, and fundamentally limited. Semantic Router enables lightning-fast and cheap tool usage that can scale to many thousands of tools. ... -
19
Guardrails
Framework for validating and controlling LLM outputs in AI apps
...Guardrails also supports generating structured data from language models, allowing developers to enforce schemas or type constraints on responses. A companion ecosystem known as a hub provides reusable validators that can be combined into input and output guards to address different reliability and safety concerns. -
20
LLaMA Models
Utilities intended for use with Llama models
This repository serves as the central hub for the Llama foundation model family, consolidating model cards, licenses and use policies, and utilities that support inference and fine-tuning across releases. It ties together other stack components (like safety tooling and developer SDKs) and provides canonical references for model variants and their intended usage. The project’s issues and releases reflect an actively used coordination point for the ecosystem, where guidance, utilities, and compatibility notes are published. It complements separate repos that carry code and demos (for example inference kernels or cookbook content) by keeping authoritative metadata and specs here. ... -
21
Serena
Agent toolkit providing semantic retrieval and editing capabilities
Serena is a coding-focused agent toolkit that turns an LLM into a practical software-engineering agent with semantic retrieval and editing over real repositories. It operates as an MCP server (and other integrations), exposing IDE-like tools so agents can locate symbols, reason about code structure, make targeted edits, and validate changes. The toolkit is LLM-agnostic and framework-agnostic, positioning itself as a drop-in capability for different chat UIs, orchestrators, or custom agent... -
22
VibeVoice
Open-source multi-speaker long-form text-to-speech model
...The model integrates a Qwen2.5-based large language model with a diffusion head to produce realistic acoustic details and capture conversational context. Training involved curriculum learning with increasing sequence lengths up to 65K tokens, allowing VibeVoice to handle very long dialogues effectively. Safety mechanisms include an audible disclaimer and imperceptible watermarking in all generated audio to mitigate misuse risks. -
23
OpenAI Agents SDK
A lightweight, powerful framework for multi-agent workflows
The OpenAI Agents Python SDK is a powerful yet lightweight framework for developing multi-agent workflows. This framework enables developers to create and manage agents that can coordinate tasks autonomously, using a set of instructions, tools, guardrails, and handoffs. The SDK allows users to configure workflows in which agents can pass control to other agents as necessary, ensuring dynamic task management. It also includes a built-in tracing system for tracking, debugging, and optimizing... -
24
Pedalboard
A Python library for audio
pedalboard is a Python library for working with audio: reading, writing, rendering, adding effects, and more. It supports the most popular audio file formats and a number of common audio effects out of the box and also allows the use of VST3® and Audio Unit formats for loading third-party software instruments and effects. pedalboard was built by Spotify’s Audio Intelligence Lab to enable using studio-quality audio effects from within Python and TensorFlow. Internally at Spotify, pedalboard... -
25
sunnypilot
Open source driver assistance system
...Smart cruise functions can use vehicle, vision, and offline OpenStreetMap data for speed-related assistance. Display customization, quiet mode, connected services, and remote device features extend the everyday driving experience. The project aims to preserve openpilot safety constraints while offering substantially more configuration.
