Search Results for "safety"
Sort By:
Showing 18 open source projects for "safety"
View related business solutions-
Compliant and Reliable File Transfers Backed by Top Security CertificationsStop relying on non-certified, legacy file transfer tools that creak under the weight of modern security demands. Get full audit trails, advanced access controls and more supported by an award-winning team of experts. Start your free 25-day trial today.
-
Stop Cyber Threats with VM-Series Next-Gen Firewall on AzureGain integrated visibility across all traffic in a single pass. Deploy Palo Alto Networks VM-Series to determine application identity and content while automating security policy updates via rich APIs.
-
1
Purple Llama
Set of tools to assess and improve LLM security
Purple Llama is an umbrella safety initiative that aggregates tools, benchmarks, and mitigations to help developers build responsibly with open generative AI. Its scope spans input and output safeguards, cybersecurity-focused evaluations, and reference shields that can be inserted at inference time. The project evolves as a hub for safety research artifacts like Llama Guard and Code Shield, along with dataset specs and how-to guides for integrating checks into applications. ... -
2
PKU Beaver
Constrained Value Alignment via Safe Reinforcement Learning
PKU Beaver is an open-source research project focused on improving the safety alignment of large language models through reinforcement learning from human feedback under explicit safety constraints. The framework introduces techniques that separate helpfulness and harmlessness signals during training, allowing models to optimize for useful responses while minimizing harmful behavior. To support this process, the project provides datasets containing human-labeled examples that encode both performance preferences and safety constraints across multiple dimensions. ... -
3
FuzzyAI Fuzzer
A powerful tool for automated LLM fuzzing
...FuzzyAI provides testing tools, datasets, and evaluation workflows that help researchers measure how well models resist harmful instructions or attempts to bypass safety mechanisms. -
4
In-The-Wild Jailbreak Prompts on LLMs
A dataset consists of 15,140 ChatGPT prompts from Reddit
In-The-Wild Jailbreak Prompts on LLMs is an open-source research repository that provides datasets and analytical tools for studying jailbreak prompts used to bypass safety restrictions in large language models. The project is part of a research effort to understand how users attempt to circumvent alignment and safety mechanisms built into modern AI systems. The repository includes a large collection of prompts gathered from real-world platforms such as Reddit, Discord, prompt-sharing communities, and other public sources. ... -
Error to trace to log to deploy. One click. No SSH.AppSignal links every error to the trace, the trace to the log, the log to the deploy that shipped it.
-
5
Safety-Prompts
Chinese safety prompts for evaluating and improving the safety of LLMs
Safety-Prompts is an open-source repository that provides a curated collection of prompts designed to evaluate and improve the safety behavior of large language models. The project focuses primarily on safety testing scenarios relevant to Chinese language models, though the concepts can be applied to other languages and systems. The prompts are structured to test whether models generate outputs that align with human values and safety guidelines when faced with potentially harmful or sensitive requests. ... -
6
Claude Code Tools
Practical productivity tools for Claude Code, Codex-CLI
...Some components enable Claude Code to interact with terminal multiplexers such as tmux so that it can run programs, debug applications, and interact with scripts that require user input. The toolkit also provides safety mechanisms that prevent potentially dangerous shell commands from being executed automatically by AI agents. -
7
Superagent
Superagent protects your AI applications
Superagent is an open-source AI safety platform built to protect applications from prompt injections, data leaks, and harmful outputs. It embeds real-time safety directly into AI workflows, helping teams secure models before threats cause damage. Superagent provides guardrails that block jailbreaks, prompt manipulation, and sensitive data exfiltration. It includes redaction tools to remove PII, PHI, and secrets automatically from text. -
8
vLLM Semantic Router
System Level Intelligent Router for Mixture-of-Models at Cloud
...The router operates as an intelligent layer between users and model infrastructure, capturing signals from prompts, responses, and contextual data to improve decision-making. It can also integrate safety and monitoring mechanisms that detect issues such as jailbreak attempts, hallucinations, or sensitive information exposure. -
9
Heretic
Fully automatic censorship removal for language models
Heretic is an open-source Python tool that automatically removes the built-in censorship or “safety alignment” from transformer-based language models so they respond to a broader range of prompts with fewer refusals. It works by applying directional ablation techniques and a parameter optimization strategy to adjust internal model behaviors without expensive post-training or altering the core capabilities. Designed for researchers and advanced users, Heretic makes it possible to study and experiment with uncensored model responses in a reproducible, automated way. ... -
Ship Agents FasterGemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.
-
10
Rogue
AI Agent Evaluator & Red Team Platform
Rogue is an open-source evaluation and red-team framework designed to test the reliability, safety, and policy compliance of AI agents. The platform automatically interacts with an AI agent by generating dynamic scenarios and multi-turn conversations that simulate real-world interactions. Instead of relying solely on static test scripts, Rogue uses an agent-as-a-judge architecture where one agent probes another agent to detect failures or unexpected behaviors. -
11
LLaMA Models
Utilities intended for use with Llama models
This repository serves as the central hub for the Llama foundation model family, consolidating model cards, licenses and use policies, and utilities that support inference and fine-tuning across releases. It ties together other stack components (like safety tooling and developer SDKs) and provides canonical references for model variants and their intended usage. The project’s issues and releases reflect an actively used coordination point for the ecosystem, where guidance, utilities, and compatibility notes are published. It complements separate repos that carry code and demos (for example inference kernels or cookbook content) by keeping authoritative metadata and specs here. ... -
12
AI Agents Papers
A collection of AI Agents papers
AI-Agent-Papers is a curated open-source repository that collects research papers related to artificial intelligence agents and agentic systems. The project organizes a large body of academic work covering topics such as planning, reasoning, tool use, self-correction, memory systems, and safety mechanisms for AI agents. The repository categorizes papers into structured themes including agent capabilities, agent architectures, and practical applications across different domains. It also includes categories for survey papers, benchmarks, and tutorials, helping researchers understand both foundational theory and emerging developments in the field. ... -
13
pgvecto.rs
Vector database plugin for Postgres, written in Rust
pgvecto.rs is a Postgres extension that provides vector similarity search functions. It is written in Rust and based on pgrx. It is currently under heavy development, please take care when using it in production. pgvecto.rs is a Postgres extension, which means that you can use it directly within your existing database. This makes it easy to integrate into your existing workflows and applications. pgvecto.rs supports filtering. You can set conditions when searching or retrieving points. This... -
14
LangChain Rust
LangChain for Rust, the easiest way to write LLM-based programs
...The library aims to provide Rust developers with a structured framework for orchestrating prompts, chains, agents, and external tools within LLM-driven workflows. By adapting LangChain concepts to the Rust programming language, the project emphasizes performance, safety, and efficient memory management. Developers can use the framework to build chatbots, autonomous agents, and knowledge-augmented AI systems that interact with external data sources. The library provides abstractions for model providers, prompt templates, conversation memory, and vector search integrations. It also enables the construction of multi-step pipelines where LLM outputs feed into subsequent actions or tool calls. -
15
Hallucination Leaderboard
Leaderboard Comparing LLM Performance at Producing Hallucinations
...The results are published as a leaderboard that allows researchers and developers to compare model reliability and factual consistency. By focusing on hallucination rates rather than traditional metrics such as accuracy or fluency, the benchmark highlights an important aspect of AI system safety and trustworthiness. The leaderboard is regularly updated as new models are released and evaluation methods evolve. -
16
The Alignment Handbook
Robust recipes to align language models with human and AI preferences
The Alignment Handbook is an open-source resource created to provide practical guidance for aligning large language models with human preferences and safety requirements. The project focuses on the post-training stage of model development, where models are refined after pre-training to behave more helpfully, safely, and reliably in real-world applications. It provides detailed training recipes that explain how to perform tasks such as supervised fine-tuning, preference modeling, and reinforcement learning from human feedback. ... -
17
Learn Prompting
This website is a free, open-source guide on prompt engineering
This website is a free, open-source guide on prompt engineering. Contributions are welcome! Harsh criticism is welcome too. We launched the first ever prompt hacking competition designed to enhance AI safety and education by challenging participants to outsmart large language models from May 5th to June 3rd! The competition featured 10 increasingly difficult levels of prompt hacking defenses and the chance to win over $35,000 in prizes. Coding is a great skill to learn alongside prompt engineering. We recommend learning Python, as it is a popular language for AI and machine learning. ... -
18
Following Instructions with Feedback
Training Language Models to Follow Instructions with Human Feedback
The following-instructions-human-feedback repository contains the code and supplementary materials underpinning OpenAI’s work in training language models (InstructGPT models) that better follow user instructions through human feedback. The repo hosts the model card, sample automatic evaluation outputs, and labeling guidelines used in the process. It is explicitly tied to the “Training language models to follow instructions with human feedback” paper, and serves as a reference for how OpenAI...
- Previous
- You're on page 1
- Next
