evalship Integrations

Langfuse is an open source LLM engineering platform to help teams collaboratively debug, analyze and iterate on their LLM Applications. Observability: Instrument your app and start ingesting traces to Langfuse Langfuse UI: Inspect and debug complex logs and user sessions Prompts: Manage, version and deploy prompts from within Langfuse Analytics: Track metrics (LLM cost, latency, quality) and gain insights from dashboards & data exports Evals: Collect and calculate scores for your LLM completions Experiments: Track and test app behavior before deploying a new version Why Langfuse? - Open source - Model and framework agnostic - Built for production - Incrementally adoptable - start with a single LLM call or integration, then expand to full tracing of complex chains/agents - Use GET API to build downstream use cases and export data

Promptfoo discovers and eliminates major LLM risks before they are shipped to production. Its founders have experience launching and scaling AI to over 100 million users using automated red-teaming and testing to overcome security, legal, and compliance issues. Promptfoo's open source, developer-first approach has made it the most widely adopted tool in this space, with over 20,000 users. Custom probes for your application that identify failures you actually care about, not just generic jailbreaks and prompt injections. Move quickly with a command-line interface, live reloads, and caching. No SDKs, cloud dependencies, or logins. Used by teams serving millions of users and supported by an active open source community. Build reliable prompts, models, and RAGs with benchmarks specific to your use case. Secure your apps with automated red teaming and pentesting. Speed up evaluations with caching, concurrency, and live reloading.

Vitest is a next-generation testing framework powered by Vite, designed to make testing fast, lightweight, and closely integrated with modern JavaScript development. Because it builds on Vite, Vitest understands existing Vite configuration and can reuse the same resolve and transform pipelines used by an application during development and builds. It can also be used without Vite and works for backend code as well. Vitest provides a Jest-compatible API, including expect assertions, snapshots, mocking, and coverage, making migration from Jest straightforward for many projects. Its smart watch mode reruns only tests related to changed files, using an experience similar to Vite’s instant Hot Module Replacement. Vitest supports ESM, TypeScript, and JSX out of the box, reducing the need for separate transformation configuration.

pytest helps you write better programs. The pytest framework makes it easy to write small tests, yet scales to support complex functional testing for applications and libraries. Due to pytest’s detailed assertion introspection, only plain assert statements are used. Detailed info on failing assert statements. Auto-discovery of test modules and functions. Modular fixtures for managing small or parametrized long-lived test resources. Can run unittest (including trial) and nose test suites out of the box. Supports Python 3.6+ and PyPy 3. Rich plugin architecture, with over 315+ external plugins and thriving community. The maintainers of pytest and thousands of other packages are working with Tidelift to deliver commercial support and maintenance for the open source dependencies you use to build your applications. Save time, reduce risk, and improve code health, while paying the maintainers of the exact dependencies you use.

Braintrust is an AI observability and evaluation platform designed to help teams build, monitor, and improve AI systems in production. It enables users to capture and inspect real-time traces of AI interactions, including prompts, responses, and tool usage. The platform allows teams to measure performance using automated and human evaluations to ensure output quality. Braintrust helps identify issues such as hallucinations, regressions, and performance drops before they impact users. It supports prompt and model comparisons, making it easier to optimize AI workflows over time. With scalable trace ingestion and real-time monitoring, teams gain full visibility into how their AI systems behave. The platform integrates with multiple programming languages and tools, allowing developers to work within their existing tech stack. Overall, Braintrust provides a comprehensive solution for maintaining and improving AI quality at scale.

Jest aims to work out of the box, config free, on most JavaScript projects. Make tests which keep track of large objects with ease. Snapshots live either alongside your tests, or embedded inline. Tests are parallelized by running them in their own processes to maximize performance. Tests are parallelized by running them in their own processes to maximize performance. By ensuring your tests have unique global state, Jest can reliably run tests in parallel. To make things quick, Jest runs previously failed tests first and re-organizes runs based on how long test files take. By ensuring your tests have unique global state, Jest can reliably run tests in parallel. To make things quick, Jest runs previously failed tests first and re-organizes runs based on how long test files take. Jest uses a custom resolver for imports in your tests, making it simple to mock any object outside of your test’s scope.

Unexpected results happen all the time. With full visibility into the entire chain sequence of calls, you can spot the source of errors and surprises in real time with surgical precision. Software engineering relies on unit testing to build performant, production-ready applications. LangSmith provides that same functionality for LLM applications. Spin up test datasets, run your applications over them, and inspect results without having to leave LangSmith. LangSmith enables mission-critical observability with only a few lines of code. LangSmith is designed to help developers harness the power–and wrangle the complexity–of LLMs. We’re not only building tools. We’re establishing best practices you can rely on. Build and deploy LLM applications with confidence. Application-level usage stats. Feedback collection. Filter traces, cost and performance measurement. Dataset curation, compare chain performance, AI-assisted evaluation, and embrace best practices.

GitHub Actions is a powerful automation tool that enables developers to streamline their software workflows directly within GitHub. It allows teams to build, test, and deploy code automatically using CI/CD pipelines triggered by events such as code pushes or pull requests. With support for multiple programming languages and environments, developers can run workflows across Linux, macOS, and Windows. GitHub Actions also provides hosted and self-hosted runners for flexible execution. It simplifies repetitive tasks like code reviews, issue management, and deployment processes. With real-time logs and built-in secret management, it ensures transparency and security. Overall, GitHub Actions helps teams automate development processes and deliver software faster.