Search Results for "inference"
Sort By:
Showing 63 open source projects for "inference"
View related business solutions-
Veeam Data Platform v13.1 - Get Your Free TrialTry Veeam Data Platform today. Experience the unified platform that's secure by design, portable by default, and proven to recover clean, fast, and anywhere.
-
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
1
wllama
WebAssembly binding for llama.cpp - Enabling on-browser LLM inference
wllama is a WebAssembly-based library that enables large language model inference directly inside a web browser. Built as a binding for the llama.cpp inference engine, the project allows developers to run LLM models locally without requiring a server backend or dedicated GPU hardware. The library leverages WebAssembly SIMD capabilities to achieve efficient execution within modern browsers while maintaining compatibility across platforms. -
2
NemoClaw
NVIDIA plugin for secure installation of OpenClaw
...It installs and configures the NVIDIA OpenShell runtime, which provides a secure environment for running autonomous AI agents. NemoClaw enables users to launch sandboxed agent environments that control network access, file permissions, and inference requests through policy-based security. The platform integrates with AI models such as NVIDIA Nemotron and supports multiple inference backends including cloud APIs, local NIM deployments, and vLLM. Through its command-line interface, developers can deploy, monitor, and manage AI assistants running inside isolated sandboxes. By combining sandbox orchestration, agent management, and AI model integration, NemoClaw provides a secure foundation for building and operating autonomous AI assistants. -
3
Operit AI
Powerful Android AI agent with tools, automation, and Linux shell
Operit is a full-featured AI assistant and agent platform designed specifically for Android devices, aiming to go far beyond traditional chat-based interfaces. It integrates deep system-level capabilities with a wide range of tools, allowing the AI to perform real tasks such as file management, automation, and system control directly on the device. A standout aspect of the project is its built-in Ubuntu 24 environment, which enables users to run Linux commands, scripts, and development tools... -
4
Frigate NVR
NVR with realtime local object detection for IP cameras
...The system uses OpenCV and TensorFlow to analyze video feeds and detect objects such as people, vehicles, and animals in real time. Frigate is optimized for efficiency and supports hardware acceleration across a wide range of devices, including GPUs and specialized inference hardware. It also provides event recording, snapshot management, and searchable video history to improve home or small-business security workflows. Overall, Frigate functions as a privacy-focused, AI-powered NVR platform for intelligent video monitoring. -
$300 Free Credits to Build on Google CloudPut your $300 in credit toward real workloads, then keep building with free monthly usage for 20+ products. No commitment and no charge until you upgrade.
-
5
BrowserAI
Run local LLMs like llama, deepseek, kokoro etc. inside your browser
BrowserAI is a cutting-edge platform that allows users to run large language models (LLMs) directly in their web browser without the need for a server. It leverages WebGPU for accelerated performance and supports offline functionality, making it a highly efficient and privacy-conscious solution. The platform provides a developer-friendly SDK with pre-configured popular models, and it allows for seamless switching between MLC and Transformer engines. Additionally, it supports features such as... -
6
MLX Engine
LM Studio Apple MLX engine
MLX Engine is the Apple MLX-based inference backend used by LM Studio to run large language models efficiently on Apple Silicon hardware. Built on top of the mlx-lm and mlx-vlm ecosystems, the engine provides a unified architecture capable of supporting both text-only and multimodal models. Its design focuses on high-performance on-device inference, leveraging Apple’s MLX stack to accelerate computation on M-series chips. -
7
RWKV Runner
A RWKV management and startup tool, full automation, only 8MB
RWKV (pronounced as RwaKuv) is an RNN with GPT-level LLM performance, which can also be directly trained like a GPT transformer (parallelizable). So it's combining the best of RNN and transformer - great performance, fast inference, fast training, saves VRAM, "infinite" ctxlen, and free text embedding. Moreover it's 100% attention-free. Default configs has enabled custom CUDA kernel acceleration, which is much faster and consumes much less VRAM. If you encounter possible compatibility issues, go to the Configs page and turn off Use Custom CUDA kernel to Accelerate. -
8
Ollama Server
No need for Termux, you can start the Ollama service
...The application includes basic model lifecycle management such as downloading official models, uploading custom GGUF files, and controlling running instances. It also enables offline inference, allowing users to run language models entirely on-device without requiring a network connection or external infrastructure. -
9
WeKnora
LLM framework for document understanding and semantic retrieval
...This approach enables the system to provide more reliable answers by grounding model reasoning in the content of uploaded documents. WeKnora is designed with a modular architecture that separates components for document processing, search strategies, and model inference, allowing developers to customize or extend different parts of the pipeline. It supports knowledge base management and conversational question answering built on top of structured and unstructured documents. -
Ship Agents FasterGemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.
-
10
Zod
TypeScript-first schema validation with static type inference
TypeScript-first schema validation with static type inference. Zod is a TypeScript-first schema declaration and validation library. I'm using the term "schema" to broadly refer to any data type, from a simple string to a complex nested object. Zod is designed to be as developer-friendly as possible. The goal is to eliminate duplicative type declarations. With Zod, you declare a validator once and Zod will automatically infer the static TypeScript type. -
11
Latent Box
A collection of awesome-lists for AI, creativity and art. AI
...The platform emphasizes usability by providing a clean user interface that allows users to load models, configure parameters, and interact with them without needing deep technical knowledge of underlying frameworks. It supports local inference workflows, which are increasingly important for privacy-conscious users and organizations seeking to reduce reliance on external APIs. latentbox also enables extensibility through plugins or integrations, allowing developers to customize model pipelines or connect additional tools. -
12
lms
LM Studio CLI
...The tool allows developers to control model execution directly from the terminal, providing programmatic access to features that are otherwise available through graphical interfaces. Through the CLI, users can load and unload models, start or stop local inference servers, and inspect the inputs and outputs generated by language models. LMS is built using the LM Studio JavaScript SDK and integrates tightly with the LM Studio runtime environment. The interface is designed to simplify automation workflows and scripting tasks related to local AI deployment. By exposing model management capabilities through command-line commands, the tool enables developers to integrate local LLM operations into development pipelines and backend services. ... -
13
TypeGPU
A modular and open-ended toolkit for WebGPU
TypeGPU is a modular TypeScript toolkit for working with WebGPU in a more type-safe and developer-friendly way. It lets developers define GPU logic with advanced type inference while still targeting WebGPU and WGSL-based execution. The project is designed for graphics, compute shaders, GPU programming, and high-performance browser or React Native experiments. It can generate WGSL from TypeScript-style functions, helping developers keep GPU code closer to the rest of their application logic. TypeGPU is useful for teams that want the power of WebGPU without losing the safety and tooling benefits of TypeScript. ... -
14
Ollama Grid Search
A multi-platform desktop application to evaluate and compare LLM
Ollama Grid Search is a desktop application designed to automate the evaluation and comparison of large language models, prompts, and inference parameters in a structured and repeatable way. Instead of manually testing combinations, the tool performs grid search experiments by iterating across different models, prompt variations, and parameter configurations, allowing users to quickly identify optimal setups for specific tasks. It provides a visual interface where experiment results can be inspected, compared, and refined, making it especially useful for prompt engineering and benchmarking workflows. ... -
15
io-ts
Runtime type system for IO decoding/encoding
...This approach is especially useful for APIs, configuration files, JSON payloads, database results, and any data that enters a program from an unknown source. io-ts integrates closely with functional programming patterns and the fp-ts ecosystem. Its main value is helping TypeScript applications keep runtime data validation aligned with compile-time type inference. -
16
lmstudio.js
LM Studio TypeScript SDK
lmstudio.js is the official TypeScript and JavaScript SDK that enables developers to programmatically interact with LM Studio’s local AI runtime. The library exposes the same capabilities used internally by the LM Studio desktop application, allowing external apps to load models, run inference, and build autonomous AI workflows. It is designed to simplify the creation of local AI tools by handling complex concerns such as dependency management, hardware compatibility, and model configuration. The SDK introduces an agent-style API that can execute multi-step tool-using workflows through a single call, enabling more advanced automation scenarios. ... -
17
Maths, CS & AI Compendium
Become a cracked AI/ML Research Engineer
...Its chapters cover vectors, matrices, calculus, statistics, probability, machine learning, language processing, computer vision, speech, multimodal learning, robotics, and graph neural networks. It also addresses operating systems, algorithms, production software, GPU programming, AI inference, and ML systems design. Readers need only elementary mathematics and basic Python knowledge to begin. A bundled MCP server lets compatible AI assistants use the locally cloned compendium as a knowledge base. The repository is useful for self-study, interview preparation, and deeper AI research engineering training. -
18
type-challenges
Collection of TypeScript type challenges with online judge
...Each challenge is a miniature kata where you implement types that transform other types—parsing strings, inferring tuples, mapping unions—without writing any runtime code. Problems are arranged from warm-ups to brain-twisters, letting developers build intuition about distributive conditional types, inference in extends, variance, and other corner cases of the type system. The repository includes tests for each puzzle so you get immediate, compiler-driven feedback when your solution is correct. As a result, it doubles as both training material and a living reference for advanced patterns used in real libraries. Many engineers report that solving a handful of these dramatically improves their ability to write safe, expressive APIs with minimal runtime overhead. -
19
Eko
Build Production-ready Agentic Workflow with Natural Language
Eko (Eko Keeps Operating) is a JavaScript framework designed for building production-ready agent-based workflows using natural language commands. It allows developers to create automated agents that can handle complex workflows in both computer and browser environments. With a focus on high development efficiency, Eko simplifies the creation of multi-step workflows, enabling users to integrate and automate tasks across platforms. It provides a unified interface for managing agents, offering... -
20
OSRS .json Hiscores
The Oldschool Runescape API wrapper that does more
OSRS .json Hiscores is a server-side API wrapper designed to transform Old School RuneScape hiscores data into structured and developer-friendly JSON objects, addressing limitations in the official API’s output formats. The official API typically returns CSV or basic arrays, which can be difficult to work with, so this project restructures the data into meaningful objects that include player stats, rankings, and inferred metadata. It also adds additional logic to determine player game modes,... -
21
Harbor LLM
Run a full local LLM stack with one command using Docker
...With a single command, users can start preconfigured tools like Ollama and Open WebUI, enabling chat, workflows, and integrations immediately. Harbor supports multiple inference engines, including llama.cpp and vLLM, and connects them seamlessly to user interfaces. It also includes tools for web retrieval, image generation, voice interaction, and workflow automation. Built on Docker, Harbor allows services to run in isolated containers while communicating over a local network. It is intended for local development and experimentation rather than production deployment, giving developers a flexible way to explore AI systems, test configurations, and manage complex LLM stacks without manual wiring or setup overhead. -
22
Generative AI JS
This SDK is now deprecated, use the new unified Google GenAI SDK
...Though marked deprecated (likely superseded by newer SDKs), the repo shows how to wrap HTTP/WS endpoints, manage streaming responses, and interoperate with browser UI or server logic. The examples include chat widgets, prompt pipelines, and generalized inference utilities. It also deals with streaming cancellation, retries, backoff logic, and message chunk assembly to help developers handle real-world use. Because it’s JavaScript, the repo supports both ESM and CommonJS contexts, making it versatile in backend and frontend setups. The deprecation label reflects that newer or official SDKs may have replaced it, but many of its patterns still serve as a useful reference to understand how streaming, chunking, and prompt logic can be implemented by hand in JS. -
23
Kodus
AI code reviews, just like your senior dev would do
Kodus-AI is a framework for building, training, and deploying intelligent agents and models, especially focusing on practical AI workflows for businesses and automation. It provides a structured set of tools and abstractions that help teams design agent behaviors, orchestrate data pipelines, optimize inference, and integrate AI capabilities with applications or services. The platform often includes model management, scalable training workflows, and orchestration patterns that help teams move from research or prototypes to production-ready AI deployments. Through configurable pipelines and a focus on modularity, it supports experimentation while maintaining reproducibility and performance. ... -
24
Clippy
Clippy, now with some AI
Clippy is an open-source desktop assistant that allows users to run modern large language models locally while presenting them through a nostalgic interface inspired by Microsoft’s classic Clippy assistant from the 1990s. The project serves as both a playful homage to the early days of personal computing and a practical demonstration of local AI inference. Clippy integrates with the llama.cpp runtime to run models directly on a user’s computer without requiring cloud-based AI services. It supports models in the GGUF format, which allows it to run many publicly available open-source LLMs efficiently on consumer hardware. Users interact with the system through a simple animated assistant interface that can answer questions, generate text, and perform conversational tasks. ... -
25
node-llama-cpp
Run AI models locally on your machine with node.js bindings for llama
node-llama-cpp is a JavaScript and Node.js binding that allows developers to run large language models locally using the high-performance inference engine provided by llama.cpp. The library enables applications built with Node.js to interact directly with local LLM models without requiring a remote API or external service. By using native bindings and optimized model execution, the framework allows developers to integrate advanced language model capabilities into desktop applications, server software, and command-line tools. ...
