Search Results for "tensorrt"
Sort By:
Showing 31 open source projects for "tensorrt"
View related business solutions-
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
Atera - an All-in-one platform for IT managementYour IT essentials, integrated & elevated. Take your IT management from automated to autonomous, download Atera's agent to start your free trial!
-
1
TensorRT
C++ library for high performance inference on NVIDIA GPUs
NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference. It includes a deep learning inference optimizer and runtime that delivers low latency and high throughput for deep learning inference applications. TensorRT-based applications perform up to 40X faster than CPU-only platforms during inference. With TensorRT, you can optimize neural network models trained in all major frameworks, calibrate for lower precision with high accuracy, and deploy to hyperscale data centers, embedded, or automotive product platforms. ... -
2
TensorRT LLM
TensorRT LLM provides users with an easy-to-use Python API
TensorRT-LLM is an open-source high-performance inference library specifically designed to optimize and accelerate large language model deployment on NVIDIA GPUs. It provides a Python-based API built on top of PyTorch that allows developers to define, customize, and deploy LLMs efficiently across a variety of hardware configurations, from single GPUs to large multi-node clusters. -
3
Torch-TensorRT
PyTorch/TorchScript/FX compiler for NVIDIA GPUs using TensorRT
Torch-TensorRT is a compiler for PyTorch/TorchScript, targeting NVIDIA GPUs via NVIDIA’s TensorRT Deep Learning Optimizer and Runtime. Unlike PyTorch’s Just-In-Time (JIT) compiler, Torch-TensorRT is an Ahead-of-Time (AOT) compiler, meaning that before you deploy your TorchScript code, you go through an explicit compile step to convert a standard TorchScript program into a module targeting a TensorRT engine. -
4
TensorRT Node for ComfyUI
Enables the best performance on NVIDIA RTX Graphics Cards
ComfyUI_TensorRT is an extension that lets ComfyUI run AI inference through NVIDIA’s TensorRT, aiming to get faster, more efficient execution on supported GPUs. It bridges the gap between ComfyUI’s flexible, node-based workflows and TensorRT’s highly optimized engine format. The result is that complex diffusion or image-processing graphs can be accelerated without the user having to rewrite the pipeline. The repo typically includes instructions for converting models to TensorRT engines and for wiring those engines into ComfyUI nodes. ... -
Build Agents and Models on One PlatformGemini Enterprise Agent Platform is Google Cloud's comprehensive platform for developers to build, scale, govern, and optimize agents and models. Choose from Google's most advanced models and third-party models like Anthropic's Claude Model Family.
-
5
TensorRT Backend For ONNX
ONNX-TensorRT: TensorRT backend for ONNX
Parses ONNX models for execution with TensorRT. Development on the main branch is for the latest version of TensorRT 8.4.1.5 with full dimensions and dynamic shape support. For previous versions of TensorRT, refer to their respective branches. Building INetwork objects in full dimensions mode with dynamic shape support requires calling the C++ and Python API. Current supported ONNX operators are found in the operator support matrix. -
6
AimAhead
The fastest AI powered Aimbot
...It captures the screen, processes the image through a selected AI model to detect enemies, and then aims towards them. Optimized for NVIDIA graphics cards, AimAhead converts ONNX models to TensorRT engine files for enhanced performance, achieving between 100 to 200 cycles per second depending on the model used. -
7
TokenSpeed
TokenSpeed is a speed-of-light LLM inference engine
TokenSpeed is an LLM inference engine designed for high-performance production agent workloads. It aims to combine TensorRT-LLM-level speed with vLLM-level usability, making it relevant for teams that need fast generation without sacrificing developer ergonomics. The project is focused on the specific needs of agentic systems, where latency, throughput, and efficient scheduling matter across many short or tool-heavy requests. It builds on ideas and components from the broader open-source inference ecosystem while presenting its own execution stack. ... -
8
WhisperLive
A nearly-live implementation of OpenAI's Whisper
...It runs as a server–client system in which the server hosts a Whisper backend and clients stream audio to be transcribed with very low delay. The project supports multiple inference backends, including Faster-Whisper, NVIDIA TensorRT, and OpenVINO, allowing you to target GPUs and different CPU architectures efficiently. It can handle microphone input, pre-recorded audio files, and network streams such as RTSP and HLS, making it flexible for live events, monitoring, or accessibility workflows. Configuration options let you control the number of clients, maximum connection time, and threading behavior so the server can be tuned for different deployment environments. ... -
9
NVIDIA Model Optimizer
A unified library of SOTA model optimization techniques
...It supports a wide range of model types, including large language models, diffusion models, and vision-language models, and integrates with deployment frameworks such as TensorRT and vLLM. By providing standardized workflows and APIs, it enables developers to experiment with different optimization strategies and select the best approach for their use case. -
Error to trace to log to deploy. One click. No SSH.AppSignal links every error to the trace, the trace to the log, the log to the deploy that shipped it.
-
10
CUDA Containers for Edge AI & Robotics
Machine Learning Containers for NVIDIA Jetson and JetPack-L4T
...The repository contains container configurations that package the latest AI frameworks and dependencies optimized for Jetson hardware. These containers simplify the deployment of complex machine learning environments by bundling libraries such as CUDA, TensorRT, and deep learning frameworks into reproducible container images. The project is particularly useful for developers building edge AI and robotics systems that rely on GPU-accelerated inference and real-time computer vision. By using containerized environments, developers can ensure that their applications run consistently across different Jetson platforms and JetPack versions. ... -
11
ChatGLM3
ChatGLM3 series: Open Bilingual Chat LLMs | Open Source Bilingual Chat
...The repo ships Python APIs, CLI and web demos (Gradio/Streamlit), an OpenAI-format API server, and a compact fine-tuning kit. Quantization (4/8-bit), CPU/MPS support, and accelerator backends (TensorRT-LLM, OpenVINO, chatglm.cpp) enable lightweight local or edge deployment. -
12
Ultralytics
Ultralytics YOLO
Ultralytics is a comprehensive computer vision framework that provides state-of-the-art implementations of the YOLO (You Only Look Once) family of models, enabling developers to perform tasks such as object detection, segmentation, classification, tracking, and pose estimation within a unified system. It is designed to be fast, accurate, and easy to use, offering both command-line and Python-based interfaces for training, validation, and deployment of machine learning models. The framework... -
13
TorchServe
Serve, optimize and scale PyTorch models in production
...Multi-model management with the optimized worker to model allocation. REST and gRPC support for batched inference. Export your model for optimized inference. Torchscript out of the box, ORT, IPEX, TensorRT, FasterTransformer. Performance Guide: built-in support to optimize, benchmark and profile PyTorch and TorchServe performance. Expressive handlers: An expressive handler architecture that makes it trivial to support inferencing for your use case with many supported out of the box. Out-of-box support for system-level metrics with Prometheus exports, custom metrics and PyTorch profiler support. -
14
Triton Inference Server
The Triton Inference Server provides an optimized cloud
Triton Inference Server is an open-source inference serving software that streamlines AI inferencing. Triton enables teams to deploy any AI model from multiple deep learning and machine learning frameworks, including TensorRT, TensorFlow, PyTorch, ONNX, OpenVINO, Python, RAPIDS FIL, and more. Triton supports inference across cloud, data center, edge, and embedded devices on NVIDIA GPUs, x86 and ARM CPU, or AWS Inferentia. Triton delivers optimized performance for many query types, including real-time, batched, ensembles, and audio/video streaming. Provides Backend API that allows adding custom backends and pre/post-processing operations. ... -
15
Infinity
Low-latency REST API for serving text-embeddings
Infinity is a high-throughput, low-latency REST API for serving vector embeddings, supporting all sentence-transformer models and frameworks. Infinity is developed under MIT License. Infinity powers inference behind Gradient.ai and other Embedding API providers. -
16
Mooncake
Mooncake is the serving platform for Kimi
Mooncake is an open-source infrastructure platform designed to optimize large language model serving by focusing on efficient management and transfer of model data and KV cache. The platform was originally developed as part of the serving infrastructure for the Kimi large language model system. Its architecture centers on a high-performance transfer engine that provides unified data transfer across different storage and networking technologies. This engine enables efficient movement of... -
17
VideoPipe
A cross-platform video structuring (video analysis) framework
...The framework is designed to be lightweight and portable, with minimal dependencies compared to other video processing systems, making it easier to deploy across different environments. It supports multiple inference backends, including OpenCV DNN, TensorRT, PaddleInference, and ONNXRuntime, allowing developers to choose the most suitable runtime for their performance and hardware requirements. VideoPipe also supports various video input sources such as RTSP, RTMP, and local files, enabling it to handle real-time streaming and batch processing scenarios. -
18
OneFlow
OneFlow is a deep learning framework designed to be user-friendly
OneFlow is a deep learning framework designed to be user-friendly, scalable and efficient. An extension for OneFlow to target third-party compiler, such as XLA, TensorRT and OpenVINO etc.CUDA runtime is statically linked into OneFlow. OneFlow will work on a minimum supported driver, and any driver beyond. For more information. Distributed performance (efficiency) is the core technical difficulty of the deep learning framework. OneFlow focuses on performance improvement and heterogeneous distributed expansion. ... -
19
AI-Aimbot
CS2, Valorant, Fortnite, APEX, every game
AI-Aimbot is a computer vision project that demonstrates how artificial intelligence can be used to automatically identify and target opponents in video games. The system uses an object detection model based on the YOLOv5 architecture to detect human-shaped characters in gameplay screenshots or video frames. Once a target is identified, the program automatically adjusts the player’s aim toward the detected target, effectively automating the aiming process in first-person shooter games. The... -
20
CLIP-as-service
Embed images and sentences into fixed-length vectors
CLIP-as-service is a low-latency high-scalability service for embedding images and text. It can be easily integrated as a microservice into neural search solutions. Serve CLIP models with TensorRT, ONNX runtime and PyTorch w/o JIT with 800QPS[*]. Non-blocking duplex streaming on requests and responses, designed for large data and long-running tasks. Horizontally scale up and down multiple CLIP models on single GPU, with automatic load balancing. Easy-to-use. No learning curve, minimalist design on client and server. Intuitive and consistent API for image and sentence embedding. ... -
21
pipeless
A computer vision framework to create and deploy apps in minutes
Pipeless is an open-source computer vision framework to create and deploy applications without the complexity of building and maintaining multimedia pipelines. It ships everything you need to create and deploy efficient computer vision applications that work in real-time in just minutes. Pipeless is inspired by modern serverless technologies. It provides the development experience of serverless frameworks applied to computer vision. You provide some functions that are executed for new... -
22
enhancr
Video Frame Interpolation & Super Resolution using NVIDIA's TensorRT
...The GUI was designed to provide a stunning experience powered by state-of-the-art technologies without feeling clunky and outdated like other alternatives. It features blazing-fast TensorRT inference by NVIDIA, which can speed up AI processes significantly. Pre-packaged, without the need to install Docker or WSL (Windows Subsystem for Linux) - and NCNN inference by Tencent which is lightweight and runs on NVIDIA, AMD and even Apple Silicon - in contrast to the mammoth of an inference PyTorch is, which only runs on NVIDIA GPUs. -
23
FasterTransformer
Transformer related optimization, including BERT, GPT
...FasterTransformer is particularly focused on inference workloads, where it significantly improves performance compared to standard framework implementations. Although development has transitioned toward TensorRT-LLM, the project remains an important reference for understanding optimized transformer execution. -
24
AI Models
A repository of trained models
All models (at least currently) are supported by chaiNNer, an upscaling GUI that allows for both very simple and very complex tasks to be completed in a nice manner where you "chain" nodes together. Highly recommended for images. If you're looking to upscale videos using the models then use enhancr simply due to the fact that it supports TensorRT, which will allow you to upscale videos at incredible speeds! The GUI is one of the best looking applications out there and is personally my go to option. While yes builds are paid, it is well worth your money and beats any other GUI out there currently. -
25
Hello AI World
Guide to deploying deep-learning inference networks
...In just a couple of hours, you can have a set of deep learning inference demos up and running for realtime image classification and object detection on your Jetson Developer Kit with JetPack SDK and NVIDIA TensorRT. The tutorial focuses on networks related to computer vision, and includes the use of live cameras. You’ll also get to code your own easy-to-follow recognition program in Python or C++, and train your own DNN models onboard Jetson with PyTorch. Ready to dive into deep learning? It only takes two days. We’ll provide you with all the tools you need, including easy to follow guides, software samples such as TensorRT code, and even pre-trained network models including ImageNet and DetectNet examples. ...