DownloadNVIDIA® TensorRT™ is an SDK for high-performance deep learning inference. It includes a deep learning inference optimizer and runtime that delivers low latency and high throughput for deep learning inference applications. TensorRT-based applications perform up to 40X faster than CPU-only platforms during inference. With TensorRT, you can optimize neural network models trained in all major frameworks, calibrate for lower precision with high accuracy, and deploy to hyperscale data centers, embedded, or automotive product platforms. TensorRT is built on CUDA®, NVIDIA’s parallel programming model, and enables you to optimize inference leveraging libraries, development tools, and technologies in CUDA-X™ for artificial intelligence, autonomous machines, high-performance computing, and graphics. With new NVIDIA Ampere Architecture GPUs, TensorRT also leverages sparse tensor cores providing an additional performance boost.
Features
- TensorRT provides INT8 using Quantization Aware Training and Post Training Quantization
- Production deployments of deep learning inference applications such as video streaming, speech recognition, recommendation, etc.
- Reduced precision inference significantly reduces application latency
- With TensorRT, developers can focus on creating novel AI-powered applications rather than performance tuning for inference deployment
- Maximizes throughput with FP16 or INT8 by quantizing models while preserving accuracy
- Optimizes use of GPU memory and bandwidth by fusing nodes in a kernel
Project Samples
