TorchServe is a performant, flexible and easy-to-use tool for serving PyTorch eager mode and torschripted models. Multi-model management with the optimized worker to model allocation. REST and gRPC support for batched inference. Export your model for optimized inference. Torchscript out of the box, ORT, IPEX, TensorRT, FasterTransformer. Performance Guide: built-in support to optimize, benchmark and profile PyTorch and TorchServe performance. Expressive handlers: An expressive handler architecture that makes it trivial to support inferencing for your use case with many supported out of the box. Out-of-box support for system-level metrics with Prometheus exports, custom metrics and PyTorch profiler support.

Features

  • REST and gRPC support for batched inference
  • Deploy complex DAGs with multiple interdependent models
  • Default way to serve PyTorch models
  • Export your model for optimized inference
  • Performance Guide
  • Metrics API

Project Samples

Project Activity

See All Activity >

License

Apache License V2.0

Follow TorchServe

TorchServe Web Site

Other Useful Business Software
Custom VMs From 1 to 96 vCPUs With 99.95% Uptime Icon
Custom VMs From 1 to 96 vCPUs With 99.95% Uptime

General-purpose, compute-optimized, or GPU/TPU-accelerated. Built to your exact specs.

Live migration and automatic failover keep workloads online through maintenance. One free e2-micro VM every month.
Start Free
Rate This Project
Login To Rate This Project

User Reviews

Be the first to post a review of TorchServe!

Additional Project Details

Operating Systems

Windows

Programming Language

Java

Related Categories

Java Machine Learning Software, Java LLM Inference Tool

Registered

2022-08-05