TorchServe Files

This is the release of TorchServe v0.11.1.

Highlights Include

• Security Updates
  • Token Authorization: TorchServe enforces token authorization by default which requires the correct token to be provided when calling a HTTP/S or gRPC API. This is a security feature which addresses the concern of unauthorized API calls. This is applicable in the scenario where an unauthorized user may try to access a running TorchServe instance. The default behavior is to enable this feature which creates a key file with the appropriate tokens to be used for API calls. Users have the option to disable this feature to prevent token authorization from being required for API calls. For more details, refer to the token authorization documentation: https://github.com/pytorch/serve/blob/master/docs/token_authorization_api.md
  • Model API Control: TorchServe disables the ability to register and delete models using HTTP/S or gRPC API calls by default once TorchServe is running. This is a security feature which addresses the concern of unintended registration and deletion of models once TorchServe has started. This is applicable in the scenario where a user may upload malicious code to the model server in the form of a model or where a user may delete a model that is being used. The default behavior prevents users from registering or deleting models once TorchServe is running. Model API control can be enabled to allow users to register and delete models using the TorchServe model load and delete APIs. For more details, refer to the model API control documentation: https://github.com/pytorch/serve/blob/master/docs/model_api_control.md
• PyTorch 2.x updates
  • Standardized torch.compile configuration
  • Added examples for tensorrt & hpu backends
• GenAI updates
  • Support continuous batching in sequence batch streaming
  • Asynchronous backend worker communication for continuous batching
  • No code LLM deployment
• Support for Intel GPUs

Security Updates

• Adding model-control-mode by @udaij12 in https://github.com/pytorch/serve/pull/3165
• Enable Token Authorization by default by @udaij12 in https://github.com/pytorch/serve/pull/3163
• Updating night CIs to account for model control and token auth by @udaij12 in https://github.com/pytorch/serve/pull/3188
• Adding token auth and model api to workflow and https by @udaij12 in https://github.com/pytorch/serve/pull/3234
• Enable token authorization and model control for gRPC by @namannandan in https://github.com/pytorch/serve/pull/3238

PyTorch 2.x Updates

• torch compile config standardization update by @agunapal in https://github.com/pytorch/serve/pull/3166
• Token Authorization fixes by @udaij12 in https://github.com/pytorch/serve/pull/3192
• Changing mar file for Bert torch compile by @udaij12 in https://github.com/pytorch/serve/pull/3175
• Fixing torch compile benchmark by @udaij12 in https://github.com/pytorch/serve/pull/3179
• Add support for hpu_backend and Resnet50 compile example by @wozna in https://github.com/pytorch/serve/pull/3182
• Update image_classifier/densenet-161 to include torch.compile by @lzcemma in https://github.com/pytorch/serve/pull/3200
• TensorRT example with torch.compile by @agunapal in https://github.com/pytorch/serve/pull/3203
• Update documentation for vgg16 to use torch.compile by @ijkilchenko in https://github.com/pytorch/serve/pull/3211
• BERT with torch.compile by @agunapal in https://github.com/pytorch/serve/pull/3201
• T5 Translation with torch.compile & TensorRT backend by @agunapal in https://github.com/pytorch/serve/pull/3223
• Adjust Resnet50 hpu example by @wozna in https://github.com/pytorch/serve/pull/3219

GenAI

• Support continuous batching in sequence batch streaming case by @lxning in https://github.com/pytorch/serve/pull/3160
• GPT-FAST-MIXTRAL-MOE integration by @alex-kharlamov in https://github.com/pytorch/serve/pull/3151
• Fix sequence continuous batching close session race condition by @namannandan in https://github.com/pytorch/serve/pull/3198
• Asynchronous worker communication and vllm integration by @mreso in https://github.com/pytorch/serve/pull/3146
• Add single command LLM deployment by @mreso in https://github.com/pytorch/serve/pull/3209
• TensorRT-LLM Engine integration by @agunapal in https://github.com/pytorch/serve/pull/3228
• Adds torch.compile documentation to alexnet example readme by @crmdias in https://github.com/pytorch/serve/pull/3227

Support for Intel GPUs

• Torchserve support for Intel GPUs by @krish-navulla in https://github.com/pytorch/serve/pull/3132
• Torchserve Metrics support for Intel GPUs enabled by @krish-navulla in https://github.com/pytorch/serve/pull/3141

Documentation

• Update supported TS version in security documentation by @namannandan in https://github.com/pytorch/serve/pull/3144
• Update performance documentation by @agunapal in https://github.com/pytorch/serve/pull/3159
• model archiver example to multi-line by @GeeCastro in https://github.com/pytorch/serve/pull/3155
• fix broken llm deployment link by @msaroufim in https://github.com/pytorch/serve/pull/3214
• Security documentation update by @udaij12 in https://github.com/pytorch/serve/pull/3183

Improvements and Bug Fixing

• workaround for compile example failure by @agunapal in https://github.com/pytorch/serve/pull/3190
• Fix Inf2 benchmark by @namannandan in https://github.com/pytorch/serve/pull/3177
• Make a copy of the torchtext utils to remove dependency by @agunapal in https://github.com/pytorch/serve/pull/3076
• Pinning setuptools version by @udaij12 in https://github.com/pytorch/serve/pull/3152
• Fixing Regression test CI GPU and CPU by @udaij12 in https://github.com/pytorch/serve/pull/3147
• Fixing docker CI by @udaij12 in https://github.com/pytorch/serve/pull/3194
• Replace pkg_resources.packaging by @udaij12 in https://github.com/pytorch/serve/pull/3187
• Kserve ci fix by @udaij12 in https://github.com/pytorch/serve/pull/3196
• Benchmark numpy fix by @udaij12 in https://github.com/pytorch/serve/pull/3197
• Add workflow dispatch trigger to nightly builds by @agunapal in https://github.com/pytorch/serve/pull/3250
• Bug fix for kserve build issue and fixing nightly tests by @agunapal in https://github.com/pytorch/serve/pull/3251
• Remove vllm dependency to not bloat docker image size by @agunapal in https://github.com/pytorch/serve/pull/3245
• Kserve fix ray & setuptools dependency issue by @udaij12 in https://github.com/pytorch/serve/pull/3205
• clean a jobGroup immediately when it finished by @lxning in https://github.com/pytorch/serve/pull/3222
• Updating examples for security tags by @udaij12 in https://github.com/pytorch/serve/pull/3224
• Fix/llm launcher disable token by @mreso in https://github.com/pytorch/serve/pull/3230
• Example update by @udaij12 in https://github.com/pytorch/serve/pull/3231
• Updating docker cuda and github branch by @udaij12 in https://github.com/pytorch/serve/pull/3233
• Reduce severity of xpu-smi logging by @namannandan in https://github.com/pytorch/serve/pull/3239
• Upgrade kserve dependencies by @agunapal in https://github.com/pytorch/serve/pull/3246
• Fix/vllm dependency by @mreso in https://github.com/pytorch/serve/pull/3249
• Copy remote branch entrypoint to compile and production image stages by @lanxih in https://github.com/pytorch/serve/pull/3213
• Fix Condition Checking for Intel GPUs Enabling by @Kanya-Mo in https://github.com/pytorch/serve/pull/3220

New Contributors

• @alex-kharlamov made their first contribution in https://github.com/pytorch/serve/pull/3151
• @lzcemma made their first contribution in https://github.com/pytorch/serve/pull/3200
• @wozna made their first contribution in https://github.com/pytorch/serve/pull/3182
• @krish-navulla made their first contribution in https://github.com/pytorch/serve/pull/3132
• @ijkilchenko made their first contribution in https://github.com/pytorch/serve/pull/3211
• @lanxih made their first contribution in https://github.com/pytorch/serve/pull/3213
• @Kanya-Mo made their first contribution in https://github.com/pytorch/serve/pull/3220
• @crmdias made their first contribution in https://github.com/pytorch/serve/pull/3227

Platform Support

Ubuntu 20.04 MacOS 10.14+, Windows 10 Pro, Windows Server 2019, Windows subsystem for Linux (Windows Server 2019, WSLv1, Ubuntu 18.0.4). TorchServe requires Python >= 3.8 and JDK17.

GPU Support Matrix

Inferentia2 Support Matrix