Releases: pytorch/serve
Releases · pytorch/serve
TorchServe v0.5.3 Release Notes
This is the release of TorchServe v0.5.3.
New Features
- KServe V2 support - Added support for KServe V2 protocol.
- Model customized metadata support - Extended managementAPI to support customized metadata from handler.
Improvements
- Upgraded log4j2 version to 2.17.1 - Added log4j upgrade to address CVE-2021-44832.
- Upgraded pillow to 9.0.0, python support upgraded to py3.8/py3.9 - Added docker, install dependency upgrade.
- GPU utilization and GPU memory usage metrics support - Added support for GPU utilization and GPU memory usage metrics in benchmarks.
- Workflow benchmark support - Added support for workflow benchmark.
- benchmark-ab.py warmup support - Added support for warmup in benchmark-ab.py.
- Multiple inputs for a model inference example - Added example to support multiple inputs for a model inference.
- Documentation refactor - Improved documention.
- Added API auto-discovery - Added support for API auto-discovery.
- Nightly build support - Added support for Github action nightly build
pip install torchserve-nightly
Platform Support
Ubuntu 16.04, Ubuntu 18.04, MacOS 10.14+, Windows 10 Pro, Windows Server 2019, Windows subsystem for Linux (Windows Server 2019, WSLv1, Ubuntu 18.0.4). TorchServe now requires Python 3.8 and above.
GPU Support
Torch 1.10+ Cuda 10.2, 11.3
Torch 1.9.0 + Cuda 11.1
Torch 1.8.1 + Cuda 9.2
Planned Improvements
TorchServe v0.5.2 Release Notes
This is a hotfix release of Log4j issue.
Log4j Fixing
- Upgrade log4j2 version to 2.17.0 - Added log4j upgrade to address CVE-2021-45105.
TorchServe v0.5.1 Release Notes
This is a hotfix release of Log4j issue.
Log4j Fixing
- Upgrade log4j2 version to 2.16.0 - Added log4j upgrade to address CVE-2021-44228 and CVE-2021-45046.
New Features
- IPEX launcher support - Added support for Intel extension for PyTorch.
TorchServe v0.5.0 Release Notes
This is the release of TorchServe v0.5.0.
New Features
- PyTorch 1.10.0 support - TorchServe is now certified working with torch 1.10.0 torchvision 0.11.1, torchtext 0.11.0 and torchaudio 0.10.0
- Kubernetes HPA support - Added support for Kubernetes HPA.
- Faster transformer example - Added example for Faster transformer for optimized transformer model inference.
- (experimental) torchprep support - Added experimental CLI tool to prepare Pytorch models for efficient inference.
- Custom metrics example - Added example for custom metrics with mtail metrics exporter and Prometheus.
- Reactjs example for Image Classifier - Added example for Reactjs Image Classifier.
Improvements
- Batching inference exception support - Optimized batching to fix a concurrent modification exception that was occurring with batch inference.
- k8s cluster creation support upgrade - Updated Kubernetes cluster creation scripts for v1.17 support.
- Nvidia devices visibility support - Added support for NVIDIA devices visibility.
- Large image support - Added support for PIL.Image.MAX_IMAGE_PIXELS.
- Custom HTTP status support - Added support to return custom http status from a model handler.
- TS_CONFIG_FILE env var support - Added support for setting
TS_CONFIG_FILEas env var. - Frontend build optimization - Optimized frontend to reduce build times by 3.7x.
- Warmup in benchmark - Added support for warmup in benchmark scripts.
Platform Support
Ubuntu 16.04, Ubuntu 18.04, MacOS 10.14+, Windows 10 Pro, Windows Server 2019, Windows subsystem for Linux (Windows Server 2019, WSLv1, Ubuntu 18.0.4)
GPU Support
Torch 1.10+ Cuda 10.2, 11.3
Torch 1.9.0 + Cuda 11.1
Torch 1.8.1 + Cuda 9.2
TorchServe v0.4.2 Release Notes
This is a hotfix release of TorchServe v0.4.2.
Improvements
TorchServe v0.4.1 Release Notes
This is the release of TorchServe v0.4.1.
New Features
- PyTorch 1.9.0 support - TorchServe is now certified working with torch 1.9.0 torchvision 0.10.0, torchtext 0.10.0 and torchaudio 0.9.0
- Model configuration support - Added support for model performance tuning on SageMaker via model configuration in config.properties.
- Serialize config snapshots to DynamoDB - Added support for serializing config snapshots to DDB.
- Prometheus metrics plugin support - Added support for Prometheus metrics plugin.
- Kubeflow Pipelines support - Added support for Kubeflow pipelines and Google Vertex AI Manages pipelines, see examples here
- KFServing docker support - Added production docker for KFServing.
- Python 3.9 support - TorchServe is now certified working with Python 3.9.
Improvements
- HF BERT models multiple GPU support - Added multi-gpu support for HuggingFace BERT models.
- Error log for customer python package installation - Added support to log error of customer python package installation.
- Workflow documentation optimization - Optimized workflow documentation.
Tooling improvements
- Mar file automation integration - Integrated mar file generation automation into pytest and postman test.
- Benchmark automation for AWS neuron support - Added support for AWS neuron benchmark automation.
- Staging binary build support - Added support for staging binary build.
Platform Support
Ubuntu 16.04, Ubuntu 18.04, MacOS 10.14+, Windows 10 Pro, Windows Server 2019, Windows subsystem for Linux (Windows Server 2019, WSLv1, Ubuntu 18.0.4)
GPU Support
Torch 1.9.0 + Cuda 10.2, 11.1
Torch 1.8.1 + Cuda 9.2, 10.1
TorchServe v0.4.0 Release Notes
This is the release of TorchServe v0.4.0.
New Features
- Workflow support - Added support for sequential and parallel ensemble models with Language Translation and Computer Vision classification examples.
- S3 Model Store SSE support - Added support for S3 server side model encryption via KMS.
- MMF-activity-recognition model example - Added example MMF-activity-recognition model
- PyTorch 1.8.1 support - TorchServe is now certified working with torch 1.8.1, torchvision 0.9.1, torchtext 0.9.1, and torchaudio 0.8.1
Improvements
- Fixed GPU memory high usage issue and updated model zoo - Fixed duplicate process on GPU device .
- gRPC max_request_size support- Added support for gRPC max_request_size configuration in config.properties.
- Non SSL request support - Added support for non SSL request.
- Benchmark automation support - Added support for benchmark automation.
- Support mar file generation automation - Added mar file generation automation.
Community Contributions
- Fairseq NMT example - Added Fairseq Neural Machine Translation example (contributed by @AshwinChafale)
- DeepLabV3 Image Segmentation example - Added DeepLabV3 Image Segmentation example (contributed by @alvarobartt)
Bug Fixes
- Huggingface_Transformers model example - Fixed Captum explanations fails with HF models.
Platform Support
Ubuntu 16.04, Ubuntu 18.04, MacOS 10.14+, Windows 10 Pro, Windows Server 2019, Windows subsystem for Linux (Windows Server 2019, WSLv1, Ubuntu 18.0.4)
GPU Support
Cuda 10.1, 10.2, 11.1
TorchServe v0.3.1 Release Notes (Beta)
TorchServe v0.3.0 Release Notes (Beta)
This is the release of TorchServe v0.3.0
Highlights:
- Native windows support - Added support for TorchServe on Windows 10 pro and Windows Server 2019
- KFServing Integration - Added support for v1 KFServing predict and explain APIs with auto-scaling and canary deployments for serving models in Kubeflow/KFServing
- MLFlow-TorchServe: New MLflow TorchServe deployment plugin for serving models for MLflow MLOps lifecycle
- Captum explanations - Added explain API for Captum model interpretability of different models
- AKS Support - Added support for TorchServe deployment on Azure Kubernetes Service
- GKE Support - Added support for TorchServe deployment on Google Kubernetes Service
- gRPC support - Added support for gRPC based management and inference APIs
- Request Envelopes - Added support for request envelopes which parses request from multiple Model serving frameworks like Seldon, KFServing, without any modifications in the handler code
- PyTorch 1.7.1 support - TorchServe is now certified working with torch 1.7.1, torchvision 0.8.2, torchtext 0.8.1, and torchaudio 0.7.2
- TorchServe Profiling - Added end-to-end profiling of inference requests. The time taken for different events by TorchServe for an inference request is captured in TorchServe metrics logs
- Serving SDK - Release TorchServe Serving SDK 0.4.0 on maven with contracts/interfaces for Metric Endpoint plugin and Snapshot plugins
- Naked DIR support - Added support for Model Archives as Naked DIRs with the
--archive-format no-archive - Local file URL support - Added support for registering model through local file (
file:///) URLs - Install dependencies - Added a more robust install dependency script certified across different OS platforms (Ubuntu 18.04, MacOS, Windows 10 Pro, Windows Server 2019)
- Link Checker - Added link checker in sanity script to report any broken links in documentation
- Enhanced model description - Added GPU usage info and worker PID in model description
- FAQ guides - Added most frequently asked questions by community users
- Troubleshooting guide - Added documentation for troubleshooting common problems related to model serving by TorchServe
- Use case guide - Provides the reference use cases i.e. different ways in which TorchServe can be deployed for serving different types of PyTorch models
Other PRs since v0.2.0
Bug Fixes:
- Fixed unbound variable issue while creating binaries from script #595
- Fixed model latency calculation logic #630
- Treat application/x-www-form-urlencoded as binary data #705
- Fix socket.send does not guarantee that all data will be send. #765
- Fixed bug in create_mar.sh script of Text_to_Speech_Synthesizer #704
- Docker fixes #709 #724 #642 #823 #839 #853 #880
- Unit and regression test fixes #774 #775 #827 #845 #858 #852
- Install scripts fixes #798 #837 #844 #836
- Benchmark fixes #768
- Dependency fixes #757 #820
- Temp path fixes #877 #638
- Migrate model urls #697 #696 #695
Others
- Added metrics endpoint to cfn templates and k8s setup #670 #747
- Environment information header in regression and sanity suite #622 #865 #863
- Documentation changes and fixes #754 #470 #816 #584 #872 #871 #879 #739
- FairSeq language translation example #592
- Additional regression tests for KFServing #855
Platform Support
Ubuntu 16.04, Ubuntu 18.04, MacOS 10.14+, Windows 10 Pro, Windows Server 2019, Windows subsystem for Linux (Windows Server 2019, WSLv1, Ubuntu 18.0.4)
Getting Started with TorchServe
Additionally, you can get started at https://pytorch.org/serve/ with installation instructions, tutorials and docs.
Lastly, if you have questions, please drop it into the PyTorch discussion forums using the ‘deployment’ tag or file an issue on GitHub with a way to reproduce.
TorchServe v0.2.0 Release Notes (Beta)
This is the release of TorchServe v0.2.0
Highlights:
- Kubernetes Support - Torchserve deployment in Kubernetes using Helm Charts and a Persistent Volume
- Prometheus metrics - Added Prometheus as the default metrics framework
- Requirements.txt support - Added support to specify model specific dependencies as a requirements file within a mar archive; Cleanup of unused parameters and addition of relevant ones for torch-model-archiver
- Pytorch Scripted Models Support - Scripted model versions added to model zoo; Added testing for scripted models
- Default Handler Refactor: (breaking changes) The default handlers have been refactored for code reuse and enhanced post-processing support. More details in Backwards Incompatible Changes section below
- Windows Support - Added support for torchserve on windows subsystem for Linux
- AWS Cloud Formation Support - Added support for multi-node AutoScaling Group deployment, behind an Elastic Load Balancer using Elastic File System as the backing store
- Benchmark and Testing Enhancements - Added models in benchmark and sanity tests, support for throughput with batch processing in benchmarking, support docker for jmeter and apache benchmark tests
- Regression Suite Enhancements - Added new POSTMAN based test cases for API and pytest based intrusive test cases
- Docker Improvements - Consolidated dev and codebuild dockerfiles
- Install and Build Script Streamlining - Unified install scripts, added code coverage and sanity script
- Python Linting - More exhaustive python linting checks across Torchserve and Model Archiver
Backwards Incompatible Changes
- Default Handler Refactor:
- The default handlers have been refactored for code reuse and enhanced post-processing support. The output format for some of the following examples/models has been enhanced to include additional details like score/class probability.
- The following default-handlers have been equipped with batch support. Due to batch support, resnet_152_batch example is not a custom handler example anymore.
- image_classifier
- object_detector
- image_segmenter
- The index_to_name.json file use for the class to name mapping has been standardized across vision/text related default handlers
- Refactoring and code reuse have resulted into reduced boilerplate code in all the
serve/examples. - Custom handler documentation has been restructured and enhanced to facilitate the different possible ways to build simple or complex custom handlers
Other PRs since v0.1.1
Bug Fixes:
- Fixed
NameErrorin default image_classifier handler #489 - Fixed timeout errors during build #420 and unit tests #493
- Fixed model loading error on cpu which was saved on gpu #444
- Fixed Snapshot not being emitted after unregistering model with no workers #491
- Inference API description conformant to OpenAPI #372
- Removed duplicate snapshot server property #318
- Fixed tag for latest CPU version in README #452
- Added check for no objects detected in object detector #447
- Fixed incorrect set up of default workers per model #513
- Fixed model-archiver to accept handler name or handler_name:entry_pnt_func combinations #472
Others
Platform Support
Ubuntu 16.04, Ubuntu 18.04, MacOS 10.14+, Windows subsystem for Linux (Windows Server 2019, WSLv1, Ubuntu 18.0.4)
Getting Started with TorchServe
Additionally, you can get started at https://pytorch.org/serve/ with installation instructions, tutorials and docs.
Lastly, if you have questions, please drop it into the PyTorch discussion forums using the ‘deployment’ tag or file an issue on GitHub with a way to reproduce.