Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for gRPC max connection age configuration #3121

Merged
merged 7 commits into from
May 9, 2024

Conversation

namannandan
Copy link
Collaborator

@namannandan namannandan commented Apr 29, 2024

Description

Add support for server side gRPC max connection age configuration

References:

  1. Netty gRPC maxConnectionAge
  2. Netty gRPC maxConnectionAgeGrace
  3. Default gRPC KeepAlive configuration

Fixes #2999

Type of change

  • New feature (non-breaking change which adds functionality)

Feature/Issue validation/testing

# gRPC management connection timeout is set to 5 seconds
$ cat custom_config.properties 
grpc_management_max_connection_age_ms=5000
grpc_management_max_connection_age_grace_ms=0

$ torchserve --ncs --start --model-store ./model_store --ts-config ./custom_config.properties

# Note: Model takes longer than 5 seconds to load
$ time python ts_scripts/torchserve_grpc_client.py register densenet161
## Check densenet161.mar in mar_set : set()
## Register marfile: https://torchserve.s3.amazonaws.com/mar_files/densenet161.mar

Failed to register model densenet161.
Socket closed
python ts_scripts/torchserve_grpc_client.py register densenet161  0.13s user 0.06s system 3% cpu 5.629 total

As can be seen from the test above, the gRPC management connection to register model is closed after the configured 5 seconds (max connection age).

Copy link
Collaborator

@mreso mreso left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, please view comments and add some comments on what grace means in this context

docs/configuration.md Show resolved Hide resolved
@namannandan namannandan enabled auto-merge May 9, 2024 18:15
@namannandan namannandan added this pull request to the merge queue May 9, 2024
Merged via the queue into pytorch:master with commit 9419344 May 9, 2024
9 of 12 checks passed
@slashvar
Copy link

Thanks ! This will allow us to use torchserve with gRPC at scale !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Configuration of gRPC MAX_CONNECTION_AGE and MAX_CONNECTION_AGE_GRACE
3 participants