support client timeout #2267
support client timeout #2267
Conversation
Codecov Report
@@ Coverage Diff @@
## master #2267 +/- ##
=======================================
Coverage 69.82% 69.82%
=======================================
Files 77 77
Lines 3420 3420
Branches 57 57
=======================================
Hits 2388 2388
Misses 1029 1029
Partials 3 3 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
frontend/archive/src/main/java/org/pytorch/serve/archive/model/ModelConfig.java
Show resolved
Hide resolved
| * the client timeout in millions second. The inference request will be dropped once it is | ||
| * timeout. | ||
| */ | ||
| private long clientTimeoutInMills; |
There was a problem hiding this comment.
is default value infinity?
EDIT: browsed further seems like yes
There was a problem hiding this comment.
default is 0 so that setClientExpireTS of RequestInput.java can be easily calculated (ie. clientExpireTS will be infinity).
| inferChannel.writeAndFlush(req); | ||
| TestUtils.getLatch().await(1, TimeUnit.SECONDS); | ||
| Assert.assertNull(TestUtils.result); | ||
|
|
There was a problem hiding this comment.
what's the expected returned error code?
There was a problem hiding this comment.
nothing is returned if client already timeout. The inference request will be dropped and warning is logged.
| @@ -281,7 +283,13 @@ public void pollBatch(String threadId, long waitTime, Map<String, Job> jobsRepo) | |||
| } | |||
| maxDelay -= end - begin; | |||
| begin = end; | |||
| jobsRepo.put(j.getJobId(), j); | |||
| if (j.getPayload().getClientExpireTS() > System.currentTimeMillis()) { | |||
| jobsRepo.put(j.getJobId(), j); | |||
There was a problem hiding this comment.
Is the client timeout w.r.t to a single inference example or a single batch?
There was a problem hiding this comment.
timeout is applied on each inference request.
Description
Please read our CONTRIBUTING.md prior to creating your first pull request.
Please include a summary of the feature or issue being fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
Fixes #(issue)
#1863
Type of change
Please delete options that are not relevant.
Feature/Issue validation/testing
Please describe the Unit or Integration tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.
Test A
Logs for Test A
Test B
Logs for Test B
Checklist: