example for using TorchInductor caching with torch.compile #2925
Conversation
| self.manifest = ctx.manifest | ||
| properties = ctx.system_properties | ||
|
|
||
| if ( |
There was a problem hiding this comment.
I'm not sure I understand the point of this PR? Why add a layer of indirection to set a torch config via the torchserve yaml config when I can just directly set an environment variable?
EDIT: Ah actually since you're using os.environ this will introduce some config isolation which is useful
There was a problem hiding this comment.
Yes, that is the idea.
There was a problem hiding this comment.
Have you also tested this for TS with multiple models? Any downside of setting Environment variable for model specific config yaml for other workers on TS? ie if two workers tried to set different values will that work?
There was a problem hiding this comment.
I haven't tried multi model, but updated the example to be a multi-worker example.
Unfortunately, no. |
|
So I'd recommend at least adding some debug statement after building pytorch from source so you're sure things work. MIght as well upstream that too |
There was a problem hiding this comment.
@ankithagunapal Great to see the speed with the cache. Let's also verify for multiple workers.
Will be good to also reference these articles for additional tips for reducing / debugging compile time issues:
https://pytorch.org/blog/training-production-ai-models/#34-controlling-just-in-time-compilation-time
https://pytorch.org/docs/stable/torch.compiler_profiling_torch_compile.html
agunapal
changed the title
@msaroufim There are counters. I added this logic in the handler |
Modified the example to work with multiple workers. And added a section for additional links you mentioned. @chauhang Merging this PR for now. |
agunapal
changed the title
Description
This PR shows 2 examples
TORCHINDUCTOR_FX_GRAPH_CACHEtorchinductorcache with a configFixes #(issue)
Type of change
Please delete options that are not relevant.
Feature/Issue validation/testing
Please describe the Unit or Integration tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.
Seeing a 40% reduction in compile time with resnet18 max-autotune
Checklist: