kwargs for torch.compile in BaseHandler

[eballesteros]

Description

Enables passing and arbitrary kwarg dict for torch.compile in BaseHandler, as discussed in issue #2783 .

Type of change

Please delete options that are not relevant.

• New feature (non-breaking change which adds functionality)
• This change requires a documentation update

Feature/Issue validation/testing

Modified test_torch_compile.py to register 2 models:

• one configured with the backend as a str
• one configured with backend and mode in a dict

>> pytest serve/test/pytest/test_torch_compile.py

=========================================================================================== test session starts ============================================================================================
platform linux -- Python 3.9.18, pytest-7.4.3, pluggy-1.3.0
rootdir: /home/model-server/serve
collected 5 items

serve/test/pytest/test_torch_compile.py .....                                                                                                                                                        [100%]

============================================================================================= warnings summary =============================================================================================
serve/test/pytest/test_torch_compile.py:11
  /home/model-server/serve/test/pytest/test_torch_compile.py:11: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
    from pkg_resources import packaging

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================================================================= 5 passed, 1 warning in 85.48s (0:01:25) ==================================================================================

Checklist:

• Did you have fun?
• Have you added tests that prove your fix is effective or that this feature works?
• Has code been commented, particularly in hard-to-understand areas?
• Have you made corresponding changes to the documentation?

[eballesteros]

a couple of notes on the testing:

• I commented out locally the skip mark in test_serve_inference to check that it was passing
• if test_serve_inference is skipped, I'm pretty sure the other tests in test_torch_compile.py will pass even if the handler initialization fails, since:
  • the status will still be healthy in test_server_status
  • the model(s) will still appear as registered in test_registered_model. The model(s) worker status would not be READY, but this is not getting checked at the moment

[msaroufim]

very cool thank you for this, if regression tests pass we should be good to go - if you're planning on contributing more lmk and share a good email. i can invite you the pytorch slack where we can chat in higher bandwidth

[msaroufim]

Errors look legit and check out details on pre-commit in our contributing.md

[eballesteros]

Errors look legit and check out details on pre-commit in our contributing.md

my bad, it looks like the type annotations I added in the test helper were not yet supported in python 3.8. I got rid of them. Also installed the pre-commit hooks and fixed the linting.

if you're planning on contributing more lmk and share a good email. i can invite you the pytorch slack where we can chat in higher bandwidth

sure! Sounds good. I think I just sent an email to one of your addresses. Let me know if you can't find it