Make handler tests callable with pytest #2173

mreso · 2023-03-11T05:30:55Z

Description

This PR refactors the torch_handler unit tests to make them callable with pytest and integrates them into torchserve_sanity.py

Fixes #(issue)
1667 (partially)

Type of change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
New feature (non-breaking change which adds functionality)
This change requires a documentation update

Feature/Issue validation/testing

Please describe the Unit or Integration tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.

Test A
Logs for Test A
Test B
Logs for Test B

Checklist:

Did you have fun?
Have you added tests that prove your fix is effective or that this feature works?
Has code been commented, particularly in hard-to-understand areas?
Have you made corresponding changes to the documentation?

codecov · 2023-03-11T05:51:29Z

Codecov Report

Merging #2173 (710ead3) into master (fd8f1b3) will increase coverage by 18.07%.
The diff coverage is 99.52%.

❗ Current head 710ead3 differs from pull request most recent head 3f5d4a5. Consider uploading reports for the commit 3f5d4a5 to get more accurate results

@@             Coverage Diff             @@
##           master    #2173       +/-   ##
===========================================
+ Coverage   53.37%   71.45%   +18.07%     
===========================================
  Files          71       73        +2     
  Lines        3226     3296       +70     
  Branches       57       57               
===========================================
+ Hits         1722     2355      +633     
+ Misses       1504      941      -563

Impacted Files	Coverage Δ
ts/torch_handler/unit_tests/models/base_model.py	`80.00% <75.00%> (+80.00%)`	⬆️
ts/torch_handler/unit_tests/conftest.py	`100.00% <100.00%> (ø)`
ts/torch_handler/unit_tests/test_base_handler.py	`100.00% <100.00%> (+100.00%)`	⬆️
ts/torch_handler/unit_tests/test_envelopes.py	`100.00% <100.00%> (+100.00%)`	⬆️
.../torch_handler/unit_tests/test_image_classifier.py	`100.00% <100.00%> (+100.00%)`	⬆️
...s/torch_handler/unit_tests/test_image_segmenter.py	`100.00% <100.00%> (+100.00%)`	⬆️
ts/torch_handler/unit_tests/test_mnist_kf.py	`100.00% <100.00%> (+100.00%)`	⬆️
...s/torch_handler/unit_tests/test_object_detector.py	`100.00% <100.00%> (+100.00%)`	⬆️
...s/torch_handler/unit_tests/test_utils/model_dir.py	`100.00% <100.00%> (ø)`

... and 12 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

msaroufim

will increase coverage by 18.03%.

:O

…with old weights version on gpu

agunapal

@mreso Could you please point to the code for integration with TS sanity .

Also, could you please attach the sanity logs to the PR

mreso · 2023-03-13T22:07:21Z

@mreso Could you please point to the code for integration with TS sanity .

Also, could you please attach the sanity logs to the PR

@ankithagunapal Sure, integration point is here:

serve/ts_scripts/backend_utils.py

Line 17 in 3f5d4a5

    
           ts_test_cmd = f"python -m pytest --cov-report xml:{report_output_dir} --cov={coverage_dir} {test_dir} {handler_test_dir}"

This is output from

python -c "import ts_scripts.backend_utils; ts_scripts.backend_utils.test_torchserve()"

============================= test session starts ==============================
platform linux -- Python 3.9.16, pytest-7.2.2, pluggy-1.0.0
rootdir: /home/ubuntu/serve
plugins: mock-3.10.0, cov-4.0.0
collected 90 items

ts/tests/unit_tests/test_backend_metric.py .                             [  1%]
ts/tests/unit_tests/test_model_loader.py ........                        [ 10%]
ts/tests/unit_tests/test_model_service_worker.py .............           [ 24%]
ts/tests/unit_tests/test_otf_codec_protocol.py .........                 [ 34%]
ts/tests/unit_tests/test_version.py .                                    [ 35%]
ts/tests/unit_tests/test_worker_service.py ....                          [ 40%]
ts/tests/unit_tests/metrics_yaml_testing/metric_cache_unit_test.py ..... [ 45%]
...................................                                      [ 84%]
ts/torch_handler/unit_tests/test_base_handler.py ..                      [ 86%]
ts/torch_handler/unit_tests/test_envelopes.py .....                      [ 92%]
ts/torch_handler/unit_tests/test_image_classifier.py ..                  [ 94%]
ts/torch_handler/unit_tests/test_image_segmenter.py .                    [ 95%]
ts/torch_handler/unit_tests/test_mnist_kf.py ...                         [ 98%]
ts/torch_handler/unit_tests/test_object_detector.py .                    [100%]

=============================== warnings summary ===============================
../miniconda3/envs/microbatching/lib/python3.9/site-packages/pkg_resources/__init__.py:121
  /home/ubuntu/miniconda3/envs/microbatching/lib/python3.9/site-packages/pkg_resources/__init__.py:121: DeprecationWarning: pkg_resources is deprecated as an API
    warnings.warn("pkg_resources is deprecated as an API", DeprecationWarning)

../miniconda3/envs/microbatching/lib/python3.9/site-packages/pkg_resources/__init__.py:2870
  /home/ubuntu/miniconda3/envs/microbatching/lib/python3.9/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

../miniconda3/envs/microbatching/lib/python3.9/site-packages/pkg_resources/__init__.py:2870
../miniconda3/envs/microbatching/lib/python3.9/site-packages/pkg_resources/__init__.py:2870
  /home/ubuntu/miniconda3/envs/microbatching/lib/python3.9/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('ruamel')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

ts/torch_handler/unit_tests/test_image_segmenter.py::test_handle
  /home/ubuntu/miniconda3/envs/microbatching/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
    warnings.warn(

ts/torch_handler/unit_tests/test_image_segmenter.py::test_handle
  /home/ubuntu/miniconda3/envs/microbatching/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=ResNet101_Weights.IMAGENET1K_V1`. You can also use `weights=ResNet101_Weights.DEFAULT` to get the most up-to-date weights.
    warnings.warn(msg)

ts/torch_handler/unit_tests/test_mnist_kf.py::test_handle
ts/torch_handler/unit_tests/test_mnist_kf.py::test_handle_kf
ts/torch_handler/unit_tests/test_mnist_kf.py::test_handle_kfv2
  /home/ubuntu/miniconda3/envs/microbatching/lib/python3.9/site-packages/torch/nn/functional.py:1331: UserWarning: dropout2d: Received a 2-D input to dropout2d, which is deprecated and will result in an error in a future release. To retain the behavior and silence this warning, please use dropout instead. Note that dropout2d exists to provide channel-wise dropout on inputs with 2 spatial dimensions, a channel dimension, and an optional batch dimension (i.e. 3D or 4D inputs).
    warnings.warn(warn_msg)

ts/torch_handler/unit_tests/test_object_detector.py::test_handle
  /home/ubuntu/miniconda3/envs/microbatching/lib/python3.9/site-packages/torchvision/models/_utils.py:135: UserWarning: Using 'backbone_name' and 'weights' as positional parameter(s) is deprecated since 0.13 and may be removed in the future. Please use keyword parameter(s) instead.
    warnings.warn(

ts/torch_handler/unit_tests/test_object_detector.py::test_handle
  /home/ubuntu/miniconda3/envs/microbatching/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=ResNet50_Weights.IMAGENET1K_V1`. You can also use `weights=ResNet50_Weights.DEFAULT` to get the most up-to-date weights.
    warnings.warn(msg)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

---------- coverage: platform linux, python 3.9.16-final-0 -----------
Coverage XML written to file ts/tests/unit_tests/coverage.xml

======================= 90 passed, 11 warnings in 5.95s ========================
## Started torchserve linting
## Started torchserve pytests
## In directory: /home/ubuntu/serve | Executing command: python -m pytest --cov-report xml:ts/tests/unit_tests/coverage.xml --cov=ts ts/tests/unit_tests ts/torch_handler/unit_tests

Let me know if you need full log.

agunapal

LGTM

mreso added 2 commits March 11, 2023 05:17

Make torch_handler tests callable with pytest

560f4eb

Add handler test to sanity check

73766a8

msaroufim self-requested a review March 11, 2023 21:48

msaroufim approved these changes Mar 11, 2023

View reviewed changes

mreso added 2 commits March 13, 2023 19:41

Replace resnet 152 against 18 to speedup unit test; also fixes issue …

b1cf99e

…with old weights version on gpu

Fix lint error

3f5d4a5

mreso requested review from agunapal, lxning and namannandan March 13, 2023 21:36

mreso changed the title ~~[WIP]Make handler tests callable with pytest~~ Make handler tests callable with pytest Mar 13, 2023

agunapal reviewed Mar 13, 2023

View reviewed changes

agunapal approved these changes Mar 13, 2023

View reviewed changes

mreso merged commit 9412c0a into master Mar 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make handler tests callable with pytest #2173

Make handler tests callable with pytest #2173

mreso commented Mar 11, 2023

codecov bot commented Mar 11, 2023 •

edited

Loading

msaroufim left a comment

agunapal left a comment

mreso commented Mar 13, 2023

agunapal left a comment

Make handler tests callable with pytest #2173

Make handler tests callable with pytest #2173

Conversation

mreso commented Mar 11, 2023

Description

Type of change

Feature/Issue validation/testing

Checklist:

codecov bot commented Mar 11, 2023 • edited Loading

Codecov Report

msaroufim left a comment

Choose a reason for hiding this comment

agunapal left a comment

Choose a reason for hiding this comment

mreso commented Mar 13, 2023

agunapal left a comment

Choose a reason for hiding this comment

codecov bot commented Mar 11, 2023 •

edited

Loading