Developer guides – Testing & Linting

This page will give an introduction into the tools and practices we use for testing & linting/formatting our code.

• Testing
  • Writing tests
  • Pytest
  • Weberver tests
  • Coverage
• Linting
  • Pre-commit

Testing

For testing, we primarily use the pytest framework. The only exception to this are the tests for the user interface and its underlying API, which are tested using unittest, through the Django framework. There is a dedicated section regarding these tests later on in this document.

Writing tests

This section is regarding only tests written explicitly for the recommerce simulation framework, not the user interface/webserver.

All of our tests live within the tests/ folder within the root of the project. There are three significant files/folder to take note of:

• conftest.py: This file includes logic used by pytest concerning the setup and teardown of the complete test suite, meaning its respective functions are run at the very start of the pytest execution, and at the very end. We currently use this functionality to set the datapath at the start of the tests and remove any results files at the end.
• utils_tests.py: This file contains utility functions used across test files, such as loading a .json file or creating mock dictionaries. Use these functions when needed.
• test_data folder: This folder contains configuration files as well as pre-trained RL-agent models to be used during testing. Add any data you need in your tests to this folder.

All of the tests in our simulation framework are ordered by the respective file they test. If adding new tests for a specific functionality, add it to the test file that belongs to that functionality's file in the project.

Within our tests, we often make use of a pytest feature called parametrize, through

@pytest.mark.parametrize()

Parametrize allows us to run a test with a multitude of different inputs. Take a look at current implementations for how to use the feature. The feature takes a list of these inputs as its argument. Please conform to our style when adding a new test and place the argument list right above the respective test (or the first test using it if the list is used by more than one test) and name it as follows:

<test_name>_testcases

Pytest

Pytest documentation

Running the tests

If you want to run tests locally you can do so in a few ways:

Simply running all tests:

pytest

Running all tests with increased verbosity:

pytest -v

Pytest Plugins

We are using a number of plugins for pytest, which are automatically installed through the anaconda environment.

Markers

We have added markers to some tests, such as slow or training. You can filter tests using the -m flag, if for example you want to exclude slow tests from the run, use:

pytest -m "not slow"

Pytest-randomly

pytest-randomly documentation

To make sure that our tests do not have hidden dependencies between each other, their order is shuffled every time the suite is run. For this, we use the pytest-randomly plugin. Before test collection, a --randomly-seed is set and printed to the terminal, which you can use in subsequent runs to repeat the same order of tests, which can be useful for debugging.

Pytest-xdist

pytest-xdist documentation

pytest-xdist is a plugin which can be used to distribute tests across multiple CPUs to speed up test execution. See their documentation for usage info.

Webserver tests

To run tests written for the Django webserver go into the webserver folder within the project directory and run

python ./manage.py test -v 2

The -v 2 flag is optional, but offers a better insight into the test process.

Coverage

Coverage.py documentation

We use coverage.py to track and report on the code coverage of our tests. Coverage.py is automatically installed through the anaconda environment. If you want to track the current test coverage within the framework use these commands:

coverage run --source=. -m pytest
coverage json
coverage-badge -f -o ./badges/coverage.svg

By running these commands Coverage.py will run all tests, collect coverage info, write it to the coverage.json (for optional source code analysis of coverage) and update the coverage.svg badge.

To see the coverage report locally run

coverage report

Linting

As with all projects, there are certain coding standards followed within the project, and we ask that new developers conform to them. However, you are of course welcome to change old standards or impose new ones as well.

One notable standard that not everyone may agree with (which is why it is explicitly mentioned here) is that the original project team has decided on using tabs instead of spaces for indentation.

Pre-commit

Pre-commit documentation

We are using pre-commit to lint and format our files before committing. Pre-commit itself is automatically installed through the anaconda environment. You need to initialize pre-commit before your first commit using

pre-commit install

To circumvent possible errors caused later on, we recommend also running pre-commit manually once using the following command:

pre-commit run --all-files

From now on, pre-commit will always be run automatically before committing. It is also run as part of our CI pipeline.

Interrogate

Interrogate documentation

We use interrogate to monitor our docstring coverage. The plugin is configured within the pyproject.toml in the root directory of the project and automatically run as part of pre-commit. However, its badge can only be updated manually.

To update the badge, you need to (locally) modify the pre-commit-config.yml file by swapping the line setting the args for the plugin and then run pre-commit locally.

WARNING: Do not commit this modified pre-commit-config.yml file to Github, as this will break the CI pipeline!

Pre-commit troubleshooting

If you get the following error:

Git: Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases

while trying to commit, the cause is most likely pre-commit trying to access a Python version not in your virtual environment.

Solution: Check the App execution Aliases, and if no Python version is present, install it from the Microsoft Store. You do not need to disable the alias.

---

If you get an error saying that the _sqlite3-module is missing, you are missing the sqlite3.dll and sqlite3.def files.

Solution: Go to the SQLite download page to download the sqlite3.dll and sqlite3.def files and drop them into the anaconda installation folder:

Path\To\anaconda3\envs\your_venv_name\DLLs