markers: avoid combinatorial explosion in certain cases #568

radoering · 2023-03-25T16:20:44Z

Added tests for changed code.
Updated documentation for changed code.

Implements (more or less) the proposal from python-poetry/poetry#7689 (comment) by replacing compatible MultiMarker/MarkerUnion with AtomicMultiMarker/AtomicMarkerUnion, which behave more like single markers than complex markers during marker simplification.

Example pyproject.toml from python-poetry/poetry#7689:

$ time poetry132 export -q
real    0m0.953s
user    0m0.853s
sys     0m0.040s

$ time poetry141 export -q
real    3m34.909s
user    3m34.617s
sys     0m0.230s

$ time poetry141_patched export -q
real    0m0.622s
user    0m0.538s
sys     0m0.020s

Why not use a normal SingleMarker with in and not in?

I assume that we are not standard compliant with our implementation of in and not in. From reading PEP 508, which does not give much detail about in and not in, and experimenting with packaging, I suspect it's just supposed to be a string operation:

>>> from packaging import markers
>>> m = markers.Marker("platform_machine in 'x86_64,amd64'")
>>> m.evaluate({'platform_machine': 'aarch64'})
False
>>> m.evaluate({'platform_machine': 'x86_64'})
True
>>> m.evaluate({'platform_machine': 'x86'})
True

Thus, in and not in will be kind of dangerous if there are values that are contained in other values.

…arker/MarkerUnion with AtomicMultiMarker/AtomicMarkerUnion

sonarcloud · 2023-03-25T16:21:29Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
2 Code Smells

No Coverage information
0.0% Duplication

github-advanced-security · 2023-03-25T16:21:32Z

You have successfully added a new SonarCloud configuration ``. As part of the setup process, we have scanned this repository and found no existing alerts. In the future, you will see all code scanning alerts on the repository Security tab.

This PR contains the following updates: | Package | Update | Change | |---|---|---| | [poetry](https://python-poetry.org/) ([source](https://github.com/python-poetry/poetry), [changelog](https://python-poetry.org/history/)) | minor | `1.4.2` -> `1.5.0` | --- ### Release Notes <details> <summary>python-poetry/poetry</summary> ### [`v1.5.0`](https://github.com/python-poetry/poetry/blob/HEAD/CHANGELOG.md#150---2023-05-19) [Compare Source](python-poetry/poetry@1.4.2...1.5.0) ##### Added - **Introduce the new source priorities `explicit` and `supplemental`** ([#7658](python-poetry/poetry#7658), [#6879](python-poetry/poetry#6879)). - **Introduce the option to configure the priority of the implicit PyPI source** ([#7801](python-poetry/poetry#7801)). - Add handling for corrupt cache files ([#7453](python-poetry/poetry#7453)). - Improve caching of URL and git dependencies ([#7693](python-poetry/poetry#7693), [#7473](python-poetry/poetry#7473)). - Add option to skip installing directory dependencies ([#6845](python-poetry/poetry#6845), [#7923](python-poetry/poetry#7923)). - Add `--executable` option to `poetry env info` ([#7547](python-poetry/poetry#7547)). - Add `--top-level` option to `poetry show` ([#7415](python-poetry/poetry#7415)). - Add `--lock` option to `poetry remove` ([#7917](python-poetry/poetry#7917)). - Add experimental `POETRY_REQUESTS_TIMEOUT` option ([#7081](python-poetry/poetry#7081)). - Improve performance of wheel inspection by avoiding unnecessary file copy operations ([#7916](python-poetry/poetry#7916)). ##### Changed - **Remove the old deprecated installer and the corresponding setting `experimental.new-installer`** ([#7356](python-poetry/poetry#7356)). - **Introduce `priority` key for sources and deprecate flags `default` and `secondary`** ([#7658](python-poetry/poetry#7658)). - Deprecate `poetry run <entry point>` if the entry point was not previously installed via `poetry install` ([#7606](python-poetry/poetry#7606)). - Only write the lock file if the installation succeeds ([#7498](python-poetry/poetry#7498)). - Do not write the unused package category into the lock file ([#7637](python-poetry/poetry#7637)). ##### Fixed - Fix an issue where Poetry's internal pyproject.toml continually grows larger with empty lines ([#7705](python-poetry/poetry#7705)). - Fix an issue where Poetry crashes due to corrupt cache files ([#7453](python-poetry/poetry#7453)). - Fix an issue where the `Retry-After` in HTTP responses was not respected and retries were handled inconsistently ([#7072](python-poetry/poetry#7072)). - Fix an issue where Poetry silently ignored invalid groups ([#7529](python-poetry/poetry#7529)). - Fix an issue where Poetry does not find a compatible Python version if not given explicitly ([#7771](python-poetry/poetry#7771)). - Fix an issue where the `direct_url.json` of an editable install from a git dependency was invalid ([#7473](python-poetry/poetry#7473)). - Fix an issue where error messages from build backends were not decoded correctly ([#7781](python-poetry/poetry#7781)). - Fix an infinite loop when adding certain dependencies ([#7405](python-poetry/poetry#7405)). - Fix an issue where pre-commit hooks skip pyproject.toml files in subdirectories ([#7239](python-poetry/poetry#7239)). - Fix an issue where pre-commit hooks do not use the expected Python version ([#6989](python-poetry/poetry#6989)). - Fix an issue where an unclear error message is printed if the project name is the same as one of its dependencies ([#7757](python-poetry/poetry#7757)). - Fix an issue where `poetry install` returns a zero exit status even though the build script failed ([#7812](python-poetry/poetry#7812)). - Fix an issue where an existing `.venv` was not used if `in-project` was not set ([#7792](python-poetry/poetry#7792)). - Fix an issue where multiple extras passed to `poetry add` were not parsed correctly ([#7836](python-poetry/poetry#7836)). - Fix an issue where `poetry shell` did not send a newline to `fish` ([#7884](python-poetry/poetry#7884)). - Fix an issue where `poetry update --lock` printed operations that were not executed ([#7915](python-poetry/poetry#7915)). - Fix an issue where `poetry add --lock` did perform a full update of all dependencies ([#7920](python-poetry/poetry#7920)). - Fix an issue where `poetry shell` did not work with `nushell` ([#7919](python-poetry/poetry#7919)). - Fix an issue where subprocess calls failed on Python 3.7 ([#7932](python-poetry/poetry#7932)). - Fix an issue where keyring was called even though the password was stored in an environment variable ([#7928](python-poetry/poetry#7928)). ##### Docs - Add information about what to use instead of `--dev` ([#7647](python-poetry/poetry#7647)). - Promote semantic versioning less aggressively ([#7517](python-poetry/poetry#7517)). - Explain Poetry's own versioning scheme in the FAQ ([#7517](python-poetry/poetry#7517)). - Update documentation for configuration with environment variables ([#6711](python-poetry/poetry#6711)). - Add details how to disable the virtualenv prompt ([#7874](python-poetry/poetry#7874)). - Improve documentation on whether to commit `poetry.lock` ([#7506](python-poetry/poetry#7506)). - Improve documentation of `virtualenv.create` ([#7608](python-poetry/poetry#7608)). ##### poetry-core ([`1.6.0`](https://github.com/python-poetry/poetry-core/releases/tag/1.6.0)) - Improve error message for invalid markers ([#569](python-poetry/poetry-core#569)). - Increase robustness when deleting temporary directories on Windows ([#460](python-poetry/poetry-core#460)). - Replace `tomlkit` with `tomli`, which changes the interface of some *internal* classes ([#483](python-poetry/poetry-core#483)). - Deprecate `Package.category` ([#561](python-poetry/poetry-core#561)). - Fix a performance regression in marker handling ([#568](python-poetry/poetry-core#568)). - Fix an issue where wildcard version constraints were not handled correctly ([#402](python-poetry/poetry-core#402)). - Fix an issue where `poetry build` created duplicate Python classifiers if they were specified manually ([#578](python-poetry/poetry-core#578)). - Fix an issue where local versions where not handled correctly ([#579](python-poetry/poetry-core#579)). </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).  Reviewed-on: https://git.walbeck.it/walbeck-it/docker-python-poetry/pulls/717 Co-authored-by: renovate-bot <bot@walbeck.it> Co-committed-by: renovate-bot <bot@walbeck.it>

radoering added 2 commits March 25, 2023 15:04

constraints: new method invert() for generic constraints

9d9e6f6

markers: avoid combinatorial explosion by replacing compatible MultiM…

e06ae6f

…arker/MarkerUnion with AtomicMultiMarker/AtomicMarkerUnion

neersighted approved these changes Mar 31, 2023

View reviewed changes

radoering merged commit ee2f92b into python-poetry:main Mar 31, 2023

This was referenced May 12, 2023

release: bump version to 1.6.0 #584

Merged

release: bump version to 1.5.0 python-poetry/poetry#7947

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

markers: avoid combinatorial explosion in certain cases #568

markers: avoid combinatorial explosion in certain cases #568

radoering commented Mar 25, 2023 •

edited

Loading

sonarcloud bot commented Mar 25, 2023

github-advanced-security bot commented Mar 25, 2023

markers: avoid combinatorial explosion in certain cases #568

markers: avoid combinatorial explosion in certain cases #568

Conversation

radoering commented Mar 25, 2023 • edited Loading

sonarcloud bot commented Mar 25, 2023

github-advanced-security bot commented Mar 25, 2023

radoering commented Mar 25, 2023 •

edited

Loading