[extension/storage/filestorage] Add CleanupOnStart flag for compaction temporary files #32863
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
djaglowski
reviewed
May 8, 2024
Co-authored-by: Daniel Jaglowski <jaglows3@gmail.com>
djaglowski
approved these changes
May 13, 2024
Co-authored-by: Daniel Jaglowski <jaglows3@gmail.com>
djaglowski
reviewed
May 14, 2024
Each extension instance can create multiple clients. Moving the logic to extension start is better aligned with the extension lifecycle and prevent potential collisions, all clients will share the same compaction folder already cleaned up.
djaglowski
reviewed
May 15, 2024
cparkins
pushed a commit
to AmadeusITGroup/opentelemetry-collector-contrib
that referenced
this pull request
Jul 11, 2024
…n temporary files (open-telemetry#32863) **Description:** This PR includes a new flag **cleanup_on_start** for the compaction section. During compaction a copy of the database is created, when the process is unexpectedly terminated that temporary file is not removed. That could lead to disk exhaustion given the following scenario: - Process is killed with a big database to be compacted - Compaction is enabled on start - Process will take longer to compact than the allotted time for the collector to reply health checks (see: open-telemetry#32456) - Process is killed while compacting - Big temporary file left This mitigates the potential risk of those temporary files left in a short period of time, by this scenario or similar ones. **Testing:** Included corner case where two instances of the extensions are spawned and one is compacting while the other would attempt to cleanup. **Documentation:** Included description in the README of the new configuration flag --------- Co-authored-by: Daniel Jaglowski <jaglows3@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
This PR includes a new flag cleanup_on_start for the compaction section.
During compaction a copy of the database is created, when the process is unexpectedly terminated that temporary file is not removed. That could lead to disk exhaustion given the following scenario:
This mitigates the potential risk of those temporary files left in a short period of time, by this scenario or similar ones.
Testing:
Included corner case where two instances of the extensions are spawned and one is compacting while the other would attempt to cleanup.
Documentation:
Included description in the README of the new configuration flag