feat: WAL Manager #13428
Merged
feat: WAL Manager #13428
+461
−0
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
benclive
reviewed
Jul 8, 2024
grobinson-grafana
force-pushed
the
grobinson/wal-manager
branch
from
d75680b
to
7e227c9
Compare
grobinson-grafana
force-pushed
the
grobinson/wal-manager
branch
6 times, most recently
from
afc79c5
to
d2fc460
Compare
This commit contains a WAL Manager that manages a pool of segments, and allows a configurable amount of data to be buffered in memory. More data allows for bursts of Append operations that exceed the rate at which segments can be flushed, but adds latency.
grobinson-grafana
force-pushed
the
grobinson/wal-manager
branch
from
d2fc460
to
095ed5d
Compare
cyriltovena
reviewed
Jul 10, 2024
benclive
reviewed
Jul 10, 2024
This commit changes the maxAge behavior from being checked using a fixed interval to when the flush loop gets the next segment to flush. This improves utilization, where as before we would have a lot of nearly empty segments.
benclive
approved these changes
Jul 11, 2024
pkg/storage/wal/manager.go
Outdated
| return it.r, nil | ||
| } | ||
|
|
||
| func (m *Manager) Get() (*PendingItem, error) { |
There was a problem hiding this comment.
mega nit: Maybe this should be called GetPending or something else?
API Clients only call Append but my mental model is that Get should return a valid WAL, not a closed one.
I don't feel strongly about it though.
There was a problem hiding this comment.
Yeah I don't have strong opinion on this either. I took the name Get and Put from sync.Pool.
There was a problem hiding this comment.
Let me think on this, because PutPending sounds weird and incorrect.
grobinson-grafana
force-pushed
the
grobinson/wal-manager
branch
from
09b9103
to
a629015
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
This commit contains a WAL Manager that manages a pool of segments, and allows a configurable amount of data to be buffered in memory. More data allows for bursts of Append operations that exceed the rate at which segments can be flushed, but adds latency.
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
Checklist
CONTRIBUTING.mdguide (required)featPRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.docs/sources/setup/upgrade/_index.mdproduction/helm/loki/Chart.yamland updateproduction/helm/loki/CHANGELOG.mdandproduction/helm/loki/README.md. Example PRdeprecated-config.yamlanddeleted-config.yamlfiles respectively in thetools/deprecated-config-checkerdirectory. Example PR