Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support retries on rate limited batches #192

Closed
acm19 opened this issue Nov 16, 2023 · 2 comments · Fixed by #193
Closed

Support retries on rate limited batches #192

acm19 opened this issue Nov 16, 2023 · 2 comments · Fixed by #193
Labels
enhancement New feature or request

Comments

@acm19
Copy link
Contributor

acm19 commented Nov 16, 2023

Allow to retry on HTTP 429, currently retries AsyncBufferPipeline#checkIfEligibleForRetry are performed on connection errors or HTTP 503. It'd be great to be able to retry when we get rate limited. This would be similar to drop_rate_limited_batches in Promtail.

Also, are you open to contributions?
@nehaev
Copy link
Contributor

nehaev commented Nov 17, 2023

Hi @acm19, thanks for reporting this!

I guess the support for retry on 429 should go in conjunction with adding a config param similar to drop_rate_limited_batches, which in our case should be true by default.

Yes, contributions are welcome. Just leave a note in this issue that you are going to implement a solution by yourself.

@nehaev nehaev added the enhancement New feature or request label Nov 17, 2023
acm19 pushed a commit to acm19/loki-logback-appender that referenced this issue Nov 23, 2023
@acocad
Add support for retries on 429 rate limited requests
3436cdd 
Adds an option to enable/disable retries on rate limit errors:
`dropRateLimitedBatches`. It's enabled by default to keep consistency
with Promtail options. Also because Loki scales less aggressively and
relies on retries when the load is spiky.

Resolves: loki4j#192
@acm19 acm19 mentioned this issue Nov 23, 2023
Merged
acm19 added a commit to acm19/loki-logback-appender that referenced this issue Nov 23, 2023
@acm19 @acocad
Add support for retries on 429 rate limited requests
26e3f09 
Adds an option to enable/disable retries on rate limit errors:
`dropRateLimitedBatches`. It's enabled by default to keep consistency
with Promtail options. Also because Loki scales less aggressively and
relies on retries when the load is spiky.

Resolves: loki4j#192
@acm19
Copy link
Contributor Author

acm19 commented Nov 23, 2023

Hey @nehaev , I've created a PR, let me know your thoughts.

acm19 added a commit to acm19/loki-logback-appender that referenced this issue Nov 24, 2023
@acm19 @acocad
Add support for retries on 429 rate limited requests
d851c91 
Adds an option to enable/disable retries on rate limit errors:
`dropRateLimitedBatches`. It's enabled by default to keep consistency
with Promtail options. Also because Loki scales less aggressively and
relies on retries when the load is spiky.

Resolves: loki4j#192
acm19 added a commit to acm19/loki-logback-appender that referenced this issue Nov 24, 2023
@acm19
Add support for retries on 429 rate limited requests
6396c9d 
Adds an option to enable/disable retries on rate limit errors:
`dropRateLimitedBatches`. It's enabled by default to keep consistency
with Promtail options. Also because Loki scales less aggressively and
relies on retries when the load is spiky.

Resolves: loki4j#192
acm19 added a commit to acm19/loki-logback-appender that referenced this issue Nov 27, 2023
@acm19
Add support for retries on 429 rate limited requests
1c7ac26 
Adds an option to enable/disable retries on rate limit errors:
`dropRateLimitedBatches`. It's enabled by default to keep consistency
with Promtail options. Also because Loki scales less aggressively and
relies on retries when the load is spiky.

Resolves: loki4j#192
acm19 added a commit to acm19/loki-logback-appender that referenced this issue Nov 28, 2023
@acm19
Add support for retries on 429 rate limited requests
1b2d791 
Adds an option to enable/disable retries on rate limit errors:
`dropRateLimitedBatches`. It's enabled by default to keep consistency
with Promtail options. Also because Loki scales less aggressively and
relies on retries when the load is spiky.

Resolves: loki4j#192
nehaev pushed a commit that referenced this issue Nov 30, 2023
@acm19
Add support for retries on 429 rate limited requests (#193)
b66ec83 
Adds an option to enable/disable retries on rate limit errors:
`dropRateLimitedBatches`. It's enabled by default to keep consistency
with Promtail options. Also because Loki scales less aggressively and
relies on retries when the load is spiky.

Resolves: #192
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add support for retries on 429 rate limited requests acm19/loki-logback-appender
2 participants
@nehaev @acm19