Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

set memory limits for all services #460

Merged
merged 6 commits into from
Oct 18, 2022
Merged

set memory limits for all services #460

merged 6 commits into from
Oct 18, 2022

Conversation

puckpuck
Copy link
Contributor

@puckpuck puckpuck commented Oct 17, 2022 •
edited
Loading

Changes

Sets reasonable memory limits for all services, based on an 80% memory target for each service, running the app for 7 days (minimum of 20M).

Screen Shot 2022-10-17 at 2 18 37 PM

Data used by this PR incorporated the memory limits shown in the chart above which is a 7-day run on EKS. Some exceptions for loadgenerator, email, and quote services where I used data from docker stats after running the app on my local machine for several days.

  • loadgenerator - the above didn't incorporate the memory leak fix
  • email - I guess we have another small bug in the Helm chart since the email service wasn't listening on the proper port and I didn't notice :(, instead I used data from docker stats. I think we also have an issue with this service since it continues to eat memory erratically after running for several hours.
  • quote - I'm not sure why the memory just runs away, but if you put a limit on it this service would stay at 99% usage and be happy, so I picked 30M.
  • Appropriate CHANGELOG.md updated for non-trivial changes

@puckpuck puckpuck requested a review from a team as a code owner October 17, 2022 22:53
mviitane
mviitane approved these changes Oct 18, 2022
View reviewed changes
Copy link
Member

@mviitane mviitane left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@austinlparker austinlparker merged commit 616dd95 into open-telemetry:main Oct 18, 2022
@puckpuck puckpuck deleted the resource-limits branch October 19, 2022 01:00
@puckpuck puckpuck mentioned this pull request Oct 19, 2022
Merged
@styblope
Copy link
Contributor

feature-flag-service limit of 160M is too strict and causes the container to be Killed in my docker compose deploy. It may be environment specific, but a minimum limit I can get this working is 1600M.

@puckpuck
Copy link
Contributor Author

feature-flag-service limit of 160M is too strict and causes the container to be Killed in my docker compose deploy. It may be environment specific, but a minimum limit I can get this working is 1600M.

@styblope 1600M is definitely not normal. Which OS is this on?

@styblope
Copy link
Contributor

styblope commented Oct 20, 2022 •
edited
Loading

I know, it's weird, but I stepped up piece by piece until the container stopped getting killed. It's on x86_64 Fedora 34 VM (vsphere), 8GB RAM, Docker CE 20.10.17. Maybe it's a docker thing? Or a startup ram hike.

@styblope styblope mentioned this pull request Nov 8, 2022
Closed
jmichalak9 pushed a commit to jmichalak9/opentelemetry-demo that referenced this pull request Mar 22, 2024
@puckpuck @austinlparker
set memory limits for all services (open-telemetry#460)
3465030 
* set limits for all services

* memory limit for services

* re-add quoteservice to frontend dependency

* Update docker-compose.yml

Co-authored-by: Austin Parker <austin@ap2.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@austinlparker austinlparker austinlparker approved these changes

@mviitane mviitane mviitane approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@puckpuck @styblope @austinlparker @mviitane