FeatureFlagService keep restarting

[glongzh]

Bug Report

Which version of the demo you are using?

IMAGE_VERSION=v1.0.0

Symptom

when deploy with docker-compose, featureflagservice keeps restarting status:

NAME                      COMMAND                  SERVICE                 STATUS              PORTS
ad-service                "./build/install/hip…"   adservice               running             0.0.0.0:32862->9555/tcp
cart-service              "./cartservice"          cartservice             running             0.0.0.0:32861->7070/tcp
checkout-service          "./checkoutservice"      checkoutservice         running             0.0.0.0:32867->5050/tcp
currency-service          "/bin/sh -c '/usr/lo…"   currencyservice         running             0.0.0.0:32864->7001/tcp
email-service             "bundle exec ruby em…"   emailservice            running             0.0.0.0:32860->6060/tcp
feature-flag-service      "/app/bin/server"        featureflagservice      restarting
frontend                  "/bin/sh -c 'npm sta…"   frontend                running             0.0.0.0:32871->8080/tcp
frontend-proxy            "/bin/sh -c 'envsubs…"   frontendproxy           running             0.0.0.0:8080->8080/tcp, 0.0.0.0:10000->10000/tcp
grafana                   "/run.sh"                grafana                 running             0.0.0.0:32852->3000/tcp
jaeger                    "/go/bin/all-in-one-…"   jaeger                  running             5775/udp, 5778/tcp, 14250/tcp, 6831-6832/udp, 14268/tcp, 0.0.0.0:32854->4317/tcp, 0.0.0.0:32853->16686/tcp
load-generator            "/bin/sh -c locust"      loadgenerator           running             0.0.0.0:32872->8089/tcp
otel-col                  "/otelcol-contrib --…"   otelcol                 running             0.0.0.0:4318->4318/tcp, 55678-55679/tcp, 0.0.0.0:32897->4317/tcp, 0.0.0.0:32896->8888/tcp, 0.0.0.0:32895->9464/tcp
payment-service           "node --require ./tr…"   paymentservice          running             0.0.0.0:32865->50051/tcp
postgres                  "docker-entrypoint.s…"   ffs_postgres            running (healthy)   5432/tcp
product-catalog-service   "./productcatalogser…"   productcatalogservice   running             0.0.0.0:32863->3550/tcp
prometheus                "/opt/bitnami/promet…"   prometheus              running             0.0.0.0:9090->9090/tcp
quoteservice              "/bin/sh -c 'php -S …"   quoteservice            running             0.0.0.0:32866->8090/tcp
recommendation-service    "opentelemetry-instr…"   recommendationservice   running             0.0.0.0:32870->9001/tcp
redis-cart                "docker-entrypoint.s…"   redis-cart              running             0.0.0.0:32855->6379/tcp
shipping-service          "/app/shippingservice"   shippingservice         running             0.0.0.0:32859->50050/tcp

and the service log:

feature-flag-service  | 06:47:36.440 [info] == Running 20220524172636 Featureflagservice.Repo.Migrations.CreateFeatureflags.change/0 forward
feature-flag-service  | 06:47:36.444 [info] create table featureflags
feature-flag-service  | 06:47:36.454 [info] create index featureflags_name_index
feature-flag-service  | 06:47:36.502 [info] == Migrated 20220524172636 in 0.0s
feature-flag-service  | 06:47:39.701 [info] Migrations already up
feature-flag-service  | 06:47:42.681 [info] Migrations already up
feature-flag-service  | 06:47:45.630 [info] Migrations already up
feature-flag-service  | 06:47:48.823 [info] Migrations already up
feature-flag-service  | 06:47:52.868 [info] Migrations already up
feature-flag-service  | {erl_prim_loader,file_error}
feature-flag-service  | "File operation error: enomem. Target: /app/lib/ecto-3.8.3/ebin/Elixir.Ecto.Repo.Schema.beam. Function: get_modules. "
feature-flag-service  | {erl_prim_loader,file_error}
feature-flag-service  | "File operation error: enomem. Target: /app/lib/ecto-3.8.3/ebin/Elixir.Ecto.Repo.Assoc.beam. Function: get_modules. "
feature-flag-service  | 06:47:58.419 [info] Migrations already up
feature-flag-service  | 06:48:07.318 [info] Migrations already up
feature-flag-service  | 06:48:22.358 [info] Migrations already up
feature-flag-service  | 06:48:50.496 [info] Migrations already up
feature-flag-service  | 06:49:44.076 [info] Migrations already up
feature-flag-service  | 06:50:46.628 [info] Migrations already up
feature-flag-service  | 06:51:49.177 [info] Migrations already up

What is the expected behavior?

featureflagservice keeps runing

Reproduce

docker compose up --no-build

Additional Context

OS: SUSE Linux Enterprise Server 12 SP5
Kernel: 4.12.14
Docker: 19.03.12
Docker Compose: v2.12.2

[austinlparker]

Out of curiosity, what happens if you build the feature flag service (run docker compose build featureflagservice) then docker compose up? Also, what's the architecture of your machine -- x86 or ARM?

[glongzh]

It took a long time to build the image and was unsuccessful due to network connectivity problems.
FYI, my machine's arch is X86.

[austinlparker]

The error message makes me think it's a memory issue -- could you try increasing the amount of memory available for the feature flag service in the compose file and see if that resolves things?

[glongzh]

After increasing the memory limit from 160M to 500M, and restarting the featureflagservice container, the same problem still exists

[fatsheep9146]

what's the last log before restarting?

[styblope]

This might be related to my earlier comment: #460 (comment). I now run the service with no limits set.

[glongzh]

Oh, it's my fault. After increasing the memory limit, you need to run docker-compose up -d to recreate the featureflagservice and the depends_on services. Not just to restart the featureflagservice container.

[cartersocha]

@puckpuck we may want to increase our memory limits for the feature flag service.

I’m not sure if 500M is needed but seems reasonable to move it up then we can close this

[cartersocha]

I can submit a pr tomorrow if we have a new number in mind besides 500. Or we can discuss briefly on Monday

[cartersocha]

@glongzh would you mind checking lower memory allocations to gauge the needed level?

[glongzh]

@cartersocha
I tried 200, still working. Maybe 200 is enough.
Here's the docker stats output in two cases:

dfc94a863969        feature-flag-service      4.27%               157MiB / 500MiB       31.40%              153MB / 288MB       0B / 0B             85

cd96c311eda4        feature-flag-service      3.12%               139.9MiB / 200MiB     69.96%              126kB / 153kB       0B / 0B             85

[cartersocha]

ty @glongzh !