failed to select an OpenTelemetry Instrumentation instance for this pod

[lifeofmoo]

Hello,

Old woes and resolutions have been mentioned here

I've recently refactored (k8s manifests) and enabled auto-instrumentation for a few apps in a NS but i'm seeing this error in my otel operator logs

(k logs -n opentelemetry-operator-system opentelemetry-operator-98f5b9c89-55q4z)

{"level":"error","ts":"2024-06-27T14:43:48Z","msg":"failed to select an OpenTelemetry Instrumentation instance for this pod","namespace":"my-namespace","name":"notifications-api-865999bd6b-nspcw","error":"multiple OpenTelemetry Instrumentation instances available, cannot determine which one to select","stacktrace":"github.com/open-telemetry/opentelemetry-operator/pkg/instrumentation.(*instPodMutator).Mutate\n\t/home/runner/work/aws-otel-operator/aws-otel-operator/pkg/instrumentation/podmutator.go:226\ngithub.com/open-telemetry/opentelemetry-operator/internal/webhook/podmutation.(*podMutationWebhook).Handle\n\t/home/runner/work/aws-otel-operator/aws-otel-operator/internal/webhook/podmutation/webhookhandler.go:92\nsigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).Handle\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.1/pkg/webhook/admission/webhook.go:169\nsigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.1/pkg/webhook/admission/http.go:119\nsigs.k8s.io/controller-runtime/pkg/webhook/internal/metrics.InstrumentedHook.InstrumentHandlerInFlight.func1\n\t/home/runner/go/pkg/mod/github.com/prometheus/client_golang@v1.18.0/prometheus/promhttp/instrument_server.go:60\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.21.8/x64/src/net/http/server.go:2136\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1\n\t/home/runner/go/pkg/mod/github.com/prometheus/client_golang@v1.18.0/prometheus/promhttp/instrument_server.go:147\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.21.8/x64/src/net/http/server.go:2136\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2\n\t/home/runner/go/pkg/mod/github.com/prometheus/client_golang@v1.18.0/prometheus/promhttp/instrument_server.go:109\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.21.8/x64/src/net/http/server.go:2136\nnet/http.(*ServeMux).ServeHTTP\n\t/opt/hostedtoolcache/go/1.21.8/x64/src/net/http/server.go:2514\nnet/http.serverHandler.ServeHTTP\n\t/opt/hostedtoolcache/go/1.21.8/x64/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/opt/hostedtoolcache/go/1.21.8/x64/src/net/http/server.go:2009"}

I don't think i've enabled java instumention on the NS as per this Stack-overflow answer

k describe ns my-namespace 
Name:         my-namespace
Labels:       app.kubernetes.io/instance=namespaces
              elbv2.k8s.aws/pod-readiness-gate-inject=enabled
              kubernetes.io/metadata.name=my-namespace
              name=my-namespace
Annotations:  <none>
Status:       Active

No resource quota.

No LimitRange resource.

EKS: v1.28
Otel ADOT: v0.94.1-eksbuild.1