Target Allocator not distributing load evenly on Collector pods

[RaviHari]

Component(s)

target allocator

What happened?

Target Allocator not distributing load evenly on Collector pods

Expected Result

Proper load balancing on otel collector pods.

Actual Result

Some otel collector pods getting higher usage:

Kubernetes Version

1.27.0

Operator version

0.97.1

Collector version

0.97.0

Environment information

No response

Log output

No response

Additional context

No response

[RaviHari]

Here is the Otel CRD:

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  labels:
    app.kubernetes.io/managed-by: opentelemetry-operator
  name: otelcol
  namespace: opentelemetry-operator-system
spec:
  autoscaler:
    behavior:
      scaleDown:
        stabilizationWindowSeconds: 15
      scaleUp:
        stabilizationWindowSeconds: 1
    maxReplicas: 10
    minReplicas: 2
    targetCPUUtilization: 30
  config: |
    receivers:
      prometheus:
        config:
          scrape_configs:[]
          
        target_allocator:
          endpoint: http://otelcol-targetallocator.opentelemetry-operator-system.svc.cluster.local
          interval: 30s
          collector_id: "${POD_NAME}"
    exporters:
      logging:
        verbosity: detailed
      prometheusremotewrite:
        endpoint: "http://<remotewrite-endpoint-url>"
        external_labels:
          label_name1: label_value1
          label_name2: label_value2
    service:
      pipelines:
        metrics:
          receivers: [prometheus]
          processors: []
          exporters: [logging, prometheusremotewrite]
  deploymentUpdateStrategy: {}
  ingress:
    route: {}
  managementState: managed
  maxReplicas: 10
  minReplicas: 2
  mode: statefulset
  observability:
    metrics: {}
  podDisruptionBudget:
    maxUnavailable: 1
  replicas: 9
  resources:
    limits:
      cpu: 300m
      memory: 1Gi
    requests:
      cpu: 50m
      memory: 400Mi
  targetAllocator:
    allocationStrategy: consistent-hashing
    enabled: true
    filterStrategy: relabel-config
    image: target-allocator:v0.97.1
    observability:
      metrics: {}
    podDisruptionBudget:
      maxUnavailable: 1
    prometheusCR:
      enabled: true
      scrapeInterval: 30s
    replicas: 1
    resources: {}
    serviceAccount: otelcol-collector
  updateStrategy: {}
  upgradeStrategy: automatic