Some Prometheus relabel configs don't work when target allocator is enabled

[swiatekm]

The following resource using a labeldrop relabel action in prometheus receiver:

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: relabel-test
spec:
  mode: statefulset
  targetAllocator:
    enabled: true
  config: |
    receivers:
      prometheus:
        config:
          scrape_configs:
            - job_name: 'otel-collector'
              scrape_interval: 5s
              static_configs:
                - targets: ['0.0.0.0:8888']
              relabel_configs:
                - action: labeldrop
                  regex: id
    processors:
    exporters:
      logging:

    service:
      pipelines:
        metrics:
          receivers: [prometheus]
          processors: []
          exporters: [logging]

gives the following receiver config in the ConfigMap:

 receivers:                                                                                                                                                                                                                                               
  prometheus:                                                                                                                                                                                                                                            
    config:                                                                                                                                                                                                                                              
      global:                                                                                                                                                                                                                                            
        scrape_interval: 1m                                                                                                                                                                                                                              
        scrape_timeout: 10s                                                                                                                                                                                                                              
        evaluation_interval: 1m                                                                                                                                                                                                                          
      scrape_configs:                                                                                                                                                                                                                                    
      - job_name: otel-collector                                                                                                                                                                                                                         
        honor_timestamps: true                                                                                                                                                                                                                           
        scrape_interval: 5s                                                                                                                                                                                                                              
        scrape_timeout: 5s                                                                                                                                                                                                                               
        metrics_path: /metrics                                                                                                                                                                                                                           
        scheme: http                                                                                                                                                                                                                                     
        follow_redirects: true                                                                                                                                                                                                                           
        enable_http2: true                                                                                                                                                                                                                               
        relabel_configs:                                                                                                                                                                                                                                 
        - separator: ;                                                                                                                                                                                                                                   
          regex: id
          replacement: $1                                                                                                                                                                                                                       
          action: labeldrop                                                                                                                                                                                                                              
        http_sd_configs:                                                                                                                                                                                                                                 
        - follow_redirects: false                                                                                                                                                                                                                        
          enable_http2: false                                                                                                                                                                                                                            
          url: http://relabel-test-targetallocator:80/jobs/otel-collector/targets?collector_id=$POD_NAME

and the following collector error:

error decoding 'receivers': error reading configuration for "prometheus": prometheus receiver failed to unmarshal yaml to prometheus config: labeldrop action requires only 'regex', and no other fields

The problem is that the default value for the replacement key is $1, which the collector treats as variable substitution, and resolves to an empty string. The fact that this causes an error here has to do with the specifics of how Prometheus validates the relabel configs - it checks for the existence of unnecessary fields by comparing them against defaults, but the underlying problem affects all relabel actions.

I think this can be fixed by simply unsetting the value in the config replace function if it's the default, and I'll submit a draft of the fix shortly.

[matej-g]

Hey @swiatekm-sumo, looks like this duplicates #958? It should be already worked on.

[swiatekm]

Thanks, closing as duplicate of #958.