replace_instance_label does not take effect on custom scrape_configs

[mateuszdrab]

Hi guys

I've been wanting to utilize the feature replace_instance_label which is enabled by default; however, in my setup, the instance label is not replaced.

A very simple example is when trying to use a vscode exporter extension with the following sample config:

server:
  log_level: info

metrics:
  wal_directory: C:\ProgramData\grafana-agent\wal
  global:
    scrape_interval: 1m
    scrape_timeout: 1m
    remote_write:
      - url: x
        basic_auth:
          username: y
          password: z
  configs:
    - name: agent
      scrape_configs:
        - job_name: vscode
          static_configs:
            - targets:
                - 127.0.0.1:9931
              labels:
                job: vscode
integrations:
  windows_exporter:
    enabled: true
  agent:
    enabled: true

I'm aware that it seems the replace_instance_label feature is scoped to integrations but I'm not seeing a way to add a custom scrape target via integrations.

Am I missing something?

P.S. Is there such feature for logs? I'd like to use a file target job to collect files from a local system and send to loki; however, I don't see a way to inject the hostname into a label without using environment variable based labels which requires an extra agent flag which can't be easily enabled via the bundled agent installer for Windows.

Perhaps an agent global setting to replace any instance of 127.0.0.1/localhost with actual host name could be introduced?

Thanks!

[tpaschalis]

Hey there 👋

Just to make sure we're on the same page, the replace_instance_label field was deprecated back in v0.21; all integrations now provide their own InstanceKey which tries to be descriptive of what the integrations is observing and cannot be overriden.

For both metrics and logs, the behavior around the instance label is inherited from the upstream Prometheus code which sets it from the contents of the __address__ label, and is either the remote address of the endpoint it's scraping, or the place that it's reading its logs from.

If what you're trying to achieve is to replace the instance label (or any other label), you can only do it via relabel_configs.
There you can either hardcode the replaced value in the YAML, make use of another existing label value (eg. from a Service Discovery mechanism) or make use of an environment variable, which I know is unwieldy but unfortunately there isn't a way to dynamically replace it with the hostname as it would break our Prometheus compatibility.

[tpaschalis]

The other alternative would be to make use of Flow mode which exposes the hostname as constants.hostname.

We don't have the tooling for automatically translating from YAML -> River config files yet, but if your config is not terribly big, making the transition shouldn't be too hard!

[mateuszdrab]

The other alternative would be to make use of Flow mode which exposes the hostname as constants.hostname.

We don't have the tooling for automatically translating from YAML -> River config files yet, but if your config is not terribly big, making the transition shouldn't be too hard!

I think that's the best way to go, as much as I love my config in yaml... It's a shame there is no such option in the yaml based config without using the environment variable.

The config itself is pretty basic, at least the file that gets centrally distributed... Snowflake installations are manually configured as each has different needs or by Ansible and I just inject the right label there but the ultimate goal is to have the config from one central place.

Thanks for your comments

[rmarops]

how are you setting via env var? I am troubleshooting a issue where var is unset

I am setting the env from systemd service, tried 2 different ways (daemon-reload and service restart after each change)

Service]
#Environment="HOSTNAME=%H"
Environment="HOSTNAME=builder-host"

grafana agent excerpt

          relabel_configs:
            - source_labels: [__address__]
              target_label: instance
              #replacement: "${HOSTNAME}:12345"
              replacement: ${HOSTNAME}:12345

example:
process_resident_memory_bytes{instance=":12345", job="agent"}

[rmarops]

coincidentally just saw there is a -config.expand-env option

[mateuszdrab]

There is, but there was no way of adding it through the installer and it would require registry key modification though gpo or an install time script.
I was expecting built in label this label.