fix issues related to prometheus relabel configs when target allocator is enabled

[anunarapureddy]

Related to following issues: #1623, #1622, #958

Cause:

The issue manifests itself in two scenarios.

1. First, when Prometheus adds a default value of the replacement key with $1, the collector interprets it as an environment variable substitution and resolves it to an empty string. When the configMap is built by the operator, the unmarshalling logic calls the Prometheus relabel config validation logic (relabel.go), which includes the additional defaults, including the $1 replacement field, appended to the original Prometheus configuration. Upon initialization, the collector pod loads the configMap and interprets $1 as an environment variable, leading to its replacement with an empty string. As a result, the same Prometheus relabel config validation logic fails when the config is unmarshalled in the collector code, as the replacement key is now replaced with an empty string due to env var substitution.

2. Second, when Prometheus configuration explicitly contains $$ for labelmap, labeldrop, labelkeep or other Prometheus actions that undergo regex validation in the prometheus validation code (relabel.go), the validation of the regex will fail.

Resolution:

1. For the first scenario, the solution involved explicitly escaping the default '$' character in the replacement key while building the configMap. This was done by replacing all occurrences of '$' with ‘$$’. Upon initialization, the collector pod loads the ConfigMap and resolves ‘$$’ as ‘$’, thus ensuring that the metric relabel config validation in Prometheus passes successfully when the configuration is unmarshalled in the collector code.

2. For the second scenario, the approach involved replacing all occurrences of '$$' with the string 'DOUBLE_DOLLAR' and '$' with the string 'SINGLE_DOLLAR' right before unmarshalling the Prometheus config while creating the configMap. After unmarshalling, the values were reset to their original values. This solution ensured that Prometheus regex validation succeeds.

[linux-foundation-easycla]

• ✅ login: anunarapureddy / name: Anusha Reddy Narapureddy (44d8b6b)
• ❌ The commit (c08d4e5). This user is missing the User's ID, preventing the EasyCLA check. Consult GitHub Help to resolve.For further assistance with EasyCLA, please submit a support request ticket.