[confmap] Add featuregate to use stable expansion rules #10259

TylerHelmuth · 2024-05-29T23:31:22Z

Description

Add a feature gate that provides access to stable expansion rules. When enabled:

env is used as the default scheme
expandconverter is not used, which means $ENV syntax is no longer expanded

Link to tracking issue

Related to #10161
Related to #8215
Related to #7111

Testing

Tested manually locally

codecov · 2024-05-30T17:42:51Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.53%. Comparing base (7dfb57b) to head (a123337).

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #10259   +/-   ##
=======================================
  Coverage   92.53%   92.53%           
=======================================
  Files         388      388           
  Lines       18317    18317           
=======================================
  Hits        16949    16949           
  Misses       1022     1022           
  Partials      346      346

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

TylerHelmuth · 2024-05-30T19:22:44Z

cmd/builder/internal/builder/templates/main.go.tmpl

+    {{ if .Distribution.SupportsConfmapFactories }}
+	if confmap.UseStableExpansionRules.IsEnabled() {
+		set.ConfigProviderSettings.ResolverSettings.DefaultScheme = "env"
+	} else {
+		set.ConfigProviderSettings.ResolverSettings.ConverterFactories = []confmap.ConverterFactory{
+			expandconverter.NewFactory(),
+		}
+	}
+    {{- end}}


@mx-psi @evan-bradley this solution has the downside that it will only apply to people that use ocb. Is that ok?

Does it need to live in otelcol, like in NewCommand, and be allowed to mess with the incoming settings, ignoring any configured expandconverter?

If possible, this is the process I think would work best:

Implement the feature gate in NewCommand/NewCollector/etc. with what you have here.

Turn the feature gate on by default.

Allow configuring the default scheme in an ocb manifest.

Remove the feature gate and the expand converter from ocb in one go, since there's no way to configure converters in the ocb manifest right now.

If the two behaviors can coexist, I don't think we should remove a configured expand converter.

Thinking about this again I propose a alternative:

Since this is a change to how CollectorSettings is configured, we handle the transition for only our own usage (ocb) - anyone using NewCommand is responsible for updating their own CollectorSettings once expandconverter is deprecated.

This approach would mean moving the featuregate into the main.go generated by ocb, deprecating expandconverter and then as part of that deprecation following a standard feature gate process.

This approach saves us the trouble of guessing how other users of NewCommand want to handle the deprecation of the expandconverter while at the same time enforcing our own Provider expectations established in the rfc.

We would first have to handle this TODO first, where we start requiring incoming CollectorSettings to have the providers/converts supplied, making NewCommand more dependent on a full set of settings when invoked.

@evan-bradley @mx-psi @codeboten I am still working towards this PR but trying to unblock/simplify it by working on #10290 first.

Thinking about this some more, I believe by moving the feature gate to builder and only controlling our own use of expandconverter allows this PR and #10359 to happen in parallel.

codeboten

I'd suggest moving the feature gate to a different package to prevent all the imports this is causing

codeboten

Lint fails:

/home/runner/work/opentelemetry-collector/opentelemetry-collector/.tools/golangci-lint run
Error: configprovider.go:145:51: SA1019: expandconverter.NewFactory is deprecated: [v0.102.0] Expanding $ENV is no longer supported. To expand ${} using env vars use DefaultScheme="env" or the otelcol.useStableExpansionRules feature gate. (staticcheck)
			ConverterFactories: []confmap.ConverterFactory{expandconverter.NewFactory()},
			                                               ^
Error: command_test.go:57:52: SA1019: expandconverter.NewFactory is deprecated: [v0.102.0] Expanding $ENV is no longer supported. To expand ${} using env vars use DefaultScheme="env" or the otelcol.useStableExpansionRules feature gate. (staticcheck)
				ConverterFactories: []confmap.ConverterFactory{expandconverter.NewFactory()},
				                                               ^
Error: command_test.go:91:52: SA1019: expandconverter.NewFactory is deprecated: [v0.102.0] Expanding $ENV is no longer supported. To expand ${} using env vars use DefaultScheme="env" or the otelcol.useStableExpansionRules feature gate. (staticcheck)

TylerHelmuth · 2024-05-31T17:42:57Z

I believe the actual deprecation depends on #10281

#### Description Allows configuring `DefaultScheme` via the builder config. #### Link to tracking issue Related to #10259 Related to #10290 #### Testing Local testing and unit tests --------- Co-authored-by: Evan Bradley <11745660+evan-bradley@users.noreply.github.com> Co-authored-by: Alex Boten <223565+codeboten@users.noreply.github.com>

mx-psi

I think it would be good to add some config provider tests that show what the change in behavior is before and after enabling the feature gate

.chloggen/confmap-add-expand-featuregate-2.yaml

.chloggen/confmap-add-expand-featuregate.yaml

TylerHelmuth · 2024-06-06T22:33:37Z

This PR is currently under the assumption that we'll stop providing default providers and converters in NewCommand. If we decide to keep doing that, I'll open a new PR that keeps the feature gate in otelcol and does the same logic as the builder when setting defaults.

TylerHelmuth · 2024-06-07T00:00:55Z

Created #10361 as an alternative

.chloggen/confmap-add-expand-featuregate.yaml

cmd/builder/internal/builder/templates/main.go.tmpl

TylerHelmuth · 2024-06-11T19:01:27Z

So it turns out featuregates don't work in main.go because it happens before flags are read in. @codeboten and I found this while writing an integration test.

So I need to rework this PR to do the manipulation inside otelcol like I originally had it 😢

…featuregate

TylerHelmuth · 2024-06-12T01:26:32Z

This solution doesn't work, need #10391 instead

#### Description This PR adds a feature gate that will handle transitioning users away from expandconverter, specifically expanding `$VAR` syntax. The wholistic strategy is: 1. create a new feature gate, `confmap.unifyEnvVarExpansion`, that will be the single feature gate to manage unifying collector configuraiton resolution. 2. Update expandconverter to return an error if the feature gate is enabled and it is about to expand `$VAR` syntax. 3. Update `otelcol.NewCommand` to set a `DefaultScheme="env"` when the feature gate is enabled and no `DefaultScheme` is set, this handles `${VAR}` syntax. 4. Separately, deprecate `expandconverter`. 5. Follow a normal feature gate cycle. 6. Removed the `confmap.unifyEnvVarExpansion` feature gates and `expandconverter` at the same time Supersedes #10259 #### Link to tracking issue Related to #10161 Related to #8215 Related to #7111 #### Testing Unit tests

TylerHelmuth force-pushed the confmap-add-expand-featuregate branch from 399d98f to 26a6d75 Compare May 30, 2024 17:32

TylerHelmuth marked this pull request as ready for review May 30, 2024 17:49

TylerHelmuth requested a review from a team as a code owner May 30, 2024 17:49

TylerHelmuth requested review from codeboten, mx-psi and evan-bradley May 30, 2024 17:49

TylerHelmuth commented May 30, 2024

View reviewed changes

codeboten reviewed May 30, 2024

View reviewed changes

Add featuregate to use stable expansion rules

50bf50a

TylerHelmuth force-pushed the confmap-add-expand-featuregate branch 2 times, most recently from 0e8fb5d to 36431dc Compare May 31, 2024 14:32

move featuregate to otelcol

e8a88ff

TylerHelmuth force-pushed the confmap-add-expand-featuregate branch from 36431dc to e8a88ff Compare May 31, 2024 14:36

TylerHelmuth added 2 commits May 31, 2024 10:57

Deprecate expandconverter

92c4021

changelog

14904b8

codeboten approved these changes May 31, 2024

View reviewed changes

codeboten reviewed May 31, 2024

View reviewed changes

Merge branch 'main' into confmap-add-expand-featuregate

ad2a310

TylerHelmuth mentioned this pull request Jun 3, 2024

[otelcol] Determine if NewCommand should supply any default providers/converters #10290

Closed

update defaults to use gate

d293aaa

TylerHelmuth mentioned this pull request Jun 3, 2024

[cmd/builder] Allow setting DefaultScheme in builder config #10296

Merged

mx-psi reviewed Jun 5, 2024

View reviewed changes

.chloggen/confmap-add-expand-featuregate-2.yaml Outdated Show resolved Hide resolved

.chloggen/confmap-add-expand-featuregate.yaml Outdated Show resolved Hide resolved

TylerHelmuth added 5 commits June 6, 2024 15:13

Move featuregate to builder

7984bf8

Update changelogs

72a5b05

merge in upstream/main

325ecdd

fix imports

46fd3d1

Deprecate expandconverter later

cd066fd

TylerHelmuth mentioned this pull request Jun 7, 2024

[otelcol] Add featuregate to use stable expansion rules #10361

Closed

Update feature gate name and desc

a215b40

TylerHelmuth requested a review from mx-psi June 10, 2024 20:12

mx-psi approved these changes Jun 11, 2024

View reviewed changes

.chloggen/confmap-add-expand-featuregate.yaml Outdated Show resolved Hide resolved

cmd/builder/internal/builder/templates/main.go.tmpl Outdated Show resolved Hide resolved

evan-bradley reviewed Jun 11, 2024

View reviewed changes

cmd/builder/internal/builder/templates/main.go.tmpl Outdated Show resolved Hide resolved

TylerHelmuth added 2 commits June 11, 2024 12:05

Update gate name

fbe5dc1

Fix issue where converter would be added is no defaultscheme was set

f7a1731

Merge remote-tracking branch 'upstream/main' into confmap-add-expand-…

a123337

…featuregate

TylerHelmuth mentioned this pull request Jun 12, 2024

[confmap] Add featuregate to use stable expansion rules #10391

Merged

TylerHelmuth closed this Jun 12, 2024

TylerHelmuth deleted the confmap-add-expand-featuregate branch June 12, 2024 01:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[confmap] Add featuregate to use stable expansion rules #10259

[confmap] Add featuregate to use stable expansion rules #10259

TylerHelmuth commented May 29, 2024

codecov bot commented May 30, 2024 •

edited

Loading

TylerHelmuth May 30, 2024

evan-bradley May 31, 2024

evan-bradley May 31, 2024

TylerHelmuth May 31, 2024 •

edited

Loading

TylerHelmuth May 31, 2024

TylerHelmuth Jun 6, 2024

TylerHelmuth Jun 6, 2024

codeboten left a comment

codeboten left a comment

TylerHelmuth commented May 31, 2024

mx-psi left a comment •

edited

Loading

TylerHelmuth commented Jun 6, 2024

TylerHelmuth commented Jun 7, 2024

TylerHelmuth commented Jun 11, 2024

TylerHelmuth commented Jun 12, 2024

[confmap] Add featuregate to use stable expansion rules #10259

[confmap] Add featuregate to use stable expansion rules #10259

Conversation

TylerHelmuth commented May 29, 2024

Description

Link to tracking issue

Testing

codecov bot commented May 30, 2024 • edited Loading

Codecov Report

TylerHelmuth May 30, 2024

Choose a reason for hiding this comment

evan-bradley May 31, 2024

Choose a reason for hiding this comment

evan-bradley May 31, 2024

Choose a reason for hiding this comment

TylerHelmuth May 31, 2024 • edited Loading

Choose a reason for hiding this comment

TylerHelmuth May 31, 2024

Choose a reason for hiding this comment

TylerHelmuth Jun 6, 2024

Choose a reason for hiding this comment

TylerHelmuth Jun 6, 2024

Choose a reason for hiding this comment

codeboten left a comment

Choose a reason for hiding this comment

codeboten left a comment

Choose a reason for hiding this comment

TylerHelmuth commented May 31, 2024

mx-psi left a comment • edited Loading

Choose a reason for hiding this comment

TylerHelmuth commented Jun 6, 2024

TylerHelmuth commented Jun 7, 2024

TylerHelmuth commented Jun 11, 2024

TylerHelmuth commented Jun 12, 2024

codecov bot commented May 30, 2024 •

edited

Loading

TylerHelmuth May 31, 2024 •

edited

Loading

mx-psi left a comment •

edited

Loading