Need to create a bean OtlpHttpSpanExporter before opentelemetry spring boot starter.

[DineshkumarVG]

Hi,
I am using oauth authentication for otel collector, The token needs to be changed one hour once. I need to set that dynamically. I have used your setHeader with supplier and it was working as expected. I am trying to implement that using @bean annotation. but springboot auto configuration injects the bean before the bean i have created. Anyone please help on this or is there anything i have missed?

It was working fine with the version 2.1.0-alpha

Sample code for reference.
build.gradle

`dependencyManagement {
imports {
mavenBom("io.opentelemetry:opentelemetry-bom:1.36.0")
mavenBom("io.opentelemetry.instrumentation:opentelemetry-instrumentation-bom-alpha:2.2.0-alpha")
}
}

implementation("io.opentelemetry.instrumentation:opentelemetry-spring-boot-starter")`

Bean File

`@AutoConfiguration
@ConditionalOnClass({OpenTelemetry.class})
@AutoConfigureOrder(Ordered.HIGHEST_PRECEDENCE)
public class RefreshToken {

@bean
public OtlpHttpSpanExporter otlpHttpSpanExporter() {
Supplier<Map<String, String>> mapSupplier = () -> {
Map<String, String> map = new HashMap<>();
try {
map.put("Authorization", "Bearer " + refreshToken());
} catch (Exception e) {
throw new RuntimeException(e);
}
return map;
};
return OtlpHttpSpanExporter.builder().setHeaders(mapSupplier)
.setEndpoint("url").build();
}
}`

[jack-berg]

Transferred to opentelemetry-java-instrumentation because this is related to the spring boot starter.

[jack-berg]

cc @jeanbisutti, @zeitlinger

[zeitlinger]

This breaking change is a consequence of #10453 - I didn't call this out as a breaking change, because I simply couldn't image a use case that would break 😞

Anyways, there's a way to get your setup running:

• You have to create a new exporter with a different name, e.g. "otlp-dynamic-header"
• You have to use "otlp-dynamic-header" instead of the default "otlp"
• You have to register a bean that supplies "otlp-dynamic-header" - similar to this integration test

[zeitlinger]

I'm keeping this issue open to document this use case.

[zeitlinger]

Note: it's also possible to use "otlp" as the key - it will replace the already configured otlp exporter

[DineshkumarVG]

@zeitlinger , Thanks for your input. Correct me if my understanding is wrong. GlobalOpenTelemetry will not set the configuration for second time. How can i make it happen?
Could you please provide the any sample code for reference to use otlp as the key?

[zeitlinger]

I've created a draft PR that demonstrates both: open-telemetry/opentelemetry-java-examples#378

Disclaimer: I haven't tested it manually.

[DineshkumarVG]

I will try it from my end the same you have shared as example code... @zeitlinger Thank you..

[DineshkumarVG]

Thank you @zeitlinger ,
I verified from my end and it was working fine. I simply used like the shared code in my Configuration class.

@Bean
ConfigurableSpanExporterProvider otlpSpanExporterProvider() {
        return new OtlpSpanExporterProvider();
}

 @Bean
 OpenTelemetryInjector registerGlobalOpenTelemetry() {
     return GlobalOpenTelemetry::set;
 }

private static class OtlpSpanExporterProvider
            implements ConfigurableSpanExporterProvider {

        @Override
        public String getName() {
            return "otlp";
        }

        @Override
        public SpanExporter createExporter(ConfigProperties config) {
            OtlpHttpSpanExporterBuilder builder = OtlpHttpSpanExporter.builder();
            Supplier<Map<String, String>> mapSupplier = () -> {
                Map<String, String> map = new HashMap<>();
                try {
                    map.put("AUTHORIZATION", "Bearer " + refreshToken());
                } catch (Exception e) {
                    throw new RuntimeException(e);
                }
                return map;
            };
            return builder.setHeaders(mapSupplier).build();
        }
		
		private String refreshToken(){
		return "token"
		}
    }

I removed AutoConfigureListener implementation but it was working fine . Is there any impact by removing AutoConfigureListener?

As per my understanding, we are just injecting the bean with dynamic header.

Could you please explain me more on this and how it work?

Is this changes will be the standard one in future also?

[zeitlinger]

Thank you @zeitlinger , I verified from my end and it was working fine. I didn't set any GlobalOpenTelemetry object. I simply used like the shared code in my Configuration class.

great 🎉

I removed AutoConfigureListener implementation but it was working fine . Is there any impact by removing AutoConfigureListener?

it creates metrics about the span export (i.e. if there were errors)

As per my understanding, we are just injecting the bean with dynamic header.

correct

Could you please explain me more on this and how it work?
Is this changes will be the standard one in future also?

I can't follow - can you make the questions more concrete or add an example?

[DineshkumarVG]

Please refer the code i just shared in the comment above, i just modified now and it was working as expected for my use case.

I need some clarification on this:

• I used your code and need to know how its working? In high level i understood that we are just injecting a bean but it would be more helpful if there is any explanation on this for more understanding.
  

• GlobalOpenTelemetry should not be set twice, but in that bean we are trying to set that. Is it okay to use like this?
  

• Can i use the shared solution as a standard one?
  

[zeitlinger]

I used your code and need to know how its working? In high level i understood that we are just injecting a bean but it would be more helpful if there is any explanation on this for more understanding.

Yes, you're injecting an exporter.

The exporter is loaded here: https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk-extensions/autoconfigure/src/main/java/io/opentelemetry/sdk/autoconfigure/SpanExporterConfiguration.java#L82-L86

There's an additional translation of spring beans to "SPI": https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/spring/spring-boot-autoconfigure/src/main/java/io/opentelemetry/instrumentation/spring/autoconfigure/OpenTelemetryAutoConfiguration.java#L140-L146

GlobalOpenTelemetry should not be set twice, but in that bean we are trying to set that. Is it okay to use like this?

You should to not to use GlobalOpenTelemetry at all (for that reason, I'll also remove that hint from the PR).

• the global causes problems in some cases that are hard to detect
• in spring, there's no reason to use a global. Just inject @Autowired OpenTelemetry

Can i use the shared solution as a standard one?

yes

[DineshkumarVG]

Thank you for your explanation @zeitlinger , As you recommened i just removed GlobalOpenTelemetry bean from my code.
Removed Code :

@Bean
 OpenTelemetryInjector registerGlobalOpenTelemetry() {
     return GlobalOpenTelemetry::set;
 }

As a result i can export the spans but at the same time i could able to see this info log in my application.

Apr 10, 2024 4:07:21 PM io.opentelemetry.api.GlobalOpenTelemetry maybeAutoConfigureAndSetGlobal
INFO: AutoConfiguredOpenTelemetrySdk found on classpath but automatic configuration is disabled. To enable, run your JVM with -Dotel.java.global-autoconfigure.enabled=true

[zeitlinger]

Ah, that means someone is trying to access the global.

Can you set a breakpoint and copy the stack trace of the caller? https://github.com/open-telemetry/opentelemetry-java/blob/cb44b2b18c474f80478184ff1665bc3fa5d2ced3/api/all/src/main/java/io/opentelemetry/api/GlobalOpenTelemetry.java#L236

[DineshkumarVG]

I dont see any stack trace there. globalOpenTelemetry is null while calling get() method, that's why i received this info log.
https://github.com/open-telemetry/opentelemetry-java/blob/cb44b2b18c474f80478184ff1665bc3fa5d2ced3/api/all/src/main/java/io/opentelemetry/api/GlobalOpenTelemetry.java#L73

[zeitlinger]

I mean, who is calling the get() method

[DineshkumarVG]

My springboot project was launched smoothly, once i try to export the spans by hitting api it was occurred.
The get() method was called by this method

  public static MeterProvider getMeterProvider() {
        return get().getMeterProvider();
    }

And this method is called by this line
https://github.com/open-telemetry/opentelemetry-java/blob/cb44b2b18c474f80478184ff1665bc3fa5d2ced3/exporters/common/src/main/java/io/opentelemetry/exporter/internal/ExporterMetrics.java#L89

[zeitlinger]

My springboot project was launched smoothly, once i try to export the spans by hitting api it was occurred. The get() method was called by this method

  public static MeterProvider getMeterProvider() {
        return get().getMeterProvider();
    }

And this method is called by this line https://github.com/open-telemetry/opentelemetry-java/blob/cb44b2b18c474f80478184ff1665bc3fa5d2ced3/exporters/common/src/main/java/io/opentelemetry/exporter/internal/ExporterMetrics.java#L89

Interesting!
I belive this is a result of removing "I removed AutoConfigureListener implementation but it was working fine . Is there any impact by removing AutoConfigureListener?"

[DineshkumarVG]

I tried by adding AutoConfigureListener again in my code, the same info log persist.

The current code..

@Bean
ConfigurableSpanExporterProvider otlpSpanExporterProvider() {
        return new OtlpSpanExporterProvider();
}

private static class OtlpSpanExporterProvider
            implements ConfigurableSpanExporterProvider, AutoConfigureListener {

        private final AtomicReference<MeterProvider> meterProviderRef =
                new AtomicReference<>(MeterProvider.noop());


        @Override
        public String getName() {
            return "otlp";
        }

        @Override
        public SpanExporter createExporter(ConfigProperties config) {
            OtlpHttpSpanExporterBuilder builder = OtlpHttpSpanExporter.builder();
            Supplier<Map<String, String>> mapSupplier = () -> {
                Map<String, String> map = new HashMap<>();
                try {
                    map.put("Authorization", "Bearer " + refreshToken());
                } catch (ObservabilityException e) {
                    throw new RuntimeException(e);
                }
                return map;
            };
            return builder.setHeaders(mapSupplier).build();
        }

        @Override
        public void afterAutoConfigure(OpenTelemetrySdk openTelemetrySdk) {
            meterProviderRef.set(openTelemetrySdk.getMeterProvider());
        }
		
		private String refreshToken(){
		return "token";
		}
    }

[zeitlinger]

Can you check the value of seenAttrs?

[DineshkumarVG]

Hey @zeitlinger I didn't set builder.setMeterProvider(meterProviderRef::get) in my application. now the log is gone.
Thanks for your support ....

[DineshkumarVG]

Hi @zeitlinger ,
Previously we are using otel java agent for instrumentation, currently we changed that to opentelemetry-springboot-starter dependency to achieve dynamic authentication. The dynamic authetication achieved but noticing that while using agent the span count is 8 for a single api , but using springboot dependency the span count is 1 (Controller level) for the same api. Is there anything i am missing or this is how the dependency behaves?

On other hand We are using microservice with 2 springboot application, Both application having otel springboot dependency with different app name, I am hitting a api and it lands on app1 and communicates with app2. While seeing the result we are seeing 2 different traces. app1 having trace1 and app2 having trace2 (i.e) nested traces is not happening. will you please share the thoughts on this, if possible.

My code :

dependencyManagement {
    imports {
        mavenBom("io.opentelemetry:opentelemetry-bom:1.36.0")
        mavenBom("io.opentelemetry.instrumentation:opentelemetry-instrumentation-bom-alpha:2.2.0-alpha")
    }
} 

implementation("io.opentelemetry.instrumentation:opentelemetry-spring-boot-starter")

The Env variables:

OTEL_EXPERIMENTAL_EXPORTER_OTLP_RETRY_ENABLED=true;
OTEL_EXPORTER_OTLP_ENDPOINT=url;
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf;
OTEL_LOGS_EXPORTER=none;
OTEL_METRICS_EXPORTER=otlp;
OTEL_PROPAGATORS=tracecontext;
OTEL_RESOURCE_ATTRIBUTE=environment=dev;
OTEL_SERVICE_NAME=app1;
OTEL_TRACES_EXPORTER=otlp;
OTEL_TRACES_SAMPLER=traceidratio;
OTEL_TRACES_SAMPLER_ARG=1

Note: The same configuration working for java agent. The web span, db span and custom spans are automatically created by the agent.

[zeitlinger]

currently we changed that to opentelemetry-springboot-starter dependency to achieve dynamic authentication.

You could also use an extension BTW

The dynamic authetication achieved but noticing that while using agent the span count is 8 for a single api , but using springboot dependency the span count is 1 (Controller level) for the same api. Is there anything i am missing or this is how the dependency behaves?

I guess it's because the starter doesn't support the same number of libraries out of the box - or because the spring starter is a bit more picky about how you use beans like RestTemplate (see #11054)

If you provide details about the spans, I can double check.

[DineshkumarVG]

Current span:

Previous span:

[zeitlinger]

please go to the span details and look for the "library name"

[DineshkumarVG]

Please refer here, If you see the image shared in the last message, you can see the stans between two different application.

[zeitlinger]

each of the entries in "previous span" (actually it's previous trace) should have such an entry. Can you add the library along with the span name for each?

[DineshkumarVG]

Can you please explain me how to do that? I don't have an idea on this..

[zeitlinger]

I don't know how to do that in your UI, but you can always turn on console logging - which was called "logging" in older releases.

[DineshkumarVG]

I am using otel to export metrics and traces and we set logs as none.
The library name of previous span is

The library name of current implementation (after migrating to springboot starter)

If i use @WithSpan on top of every method, I can get the spans but in my case, to make this change to existing app is quite difficult. Java agent instrumentation collecting spans without @Withspan but spring boot starter can't. Is there any dependency we are misssing to achieve that?

Since it was a very nice feature in java agent, even it produces spans for internal class files also, but that was missing in opentelemetry-springboot-starter dependency..

[DineshkumarVG]

@zeitlinger For your information,
While exploring otel java agent i could see these package inside opentelemetry/instrumentation/api folder

While exploring opentelemetry-instrumentation-api provided by springboot starter I could see the shared packages

Do you think is this because of this changes?

[zeitlinger]

@DineshkumarVG let's do a call to investigate the issue.

[DineshkumarVG]

Hi @zeitlinger , Sounds great.. Shall we connect through slack? Or any other medium are you comfortable with?

[zeitlinger]

CNCF slack sounds good 😄

[DineshkumarVG]

Hi @zeitlinger , when can i schedule a meet. Since i am having some login issue with slack CNCF. Planned to connect on zoom.
Are you okay with that? If yes we will share the meet link here and please give me a time of your availability..

[zeitlinger]

sure - I'm still pretty free tomorrow

[zeitlinger]

Here are the expected spans (from the call)

1: the server span.

The server span has a different library name - but the functionality is equivalent.

2: Different spans related to database queries. The spring starter needs to be set up to support DB queries: https://opentelemetry.io/docs/languages/java/automatic/spring-boot/#jdbc-instrumentation

2.1: spring data adds more details about the query - and is not supported in the starter

2.2: You should see the DB query once you follow the instructions (2).

2.3: Hibernate adds more details about the query - and is not supported in the starter

3: HTTP URL connection

HTTP URL connection is not supported in the starter - and I have no idea how to make it work outside a java agent.

Final thoughts:

• For any missing feature, feel free to create a ticket. The team might not have time, but can give you feedback if/how you can contribute missing features.
• The spring starter seems to be less verbose regarding the DB queries. In recent versions of the java agent, the default verbosity has been reduced to create less spans based on user feedback that it was too much.

[DineshkumarVG]

Sounds Great @zeitlinger , Thank you for dedicating your time and attention to this matter. Your engagement is greatly appreciated. 👏 . Will proceed next steps on this.

[zeitlinger]

OK, closing then - please re-open if needed.