io.opentelemetry.extension.trace.propagation.B3 Propagator ExtractorMultipleHeaders - Invalid TraceId in B3 header:null, Returning INVALID span context

[eottabom]

Describe the bug

It is used as a combination of armeria and spring,
Enable and use option to change REST request to gRPC in armeria.
This problem occurs when you enable the otel java agent debug log,
io.opentelemetry.extension.trace.propagation.B3 Propagator ExtractorMultipleHeaders - Invalid TraceId in B3 header:null, Returning INVALID span context
Is there a phenomenon like memory leak in this case?

Steps to reproduce

Change REST request to gRPC using armeria httpJson Transcoder
gRPC <-> gRPC

Expected behavior

I hope there is no memory leak.

Actual behavior

io.opentelemetry.extension.trace.propagation.B3 Propagator ExtractorMultipleHeades - Invalid TraceId in B3 header:null, Returning INVALID span context

Javaagent or library instrumentation version

1.32.0

Environment

JDK: 11.0.7
Spring Boot: 2.7.18
armeria: 1.20.3

Additional context

No response

[trask]

hi @eottabom! is this a duplicate of #11618?

[eottabom]

Well, it would be nice if you could understand that it's similar, but a little different.

At first, I was going to ask you to close this issue by leaving it as a Q&A.
By the way, talking about it, I think there should be additional settings.

Can you tell me if there is any other information that needs to be set up?

On the application side, no other settings have been added, but in this case, such debug logs do not come out when it is REST.

However, when it was gRPC, I wondered if it was strange because such a log came out.

The x-b3-traceid is in the header when the client sends the request.
But when I turn on the java agent log on the client side with debug, that kind of log comes out, and I was wondering if I need another setting.

If you need any other information, please let me know.

[laurit]

We'd need a minimal application that reproduces the issue to determine why exactly it is logged.

[eottabom]

Flow
REST Server -> (REST) -> Armeria server (trancscoding REST -> gRPC) -> (gRPC) -> Armeria trancscoding envoy sidecar -> (gRPC) -> Armeria gRPC Server envoy sidecar -> (gRPC) -> Armeria gRPC Server ...

The overall flow is as above.

According to the b3 specification, when it is a gRPC, all b3 headers are changed to lowercase, which is actually accepted as lowercase.
https://github.com/openzipkin/b3-propagation?tab=readme-ov-file#multiple-headers

Would it be a problem with the open telemetry java library being case sensitive?
https://github.com/open-telemetry/opentelemetry-java/blob/0750660f6f0ab4130b81647921b21dafd48cf417/extensions/trace-propagators/src/main/java/io/opentelemetry/extension/trace/propagation/B3Propagator.java#L48

The envoy is pasting the b3 header

I think it will be similar to this issue.
open-telemetry/opentelemetry-js#3150

[wgy035]

The reason why headers are converted to lowercase when using the gRPC protocol is because all headers in HTTP/2 must be in lowercase(RFC7540)
So it happens in all protocols use HTTP/2, such as HTTP/2 server, gRPC, apache dubbo triple

[wgy035]

Because OTel spec does not clarify case sensitive, it can be discuss in opentelemetry-java-contrib

[laurit]

If you believe you have found a bug then please provide a minimal sample application that reproduces the bug.