[Vert.x project] http.route not resolving to complete url

[kintan-singh-backend]

Describe the bug

Need help with http.route resolution in vert.x project:
Recently we started using Opentelemetry java agent instrumentation for metrics and traces to be passed to our datadog agent and then send them to Datadog server using below reference doc. But we are getting http.route as "/supply-service" in our vert.x services while in SpringBoot service we are getting it as "supply-service/health-check".
Can someone please help us in getting this http.route value resolved for our Opentelemetry java agent automatic instrumentation in our vert.x project.

Also in the semantic conventions we can see that the http.route will be sent as complete in http spans as given here: https://opentelemetry.io/docs/specs/semconv/http/http-spans/

Steps to reproduce

Referece doc for setup: https://github.com/open-telemetry/opentelemetry-java-instrumentation?tab=readme-ov-file#configuring-the-agent
https://docs.datadoghq.com/opentelemetry/interoperability/otlp_ingest_in_the_agent/?tab=kuberneteshelmvaluesyaml

Steps to reproduce:

Using sample application:

Please use below minimal application to replicate the issue:

1.) Clone the sample vert.x project from here:
https://github.com/kintan123/openTelemetryVertx.git

2.) Build the jar from the above vert.x application using below command:
./gradlew jar publishToMavenLocal
This will publish the jar in m2 repository under this path:
/Users/<YOUR_USER>/.m2/repository/com/example/supply-service/1.0.0/supply-service-1.0.0.jar

3.) Run the published application jar from its location (/.m2) with opentelemetry javaagent jar version: 2.3.0 using below command:
java -javaagent:/Users/<PATH_TO_JAVA_AGENT>/opentelemetry-javaagent.jar -Dotel.logs.exporter=none -Dotel.traces.exporter=logging-otlp -Dotel.metrics.exporter=logging-otlp -Dotel.resource.attributes=service.name=Test-service -Dotel.javaagent.enabled=true -jar supply-service-1.0.0.jar

4.) Call the API curl with endpoint /supply-service/health-check for http.route resolution check:
curl --location --request GET 'http://localhost:8082/supply-service/health-check'

5.) Check the logs published in /opt/app/logs/supply-service.log for opentelemetry generated http.route value which shows the only the application context path(/supply-service) rather than api path (/supply-service/health-check):
Actual

{
                            "key": "http.route",
                            "value": {
                                "stringValue": "/supply-service"
                            }
}

Expected

{
                            "key": "http.route",
                            "value": {
                                "stringValue": "/supply-service/health-check"
                            }
}

DATADOG PRODUCTION CONFIGURATION STEPS:

1. Adding OpenTelemetry Java Agent
To instrument the application using the OpenTelemetry Java agent, we use the following command:

CMD ["java", "-javaagent:/opt/app/opentelemetry-javaagent.jar", "-jar", "/opt/app/our-service-1.0.0.jar"]
2. Configuring OTLP Ingestion
a. OTLP Receiver Configuration
Add the following configuration to enable OTLP ingestion in the Datadog agent:

Datadog agent version - 7.51.0 and configuration

## OTLP ingest related configuration
  otlp:
    receiver:
      protocols:
        # datadog.otlp.receiver.protocols.grpc - OTLP/gRPC configuration
        grpc:
          # datadog.otlp.receiver.protocols.grpc.enabled -- Enable the OTLP/gRPC endpoint
          enabled: false
          # datadog.otlp.receiver.protocols.grpc.endpoint -- OTLP/gRPC endpoint
          endpoint: "0.0.0.0:4317"
          # datadog.otlp.receiver.protocols.grpc.useHostPort -- Enable the Host Port for the OTLP/gRPC endpoint
          useHostPort: true

        # datadog.otlp.receiver.protocols.http - OTLP/HTTP configuration
        http:
          # datadog.otlp.receiver.protocols.http.enabled -- Enable the OTLP/HTTP endpoint
          enabled: true
          # datadog.otlp.receiver.protocols.http.endpoint -- OTLP/HTTP endpoint
          endpoint: "0.0.0.0:4318"
          # datadog.otlp.receiver.protocols.http.useHostPort -- Enable the Host Port for the OTLP/HTTP endpoint
          useHostPort: true
    logs:
      # datadog.otlp.logs.enabled -- Enable logs support in the OTLP ingest endpoint
      enabled: false

b. Environment Variables
Set the following environment variables for the application deployment to configure the OTLP exporter:

Configuration for opentelementry java agent(Version 2.3.0) is

 - name: OTEL_EXPORTER_OTLP_ENDPOINT
            value: "http://$(DD_AGENT_HOST):4318"
          - name: OTEL_SERVICE_NAME
            value: "$(DD_SERVICE)"
          - name: OTEL_RESOURCE_ATTRIBUTES
            value: "env=prod,version=$(DD_VERSION)"
          - name: OTEL_LOGS_EXPORTER
            value: "none"

Expected behavior

Attaching before and after images for resource info in our current datadog server with route info coming up as only the context path for vert.x project while coming up as complete path for other projects.
Attaching complete span info details for requests in vert.x project from both earlier datadog java agent instrumentation and new opentelemetry java agent instrumentation for reference.

previousDatadogJavaAgentSpanInfo.docx

Actual behavior

The span info http.route is not having complete url and resource endpoint url is also not showing in Datadog server as shown above in expected behaviour.

openTelemetrySpanInfo.docx

Javaagent or library instrumentation version

v2.3.0

Environment

JDK:
OS:

Additional context

No response

[laurit]

Please provide a minimal application that reproduces the issue along with any necessary instructions.

[kintan-singh-backend]

@laurit Please use below minimal application to replicate the issue:

1.) Clone the sample vert.x project from here:
https://github.com/kintan123/openTelemetryVertx.git

2.) Build the jar from the above vert.x application using below command:
./gradlew jar publishToMavenLocal
This will publish the jar in m2 repository under this path:
/Users/<YOUR_USER>/.m2/repository/com/example/supply-service/1.0.0/supply-service-1.0.0.jar

3.) Run the published application jar from its location (/.m2) with opentelemetry javaagent jar version: 2.3.0 using below command:
java -javaagent:/Users/<PATH_TO_JAVA_AGENT>/opentelemetry-javaagent.jar -Dotel.logs.exporter=none -Dotel.traces.exporter=logging-otlp -Dotel.metrics.exporter=logging-otlp -Dotel.resource.attributes=service.name=Test-service -Dotel.javaagent.enabled=true -jar supply-service-1.0.0.jar

4.) Call the API curl with endpoint /supply-service/health-check for http.route resolution check:
curl --location --request GET 'http://localhost:8082/supply-service/health-check'

5.) Check the logs published in /opt/app/logs/supply-service.log for opentelemetry generated http.route value which shows the only the application context path(/supply-service) rather than api path (/supply-service/health-check):
Actual

{
                            "key": "http.route",
                            "value": {
                                "stringValue": "/supply-service"
                            }
}

Expected

{
                            "key": "http.route",
                            "value": {
                                "stringValue": "/supply-service/health-check"
                            }
}

[laurit]

@kintan-singh-backend I was not able to reproduce this. Firstly using the jar from the maven repository didn't work for me. That jar does not have a manifest and it does not include dependencies, executing it with java -jar won't work so I ran the application form the IDE instead. Output from logging exporter is

[otel.javaagent 2024-06-05 10:21:06:158 +0300] [vert.x-eventloop-thread-0] INFO io.opentelemetry.exporter.logging.LoggingSpanExporter - 'GET /supply-service/health-check' : ae198fd183117a758450db282d0c6376 db585d2fd1bad1d4 SERVER [tracer: io.opentelemetry.netty-4.1:2.3.0-alpha] AttributesMap{data={url.scheme=http, thread.name=vert.x-eventloop-thread-0, network.protocol.version=1.1, user_agent.original=curl/8.6.0, network.peer.port=54708, http.request.method=GET, thread.id=17, http.response.status_code=200, http.route=/supply-service/health-check, network.peer.address=0:0:0:0:0:0:0:1, server.address=localhost, client.address=0:0:0:0:0:0:0:1, url.path=/supply-service/health-check, server.port=8082}, capacity=128, totalAddedValues=14}

http.route=/supply-service/health-check as expected.

[kintan-singh-backend]

@laurit can you please try cloning the project again from here and rerun for logs from local IDE: https://github.com/kintan123/openTelemetryVertx
updated route path according to our project with mountSubRouter path.

[laurit]

@kintan-singh-backend now it does not compile