Some spans are not appearing in X-Ray

[rcdailey]

I am using a library named Rebus to publish messages to an SQS queue in an ASP.NET C# WebApi application. The receiver for these messages is a BackgroundService in the same process. I initialize Open Telemetry for .NET once in the main Program.cs.

I noticed in X-Ray, that certain spans are not appearing in the service map or in the traces list. I do, however, see the spans exported from the ADOT otel collector. Given the multi-faceted nature of this issue, I'm not sure if this is the best place for my question, but it's a start.

My expectation is that I see all spans in the trace event (which is a call to one of the endpoints in my web api). I also would like a better understanding of why these spans do not appear in X-Ray but show up fine in applications like Jaeger.

My environment:

ECS to host my WebApi microservice. In the same ECS task, I also have the adot otel collector running, which my webapi sends telemetry to. From there, my collector is configured to export to xray.

Here is my collector config:

receivers:
  otlp:
    protocols:
      grpc:

processors:
  batch:

exporters:
  logging:
    loglevel: debug
  awsxray:

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging, awsxray]

My initialization code in C#:

public sealed class OpenTelemetryProviderBuilder : IDisposable
{
    private TracerProvider? _traceBuilder;

    private readonly TextMapPropagator _defaultPropagator = new CompositeTextMapPropagator(new TextMapPropagator[]
    {
        new BaggagePropagator(),
        new TraceContextPropagator(),
        new AWSXRayPropagator()
    });

    public void SetupTracing(string serviceName, string version)
    {
        var resourceBuilder = ResourceBuilder.CreateDefault()
            .AddService(serviceName, serviceVersion: version)
            .AddDetector(new AWSECSResourceDetector());

        var builder = Sdk.CreateTracerProviderBuilder()
            .SetResourceBuilder(resourceBuilder)
            .AddHttpClientInstrumentation()
            .AddAspNetCoreInstrumentation()
            .AddXRayTraceId()
            .AddAWSInstrumentation()
            .AddRebusInstrumentation()
            .AddOtlpExporter();

        Sdk.SetDefaultTextMapPropagator(_defaultPropagator);
        _traceBuilder = builder.Build();
    }

    public void Dispose()
    {
        _traceBuilder?.Dispose();
    }
}

As far as Rebus goes, I'm using Rebus.OpenTelemetry package which ensures that baggage/headers are propagated with messages sent over SQS so that traces on the receiving end can be correlated. Maybe the issue is in this package, but I'm not sure.

And I'll attach a log file containing debug output from the collector for trace information. Notice that you'll see spans with instrumentation scope "Rebus.Diagnostics". These do not appear in X-Ray. Nor do the S3.DeleteObject HTTP calls. All of these show up in the attached log though, so I know the instrumentation itself is working.

otel-collector-debug-output.txt

I've spent hours combing over the logs but there's a lot of information and I'm not sure how to identify what is the cause.

[srprash]

Hi @rcdailey
Regarding the SQS.ReceiveMessage spans not appearing in the trace - looking at one such span, I found that they don't have a Parent ID (see example span below). Such Client spans are dropped in X-Ray. I suspect the reason it doesn't have a parent is because the trace context is not propagated from main thread the background thread polling the messages.

ScopeSpans #0
ScopeSpans SchemaURL: 
InstrumentationScope Amazon.AWS.AWSClientInstrumentation 
Span #0
    Trace ID       : 64fa4ae3ae37e44d0b2c151af31078d3
    Parent ID      : 
    ID             : 3a67024376b6031f
    Name           : SQS.ReceiveMessage
    Kind           : Client
    Start time     : 2023-09-07 22:12:51.6823342 +0000 UTC
    End time       : 2023-09-07 22:13:11.9347381 +0000 UTC
    Status code    : Unset
    Status message : 
Attributes:
     -> aws.service: Str(SQS)
     -> aws.operation: Str(ReceiveMessage)
     -> aws.region: Str(us-east-1)
     -> aws.queue_url: Str(https://sqs.us-east-1.amazonaws.com/741061989950/CaaSMonitor-DeploymentService-MonitorDeploymentOperations26D12DE2-eWBIw5b8bgqp)
     -> aws.requestId: Str(b7be11d1-ce3f-5b1d-9c05-4ca5831c935c)
     -> http.status_code: Int(200)
     -> http.response_content_length: Int(240)

Notice that you'll see spans with instrumentation scope "Rebus.Diagnostics". These do not appear in X-Ray. Nor do the S3.DeleteObject HTTP calls.

Not sure why these spans are not showing up in X-Ray, and I need more information from you. I tried mapping out the spans for trace id 64fa4b15b730d0d5b5e59cbdd0c42253 from the provided log. Here's what I think the flow looks like. please correct me if I got anything incorrect.

[0199d929abeb4492] DELETE api/deployments/{id}
|
|----- [d4db25fbb164db32] DynamoDBv2.DescribeTable
|----- [6e8c3bdc1b6e2e35] DynamoDBv2.GetItem
|----- [efc619a26117efbc] DynamoDBv2.Scan
|----- [81a64e18d124d5b0] DynamoDBv2.GetItem
|----- [7fdd5b2f71f64b7f] DynamoDBv2.UpdateItem
|----- [b97169fbfda309ae] Monitor.Deployment.Operations.DeleteDeploymentMessage, Monitor.Deployment.Operations send (sqs queue)
|          |
|          |----- [9323dc2afc47d113] Monitor.Deployment.Operations.DeleteDeploymentMessage, Monitor.Deployment.Operations receive
|                         |
|                         |----- [dcfbe7036f27fec3] Monitor.Deployment.Operations.DeleteDeploymentMessage, Monitor.Deployment.Operations process
|                         				|
|                         				|----- [b1a21b791feb8a0d] S3.ListObjects
|                         				|----- [436fcb2a32c5c74a] S3.DeleteObjects
|                         				|----- [b3decbea22346625] DynamoDBv2.DeleteItem
|
|----- [a198796b8fc3855f] HTTP POST (sqs queue)

If you search in X-Ray console for trace 1-64fa4b15-b730d0d5b5e59cbdd0c42253, do you see every segment/subsegment mentioned above except for segment/subsegment ids b97169fbfda309ae, 9323dc2afc47d113, dcfbe7036f27fec3, and 436fcb2a32c5c74a, is it?

If possible, can you send me the trace screenshot and raw json data for the above trace id from your X-Ray console?

[krissrex]

Perhaps the Span Kind is not being exported as Segments to Xray. See
open-telemetry/opentelemetry-collector-contrib#20267 and #1773, where Xray only uses Server to generate Segments.

[rcdailey]

I apologize for the delay in my response. I no longer work at the company where I ran into this issue, so I unfortunately am not able to follow up with more details.

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.