[spanmetrics] Need a feature to add Event attributes to dimensions for spanmetrics

[aishyandapalli]

Component(s)

No response

Is your feature request related to a problem? Please describe.

Business Justification - We have exception data recorded as span events. There are exception.type and exception.message recorded as span event attributes. There should be a way to count the total number of errors for each service name, operation, span kind, status code, exception.type and exception.message. So basically there should be a way to add exception.type and exception.message dimensions to RED metrics.

We should also be able to calculate the latency metric for the list of dimensions - service name, operation, span kind, status code, exception.type and exception.message

We would love to see a feature where we can add the exception data from Events to spanmetrics dimensions.

Describe the solution you'd like

We want a way to add exception data from span Events to spanmetric dimensions.

Describe alternatives you've considered

No response

Additional context

No response

[github-actions]

Pinging code owners for connector/spanmetrics: @albertteoh. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[djaglowski]

This makes sense to me. WDYT @albertteoh?

[vjsamuel]

@djaglowski @albertteoh , few things that we need to figure out is how do we handle span metrics for spans that have more than one exception recorded. however, I would agree that being able to generate additional labels to record errors as labels and generate exemplars for the same would make the metrics being generated very very useful.

[albertteoh]

I just want to be clear that the ask is to add opinionated support specifically for the exception.type and exception.message (as described in semantic conventions) attributes of span events (if any).

If that's the ask, I'm generally supportive of this new feature.

Additional questions:

• Should it be an opt-in feature or always on? I'm okay either way, but slightly prefer the former only because it can save some cpu cycles iterating over events, while also reducing the number of new metrics created if the user won't use these labels.
• Are there any requirements around a span event's Name()?

how do we handle span metrics for spans that have more than one exception recorded

I'm thinking we could assign each set of exception.type and exception.message to a new metric key (new set of dimensions), so a single span could potentially result in updates on multiple timeseries within a single metric name (e.g. calls).

[vjsamuel]

• I think its a great idea to have this as an opt in feature given that it will be expensive to traverse records.
• the question on creating unique metric keys for type and exception resulting in multiple time series would be, is it a fair expectation to say that count(calls) should be count of all spans? if yes then when we record each exception, if there is more than one in a span then that premise would be broken. same goes for latency if we record the same duration for all exceptions seen, we would end up with skewed histograms.

would love to hear your thoughts around the latter.

[vjsamuel]

or should we create an exceptions or errors metric alone given that we don't have the notion of a latency for such spans? what we care for the most is a series denoting the trends of errors and exemplars for the same.

[albertteoh]

is it a fair expectation to say that count(calls) should be count of all spans?

Yeah, great point, @vjsamuel.

given that we don't have the notion of a latency for such spans?

Why wouldn't exception spans have a notion of latency, or perhaps I've misunderstood what you meant? I can imagine it would be quite useful to know; say if a request times out. This requirement was also mentioned in the description:

"We should also be able to calculate the latency metric for the list of dimensions - service name, operation, span kind, status code, exception.type and exception.message."

One alternative is for the spanmetrics connector to only select the first exception.type/message it sees and ignore any other exception events present in the span. The thinking behind that is that usually for a single unit of work (span), there should typically be a single error, if any, that would cause it to fail. It would be unusual for there to be more than one error IMO.

I'm interested to get your thoughts on this @vjsamuel.

Also @aishyandapalli, do you have any thoughts around how you'd like to see metrics for multiple exceptions per span in your use case?

[djaglowski]

I just want to be clear that the ask is to add opinionated support specifically for the exception.type and exception.message (as described in semantic conventions) attributes of span events (if any).

I am admittedly not very tuned in to the details of this connector, but I'd like to encourage that we look first for solutions which are not hardcoded to specific attributes. To that end, is there strictly a need to count by exception.type and exception.message, or could these be configurable dimensions in a more generic solution?

[vjsamuel]

@djaglowski https://opentelemetry.io/docs/specs/otel/trace/semantic_conventions/exceptions/ records the specification of exceptions. they must have type and message. while i am open to configurability, we must mandate at least type. message can get tricky if the message has uniqueness associated to it.

[djaglowski]

@vjsamuel, are there events which are not exceptions, which could use the same functionality but based on different attributes?

[vjsamuel]

maybe yes? :) @aishyandapalli and i were razor focused on the errors aspect of things but there could be value.

[albertteoh]

How will we represent events in metrics if we support the more general case. Presumably, each attribute key-value would be included as part of metric's dimensions which would break this invariant called out by @vjsamuel:

is it a fair expectation to say that count(calls) should be count of all spans?

Perhaps we could set this expectation as a condition of enabling event attributes in spanmetrics; or maybe there is a solution to this problem that I haven't thought of?

[vjsamuel]

lets maybe create two new metrics events and exceptions. the former can have free form attribute support + whatever is defined for the span attributes. the latter is razor focused on type and message. we can allow users to decide if they want just type or type and message. this will keep the feature separate, not diminish the meaning of the current metrics and also allow users to draw graphs out of things found in span events. thoughts?

[albertteoh]

I like the idea of creating new metrics (sorry, you mentioned it earlier in this thread); would we also need events_duration, etc. equivalents?

What is the benefit of having a separate exceptions metric?

[vjsamuel]

span events dont have any duration attached to them specifically. so we would only do counters. keeping the metric separate is just making the representation prettier. nothing else. also exceptions have dedicated semantic conventions.

[vjsamuel]

@albertteoh any thoughts on how we should move forward?

[albertteoh]

@albertteoh any thoughts on how we should move forward?

We can go ahead with adding this feature. I propose starting with the following spec:

1. Opt-in via configuration: events.enabled (defaults to false).
2. If enabled, it will produce a new metric called events which will add a dimension value per configured selected event attribute key-value.
3. The events dimensions must be identical in configuration (and its semantics) to the existing dimensions configuration used to select "top-level" span attributes.
4. Similar to dimensions, if an event's attributes are repeated in another event with a different value, it will result in a new timeseries. It will also contain all of the span's "top-level" dimensions. Example:
   1. If the following events are present in a span:
      1. { "event0": { "attr": "val0" } }
      2. { "event1": { "attr": "val1"} }
   2. The following two timeseries will be created:
      1. events{service.name="foo",..., attr="val0"}
      2. events{service.name="foo",..., attr="val1"}
5. dimensions must be non-empty if events.enabled is true. An empty dimensions list should return a configuration error.

The following example demonstrates all of the above spec:

connectors:
  spanmetrics:
    dimensions:
      - name: http.method
        default: GET
    ...
    events:
      enabled: true
      dimensions:
        - name: attr
          default: val
        - name: exceptions.type
        - name: exceptions.message 

I've considered including exceptions, however, I'm not yet convinced it's worth creating a new metric for it. If we want a count of errors, users should use the status.code dimension. Adding a new exceptions metric also either introduces data redundancy/duplication, or requires logic in code to specifically prevent this.

However, I'm open to being convinced otherwise. 😄

Please let me know what you think of that proposed spec.

[aishyandapalli]

Wow, this is such a detailed explanation. Thanks for that @albertteoh

@vjsamuel for final approval on the above logic

[vjsamuel]

@aishyandapalli, @albertteoh owns this :) he is the final approver. i think the proposal works and we should build it out quickly and submit a PR.

[aishyandapalli]

@vjsamuel I meant for our use case :) I will start working on this

[vjsamuel]

@albertteoh is it possible today for calls and in future events to have exemplars attached to them? if not @aishyandapalli can add support for that as well.

[albertteoh]

@albertteoh is it possible today for calls and in future events to have exemplars attached to them? if not @aishyandapalli can add support for that as well.

I think so, I can't see a reason why we shouldn't add exemplars to calls and events.

At the time when it was added to spanmetrics processor, the focus was just on histograms. cc @anneelisabethlelievre

Thank you very much @vjsamuel @aishyandapalli! 🙏🏼

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[djaglowski]

Closed by #27811