Add kafka client metrics

[jack-berg]

Proposal to bridge metrics from the kafka client library into OpenTelemetry.

As this PR currently stands, only metrics which align with those already declared in the kafka metric semantic convention are bridged. However, the semantic conventions are focused around the use case of monitoring kafka brokers, rather than clients. There's a lot more useful data available, which I'd like the semantic conventions to be extended with.

I've added a utility that makes it easier to explore which metrics are available, and if / how they're mapped into OpenTelemetry metrics. The utility prints out all this info in a markdown table, which I've included in the README.

I could use some feedback about:

• The best place to include this code from an artifact standpoint.

** Updated 6/8/22 **
I've updated the PR to be a general purpose bridge between kafka client metrics and opentelemetry metric, rather than implementing as an allow list of known metrics. My reasoning is as follows:

• There are 202 metrics exposed by kafka client metrics. I made it through mapping about 60% of them before I decided that manually mapping all the metrics is madness. Its error prone, brittle against changes between kafka client versions, and adds little value.
• The metrics kafka client exposes are conceptually similar what we do in our micrometer shim: Kafka client has its own internal metrics system. Its authors have already done the analysis to decide what is important to monitor when using the kafka client, and they've provided a hook for to access the metrics in a generic format. In bridging these metrics to opentelemetry we should use a light hand. We should map the metrics to the appropriate opentelemetry instrument types, but should minimize changes to metric names, descriptions, etc. Trying to map these to semantic conventions is folly, because we don't control the instrumentation at its source and the authors could change the metrics drastically in the next version.
• The lack of mapping to semantic conventions doesn't mean this data is not useful. This data is very useful to folks operating applications that produce and consume from kafka. We'd be doing a disservice to opentelemetry users and hampering the adoption of opentelemetry by rejecting good telemetry data because of a lack of semantic conventions. Furthermore, I don't think we should try to codify these metrics in the semantic conventions: 202 metrics is just too much data to review. Additionally, there's likely to be significant differences in the metrics that are exposed in kafka clients from language to language. Does that mean we should withhold this data that's available to java kafka client users? No. There's a place for semantic conventions, and there's a place for bridging in telemetry data that is available but not under our control.

[brunobat]

First review in the project

[mateuszrzeszutek]

Kafka client has its own internal metrics system. Its authors have already done the analysis to decide what is important to monitor when using the kafka client, and they've provided a hook for to access the metrics in a generic format. In bridging these metrics to opentelemetry we should use a light hand. We should map the metrics to the appropriate opentelemetry instrument types, but should minimize changes to metric names, descriptions, etc. Trying to map these to semantic conventions is folly, because we don't control the instrumentation at its source and the authors could change the metrics drastically in the next version.

100% agree with that 👍

The only thing that I'm slightly worried about is that we won't expose metrics that are common to all messaging clients (like the HTTP client/server metrics; think http.server.duration but across all messaging instrumentations) - but since these are not defined yet in the spec I suppose we'll have to put adding them off until later.

[jack-berg]

Thanks for the review @mateuszrzeszutek!

The only thing that I'm slightly worried about is that we won't expose metrics that are common to all messaging clients (like the HTTP client/server metrics; think http.server.duration but across all messaging instrumentations) - but since these are not defined yet in the spec I suppose we'll have to put adding them off until later.

👍 My thinking is that bridging these metrics does not prevent us from later adding additional metrics which have consistency with other messaging systems.

[jack-berg]

I just added a commit with new logic that does what's described in this comment. I think its a fairly important improvement.

[jack-berg]

Here's how I detect whether a metric exists with less granular set of attribute keys, as explained in this part of the readme.

In the process of adding support for this, I shuffled some stuff around for better organization, including moving the table print method to OpenTelemetryKafkaMetricTest. It turns out that all the information needed to print the table can be obtained without any additional surface area.

[trask]

thx!

[jack-berg]

@anuraaga unfortunately this has to be public, so I've moved it to an internal class.

[trask]

thx @jack-berg!