Tomcat JMX metrics

[SylvainJuge]

We can capture Tomcat metrics thanks to the JMX Insights (doc) feature.

While working as expected, I noticed something quite surprising when trying to use the metrics, in particular http.server.tomcat.threads described here.

There are multiple related aspects, so I will try to describe this in a structured manner.

Small bug ?

The http.server.tomcat.threads metric is captured from two MBean attributes of Catalina:type=ThreadPool,name=*: currentThreadCount and currentThreadsBusy (yaml definition).

The mapping of the state attribute is the following

• currentThreadsBusy is mapped to state = "busy".
• currentThreadCount is mapped to state = "idle", but it actually represents the total number of threads so this is definitely not the number of "idle" threads, so here maybe a better attribute value would be something like "total", but that does not fit the semconv recommendations.

Compliance with semantic convention metrics definitions

The general guidelines for metrics clearly state the following: "aggregations over all the attributes of a given metric SHOULD be meaningful", so here the metric does not fit because removing the state attribute and adding the values would result in double-counting the threads that are busy.

Also, using the "total" in metric name as I suggested is also not recommended for counters.

I see two possible approaches to fixing this:

• capture the number of "idle" threads by subtracting the "busy" from the total number of threads, that would allow to keep a single http.server.tomcat.threads metric with state = "idle" for idle threads and state = "busy" for busy threads, aggregating by removing the state attribute would provide the total number of threads.
• use distinct metrics, for example:
  • http.server.tomcat.threads.count for the total number of threads
  • http.server.tomcat.threads.busy.count for the busy threads

I think that the first approach would be better, even if more complex to implement as this kind of issues are likely to happen with many other JMX metrics.

Using the second option is simpler, but it's a compromise to work around the way that JMX metrics are exposed.

Should we define those in semantic conventions ?

I think those metric definitions would be better if they were part of the semantic conventions like the now stabilized JVM metrics. I understand there is a non-trivial effort required for this, but from the perspective of the consumer of those metrics the fact that they are captured from JMX and relate to Java and the JVM is only an implementation detail.

Defining those in the semantic conventions will also require that we address the points above.

[trask]

cc @PeterF778

For the first option, are you thinking to extend the jmx metrics yaml syntax, or replace the current yaml-based tomcat metrics with a programmatic implementation?

The second option doesn't sound that bad to me. We are just bridging the JMX metrics that Tomcat produces.

I think those metric definitions would be better if they were part of the semantic conventions like the now stabilized JVM metrics.

Here's some related discussion: open-telemetry/semantic-conventions#269

[SylvainJuge]

For the first option, are you thinking to extend the jmx metrics yaml syntax, or replace the current yaml-based tomcat metrics with a programmatic implementation?

I was thinking not to completely replace the convenient YAML syntax, but to allow for some simple post-processing of the metrics before they are sent, but that's a bad idea for maintenance as it would make it way more complex.

The second option doesn't sound that bad to me. We are just bridging the JMX metrics that Tomcat produces.

For me this sounds the best option so far, while it does not leverage the full expressiveness allowed by metrics and their attributes to represent things in a concise and consistent way, it would at least provide metrics that fit the semantic convention guidelines.

What I understand from open-telemetry/semantic-conventions#269 is the following:

• mapping JMX metrics is tricky by nature as it depends on the underlying implementation, so the specification would also be very brittle and tedious.
• metrics produced from JMX can be slightly different from the ones exposed through other interfaces (prometheus, native otel metrics, ...), so maybe having them in a specific jmx namespace could help clarify this.
• It's hard to define a real ownership of those metrics in the semantic conventions, thus it can lead to "forever beta" situation.

So to summarize, for now I think the probable best option is to:

• keep the JMX metrics definitions only in this repository, they are also conveniently documented through the YAML configuration.
• modify the tomcat JMX metrics to make them fit the general semantic conventions guidelines.

[trask]

So to summarize, for now I think the probable best option is to:

• keep the JMX metrics definitions only in this repository, they are also conveniently documented through the YAML configuration.
• modify the tomcat JMX metrics to make them fit the general semantic conventions guidelines.

makes sense to me

[SylvainJuge]

Thanks @trask , I'll likely open a draft PR to "fix" the tomcat metrics then.

[SylvainJuge]

I've opened #11621 which contains a fix for Tomcat JMX metrics.