Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[exporter/elasticsearch] Group metrics in documents based on timestamp and attributes #33756

Closed
andrzej-stencel opened this issue Jun 25, 2024 · 2 comments · Fixed by #33794
Closed
Labels
enhancement New feature or request exporter/elasticsearch needs triage New item requiring triage

Comments

@andrzej-stencel
Copy link
Member

Component(s)

exporter/elasticsearch

Is your feature request related to a problem? Please describe.

As implemented in #33755, metrics are exported to Elasticsearch so that a separate Elasticsearch document is created for each metric data point. We could improve the performance by grouping metrics that share the timestamp and the set of attributes in one document. For example, the following three metrics:

system.cpu.load_average.1m
system.cpu.load_average.5m
system.cpu.load_average.15m

are currently sent in three separate documents, but they could be sent in one document.

Describe the solution you'd like

Instead of sending each metric data point in a separate document, let's group the metrics that share the timestamp and attributes.

Describe alternatives you've considered

No response

Additional context

No response

@andrzej-stencel andrzej-stencel added enhancement New feature or request needs triage New item requiring triage labels Jun 25, 2024
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@strawgate
Copy link
Contributor

Here was the implementation of metric merging we used in the initial metrics exporter POC strawgate@5d1c3ff

tomasmota pushed a commit to SpringerPE/opentelemetry-collector-contrib that referenced this issue Jul 1, 2024
… attributes (open-telemetry#33794)

**Description:** <Describe what has changed.>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
Taking over from open-telemetry#33755
- Add data stream routing based on `data_stream.*` attributes
- Refine metrics grouping to work with DS routing

**Link to tracking Issue:**
Closes open-telemetry#33755 
Fixes open-telemetry#33756

**Testing:** <Describe what testing was performed and which tests were
added.>

See unit tests

**Documentation:** <Describe the documentation added.>

Updated readme

---------

Co-authored-by: Andrzej Stencel <andrzej.stencel@elastic.co>
Co-authored-by: Andrew Wilkins <axwalk@gmail.com>
lalith47 pushed a commit to lalith47/opentelemetry-collector-contrib that referenced this issue Jul 1, 2024
… attributes (open-telemetry#33794)

**Description:** <Describe what has changed.>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
Taking over from open-telemetry#33755
- Add data stream routing based on `data_stream.*` attributes
- Refine metrics grouping to work with DS routing

**Link to tracking Issue:**
Closes open-telemetry#33755 
Fixes open-telemetry#33756

**Testing:** <Describe what testing was performed and which tests were
added.>

See unit tests

**Documentation:** <Describe the documentation added.>

Updated readme

---------

Co-authored-by: Andrzej Stencel <andrzej.stencel@elastic.co>
Co-authored-by: Andrew Wilkins <axwalk@gmail.com>
cparkins pushed a commit to AmadeusITGroup/opentelemetry-collector-contrib that referenced this issue Jul 11, 2024
… attributes (open-telemetry#33794)

**Description:** <Describe what has changed.>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
Taking over from open-telemetry#33755
- Add data stream routing based on `data_stream.*` attributes
- Refine metrics grouping to work with DS routing

**Link to tracking Issue:**
Closes open-telemetry#33755 
Fixes open-telemetry#33756

**Testing:** <Describe what testing was performed and which tests were
added.>

See unit tests

**Documentation:** <Describe the documentation added.>

Updated readme

---------

Co-authored-by: Andrzej Stencel <andrzej.stencel@elastic.co>
Co-authored-by: Andrew Wilkins <axwalk@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request exporter/elasticsearch needs triage New item requiring triage
Projects
None yet
2 participants