Skip to content

Commit

Permalink
Data prepper pipelines (#101)
Browse files Browse the repository at this point in the history
* Initial setup of Observability OTEL demo using Opensearch as store
* add curl to the basic docker O/S
* add monitoring & opinionated assets for this demo
* export F/E & Load-Generator port services
* fix jaeger service name
* adding nginx as frontend proxy, fluent-bit.conf as log emitter

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* adding nginx & fluent-bit docs

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* adding integrationbs docker-compose service for assets loading into opensearch / dashboards

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* adding datasource for query prometheus

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* adding architecture.md including images

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* add images and service links

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* add jaeger-agent container name

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* add data-prepper support for traces ingestion & serviceMap creation

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* fix auth issues for data-prepper

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* add documentation for different ingestion capabilities

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* Initial setup of Observability OTEL demo using Opensearch as store

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* Initial setup of Observability OTEL demo using Opensearch as store

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* add curl to the basic docker O/S

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* add monitoring & opinionated assets for this demo

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* [shippingservice] update rust version and dependencies (open-telemetry#865)

* update rust version and dependencies

* Changelog

---------

Co-authored-by: Carter Socha <43380952+cartersocha@users.noreply.github.com>

* [load generator] Bump loagen dependencies (open-telemetry#869)

* Bump loagen dependencies

* Changelog

* [grafana] fix demo dashboard to be compatible with spanmetrics connector (open-telemetry#874)

* [grafana] fix demo dashboard to be compatible with spanmetrics connector

Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>

* add changelog entry

Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>

* fix some naming

Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>

---------

Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>

* export F/E & Load-Generator port services

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* [kafka] remove KRaft mode support workarounds (open-telemetry#880)

as the latest confluentinc/cp-kafka:7.4.0-1-ubi8
already provides built-in support

* enabling batch span processor metrics for quoteservice (open-telemetry#878)

* enabling batch span processor metrics for quoteservice
PHP's batch span processor can emit metrics. There is not yet an async implementation,
so use react's event loop to periodically flush traces and metrics.
Add a grafana chart for some of the batch span processor metrics: dropped, pending, processed, queued

* update changelog

* revert change in selected graph

* Update dependencies

---------

Co-authored-by: Juliano Costa <julianocosta89@outlook.com>

* Generate random errors in cartservice (open-telemetry#824)

* Generate random errors from cart service

* Update migration script?

* Update changelog

* Update changelog

* Add Copyright to FeatureFlagHelper.cs

---------

Co-authored-by: Carter Socha <43380952+cartersocha@users.noreply.github.com>
Co-authored-by: Juliano Costa <julianocosta89@outlook.com>

* fix jaeger service name

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* adding nginx as frontend proxy, fluent-bit.conf as log emitter

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* adding nginx & fluent-bit docs

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* adding integrationbs docker-compose service for assets loading into opensearch / dashboards

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* [currencyservice] Fix OTel C++ build and update OTel version (open-telemetry#886)

* Fix OTel C++ build and update OTel version

* changelog

* adding datasource for query prometheus

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* adding architecture.md including images

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* add images and service links

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* add jaeger-agent container name

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* add data-prepper support for traces ingestion & serviceMap creation

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* fix auth issues for data-prepper

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* add documentation for different ingestion capabilities

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* add assets manager flast app for loading the integration assets

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* remove experimental assets mgr app

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* Opensearch otel demo observability tutorial (#5)

* add tutorial for using observability and diagnostic of a real use case



* adding otel landing page markdown dashboard


* adding tutorial markdown dashboards and documents


* adding Observability Introduction.md tutorial markdown dashboards and documents


* add dependencies to the docker-compose.yml


* exposing docker ports locally

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

---------

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* Add .whitesource configuration file

* Update local-config.md

Co-authored-by: Joshua Li <joshuali925@gmail.com>
Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* Update README.md

update fluent-bit README.md file reference to the config

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* chore(deps): update dependency com.fasterxml.jackson.core:jackson-databind to v2.13.4.1 (#4)

Co-authored-by: mend-for-github-com[bot] <50673670+mend-for-github-com[bot]@users.noreply.github.com>

* chore(deps): update dependency io.grpc:grpc-protobuf to v1.45.2 (#5)

Co-authored-by: mend-for-github-com[bot] <50673670+mend-for-github-com[bot]@users.noreply.github.com>

* chore(deps): update dependency org.apache.kafka:kafka-clients to v3.4.0 (#6)

Co-authored-by: mend-for-github-com[bot] <50673670+mend-for-github-com[bot]@users.noreply.github.com>

* chore(deps): update dependency requests to v2.31.0 (#7)

Co-authored-by: mend-for-github-com[bot] <50673670+mend-for-github-com[bot]@users.noreply.github.com>

* update url for the new opensearch otel demo repo
update integration script to log into a file
update the README.md for the otel-collector pipline descrition
update ports for the otel-col health-check
add memory_limiter / servicegraph processors for otel-col

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* update url for the new opensearch otel demo repo
update integration script to log into a file
update the README.md for the otel-collector pipline descrition
update ports for the otel-col health-check
add memory_limiter / servicegraph processors for otel-col

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* Update docker-compose.yml

fixing the issues with latest OS images that cause the demo to break

Signed-off-by: Anirudha (Ani) Jadhav <anirudha@nyu.edu>

* fix integrations script connections retry
add integrations docker service dependencies on the O/S cluster nodes

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* update bytes map to long and status to int (#11)

Signed-off-by: Shenoy Pratik <sgguruda@amazon.com>

* Update README.md

update the ports according to the .env variables 

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* Integration config parser support (#12)

* update integration script to load all assets from data.ini file

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* update url to support both text & keyword

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

---------

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* Initial setup of Observability OTEL demo using Opensearch as store

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* Initial setup of Observability OTEL demo using Opensearch as store

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* add curl to the basic docker O/S

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* add monitoring & opinionated assets for this demo

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* export F/E & Load-Generator port services

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* fix jaeger service name

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* adding nginx as frontend proxy, fluent-bit.conf as log emitter

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* adding nginx & fluent-bit docs

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* adding integrationbs docker-compose service for assets loading into opensearch / dashboards

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* adding datasource for query prometheus

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* adding architecture.md including images

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* add images and service links

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* add jaeger-agent container name

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* add data-prepper support for traces ingestion & serviceMap creation

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* fix auth issues for data-prepper

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* add documentation for different ingestion capabilities

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* Initial setup of Observability OTEL demo using Opensearch as store

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* Initial setup of Observability OTEL demo using Opensearch as store

* Initial setup of Observability OTEL demo using Opensearch as store

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* add monitoring & opinionated assets for this demo

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* export F/E & Load-Generator port services

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* fix jaeger service name

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* adding nginx as frontend proxy, fluent-bit.conf as log emitter

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* adding nginx & fluent-bit docs

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* adding integrationbs docker-compose service for assets loading into opensearch / dashboards

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* adding datasource for query prometheus

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* adding architecture.md including images

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* add images and service links

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* add jaeger-agent container name

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* add data-prepper support for traces ingestion & serviceMap creation

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* fix auth issues for data-prepper

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* add documentation for different ingestion capabilities

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* add assets manager flast app for loading the integration assets

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* remove experimental assets mgr app

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* Opensearch otel demo observability tutorial (#5)

* add tutorial for using observability and diagnostic of a real use case



* adding otel landing page markdown dashboard


* adding tutorial markdown dashboards and documents


* adding Observability Introduction.md tutorial markdown dashboards and documents


* add dependencies to the docker-compose.yml


* exposing docker ports locally

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

---------

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* Update local-config.md

Co-authored-by: Joshua Li <joshuali925@gmail.com>
Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* Update README.md

update fluent-bit README.md file reference to the config

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* Add .whitesource configuration file

* chore(deps): update dependency com.fasterxml.jackson.core:jackson-databind to v2.13.4.1 (#4)

Co-authored-by: mend-for-github-com[bot] <50673670+mend-for-github-com[bot]@users.noreply.github.com>

* chore(deps): update dependency io.grpc:grpc-protobuf to v1.45.2 (#5)

Co-authored-by: mend-for-github-com[bot] <50673670+mend-for-github-com[bot]@users.noreply.github.com>

* chore(deps): update dependency org.apache.kafka:kafka-clients to v3.4.0 (#6)

Co-authored-by: mend-for-github-com[bot] <50673670+mend-for-github-com[bot]@users.noreply.github.com>

* chore(deps): update dependency requests to v2.31.0 (#7)

Co-authored-by: mend-for-github-com[bot] <50673670+mend-for-github-com[bot]@users.noreply.github.com>

* update url for the new opensearch otel demo repo
update integration script to log into a file
update the README.md for the otel-collector pipline descrition
update ports for the otel-col health-check
add memory_limiter / servicegraph processors for otel-col

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* Update docker-compose.yml

fixing the issues with latest OS images that cause the demo to break

Signed-off-by: Anirudha (Ani) Jadhav <anirudha@nyu.edu>

* fix integrations script connections retry
add integrations docker service dependencies on the O/S cluster nodes

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* update bytes map to long and status to int (#11)

Signed-off-by: Shenoy Pratik <sgguruda@amazon.com>

* Update README.md

update the ports according to the .env variables 

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* Integration config parser support (#12)

* update integration script to load all assets from data.ini file

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* update url to support both text & keyword

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

---------

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* upstream updates

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* update http.template mapping (#15)

fix dependency in the docker-compose missing env var for cartservices
add extra logging into the integration.py

* adding standard opensearch github dev / maintenance spec: (#16)

* adding standard opensearch github dev / maintenance spec:
 - CONTRIBUTING.md
 - MAINTAINERS.md
 - CODE_OF_CONDUCT.md
 ...

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

---------

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* fixing opensearch demo link for OpenTelemetry (#14)

Signed-off-by: Mitchell Gale <Mitchell.Gale@improving.com>

* upstream updates (#17)

* Regenerate kubernetes manifest and add autogenerate comment (open-telemetry#909)

Co-authored-by: Pierre Tessier <pierre@pierretessier.com>

* [loadgenerator] run load through frontend proxy (Envoy) (open-telemetry#914)

* Run load through frontendProxy

Signed-off-by: Pierre Tessier <pierre@pierretessier.com>

* Run load through frontendProxy

Signed-off-by: Pierre Tessier <pierre@pierretessier.com>

---------

Signed-off-by: Pierre Tessier <pierre@pierretessier.com>
Co-authored-by: Austin Parker <austin@ap2.io>

* Fix error sh in README.md (open-telemetry#926)

---------

Signed-off-by: Pierre Tessier <pierre@pierretessier.com>
Co-authored-by: isabel <isaschm@users.noreply.github.com>
Co-authored-by: Pierre Tessier <pierre@pierretessier.com>
Co-authored-by: Austin Parker <austin@ap2.io>
Co-authored-by: Lancelot <lancel.huang.cs@gmail.com>

* add untriaged gh workflow (#18)

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* [cartservice] Update .NET OTel to 1.5.0 (open-telemetry#935) (#23)

* [cartservice] Update OTel .NET to 1.5.0

* [cartservice] Simplify resource configuration

TelemetrySdk and EnvVariable are defaults

* Update changelog

Co-authored-by: Piotr Kiełkowicz <pkiekowicz@splunk.com>

* add red metrics from spans (#21)

* add spanmetrics namespace (span.metrics)
add description for all metrics in prometheus

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* add spanmetrics namespace (span.metrics)
add description for all metrics in prometheus

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

---------

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* add integration dependency on the catalog repository for getting all the schema related index and component templates mapping (#29)

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* fix data-prepper image to a stable version
update fluent-bit.conf parser to emmit a different field name
adding support for prometheus-opensearch exporter

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* add error wrapping for integration script
Signed-off-by: YANGDB <yang.db.dev@gmail.com>

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* Upstream changes needed merge  (#41)

* Commit generated Go code and bump dependencies (open-telemetry#946)

* accountingservice: commit generated code

bump deps

* license-checker to ignore genproto

* Update README.md

* Add comment to tools.go

* checkoutservice: commit generated code

bump dependencies

* productcatalogservice: commit generated code

bump dependencies

* Configure Renovate (open-telemetry#931)

* Add renovate.json

* only update otel deps

* update renovate config

* update renovate config

---------

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: austinlparker <austin@ap2.io>
Co-authored-by: Carter Socha <43380952+cartersocha@users.noreply.github.com>

* [cartservice] code cleanup (open-telemetry#943)

* File scope namespaces

* Sort modifiers

* Remove redundant type declaration

* Avoid hiding variables

* join declaration and assignment

* Use standard .NET convention for fields and consts

* inline out variable

* object initializer

* collection initializer

* drop unused using

* drop unused parameters

* remove redundant field initializer

---------

Co-authored-by: Austin Parker <austin@ap2.io>

* Add Axiom to list of vendors (open-telemetry#963)

We would very much like to add [Axiom](axiom.co) to the list of vendors in the README. I linked a public demo dashboard that is feed by the demo.

---------

Signed-off-by: YANGDB <yang.db.dev@gmail.com>
Co-authored-by: Robert Pająk <pellared@hotmail.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: austinlparker <austin@ap2.io>
Co-authored-by: Carter Socha <43380952+cartersocha@users.noreply.github.com>
Co-authored-by: Piotr Kiełkowicz <pkiekowicz@splunk.com>
Co-authored-by: Lukas Malkmus <mail@lukasmalkmus.com>

* Update dependency io.grpc:grpc-netty to v1.50.3 (#47)

Co-authored-by: mend-for-github-com[bot] <50673670+mend-for-github-com[bot]@users.noreply.github.com>

* upstream updates (#52)

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* Update frauddetectionservice (#55)

Co-authored-by: mend-for-github-com[bot] <50673670+mend-for-github-com[bot]@users.noreply.github.com>

* Update adservice (#48)

Co-authored-by: mend-for-github-com[bot] <50673670+mend-for-github-com[bot]@users.noreply.github.com>

* update upstream changes

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* update upstream changes

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* update opensearch for 2.9

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* fix: Fixing tracetest configuration (#65)

* fix: Fixing tracetest configuration

* fix: Fixing tracetest configuration

Signed-off-by: Oscar Reyes <oscar-rreyes1@hotmail.com>

---------

Signed-off-by: Oscar Reyes <oscar-rreyes1@hotmail.com>

* chore(deps): update loadgenerator (#67)

Co-authored-by: mend-for-github-com[bot] <50673670+mend-for-github-com[bot]@users.noreply.github.com>

* update upstream changes
 - remove 'opensearch integration' service
 - update the opensearch dashboard
 - add opensearch+dashboard to the minimized docker-compose.yml

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* update upstream changes
 - remove 'opensearch integration' service
 - update the opensearch dashboard
 - add opensearch+dashboard to the minimized docker-compose.yml

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* update upstream changes
 - remove 'opensearch integration' service
 - update the opensearch dashboard
 - add opensearch+dashboard to the minimized docker-compose.yml
 - update opensearch to use a single node

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* update upstream changes
 - remove 'opensearch integration' service
 - update the opensearch dashboard
 - add opensearch+dashboard to the minimized docker-compose.yml
 - update opensearch to use a single node

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* update upstream changes
 - remove 'opensearch integration' service
 - update the opensearch dashboard
 - add opensearch+dashboard to the minimized docker-compose.yml
 - update opensearch to use a single node

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* update upstream changes
 - remove 'opensearch integration' service
 - update the opensearch dashboard
 - add opensearch+dashboard to the minimized docker-compose.yml
 - update opensearch to use a single node

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* update upstream changes
 - remove 'opensearch integration' service
 - update the opensearch dashboard
 - add opensearch+dashboard to the minimized docker-compose.yml
 - update opensearch to use a single node

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* remove https in favor of http

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* restore https

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* update data-prepper config files

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* fix opensearch service name for jaeger-query

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* remove jaeger-query

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* update data-prepper pipelines

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* update data-prepper pipelines

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

* update data-prepper pipelines

Signed-off-by: YANGDB <yang.db.dev@gmail.com>

---------

Signed-off-by: YANGDB <yang.db.dev@gmail.com>
Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>
Signed-off-by: Anirudha (Ani) Jadhav <anirudha@nyu.edu>
Signed-off-by: Shenoy Pratik <sgguruda@amazon.com>
Signed-off-by: Mitchell Gale <Mitchell.Gale@improving.com>
Signed-off-by: Pierre Tessier <pierre@pierretessier.com>
Signed-off-by: Oscar Reyes <oscar-rreyes1@hotmail.com>
Co-authored-by: Juliano Costa <julianocosta89@outlook.com>
Co-authored-by: Carter Socha <43380952+cartersocha@users.noreply.github.com>
Co-authored-by: Ziqi Zhao <zhaoziqi9146@gmail.com>
Co-authored-by: Petr Styblo <styblope@gmail.com>
Co-authored-by: Brett McBride <brett@deakin.edu.au>
Co-authored-by: Alan West <3676547+alanwest@users.noreply.github.com>
Co-authored-by: mend-for-github-com[bot] <50673670+mend-for-github-com[bot]@users.noreply.github.com>
Co-authored-by: Joshua Li <joshuali925@gmail.com>
Co-authored-by: Anirudha (Ani) Jadhav <anirudha@nyu.edu>
Co-authored-by: Shenoy Pratik <sgguruda@amazon.com>
Co-authored-by: Mitchell Gale <Mitchell.gale@improving.com>
Co-authored-by: isabel <isaschm@users.noreply.github.com>
Co-authored-by: Pierre Tessier <pierre@pierretessier.com>
Co-authored-by: Austin Parker <austin@ap2.io>
Co-authored-by: Lancelot <lancel.huang.cs@gmail.com>
Co-authored-by: Piotr Kiełkowicz <pkiekowicz@splunk.com>
Co-authored-by: Robert Pająk <pellared@hotmail.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Lukas Malkmus <mail@lukasmalkmus.com>
Co-authored-by: Oscar Reyes <oscar-rreyes1@hotmail.com>
  • Loading branch information
1 parent 7c6f9c8 commit 59ec0e9
Show file tree
Hide file tree
Showing 5 changed files with 66 additions and 33 deletions.
31 changes: 2 additions & 29 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -634,33 +634,6 @@ services:
depends_on:
- opensearch

jaeger-query:
image: jaegertracing/jaeger-query:latest
container_name: jaeger-query
command:
- "--metrics-backend=prometheus"
- "--es.server-urls=https://opensearch-node1:9200"
- "--es.tls.enabled=true"
deploy:
resources:
limits:
memory: 300M
restart: unless-stopped
ports:
- 16685
environment:
- COLLECTOR_OTLP_ENABLED=true
- METRICS_STORAGE_TYPE=prometheus
- SPAN_STORAGE_TYPE=opensearch
- ES_TAGS_AS_FIELDS_ALL=true
- ES_USERNAME=admin
- ES_PASSWORD=admin
- ES_TLS_SKIP_HOST_VERIFY=true
depends_on:
- opensearch-node1
- opensearch-node2
logging: *logging

jaeger-agent:
image: jaegertracing/jaeger-agent:latest
container_name: jaeger-agent
Expand Down Expand Up @@ -716,7 +689,6 @@ services:
- "8888" # metrics endpoint

Check warning on line 689 in docker-compose.yml

View workflow job for this annotation

GitHub Actions / yamllint

689:16 [comments] too few spaces before comment
depends_on:
- jaeger-agent
- jaeger-query
- data-prepper
logging: *logging
environment:
Expand Down Expand Up @@ -781,8 +753,9 @@ services:
# OpenSearch store - dashboard
data-prepper:
image: opensearchproject/data-prepper:latest
container_name: dataprepper
volumes:
- ./src/dataprepper/pipelines.yaml:/usr/share/data-prepper/pipelines/pipelines.yaml
- ./src/dataprepper/pipelines.yaml:/usr/share/data-prepper/pipelines/trace_analytics_no_ssl_2x.yml
- ./src/dataprepper/data-prepper-config.yaml:/usr/share/data-prepper/config/data-prepper-config.yaml
ports:
- "21892"
Expand Down
61 changes: 60 additions & 1 deletion src/dataprepper/trace_analytics_no_ssl_2x.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,4 +34,63 @@ service-map-pipeline:
insecure: true
username: "admin"
password: "admin"
index_type: trace-analytics-service-map
index_type: trace-analytics-service-map
trace-error-metrics-pipeline:
source:
pipeline:
name: "span-pipeline"
processor:
- aggregate:
identification_keys: ["serviceName", "traceId"]
action:
count:
group_duration: "20s"
aggregate_when: "/status_code == 2"
sink:
- opensearch:
hosts: ["https://opensearch:9200"]
insecure: true
username: "admin"
password: "admin"
index: trace-error-metrics
trace-high-latency-metrics-pipeline:
source:

Check failure on line 57 in src/dataprepper/trace_analytics_no_ssl_2x.yml

View workflow job for this annotation

GitHub Actions / yamllint

57:5 [indentation] wrong indentation: expected 2 but found 4
pipeline:
name: "span-pipeline"
processor:
- aggregate:
identification_keys: ["serviceName", "traceId"]
action:
histogram:
key: "durationInNaos"
record_minmax: true
units: "nanoseconds"
buckets: [1000000000, 1500000000, 2000000000]
group_duration: "20s"
aggregate_when: "/durationInNanos > 1000000000"
sink:
- opensearch:
hosts: ["https://opensearch:9200"]
insecure: true
username: "admin"
password: "admin"
index: trace-high-latency-metrics
trace-normal-pipeline:
source:
pipeline:
name: "span-pipeline"
processor:
- aggregate:
identification_keys: ["serviceName"]
action:
percent_sampler:
percent: 60
group_duration: "30s"
aggregate_when: "/status_code != 2 and /durationInNanos <= 1000000000"
sink:
- opensearch:
hosts: ["https://opensearch:9200"]
insecure: true
username: "admin"
password: "admin"
index: sampled-traces

Check failure on line 96 in src/dataprepper/trace_analytics_no_ssl_2x.yml

View workflow job for this annotation

GitHub Actions / yamllint

96:30 [new-line-at-end-of-file] no new line character at the end of file
2 changes: 1 addition & 1 deletion src/otelcollector/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ After processing, the data is sent to the configured exporters:

3. **spanmetrics:** This is likely a custom exporter defined as a connector in your configuration. It seems to be designed to create metrics from spans, but this is mostly conjecture based on the name.

4. **otlp/2:** This exporter sends data to an endpoint configured as data-prepper:21890 over OTLP with TLS encryption in insecure mode.
4. **otlp/logs:** This exporter sends data to an endpoint configured as dataprepper:21892 over OTLP with TLS encryption in insecure mode.

### Metrics
**Metrics Pipeline**
Expand Down
3 changes: 2 additions & 1 deletion src/otelcollector/otelcol-config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ exporters:
endpoint: "dataprepper:21892"
tls:
insecure: true
insecure_skip_verify: true
otlphttp/prometheus:
endpoint: "http://prometheus:9090/api/v1/otlp"
tls:
Expand Down Expand Up @@ -53,7 +54,7 @@ service:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlp, debug, spanmetrics]
exporters: [otlp, debug, spanmetrics, otlp/logs]
metrics:
receivers: [otlp, spanmetrics]
processors: [filter/ottl, transform, batch]
Expand Down
2 changes: 1 addition & 1 deletion test/tracetesting/tracetest-provision.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ spec:
name: Jaeger
type: jaeger
jaeger:
endpoint: jaeger-query:16685
endpoint: jaeger:16685
tls:
insecure: true
---
Expand Down

0 comments on commit 59ec0e9

Please sign in to comment.