Grafana Distributed Tracing Example with: Tempo, Prometheus, Loki, Grafana and Python Flask
Boot the stack:
docker compose up --build -d || docker-compose up -d
The datasources will be auto-configured defined in configs/grafana/datasources.yaml
.
Requests:
$ curl http://localhost:5002/process_payment
Payment processed
$ curl http://localhost:5001/create_order
Order created with id 1
$ curl http://localhost:5001/list_payments
Payments listed
Logs:
order-service | [2023-11-30 05:33:07,334] INFO in app: Trace ID: 1660e64b3807719aa4898445766895b8
order-service | 172.18.0.1 - - [30/Nov/2023 05:33:07] "GET /create_order HTTP/1.1" 200 -
payment-service | [2023-11-30 05:33:18,883] INFO in app: Trace ID: 335c0cd1cd947c3de92b7cc9a06386e9
payment-service | 172.18.0.1 - - [30/Nov/2023 05:33:18] "GET /process_payment HTTP/1.1" 200 -
Explore traces:
Query:
Node Graph:
When we use the payment-service
container logs in Loki:
With span.set_attribute
we can enrich some of the visuals:
with tracer.start_as_current_span("database_operation") as span:
sql_query = "INSERT INTO order (description) VALUES ('New Order')"
span.set_attribute("db.statement", sql_query) # Add SQL query to span
Produces: