flowchart LR

Fluent-bit -->|push| c-p3030
App -->|push metrics & traces|c-p8889

Prometheus -.->|pull metrics| c-p8889

collector -->|push logs| Loki-3100

Grafana -..->|query logs| Loki-3100

subgraph Loki
    Loki-3100(:3100)
end

subgraph collector[Otel Collector]
    c-p3030(:3030)
    c-p8889(:8889)
end

subgraph Legend
    direction LR
    start1[ ] --->|push| stop1[ ]
    style start1 height:0px;
    style stop1 height:0px;
    start2[ ] -..->|pull| stop2[ ]
    style start2 height:0px;
    style stop2 height:0px; 
end

style Legend fill:none

Loading

Debug

kubectl run curl --image=radial/busyboxplus:curl -i --tty --rm

Cluster Settings

Enable storage for Loki:

microk8s enable hostpath-storage

Loki

kubectl port-forward --namespace default svc/loki-gateway 3100:80

export LOKI_ADDR=http://127.0.0.1:3100

logcli labels

logcli query '{exporter="OTLP"}'

Grafana

# Get admin password
kubectl get secret --namespace default grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

# Port-forwarding
export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace default port-forward $POD_NAME 3000

Prometheus

export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=prometheus,app.kubernetes.io/instance=prometheus" -o jsonpath="{.items[0].metadata.name}")

kubectl --namespace default port-forward $POD_NAME 9090

Resources

Fluent-bit

• Filters | Kubernetes
• Helm values.yaml
• Inputs | Tail
• Outputs | OpenTelemetry

OpenTelemetry Collector

• Loki Exporter
• Recommended Processors
  • Batch Processor
• Helm values.yaml

Grafana Loki

• Loki overview
• Helm values.yaml

Grafana

• Helm values.yaml