Convert kubernetes events logs from logfmt to json

[z0rc]

I have simple agent deployment with following config:

loki.source.kubernetes_events "events" {
  forward_to = [loki.write.central.receiver]
}

loki.write "central" {
  endpoint {
    url = "..."
  }
  external_labels = {
    "cluster" = "dev",
  }
}

It works okay and as expected, Kubernetes events are available in Loki.

My problem is that events posted to loki are formatted with logfmt. And this format has nasty edge cases around multiline strings, for example event for failed readiness probe includes multiline python traceback. I hope that by using json instead logfmt this edge case can be mitigated. But I just cannot find a way to express this in configuration logic. Is there a way to convert loki.source.kubernetes_events to json somewhere in pipeline?

[tpaschalis]

Hey there @z0rc, apologies for the belated response!

To process log entries, take a look at the loki.process component. More specifically, you can make use of the stage.multiline block to collapse these tracebacks into a single log entry.

The underlying issue would not be fixed just by using JSON since they'd still be separate lines in the log file the Agent is tailing, and by default would be separate entries.

[z0rc]

Thanks, I'll check it out.

Though probably I must clarify the issue. Even with logfmt format the string is multiline already, so there is no need to join multiple events. It's a case where having multiline string in logfmt formatted events can break Loki's logfmt processor, leading to inability parse event and extract fields.

I guess what I need it is loki.source.kubernetes_events to emit events formatted as json.

Also see #3313