Namespaced FluentBit CRDs and configs

[adiforluls]

Is your feature request related to a problem? Please describe.

In #521 I raised the request for having namespaced FluentBit CRDs and configs.

Following up on the discussion we had there, I don't think an external plugin is needed for namespace based multiple log routing in FluentBit daemonset. Namespace isolation of logs is a critical feature in my opinion. The two basic requirements being:

• A namespace admin/user should be able to create CRs to process logs coming out of their application namespace.
• The CRs should not affect logs coming out of other namespaces (i.e. a malicious CR should not be able to gain unwanted access of logs coming out of other namespace).

For a k8s based input, I have a proposal to achieve namespace logging with FluentBit Daemonset.

Describe the solution you'd like

The basic idea is to process the CR configs and mutate certain fields that make use of k8s metadata. Tags and match rules make the log routing possible in FluentBit, and for k8s based input (tail) tags hold k8s metadata in the form of <tag_prefix>.var.log.containers.<pod_name><namespace_name><container_name>.log.

The proposal is to form a hash based on namespace name per namespace and add it to the tag as prefix. The match rule will also be mutated accordingly. For this task the rewrite_tag filter plugin will be used.

Below is the pseudo-code.

• Controller detects the namespaced resources.

• Controller creates a rewrite_tag plugin filter config to mutate the tag for log records coming out of the target namespace to include hash of namespace as the tag prefix.

• The rewrite_tag filter config should be out of control of any user and will be handled by the operator (the user should not be able to mutate it).

• Mutate the match rule to add target namespace’s hash prefix.

• Security:

  • Since the tag and match rules will be mutated to include the hashed namespace prefix by the operator, it's not possible for a user to create a malicious CR in another namespace since they can't set the prefix for tag and match that dictate the log routing for a namespace.

  • E.g.: If a malicious user creates a CR in namespace ns2 with a match rule h[ns1]., the controller will mutate this match to h[ns2].h[ns1]., so the user won't be able to get unwanted access to other namespace logs.

  • It might make sense to allow these tags and rules in Cluster level CRs though.

• Namespaced Parsers:

  • Append the namespace hash to the parser name as a suffix in the actual FluentBit configs.

    • my-regex → my-regex-h[ns1]

  • Filters making use of parsers by name will be updated accordingly in the final config by the controller.

    • If the filter has specified a parser named my-regex, search for a parser with this name in the same namespace. If the parser exists, the parser section of the filter will be updated in the final fluentbit config to use my-regex-h[ns1] by the controller.

    • If it does not exist, then check if it's a Cluster level resource. Include this parser if it is, ignore/omit otherwise.

Below is an example of the final rewrite_tag config to mutate the tag coming from namespace ns1 by the controller.

[FILTER]
    Name          rewrite_tag
    Match         kube.*.*_ns1_*.log
    Rule          $log "^.*\[ns1\].*$"  h[ns1].$TAG false

The requirement will be to make Filters, Outputs and Parsers to be namespaced at least. The below examples assumes input to be Cluster level.

Example CRs and corresponding FluentBit configs:

apiVersion: fluentbit.fluent.io/v1alpha2
kind: Filter
metadata:
  name: kubernetes
  namespace: ns1
  labels:
    fluentbit.fluent.io/enabled: "true"
    fluentbit.fluent.io/mode: "k8s"
spec:
  match: kube.*
  filters:
  - parser:
      keyName: log
      parser: my-regex
---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: Output
metadata:
  name: ns1-output
  namespace: ns1
  labels:
    fluentbit.fluent.io/enabled: "true"
    fluentbit.fluent.io/mode: "k8s"
spec:
  match: kube.*
  openSearch:
  	host: foo.bar
	port: 9200
---
apiVersion: fluentbit.fluent.io/v1alpha2
kind: Parser
metadata:
  name: my-regex
  namespace: ns1
  labels:
    fluentbit.fluent.io/enabled: "true"
spec:
  regex:
    timeKey: time
    timeFormat: "%d/%b/%Y:%H:%M:%S %z"
    types: "code:integer size:integer"
    regex: '^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$'

[FILTER]
    Name parser
    Match h[ns1].kube.*
    Parser my_regex-h[ns1]
    KeyName log
 
[Output]
    Name opensearch
    Match h[ns1].kube.*
    Host foo.bar
    Port 9200
 
[PARSER]
    Name my-regex-h[ns1]
    TimeKey time
    TimeFormat "%d/%b/%Y:%H:%M:%S %z"
    Types "code:integer size:integer"
    Regex '^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$'

Additional context

This is a possible approach/algorithm, we can definitely discuss, refine and enhance the idea. Suggestions are welcome.
I'm open to contribute and willing to make this enhancement if there's an agreement.

[adiforluls]

@benjaminhuo et al... any feedback on the proposal? The proposed namespace algorithm and controller design can be put into a separate controller (fluentbit_namespace_controller.go) or the existing fluentbitconfig_controller.go.

I'm willing to contribute and can get started on implementing the proposal if we have an agreement. If you'd like to see any revisions or adjustments in the proposal let me know.

[benjaminhuo]

@benjaminhuo et al... any feedback on the proposal? The proposed namespace algorithm and controller design can be put into a separate controller (fluentbit_namespace_controller.go) or the existing fluentbitconfig_controller.go.

I'm willing to contribute and can get started on implementing the proposal if we have an agreement. If you'd like to see any revisions or adjustments in the proposal let me know.

@adiforluls I'm sorry for my late response to this extraordinary proposal. The logic is solid and strong.
Would you please list all the namespaced CRDs you want to create? Looks like you're going to combine cluster CRDs with namespaced CRDs?

Previously in our design, the namespace isolation logic relies on Fluentd. You'll need to forward the log fluent bit collected to Fluentd and Fluentd will do the namespaced log forwarding.

Implementing this in FluentBit is good. The fluentbit process in the fluentbit pod will be restarted whenever the config changes to pick up the new config, the fluentbit pod will not be restarted.

@wanjunlei what do you think about this proposal?

[benjaminhuo]

The proposed namespace algorithm and controller design can be put into a separate controller (fluentbit_namespace_controller.go) or the existing fluentbitconfig_controller.go.

In the existing controller might be ok just like fluentd does https://github.com/fluent/fluent-operator/blob/master/controllers/fluentdconfig_controller.go#L529

You can see the single controller is responsible for reconcile both namespaced and clustered CRD

[wanjunlei]

@benjaminhuo et al... any feedback on the proposal? The proposed namespace algorithm and controller design can be put into a separate controller (fluentbit_namespace_controller.go) or the existing fluentbitconfig_controller.go.
I'm willing to contribute and can get started on implementing the proposal if we have an agreement. If you'd like to see any revisions or adjustments in the proposal let me know.

@adiforluls I'm sorry for my late response to this extraordinary proposal. The logic is solid and strong. Would you please list all the namespaced CRDs you want to create? Looks like you're going to combine cluster CRDs with namespaced CRDs?

Previously in our design, the namespace isolation logic relies on Fluentd. You'll need to forward the log fluent bit collected to Fluentd and Fluentd will do the namespaced log forwarding.

Implementing this in FluentBit is good. The fluentbit process in the fluentbit pod will be restarted whenever the config changes to pick up the new config, the fluentbit pod will not be restarted.

@wanjunlei what do you think about this proposal?

Maybe we can allow users to create a standalone daemonset in their namespace to avoid this?

[adiforluls]

@benjaminhuo et al... any feedback on the proposal? The proposed namespace algorithm and controller design can be put into a separate controller (fluentbit_namespace_controller.go) or the existing fluentbitconfig_controller.go.
I'm willing to contribute and can get started on implementing the proposal if we have an agreement. If you'd like to see any revisions or adjustments in the proposal let me know.

@adiforluls I'm sorry for my late response to this extraordinary proposal. The logic is solid and strong. Would you please list all the namespaced CRDs you want to create? Looks like you're going to combine cluster CRDs with namespaced CRDs?

Previously in our design, the namespace isolation logic relies on Fluentd. You'll need to forward the log fluent bit collected to Fluentd and Fluentd will do the namespaced log forwarding.

Implementing this in FluentBit is good. The fluentbit process in the fluentbit pod will be restarted whenever the config changes to pick up the new config, the fluentbit pod will not be restarted.

@wanjunlei what do you think about this proposal?

Thanks for the feedback Benjamin! Namespace Filter, Parser, Output, and probably a namespace FluentBitConfig which selects these 3 namespace CRs via labels (and FluentBit CR selects FluentBitConfig CR via labels as well), and additionally selects a ClusterInput (again via labels).

Unlike FluentdConfig which selects both cluster and namespace level resources, I'm thinking FluentBitConfig only selects namespace resources. ClusterInput will be selected only to decide the tag and match rule mutation and generate the rewrite_tag config. The Cluster level configs would work anyway for all the logs when specified with a wildcard match rule.

[adiforluls]

@benjaminhuo et al... any feedback on the proposal? The proposed namespace algorithm and controller design can be put into a separate controller (fluentbit_namespace_controller.go) or the existing fluentbitconfig_controller.go.
I'm willing to contribute and can get started on implementing the proposal if we have an agreement. If you'd like to see any revisions or adjustments in the proposal let me know.

@adiforluls I'm sorry for my late response to this extraordinary proposal. The logic is solid and strong. Would you please list all the namespaced CRDs you want to create? Looks like you're going to combine cluster CRDs with namespaced CRDs?
Previously in our design, the namespace isolation logic relies on Fluentd. You'll need to forward the log fluent bit collected to Fluentd and Fluentd will do the namespaced log forwarding.
Implementing this in FluentBit is good. The fluentbit process in the fluentbit pod will be restarted whenever the config changes to pick up the new config, the fluentbit pod will not be restarted.
@wanjunlei what do you think about this proposal?

Maybe we can allow users to create a standalone daemonset in their namespace to avoid this?

That would just bump up the resource requirements, which goes against the idea of running only FluentBit since it's less resource intensive. Imagine if, say, 100 namespace admins/users spin up a daemonset for their namespace.

It's ideal to have one daemonset that handles and isolates logs coming out of hundreds of namespaces than having a separate daemonset for each namespace.

[benjaminhuo]

That would just bump up the resource requirements, which goes against the idea of running FluentBit since it's less resource intensive. Imagine if, say, 100 namespace admins/users spin up a daemonset for their namespace.

It's ideal to have one daemonset that handles and isolates logs coming out of hundreds of namespaces than having a separate daemonset for each namespace.

Yep, agree

[adiforluls]

The proposed namespace algorithm and controller design can be put into a separate controller (fluentbit_namespace_controller.go) or the existing fluentbitconfig_controller.go.

In the existing controller might be ok just like fluentd does https://github.com/fluent/fluent-operator/blob/master/controllers/fluentdconfig_controller.go#L529

You can see the single controller is responsible for reconcile both namespaced and clustered CRD

Yeah sounds good, thanks

[adiforluls]

@benjaminhuo et al... any feedback on the proposal? The proposed namespace algorithm and controller design can be put into a separate controller (fluentbit_namespace_controller.go) or the existing fluentbitconfig_controller.go.
I'm willing to contribute and can get started on implementing the proposal if we have an agreement. If you'd like to see any revisions or adjustments in the proposal let me know.

@adiforluls I'm sorry for my late response to this extraordinary proposal. The logic is solid and strong. Would you please list all the namespaced CRDs you want to create? Looks like you're going to combine cluster CRDs with namespaced CRDs?
Previously in our design, the namespace isolation logic relies on Fluentd. You'll need to forward the log fluent bit collected to Fluentd and Fluentd will do the namespaced log forwarding.
Implementing this in FluentBit is good. The fluentbit process in the fluentbit pod will be restarted whenever the config changes to pick up the new config, the fluentbit pod will not be restarted.
@wanjunlei what do you think about this proposal?

Thanks for the feedback Benjamin! Namespace Filter, Parser, Output, and probably a namespace FluentBitConfig which selects these 3 namespace CRs via labels (and have FluentBit CR select FluentBitConfig CR via labels), and additionally selects a ClusterInput (again via labels).

Unlike FluentdConfig which selects both cluster and namespace level resources, I'm thinking FluentBitConfig only selects namespace resources. ClusterInput will be selected only to decide the tag and match rule mutation and generate the rewrite_tag config. The Cluster level configs would work anyway for all the logs when specified with a wildcard match rule.

@benjaminhuo this is the only addition to the original proposal. So before I start generating the CRDs, I'll post the new CRDs here for folks to review by tomorrow.

[benjaminhuo]

@benjaminhuo this is the only addition to the original proposal. So before I start generating the CRDs, I'll post the new CRDs here for folks to review by tomorrow.

That'll be great!

[adiforluls]

@benjaminhuo Here's a list of proposed custom resources:

• Filter: Namespaced resource to filter logs coming out of the namespace where the resource exists. Spec identical to ClusterFilter's spec. Namespace Filter can use namespace and Cluster level parsers.
• Output: Namespaced resource to send logs coming out of the namespace where the resource exists to outputs. Spec identical to ClusterOutput's spec.
• Parser: Namespaced resource to parse logs coming out of the same namespace as the resource. Usable by Filter resource in the same namespace. Spec identical to ClusterParser's spec.
• FluentBitConfig: Namespaced resource to select namespace and required cluster level resources. Spec
  • Namespace resources to select: Parser, Filter and Output
  • Cluster level resources to select: ClusterParser (to be used by namespace Filter).

Example spec of FluentBitConfig (had to add Namespace prefix since FluentBitConfigSpec was already in use).

type NamespaceFluentBitConfigSpec struct {
	// Select filter plugins
	FilterSelector metav1.LabelSelector `json:"filterSelector,omitempty"`
	// Select output plugins
	OutputSelector metav1.LabelSelector `json:"outputSelector,omitempty"`
	// Select parser plugins
	ParserSelector metav1.LabelSelector `json:"parserSelector,omitempty"`
        // Select cluster level parser config
        ClusterParserSelector metav1.LabelSelector `json:"clusterParserSelector,omitempty"`
}

The FluentBitConfig CR will be selected via labelSelector in FluentBit CR.

The ClusterInput will be analyzed using the clusterFbCfg that maps to the FluentBit instance and should be of type Tail (since tail populates the required kubernetes metadata in its Tag).

Let me know your thoughts.

[benjaminhuo]

FluentBitConfig: Namespaced resource to select namespace and required cluster level resources. Spec

• Namespace resources to select: Parser, Filter and Output
• Cluster level resources to select: ClusterParser (to be used by namespace Filter).

Basically, your proposed types are ok.
We have FluentBitConfigSpec defined for the cluster config, using NamespaceFluentBitConfigSpec is ok.
@wanjunlei what do you think?