feat(plugin): support Prometheus Scrape Metrics (#355) and Prometheus… #362
Conversation
… Remote Write plugins(#348) Signed-off-by: tl <361944459@qq.com>
| HTTPUser *plugins.Secret `json:"httpUser,omitempty"` | ||
| // Basic Auth Password. | ||
| // Requires HTTP_user to be se | ||
| HTTPPasswd *plugins.Secret `json:"httpPassword,omitempty"` |
There was a problem hiding this comment.
| HTTPPasswd *plugins.Secret `json:"httpPassword,omitempty"` | |
| HTTPPasswd *plugins.Secret `json:"httpPasswd,omitempty"` |
should be consistent with http_passwd https://docs.fluentbit.io/manual/pipeline/outputs/prometheus-remote-write
| //Log the response payload within the Fluent Bit log,default: false | ||
| LogResponsePayload *bool `json:"logResponsePayload,omitempty"` | ||
| //This allows you to add custom labels to all metrics exposed through the prometheus exporter. You may have multiple of these fields | ||
| AddLabels []string `json:"addLabels,omitempty"` |
There was a problem hiding this comment.
The AddLabels should be similar to Headers, allowing to add multiple kv pairs just like opentelemetry plugin did:
https://docs.fluentbit.io/manual/pipeline/outputs/prometheus-remote-write
https://docs.fluentbit.io/manual/pipeline/outputs/opentelemetry
| // +kubebuilder:validation:Maximum:=65535 | ||
| Port *int32 `json:"port,omitempty"` | ||
| // The interval to scrape metrics, default: 10s | ||
| ScrapeInterval *int32 `json:"scrapeInterval,omitempty"` |
There was a problem hiding this comment.
Is this should be a string or duration?
There was a problem hiding this comment.
I mimicked the node exporter Metrics plug-in, which defines Int32
There was a problem hiding this comment.
After testing to determine it
There was a problem hiding this comment.
It should be a string and the support value format is 1, 1s, 1D, 1d, 1H, 1H, 1M, 1m
There was a problem hiding this comment.
Maybe you're right. Here's the example
There was a problem hiding this comment.
I mimicked the node exporter Metrics plug-in, which defines Int32
@tonyzaizai Would you help to update the node exporter input plugin as well?
https://github.com/fluent/fluent-operator/blob/master/apis/fluentbit/v1alpha2/plugins/input/node_exporter_metrics_types.go#L18
According to the discussion we've had with the fluent community, it should be like 10s, 1m https://fluent-all.slack.com/archives/C02642TE9HR/p1658215209357549
And we need some test for this plugin too. Maybe you can run a fluentibt binary to collect node metrics and see if config like 15s works
There was a problem hiding this comment.
I tested it with the fluent-bit binary and it works for 1 or 1s or 1M
The Node Expoter input plugin should be the same
There was a problem hiding this comment.
The interval is indeed one minute
There was a problem hiding this comment.
sounds good. It's in format 10s, 1m...
| // Specify an HTTP Proxy. The expected format of this value is http://HOST:PORT. | ||
| Proxy string `json:"proxy,omitempty"` | ||
| //Specify an optional HTTP URI for the target web server, e.g: /something ,default: / | ||
| Uri string `json:"uri,omitempty"` |
There was a problem hiding this comment.
Uri string `json:"uri,omitempty"`
=>
URI string `json:"uri,omitempty"`
| // +kubebuilder:object:generate:=true | ||
|
|
||
| // An output plugin to submit Prometheus Metrics using the remote write protocol | ||
| type PrometheusRemoteWrite struct { |
There was a problem hiding this comment.
Why does it not support workers?
…_write plugins Signed-off-by: tl <361944459@qq.com>
…_metrics plugins Signed-off-by: tl <361944459@qq.com>
| } | ||
|
|
||
| func (_ *PrometheusScrapeMetrics) Name() string { | ||
| return "prometheus_scrape" |
There was a problem hiding this comment.
prometheus_scrape_metrics is ok instead of prometheus_scrape
| @@ -26,6 +27,14 @@ var inputExpected = `[Input] | |||
| Tag logs.foo.bar | |||
| Rate 3 | |||
| Samples 5 | |||
| [Input] | |||
| Name prometheus_scrape | |||
There was a problem hiding this comment.
prometheus_scrape_metrics is ok instead of prometheus_scrape
There was a problem hiding this comment.
It seems that it must be the "prometheus_scrape" field
There was a problem hiding this comment.
What's the content of the configuration file you used?
Does the name of the plugin in the configuration file is set to prometheus_scape?
There was a problem hiding this comment.
"prometheus_scrape_metrics" is not effective
"prometheus_scrape" is ok
… Remote Write plugins(#348)
Signed-off-by: tl 361944459@qq.com
What this PR does / why we need it:
Which issue(s) this PR fixes:
Fixes #348 and #355
Does this PR introduced a user-facing change?
Additional documentation, usage docs, etc.: