[exporter/debug] Debug exporter has sampling enabled by default

[andrzej-stencel]

Describe the bug

The Debug exporter has sampling enabled by default. I don't think this behavior matches the intended usage of the component.

Whenever I use the Debug exporter, it's to see all the information that goes into it. The debug exporter sampling things out is misleading to me.

What version did you use?

v0.97.0

Steps to reproduce

1. Prepare empty input file input.log

$ rm input.log; touch input.log

2. Start the collector (in the background, to reuse the console)

$ ./otelcol-contrib-0.97.0-linux_amd64 --config ./config.yaml &
2024-04-09T09:49:11.057+0200    info    service@v0.97.0/telemetry.go:97 Serving metrics {"address": ":8888", "level": "Basic"}
2024-04-09T09:49:11.057+0200    info    exporter@v0.97.0/exporter.go:275        Development component. May change in the future.        {"kind": "exporter", "data_type": "logs", "name": "debug"}
2024-04-09T09:49:11.059+0200    info    service@v0.97.0/service.go:143  Starting otelcol-contrib...     {"Version": "0.97.0", "NumCPU": 16}
2024-04-09T09:49:11.059+0200    info    extensions/extensions.go:34     Starting extensions...
2024-04-09T09:49:11.059+0200    info    adapter/receiver.go:45  Starting stanza receiver        {"kind": "receiver", "name": "filelog", "data_type": "logs"}
2024-04-09T09:49:11.059+0200    info    service@v0.97.0/service.go:169  Everything is ready. Begin running and processing data.
2024-04-09T09:49:11.059+0200    warn    localhostgate/featuregate.go:63 The default endpoints for all servers in components will change to use localhost instead of 0.0.0.0 in a future version. Use the feature gate to preview the new default.       {"feature gate ID": "component.UseLocalHostAsDefaultHost"}

$

3. Add 30 lines to the input.log file, 1 line every 0.1 seconds. This results in 30 lines written to the input.log file across 3 seconds, 10 lines every second.

$ for i in $(seq 1 30); do echo "log" >> input.log; sleep 0.1; done
2024-04-09T09:49:15.939+0200    info    fileconsumer/file.go:260        Started watching file   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "0408.log"}
2024-04-09T09:49:15.960+0200    info    LogsExporter    {"kind": "exporter", "data_type": "logs", "name": "debug", "resource logs": 1, "log records": 1}
2024-04-09T09:49:16.160+0200    info    LogsExporter    {"kind": "exporter", "data_type": "logs", "name": "debug", "resource logs": 1, "log records": 2}
2024-04-09T09:49:17.060+0200    info    LogsExporter    {"kind": "exporter", "data_type": "logs", "name": "debug", "resource logs": 1, "log records": 1}
2024-04-09T09:49:17.159+0200    info    LogsExporter    {"kind": "exporter", "data_type": "logs", "name": "debug", "resource logs": 1, "log records": 1}
2024-04-09T09:49:18.159+0200    info    LogsExporter    {"kind": "exporter", "data_type": "logs", "name": "debug", "resource logs": 1, "log records": 1}
2024-04-09T09:49:18.259+0200    info    LogsExporter    {"kind": "exporter", "data_type": "logs", "name": "debug", "resource logs": 1, "log records": 1}

What did you expect to see?

• I would expect to see all 30 entries output by the Debug exporter.

What did you see instead?

• I can only see 6 entries output by the Debug exporter - 2 entries per second. The rest is sampled out.

What config did you use?

exporters:
  debug:
    verbosity: basic
receivers:
  filelog:
    include:
    - input.log
    poll_interval: 40ms
    start_at: beginning
service:
  pipelines:
    logs:
      exporters:
      - debug
      receivers:
      - filelog

Environment

OS: Ubuntu 23.10

Workaround

I was able to get all 30 entries output by the Debug exporter with the following configuration:

exporters:
  debug:
    sampling_thereafter: 1
    verbosity: basic
receivers:
  filelog:
    include:
    - 0408.log
    poll_interval: 40ms
    start_at: beginning
service:
  pipelines:
    logs:
      exporters:
      - debug
      receivers:
      - filelog
  telemetry:
    logs:
      sampling:
        enabled: false

Note that just adding the exporters.debug.sampling_thereafter: 1 to the Debug exporter's configuration is not enough. I also had to add the service.telemetry.sampling.enabled: false to get the correct result. Confusing!

Additional context

[andrzej-stencel]

I propose to disable sampling by default:

• [exporter/debug] disable sampling by default #10250

I considered proposing removing sampling options from the exporter altogether, but ultimately decided against it. I've never used these options, but maybe someone does find them useful? I believe changing the default to not sample is what should satisfy everybody.

[andrzej-stencel]

With this PR merged:

• [exporter/debug] disable sampling by default #10250

sampling is now disabled by default in the exporter.

The other issue is that the collector's internal logger's sampling (configured in service::telemetry::sampling::enabled, enabled by default) can affect the output of the exporter. The workaround for this is to configure the Debug exporter to not use the collector's internal logger with use_internal_logger: false. This feature was added in:

• [exporter/debug] add option use_internal_logger #10227