Add compressed file collection support to filelog receiver #2328
Comments
|
Are there any updates around collection of compressed .gz files and processing that data to be readable? |
|
This is definitely a feature that needs to be added, but the priority has been on stabilizing the log data model and reworking the internals of this library to reflect the latest data model. My best guess, barring a contribution from someone else, is that this will be completed in the next quarter. |
|
This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping |
|
Any update on this? Would love to see compressed file log support |
|
Removing stale and inviting contributions for this issue. @SaxyPandaBear please feel free to help and send a patch if you'd like. |
|
Pinging code owners for receiver/filelog: @djaglowski. See Adding Labels via Comments if you do not have permissions to add labels yourself. |
|
This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
|
This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
|
This issue has been closed as inactive because it has been stale for 120 days with no activity. |
|
I find myself needing this again. Reopening. |
|
If this issue is still available, I would like to work on this. CC @evan-bradley |
|
@bacherfl, thank you. I'm curious to see how the implementation could look. Feel free to bring a rough proposal here first if you have a clear idea. |
Hi @djaglowski, I was looking into this yesterday and tried out a couple of approaches - I now have a (very rough) proof of concept that seems to do the trick: #33406 - Feel free to have a look and see if this makes sense to you - I'm sure there's a lot that can be improved or done differently so I appreciate any feedback :) |
…33406) **Description:** This PR adds support for reading gzip compressed log files for the file log receiver. This is done by, if enabled via the `gzip_file_suffix` parameter, creating a `gzip.Reader` on top of the file handle of a compressed file. **Link to tracking Issue:** #2328 **Testing:** Added unit tests for the new functionality. Manually tested using the following configuration for the filelog receiver: ``` filelog: include: [ ./simple.log*.gz ] start_at: beginning gzip_file_suffix: ".gz" operators: - type: regex_parser regex: '^(?P<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (?P<sev>[A-Z]*) (?P<msg>.*)$' timestamp: parse_from: attributes.time layout: '%Y-%m-%d %H:%M:%S' severity: parse_from: attributes.sev ``` **Documentation:** Added documentation in the readme of the file log receiver --------- Signed-off-by: Florian Bacher <florian.bacher@dynatrace.com> Co-authored-by: Dan Jaglowski <jaglows3@gmail.com>
…pen-telemetry#33406) **Description:** This PR adds support for reading gzip compressed log files for the file log receiver. This is done by, if enabled via the `gzip_file_suffix` parameter, creating a `gzip.Reader` on top of the file handle of a compressed file. **Link to tracking Issue:** open-telemetry#2328 **Testing:** Added unit tests for the new functionality. Manually tested using the following configuration for the filelog receiver: ``` filelog: include: [ ./simple.log*.gz ] start_at: beginning gzip_file_suffix: ".gz" operators: - type: regex_parser regex: '^(?P<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (?P<sev>[A-Z]*) (?P<msg>.*)$' timestamp: parse_from: attributes.time layout: '%Y-%m-%d %H:%M:%S' severity: parse_from: attributes.sev ``` **Documentation:** Added documentation in the readme of the file log receiver --------- Signed-off-by: Florian Bacher <florian.bacher@dynatrace.com> Co-authored-by: Dan Jaglowski <jaglows3@gmail.com>
…pen-telemetry#33406) **Description:** This PR adds support for reading gzip compressed log files for the file log receiver. This is done by, if enabled via the `gzip_file_suffix` parameter, creating a `gzip.Reader` on top of the file handle of a compressed file. **Link to tracking Issue:** open-telemetry#2328 **Testing:** Added unit tests for the new functionality. Manually tested using the following configuration for the filelog receiver: ``` filelog: include: [ ./simple.log*.gz ] start_at: beginning gzip_file_suffix: ".gz" operators: - type: regex_parser regex: '^(?P<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (?P<sev>[A-Z]*) (?P<msg>.*)$' timestamp: parse_from: attributes.time layout: '%Y-%m-%d %H:%M:%S' severity: parse_from: attributes.sev ``` **Documentation:** Added documentation in the readme of the file log receiver --------- Signed-off-by: Florian Bacher <florian.bacher@dynatrace.com> Co-authored-by: Dan Jaglowski <jaglows3@gmail.com>
No description provided.
The text was updated successfully, but these errors were encountered: