Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[configgrpc] Use own compressors for zstd #10323

Merged
merged 3 commits into from
Jun 5, 2024
Merged

[configgrpc] Use own compressors for zstd #10323

merged 3 commits into from
Jun 5, 2024

Conversation

jpkrohling
Copy link
Member

Uses our own version of the zstd compressor for gRPC servers. The code for it is based on the gzip compressor that comes built-in with gRPC.

Benchmarks before this PR:

Running tool: /usr/bin/go test -benchmem -run=^$ -bench ^BenchmarkCompressors$ go.opentelemetry.io/collector/config/configgrpc

sm_log_requestgoos: linux
goarch: amd64
pkg: go.opentelemetry.io/collector/config/configgrpc
cpu: 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz
BenchmarkCompressors/sm_log_request/raw_bytes_160/compressed_bytes_162/compressor_gzip-16         	   71594	     19066 ns/op	     615 B/op	       4 allocs/op
sm_log_requestBenchmarkCompressors/sm_log_request/raw_bytes_160/compressed_bytes_159/compressor_zstd-16         	  151503	      8544 ns/op	     640 B/op	       6 allocs/op
sm_log_requestBenchmarkCompressors/sm_log_request/raw_bytes_160/compressed_bytes_178/compressor_snappy-16       	 3632570	       303.8 ns/op	     304 B/op	       3 allocs/op
md_log_requestBenchmarkCompressors/md_log_request/raw_bytes_242/compressed_bytes_219/compressor_gzip-16         	   68114	     16938 ns/op	     748 B/op	       4 allocs/op
md_log_requestBenchmarkCompressors/md_log_request/raw_bytes_242/compressed_bytes_209/compressor_zstd-16         	  138091	      8047 ns/op	     896 B/op	       6 allocs/op
md_log_requestBenchmarkCompressors/md_log_request/raw_bytes_242/compressed_bytes_260/compressor_snappy-16       	 3081198	       402.5 ns/op	     400 B/op	       3 allocs/op
lg_log_requestBenchmarkCompressors/lg_log_request/raw_bytes_4850/compressed_bytes_253/compressor_gzip-16        	   43414	     27174 ns/op	     386 B/op	       3 allocs/op
lg_log_requestBenchmarkCompressors/lg_log_request/raw_bytes_4850/compressed_bytes_216/compressor_zstd-16        	  117534	      9903 ns/op	   10112 B/op	       6 allocs/op
lg_log_requestBenchmarkCompressors/lg_log_request/raw_bytes_4850/compressed_bytes_454/compressor_snappy-16      	 1000000	      1190 ns/op	     528 B/op	       2 allocs/op
sm_trace_requestBenchmarkCompressors/sm_trace_request/raw_bytes_231/compressed_bytes_203/compressor_gzip-16       	   67275	     17508 ns/op	     700 B/op	       4 allocs/op
sm_trace_requestBenchmarkCompressors/sm_trace_request/raw_bytes_231/compressed_bytes_201/compressor_zstd-16       	  196862	      6137 ns/op	     848 B/op	       6 allocs/op
sm_trace_requestBenchmarkCompressors/sm_trace_request/raw_bytes_231/compressed_bytes_220/compressor_snappy-16     	 3595815	       331.7 ns/op	     272 B/op	       2 allocs/op
md_trace_requestBenchmarkCompressors/md_trace_request/raw_bytes_329/compressed_bytes_249/compressor_gzip-16       	   64105	     19104 ns/op	     844 B/op	       4 allocs/op
md_trace_requestBenchmarkCompressors/md_trace_request/raw_bytes_329/compressed_bytes_256/compressor_zstd-16       	  169221	      6929 ns/op	    1120 B/op	       6 allocs/op
md_trace_requestBenchmarkCompressors/md_trace_request/raw_bytes_329/compressed_bytes_279/compressor_snappy-16     	 2602239	       473.0 ns/op	     336 B/op	       2 allocs/op
lg_trace_requestBenchmarkCompressors/lg_trace_request/raw_bytes_7025/compressed_bytes_303/compressor_gzip-16      	   33861	     36473 ns/op	     904 B/op	       4 allocs/op
lg_trace_requestBenchmarkCompressors/lg_trace_request/raw_bytes_7025/compressed_bytes_258/compressor_zstd-16      	  107828	     10596 ns/op	   16832 B/op	       6 allocs/op
lg_trace_requestBenchmarkCompressors/lg_trace_request/raw_bytes_7025/compressed_bytes_591/compressor_snappy-16    	  725080	      1540 ns/op	     689 B/op	       2 allocs/op
sm_metric_requestBenchmarkCompressors/sm_metric_request/raw_bytes_183/compressed_bytes_140/compressor_gzip-16      	   76315	     16394 ns/op	     496 B/op	       4 allocs/op
sm_metric_requestBenchmarkCompressors/sm_metric_request/raw_bytes_183/compressed_bytes_137/compressor_zstd-16      	  193314	      5957 ns/op	     688 B/op	       6 allocs/op
sm_metric_requestBenchmarkCompressors/sm_metric_request/raw_bytes_183/compressed_bytes_152/compressor_snappy-16    	 3558649	       345.2 ns/op	     208 B/op	       2 allocs/op
md_metric_requestBenchmarkCompressors/md_metric_request/raw_bytes_376/compressed_bytes_194/compressor_gzip-16      	   68497	     18413 ns/op	     699 B/op	       4 allocs/op
md_metric_requestBenchmarkCompressors/md_metric_request/raw_bytes_376/compressed_bytes_198/compressor_zstd-16      	  177841	      6520 ns/op	    1136 B/op	       6 allocs/op
md_metric_requestBenchmarkCompressors/md_metric_request/raw_bytes_376/compressed_bytes_222/compressor_snappy-16    	 2354102	       497.4 ns/op	     272 B/op	       2 allocs/op
lg_metric_requestBenchmarkCompressors/lg_metric_request/raw_bytes_10991/compressed_bytes_601/compressor_gzip-16    	   21943	     54603 ns/op	    1941 B/op	       5 allocs/op
lg_metric_requestBenchmarkCompressors/lg_metric_request/raw_bytes_10991/compressed_bytes_559/compressor_zstd-16    	   71260	     16077 ns/op	   25312 B/op	       6 allocs/op
lg_metric_requestBenchmarkCompressors/lg_metric_request/raw_bytes_10991/compressed_bytes_1055/compressor_snappy-16 	  335415	      3026 ns/op	    1200 B/op	       2 allocs/op
PASS
ok  	go.opentelemetry.io/collector/config/configgrpc	37.766s

After this version:

Running tool: /usr/bin/go test -benchmem -run=^$ -bench ^BenchmarkCompressors$ go.opentelemetry.io/collector/config/configgrpc

sm_log_requestgoos: linux
goarch: amd64
pkg: go.opentelemetry.io/collector/config/configgrpc
cpu: 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz
BenchmarkCompressors/sm_log_request/raw_bytes_160/compressed_bytes_162/compressor_gzip-16         	   74952	     15710 ns/op	     603 B/op	       4 allocs/op
sm_log_requestBenchmarkCompressors/sm_log_request/raw_bytes_160/compressed_bytes_159/compressor_zstd-16         	  156784	      6966 ns/op	     208 B/op	       2 allocs/op
sm_log_requestBenchmarkCompressors/sm_log_request/raw_bytes_160/compressed_bytes_178/compressor_snappy-16       	 2216174	       510.4 ns/op	     308 B/op	       3 allocs/op
md_log_requestBenchmarkCompressors/md_log_request/raw_bytes_242/compressed_bytes_219/compressor_gzip-16         	   68095	     18569 ns/op	     736 B/op	       4 allocs/op
md_log_requestBenchmarkCompressors/md_log_request/raw_bytes_242/compressed_bytes_209/compressor_zstd-16         	  150705	      8849 ns/op	     294 B/op	       2 allocs/op
md_log_requestBenchmarkCompressors/md_log_request/raw_bytes_242/compressed_bytes_260/compressor_snappy-16       	 2149710	       556.8 ns/op	     406 B/op	       3 allocs/op
lg_log_requestBenchmarkCompressors/lg_log_request/raw_bytes_4850/compressed_bytes_253/compressor_gzip-16        	   40040	     26159 ns/op	     368 B/op	       3 allocs/op
lg_log_requestBenchmarkCompressors/lg_log_request/raw_bytes_4850/compressed_bytes_216/compressor_zstd-16        	  123043	     10254 ns/op	     299 B/op	       2 allocs/op
lg_log_requestBenchmarkCompressors/lg_log_request/raw_bytes_4850/compressed_bytes_454/compressor_snappy-16      	  726780	      1457 ns/op	     533 B/op	       2 allocs/op
sm_trace_requestBenchmarkCompressors/sm_trace_request/raw_bytes_231/compressed_bytes_203/compressor_gzip-16       	   64660	     18186 ns/op	     701 B/op	       4 allocs/op
sm_trace_requestBenchmarkCompressors/sm_trace_request/raw_bytes_231/compressed_bytes_201/compressor_zstd-16       	  193225	      6267 ns/op	     273 B/op	       2 allocs/op
sm_trace_requestBenchmarkCompressors/sm_trace_request/raw_bytes_231/compressed_bytes_220/compressor_snappy-16     	 2925073	       418.2 ns/op	     276 B/op	       2 allocs/op
md_trace_requestBenchmarkCompressors/md_trace_request/raw_bytes_329/compressed_bytes_249/compressor_gzip-16       	   61320	     20641 ns/op	     846 B/op	       4 allocs/op
md_trace_requestBenchmarkCompressors/md_trace_request/raw_bytes_329/compressed_bytes_256/compressor_zstd-16       	  190965	      6440 ns/op	     321 B/op	       2 allocs/op
md_trace_requestBenchmarkCompressors/md_trace_request/raw_bytes_329/compressed_bytes_279/compressor_snappy-16     	 2051575	       656.8 ns/op	     341 B/op	       2 allocs/op
lg_trace_requestBenchmarkCompressors/lg_trace_request/raw_bytes_7025/compressed_bytes_303/compressor_gzip-16      	   30097	     40680 ns/op	     907 B/op	       4 allocs/op
lg_trace_requestBenchmarkCompressors/lg_trace_request/raw_bytes_7025/compressed_bytes_258/compressor_zstd-16      	  127027	      8437 ns/op	     363 B/op	       2 allocs/op
lg_trace_requestBenchmarkCompressors/lg_trace_request/raw_bytes_7025/compressed_bytes_591/compressor_snappy-16    	  716541	      1803 ns/op	     694 B/op	       2 allocs/op
sm_metric_requestBenchmarkCompressors/sm_metric_request/raw_bytes_183/compressed_bytes_140/compressor_gzip-16      	   82287	     15054 ns/op	     496 B/op	       4 allocs/op
sm_metric_requestBenchmarkCompressors/sm_metric_request/raw_bytes_183/compressed_bytes_137/compressor_zstd-16      	  230558	      5470 ns/op	     221 B/op	       2 allocs/op
sm_metric_requestBenchmarkCompressors/sm_metric_request/raw_bytes_183/compressed_bytes_152/compressor_snappy-16    	 2759403	       417.1 ns/op	     211 B/op	       2 allocs/op
md_metric_requestBenchmarkCompressors/md_metric_request/raw_bytes_376/compressed_bytes_194/compressor_gzip-16      	   58208	     18925 ns/op	     702 B/op	       4 allocs/op
md_metric_requestBenchmarkCompressors/md_metric_request/raw_bytes_376/compressed_bytes_198/compressor_zstd-16      	  199226	      6247 ns/op	     256 B/op	       2 allocs/op
md_metric_requestBenchmarkCompressors/md_metric_request/raw_bytes_376/compressed_bytes_222/compressor_snappy-16    	 2065202	       609.8 ns/op	     276 B/op	       2 allocs/op
lg_metric_requestBenchmarkCompressors/lg_metric_request/raw_bytes_10991/compressed_bytes_601/compressor_gzip-16    	   20583	     59762 ns/op	    1945 B/op	       5 allocs/op
lg_metric_requestBenchmarkCompressors/lg_metric_request/raw_bytes_10991/compressed_bytes_559/compressor_zstd-16    	   98254	     13152 ns/op	     728 B/op	       2 allocs/op
lg_metric_requestBenchmarkCompressors/lg_metric_request/raw_bytes_10991/compressed_bytes_1055/compressor_snappy-16 	  389401	      3976 ns/op	    1209 B/op	       2 allocs/op
PASS
ok  	go.opentelemetry.io/collector/config/configgrpc	40.394s

Signed-off-by: Juraci Paixão Kröhling juraci@kroehling.de

@jpkrohling
[configgrpc] Use own compressors for zstd
cc9fcb1 
Signed-off-by: Juraci Paixão Kröhling <juraci@kroehling.de>
@jpkrohling jpkrohling requested a review from a team as a code owner June 5, 2024 11:59
@jpkrohling
fmt, changelog
26e5a40 
Signed-off-by: Juraci Paixão Kröhling <juraci@kroehling.de>
@jpkrohling
lint
76f0f90 
Signed-off-by: Juraci Paixão Kröhling <juraci@kroehling.de>
@codecov Codecov
Copy link

codecov bot commented Jun 5, 2024

Codecov Report

Attention: Patch coverage is 82.35294% with 6 lines in your changes missing coverage. Please review.

Project coverage is 92.58%. Comparing base (d9dc6eb) to head (76f0f90).
Report is 1 commits behind head on main.

Files Patch % Lines
config/configgrpc/internal/zstd.go 81.81% 5 Missing and 1 partial ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main   #10323      +/-   ##
==========================================
- Coverage   92.62%   92.58%   -0.05%     
==========================================
  Files         386      387       +1     
  Lines       18221    18254      +33     
==========================================
+ Hits        16877    16900      +23     
- Misses       1000     1008       +8     
- Partials      344      346       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

mx-psi
mx-psi approved these changes Jun 5, 2024
View reviewed changes
Copy link
Member

@mx-psi mx-psi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@mx-psi mx-psi merged commit 760f773 into open-telemetry:main Jun 5, 2024
48 of 49 checks passed
@github-actions github-actions bot added this to the next release milestone Jun 5, 2024
mx-psi pushed a commit to mx-psi/opentelemetry-collector that referenced this pull request Jun 5, 2024
@jpkrohling @mx-psi
[configgrpc] Use own compressors for zstd (open-telemetry#10323)
699107d 
Uses our own version of the zstd compressor for gRPC servers. The code
for it is based on the gzip compressor that comes built-in with gRPC.

Benchmarks before this PR:
```
Running tool: /usr/bin/go test -benchmem -run=^$ -bench ^BenchmarkCompressors$ go.opentelemetry.io/collector/config/configgrpc

sm_log_requestgoos: linux
goarch: amd64
pkg: go.opentelemetry.io/collector/config/configgrpc
cpu: 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz
BenchmarkCompressors/sm_log_request/raw_bytes_160/compressed_bytes_162/compressor_gzip-16         	   71594	     19066 ns/op	     615 B/op	       4 allocs/op
sm_log_requestBenchmarkCompressors/sm_log_request/raw_bytes_160/compressed_bytes_159/compressor_zstd-16         	  151503	      8544 ns/op	     640 B/op	       6 allocs/op
sm_log_requestBenchmarkCompressors/sm_log_request/raw_bytes_160/compressed_bytes_178/compressor_snappy-16       	 3632570	       303.8 ns/op	     304 B/op	       3 allocs/op
md_log_requestBenchmarkCompressors/md_log_request/raw_bytes_242/compressed_bytes_219/compressor_gzip-16         	   68114	     16938 ns/op	     748 B/op	       4 allocs/op
md_log_requestBenchmarkCompressors/md_log_request/raw_bytes_242/compressed_bytes_209/compressor_zstd-16         	  138091	      8047 ns/op	     896 B/op	       6 allocs/op
md_log_requestBenchmarkCompressors/md_log_request/raw_bytes_242/compressed_bytes_260/compressor_snappy-16       	 3081198	       402.5 ns/op	     400 B/op	       3 allocs/op
lg_log_requestBenchmarkCompressors/lg_log_request/raw_bytes_4850/compressed_bytes_253/compressor_gzip-16        	   43414	     27174 ns/op	     386 B/op	       3 allocs/op
lg_log_requestBenchmarkCompressors/lg_log_request/raw_bytes_4850/compressed_bytes_216/compressor_zstd-16        	  117534	      9903 ns/op	   10112 B/op	       6 allocs/op
lg_log_requestBenchmarkCompressors/lg_log_request/raw_bytes_4850/compressed_bytes_454/compressor_snappy-16      	 1000000	      1190 ns/op	     528 B/op	       2 allocs/op
sm_trace_requestBenchmarkCompressors/sm_trace_request/raw_bytes_231/compressed_bytes_203/compressor_gzip-16       	   67275	     17508 ns/op	     700 B/op	       4 allocs/op
sm_trace_requestBenchmarkCompressors/sm_trace_request/raw_bytes_231/compressed_bytes_201/compressor_zstd-16       	  196862	      6137 ns/op	     848 B/op	       6 allocs/op
sm_trace_requestBenchmarkCompressors/sm_trace_request/raw_bytes_231/compressed_bytes_220/compressor_snappy-16     	 3595815	       331.7 ns/op	     272 B/op	       2 allocs/op
md_trace_requestBenchmarkCompressors/md_trace_request/raw_bytes_329/compressed_bytes_249/compressor_gzip-16       	   64105	     19104 ns/op	     844 B/op	       4 allocs/op
md_trace_requestBenchmarkCompressors/md_trace_request/raw_bytes_329/compressed_bytes_256/compressor_zstd-16       	  169221	      6929 ns/op	    1120 B/op	       6 allocs/op
md_trace_requestBenchmarkCompressors/md_trace_request/raw_bytes_329/compressed_bytes_279/compressor_snappy-16     	 2602239	       473.0 ns/op	     336 B/op	       2 allocs/op
lg_trace_requestBenchmarkCompressors/lg_trace_request/raw_bytes_7025/compressed_bytes_303/compressor_gzip-16      	   33861	     36473 ns/op	     904 B/op	       4 allocs/op
lg_trace_requestBenchmarkCompressors/lg_trace_request/raw_bytes_7025/compressed_bytes_258/compressor_zstd-16      	  107828	     10596 ns/op	   16832 B/op	       6 allocs/op
lg_trace_requestBenchmarkCompressors/lg_trace_request/raw_bytes_7025/compressed_bytes_591/compressor_snappy-16    	  725080	      1540 ns/op	     689 B/op	       2 allocs/op
sm_metric_requestBenchmarkCompressors/sm_metric_request/raw_bytes_183/compressed_bytes_140/compressor_gzip-16      	   76315	     16394 ns/op	     496 B/op	       4 allocs/op
sm_metric_requestBenchmarkCompressors/sm_metric_request/raw_bytes_183/compressed_bytes_137/compressor_zstd-16      	  193314	      5957 ns/op	     688 B/op	       6 allocs/op
sm_metric_requestBenchmarkCompressors/sm_metric_request/raw_bytes_183/compressed_bytes_152/compressor_snappy-16    	 3558649	       345.2 ns/op	     208 B/op	       2 allocs/op
md_metric_requestBenchmarkCompressors/md_metric_request/raw_bytes_376/compressed_bytes_194/compressor_gzip-16      	   68497	     18413 ns/op	     699 B/op	       4 allocs/op
md_metric_requestBenchmarkCompressors/md_metric_request/raw_bytes_376/compressed_bytes_198/compressor_zstd-16      	  177841	      6520 ns/op	    1136 B/op	       6 allocs/op
md_metric_requestBenchmarkCompressors/md_metric_request/raw_bytes_376/compressed_bytes_222/compressor_snappy-16    	 2354102	       497.4 ns/op	     272 B/op	       2 allocs/op
lg_metric_requestBenchmarkCompressors/lg_metric_request/raw_bytes_10991/compressed_bytes_601/compressor_gzip-16    	   21943	     54603 ns/op	    1941 B/op	       5 allocs/op
lg_metric_requestBenchmarkCompressors/lg_metric_request/raw_bytes_10991/compressed_bytes_559/compressor_zstd-16    	   71260	     16077 ns/op	   25312 B/op	       6 allocs/op
lg_metric_requestBenchmarkCompressors/lg_metric_request/raw_bytes_10991/compressed_bytes_1055/compressor_snappy-16 	  335415	      3026 ns/op	    1200 B/op	       2 allocs/op
PASS
ok  	go.opentelemetry.io/collector/config/configgrpc	37.766s
```

After this version:
```
Running tool: /usr/bin/go test -benchmem -run=^$ -bench ^BenchmarkCompressors$ go.opentelemetry.io/collector/config/configgrpc

sm_log_requestgoos: linux
goarch: amd64
pkg: go.opentelemetry.io/collector/config/configgrpc
cpu: 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz
BenchmarkCompressors/sm_log_request/raw_bytes_160/compressed_bytes_162/compressor_gzip-16         	   74952	     15710 ns/op	     603 B/op	       4 allocs/op
sm_log_requestBenchmarkCompressors/sm_log_request/raw_bytes_160/compressed_bytes_159/compressor_zstd-16         	  156784	      6966 ns/op	     208 B/op	       2 allocs/op
sm_log_requestBenchmarkCompressors/sm_log_request/raw_bytes_160/compressed_bytes_178/compressor_snappy-16       	 2216174	       510.4 ns/op	     308 B/op	       3 allocs/op
md_log_requestBenchmarkCompressors/md_log_request/raw_bytes_242/compressed_bytes_219/compressor_gzip-16         	   68095	     18569 ns/op	     736 B/op	       4 allocs/op
md_log_requestBenchmarkCompressors/md_log_request/raw_bytes_242/compressed_bytes_209/compressor_zstd-16         	  150705	      8849 ns/op	     294 B/op	       2 allocs/op
md_log_requestBenchmarkCompressors/md_log_request/raw_bytes_242/compressed_bytes_260/compressor_snappy-16       	 2149710	       556.8 ns/op	     406 B/op	       3 allocs/op
lg_log_requestBenchmarkCompressors/lg_log_request/raw_bytes_4850/compressed_bytes_253/compressor_gzip-16        	   40040	     26159 ns/op	     368 B/op	       3 allocs/op
lg_log_requestBenchmarkCompressors/lg_log_request/raw_bytes_4850/compressed_bytes_216/compressor_zstd-16        	  123043	     10254 ns/op	     299 B/op	       2 allocs/op
lg_log_requestBenchmarkCompressors/lg_log_request/raw_bytes_4850/compressed_bytes_454/compressor_snappy-16      	  726780	      1457 ns/op	     533 B/op	       2 allocs/op
sm_trace_requestBenchmarkCompressors/sm_trace_request/raw_bytes_231/compressed_bytes_203/compressor_gzip-16       	   64660	     18186 ns/op	     701 B/op	       4 allocs/op
sm_trace_requestBenchmarkCompressors/sm_trace_request/raw_bytes_231/compressed_bytes_201/compressor_zstd-16       	  193225	      6267 ns/op	     273 B/op	       2 allocs/op
sm_trace_requestBenchmarkCompressors/sm_trace_request/raw_bytes_231/compressed_bytes_220/compressor_snappy-16     	 2925073	       418.2 ns/op	     276 B/op	       2 allocs/op
md_trace_requestBenchmarkCompressors/md_trace_request/raw_bytes_329/compressed_bytes_249/compressor_gzip-16       	   61320	     20641 ns/op	     846 B/op	       4 allocs/op
md_trace_requestBenchmarkCompressors/md_trace_request/raw_bytes_329/compressed_bytes_256/compressor_zstd-16       	  190965	      6440 ns/op	     321 B/op	       2 allocs/op
md_trace_requestBenchmarkCompressors/md_trace_request/raw_bytes_329/compressed_bytes_279/compressor_snappy-16     	 2051575	       656.8 ns/op	     341 B/op	       2 allocs/op
lg_trace_requestBenchmarkCompressors/lg_trace_request/raw_bytes_7025/compressed_bytes_303/compressor_gzip-16      	   30097	     40680 ns/op	     907 B/op	       4 allocs/op
lg_trace_requestBenchmarkCompressors/lg_trace_request/raw_bytes_7025/compressed_bytes_258/compressor_zstd-16      	  127027	      8437 ns/op	     363 B/op	       2 allocs/op
lg_trace_requestBenchmarkCompressors/lg_trace_request/raw_bytes_7025/compressed_bytes_591/compressor_snappy-16    	  716541	      1803 ns/op	     694 B/op	       2 allocs/op
sm_metric_requestBenchmarkCompressors/sm_metric_request/raw_bytes_183/compressed_bytes_140/compressor_gzip-16      	   82287	     15054 ns/op	     496 B/op	       4 allocs/op
sm_metric_requestBenchmarkCompressors/sm_metric_request/raw_bytes_183/compressed_bytes_137/compressor_zstd-16      	  230558	      5470 ns/op	     221 B/op	       2 allocs/op
sm_metric_requestBenchmarkCompressors/sm_metric_request/raw_bytes_183/compressed_bytes_152/compressor_snappy-16    	 2759403	       417.1 ns/op	     211 B/op	       2 allocs/op
md_metric_requestBenchmarkCompressors/md_metric_request/raw_bytes_376/compressed_bytes_194/compressor_gzip-16      	   58208	     18925 ns/op	     702 B/op	       4 allocs/op
md_metric_requestBenchmarkCompressors/md_metric_request/raw_bytes_376/compressed_bytes_198/compressor_zstd-16      	  199226	      6247 ns/op	     256 B/op	       2 allocs/op
md_metric_requestBenchmarkCompressors/md_metric_request/raw_bytes_376/compressed_bytes_222/compressor_snappy-16    	 2065202	       609.8 ns/op	     276 B/op	       2 allocs/op
lg_metric_requestBenchmarkCompressors/lg_metric_request/raw_bytes_10991/compressed_bytes_601/compressor_gzip-16    	   20583	     59762 ns/op	    1945 B/op	       5 allocs/op
lg_metric_requestBenchmarkCompressors/lg_metric_request/raw_bytes_10991/compressed_bytes_559/compressor_zstd-16    	   98254	     13152 ns/op	     728 B/op	       2 allocs/op
lg_metric_requestBenchmarkCompressors/lg_metric_request/raw_bytes_10991/compressed_bytes_1055/compressor_snappy-16 	  389401	      3976 ns/op	    1209 B/op	       2 allocs/op
PASS
ok  	go.opentelemetry.io/collector/config/configgrpc	40.394s
```

Signed-off-by: Juraci Paixão Kröhling <juraci@kroehling.de>

---------

Signed-off-by: Juraci Paixão Kröhling <juraci@kroehling.de>
@mx-psi mx-psi mentioned this pull request Jun 5, 2024
Merged
mx-psi pushed a commit to mx-psi/opentelemetry-collector that referenced this pull request Jun 5, 2024
@jpkrohling @mx-psi
[configgrpc] Use own compressors for zstd (open-telemetry#10323)
f5f1866 
Uses our own version of the zstd compressor for gRPC servers. The code
for it is based on the gzip compressor that comes built-in with gRPC.

Benchmarks before this PR:
```
Running tool: /usr/bin/go test -benchmem -run=^$ -bench ^BenchmarkCompressors$ go.opentelemetry.io/collector/config/configgrpc

sm_log_requestgoos: linux
goarch: amd64
pkg: go.opentelemetry.io/collector/config/configgrpc
cpu: 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz
BenchmarkCompressors/sm_log_request/raw_bytes_160/compressed_bytes_162/compressor_gzip-16         	   71594	     19066 ns/op	     615 B/op	       4 allocs/op
sm_log_requestBenchmarkCompressors/sm_log_request/raw_bytes_160/compressed_bytes_159/compressor_zstd-16         	  151503	      8544 ns/op	     640 B/op	       6 allocs/op
sm_log_requestBenchmarkCompressors/sm_log_request/raw_bytes_160/compressed_bytes_178/compressor_snappy-16       	 3632570	       303.8 ns/op	     304 B/op	       3 allocs/op
md_log_requestBenchmarkCompressors/md_log_request/raw_bytes_242/compressed_bytes_219/compressor_gzip-16         	   68114	     16938 ns/op	     748 B/op	       4 allocs/op
md_log_requestBenchmarkCompressors/md_log_request/raw_bytes_242/compressed_bytes_209/compressor_zstd-16         	  138091	      8047 ns/op	     896 B/op	       6 allocs/op
md_log_requestBenchmarkCompressors/md_log_request/raw_bytes_242/compressed_bytes_260/compressor_snappy-16       	 3081198	       402.5 ns/op	     400 B/op	       3 allocs/op
lg_log_requestBenchmarkCompressors/lg_log_request/raw_bytes_4850/compressed_bytes_253/compressor_gzip-16        	   43414	     27174 ns/op	     386 B/op	       3 allocs/op
lg_log_requestBenchmarkCompressors/lg_log_request/raw_bytes_4850/compressed_bytes_216/compressor_zstd-16        	  117534	      9903 ns/op	   10112 B/op	       6 allocs/op
lg_log_requestBenchmarkCompressors/lg_log_request/raw_bytes_4850/compressed_bytes_454/compressor_snappy-16      	 1000000	      1190 ns/op	     528 B/op	       2 allocs/op
sm_trace_requestBenchmarkCompressors/sm_trace_request/raw_bytes_231/compressed_bytes_203/compressor_gzip-16       	   67275	     17508 ns/op	     700 B/op	       4 allocs/op
sm_trace_requestBenchmarkCompressors/sm_trace_request/raw_bytes_231/compressed_bytes_201/compressor_zstd-16       	  196862	      6137 ns/op	     848 B/op	       6 allocs/op
sm_trace_requestBenchmarkCompressors/sm_trace_request/raw_bytes_231/compressed_bytes_220/compressor_snappy-16     	 3595815	       331.7 ns/op	     272 B/op	       2 allocs/op
md_trace_requestBenchmarkCompressors/md_trace_request/raw_bytes_329/compressed_bytes_249/compressor_gzip-16       	   64105	     19104 ns/op	     844 B/op	       4 allocs/op
md_trace_requestBenchmarkCompressors/md_trace_request/raw_bytes_329/compressed_bytes_256/compressor_zstd-16       	  169221	      6929 ns/op	    1120 B/op	       6 allocs/op
md_trace_requestBenchmarkCompressors/md_trace_request/raw_bytes_329/compressed_bytes_279/compressor_snappy-16     	 2602239	       473.0 ns/op	     336 B/op	       2 allocs/op
lg_trace_requestBenchmarkCompressors/lg_trace_request/raw_bytes_7025/compressed_bytes_303/compressor_gzip-16      	   33861	     36473 ns/op	     904 B/op	       4 allocs/op
lg_trace_requestBenchmarkCompressors/lg_trace_request/raw_bytes_7025/compressed_bytes_258/compressor_zstd-16      	  107828	     10596 ns/op	   16832 B/op	       6 allocs/op
lg_trace_requestBenchmarkCompressors/lg_trace_request/raw_bytes_7025/compressed_bytes_591/compressor_snappy-16    	  725080	      1540 ns/op	     689 B/op	       2 allocs/op
sm_metric_requestBenchmarkCompressors/sm_metric_request/raw_bytes_183/compressed_bytes_140/compressor_gzip-16      	   76315	     16394 ns/op	     496 B/op	       4 allocs/op
sm_metric_requestBenchmarkCompressors/sm_metric_request/raw_bytes_183/compressed_bytes_137/compressor_zstd-16      	  193314	      5957 ns/op	     688 B/op	       6 allocs/op
sm_metric_requestBenchmarkCompressors/sm_metric_request/raw_bytes_183/compressed_bytes_152/compressor_snappy-16    	 3558649	       345.2 ns/op	     208 B/op	       2 allocs/op
md_metric_requestBenchmarkCompressors/md_metric_request/raw_bytes_376/compressed_bytes_194/compressor_gzip-16      	   68497	     18413 ns/op	     699 B/op	       4 allocs/op
md_metric_requestBenchmarkCompressors/md_metric_request/raw_bytes_376/compressed_bytes_198/compressor_zstd-16      	  177841	      6520 ns/op	    1136 B/op	       6 allocs/op
md_metric_requestBenchmarkCompressors/md_metric_request/raw_bytes_376/compressed_bytes_222/compressor_snappy-16    	 2354102	       497.4 ns/op	     272 B/op	       2 allocs/op
lg_metric_requestBenchmarkCompressors/lg_metric_request/raw_bytes_10991/compressed_bytes_601/compressor_gzip-16    	   21943	     54603 ns/op	    1941 B/op	       5 allocs/op
lg_metric_requestBenchmarkCompressors/lg_metric_request/raw_bytes_10991/compressed_bytes_559/compressor_zstd-16    	   71260	     16077 ns/op	   25312 B/op	       6 allocs/op
lg_metric_requestBenchmarkCompressors/lg_metric_request/raw_bytes_10991/compressed_bytes_1055/compressor_snappy-16 	  335415	      3026 ns/op	    1200 B/op	       2 allocs/op
PASS
ok  	go.opentelemetry.io/collector/config/configgrpc	37.766s
```

After this version:
```
Running tool: /usr/bin/go test -benchmem -run=^$ -bench ^BenchmarkCompressors$ go.opentelemetry.io/collector/config/configgrpc

sm_log_requestgoos: linux
goarch: amd64
pkg: go.opentelemetry.io/collector/config/configgrpc
cpu: 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz
BenchmarkCompressors/sm_log_request/raw_bytes_160/compressed_bytes_162/compressor_gzip-16         	   74952	     15710 ns/op	     603 B/op	       4 allocs/op
sm_log_requestBenchmarkCompressors/sm_log_request/raw_bytes_160/compressed_bytes_159/compressor_zstd-16         	  156784	      6966 ns/op	     208 B/op	       2 allocs/op
sm_log_requestBenchmarkCompressors/sm_log_request/raw_bytes_160/compressed_bytes_178/compressor_snappy-16       	 2216174	       510.4 ns/op	     308 B/op	       3 allocs/op
md_log_requestBenchmarkCompressors/md_log_request/raw_bytes_242/compressed_bytes_219/compressor_gzip-16         	   68095	     18569 ns/op	     736 B/op	       4 allocs/op
md_log_requestBenchmarkCompressors/md_log_request/raw_bytes_242/compressed_bytes_209/compressor_zstd-16         	  150705	      8849 ns/op	     294 B/op	       2 allocs/op
md_log_requestBenchmarkCompressors/md_log_request/raw_bytes_242/compressed_bytes_260/compressor_snappy-16       	 2149710	       556.8 ns/op	     406 B/op	       3 allocs/op
lg_log_requestBenchmarkCompressors/lg_log_request/raw_bytes_4850/compressed_bytes_253/compressor_gzip-16        	   40040	     26159 ns/op	     368 B/op	       3 allocs/op
lg_log_requestBenchmarkCompressors/lg_log_request/raw_bytes_4850/compressed_bytes_216/compressor_zstd-16        	  123043	     10254 ns/op	     299 B/op	       2 allocs/op
lg_log_requestBenchmarkCompressors/lg_log_request/raw_bytes_4850/compressed_bytes_454/compressor_snappy-16      	  726780	      1457 ns/op	     533 B/op	       2 allocs/op
sm_trace_requestBenchmarkCompressors/sm_trace_request/raw_bytes_231/compressed_bytes_203/compressor_gzip-16       	   64660	     18186 ns/op	     701 B/op	       4 allocs/op
sm_trace_requestBenchmarkCompressors/sm_trace_request/raw_bytes_231/compressed_bytes_201/compressor_zstd-16       	  193225	      6267 ns/op	     273 B/op	       2 allocs/op
sm_trace_requestBenchmarkCompressors/sm_trace_request/raw_bytes_231/compressed_bytes_220/compressor_snappy-16     	 2925073	       418.2 ns/op	     276 B/op	       2 allocs/op
md_trace_requestBenchmarkCompressors/md_trace_request/raw_bytes_329/compressed_bytes_249/compressor_gzip-16       	   61320	     20641 ns/op	     846 B/op	       4 allocs/op
md_trace_requestBenchmarkCompressors/md_trace_request/raw_bytes_329/compressed_bytes_256/compressor_zstd-16       	  190965	      6440 ns/op	     321 B/op	       2 allocs/op
md_trace_requestBenchmarkCompressors/md_trace_request/raw_bytes_329/compressed_bytes_279/compressor_snappy-16     	 2051575	       656.8 ns/op	     341 B/op	       2 allocs/op
lg_trace_requestBenchmarkCompressors/lg_trace_request/raw_bytes_7025/compressed_bytes_303/compressor_gzip-16      	   30097	     40680 ns/op	     907 B/op	       4 allocs/op
lg_trace_requestBenchmarkCompressors/lg_trace_request/raw_bytes_7025/compressed_bytes_258/compressor_zstd-16      	  127027	      8437 ns/op	     363 B/op	       2 allocs/op
lg_trace_requestBenchmarkCompressors/lg_trace_request/raw_bytes_7025/compressed_bytes_591/compressor_snappy-16    	  716541	      1803 ns/op	     694 B/op	       2 allocs/op
sm_metric_requestBenchmarkCompressors/sm_metric_request/raw_bytes_183/compressed_bytes_140/compressor_gzip-16      	   82287	     15054 ns/op	     496 B/op	       4 allocs/op
sm_metric_requestBenchmarkCompressors/sm_metric_request/raw_bytes_183/compressed_bytes_137/compressor_zstd-16      	  230558	      5470 ns/op	     221 B/op	       2 allocs/op
sm_metric_requestBenchmarkCompressors/sm_metric_request/raw_bytes_183/compressed_bytes_152/compressor_snappy-16    	 2759403	       417.1 ns/op	     211 B/op	       2 allocs/op
md_metric_requestBenchmarkCompressors/md_metric_request/raw_bytes_376/compressed_bytes_194/compressor_gzip-16      	   58208	     18925 ns/op	     702 B/op	       4 allocs/op
md_metric_requestBenchmarkCompressors/md_metric_request/raw_bytes_376/compressed_bytes_198/compressor_zstd-16      	  199226	      6247 ns/op	     256 B/op	       2 allocs/op
md_metric_requestBenchmarkCompressors/md_metric_request/raw_bytes_376/compressed_bytes_222/compressor_snappy-16    	 2065202	       609.8 ns/op	     276 B/op	       2 allocs/op
lg_metric_requestBenchmarkCompressors/lg_metric_request/raw_bytes_10991/compressed_bytes_601/compressor_gzip-16    	   20583	     59762 ns/op	    1945 B/op	       5 allocs/op
lg_metric_requestBenchmarkCompressors/lg_metric_request/raw_bytes_10991/compressed_bytes_559/compressor_zstd-16    	   98254	     13152 ns/op	     728 B/op	       2 allocs/op
lg_metric_requestBenchmarkCompressors/lg_metric_request/raw_bytes_10991/compressed_bytes_1055/compressor_snappy-16 	  389401	      3976 ns/op	    1209 B/op	       2 allocs/op
PASS
ok  	go.opentelemetry.io/collector/config/configgrpc	40.394s
```

Signed-off-by: Juraci Paixão Kröhling <juraci@kroehling.de>

---------

Signed-off-by: Juraci Paixão Kröhling <juraci@kroehling.de>
mx-psi added a commit that referenced this pull request Jun 5, 2024
@mx-psi @jpkrohling
[configgrpc] Use own compressors for zstd (#10323) (#10324)
10e89bd 
Backport of #10323

Signed-off-by: Juraci Paixão Kröhling <juraci@kroehling.de>
Co-authored-by: Juraci Paixão Kröhling <juraci@kroehling.de>
This was referenced Jun 5, 2024
Closed
Merged
Aneurysm9 added a commit to aws-observability/aws-otel-collector that referenced this pull request Jun 5, 2024
@Aneurysm9
Backport changes to confighttp and configgrpc
d5ff217 
Upstream changes to address potential decompression-related failures in
the `confighttp` and `configgrpc` modules are backported.  See
open-telemetry/opentelemetry-collector#10289 and
open-telemetry/opentelemetry-collector#10323 for
more details.

Signed-off-by: Anthony J Mirabella <a9@aneurysm9.com>
@Aneurysm9 Aneurysm9 mentioned this pull request Jun 5, 2024
Merged
Aneurysm9 added a commit to aws-observability/aws-otel-collector that referenced this pull request Jun 6, 2024
@Aneurysm9
Backport changes to confighttp and configgrpc
8e5ff8f 
Upstream changes to address potential decompression-related failures in
the `confighttp` and `configgrpc` modules are backported.  See
open-telemetry/opentelemetry-collector#10289 and
open-telemetry/opentelemetry-collector#10323 for
more details.

Signed-off-by: Anthony J Mirabella <a9@aneurysm9.com>
@Aneurysm9 Aneurysm9 mentioned this pull request Jun 6, 2024
Merged
Aneurysm9 added a commit to aws-observability/aws-otel-collector that referenced this pull request Jun 6, 2024
@Aneurysm9
Backport changes to confighttp and configgrpc (#2743)
b89839e 
Upstream changes to address potential decompression-related failures in
the `confighttp` and `configgrpc` modules are backported.  See
open-telemetry/opentelemetry-collector#10289 and
open-telemetry/opentelemetry-collector#10323 for
more details.

Signed-off-by: Anthony J Mirabella <a9@aneurysm9.com>
@mostynb mostynb mentioned this pull request Jun 6, 2024
Closed
@tatianab tatianab mentioned this pull request Jun 7, 2024
Open
steves-canva pushed a commit to Canva/opentelemetry-collector that referenced this pull request Jun 14, 2024
@mx-psi @jpkrohling @steves-canva
[configgrpc] Use own compressors for zstd (open-telemetry#10323) (ope…
714289e 
…n-telemetry#10324)

Backport of open-telemetry#10323

Signed-off-by: Juraci Paixão Kröhling <juraci@kroehling.de>
Co-authored-by: Juraci Paixão Kröhling <juraci@kroehling.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@djaglowski djaglowski djaglowski approved these changes

@mx-psi mx-psi mx-psi approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
v0.102.1
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@jpkrohling @djaglowski @mx-psi