perf(p2p/conn): Buffer secret connection writes

[ValarDragon]

component of #3198 , this PR buffers writes. What happens is secret conn will often receive a write of (say) 65kb. But it will then split this into 64 1024 byte frames. It does a write on each frame, which is a syscall write. Then starts the next frame. Instead here, we buffer these writes, and do a single syscall write at the end.

I'll come back with benchmarks from running on mainnet. The baseline for this is netconn.Write taking 33% of the time in sendroutine.

We should do something similar for reads, but due to how the evil_secret_connection test makes non-black-box use of conn, that will require notable test refactors.

---

PR checklist

• Tests written/updated
• Changelog entry added in .changelog (we use unclog to manage our changelog)
• Updated relevant documentation (docs/ or spec/) and code comments
• Title follows the Conventional Commits spec

[zmanian]

I think at very least we should also make a breaking change to increase the frame size from 1k.

If we aren't going to adaptive frames....

[melekes]

Thanks @ValarDragon ❤️

[cason]

I would request to make this PR simple by just adding the connWriter to buffer writes, which is good in general.

[ValarDragon]

I think at very least we should also make a breaking change to increase the frame size from 1k.
If we aren't going to adaptive frames....

Agreed! I'm down to (in separate PRs) increase frame size, at least to have that in next coordinate release if putting in another secret transport layer doesn't work out. (Or at minimum raising frame size) The amount of chacha20poly1305 overhead was really surprising to me though, I hope that improves proportionately with larger frames

[cason]

Good.

Can we really measure the improvements that this provides?

[ValarDragon]

Yes will get it! Sorry for delay, two silly issues got in the way. (Snapshot service was down when I first did it, and I didn't download result in time during my second profile so it get retented. Re-running mainnet benchmarks now to get the ratio here)

[ValarDragon]

This successfully eliminates the overhead coming from the write packet, but not flush (which makes sense as flush is dealing with the case where we can't fill one frame anyway. This is because the flush size is parameterized to be equal to the frame size right now. I don't know if this is coincidence or design today, they are both freely variable parameters)

We should expect the ratio of seal to file.Write in the sendPacketMsg call (the non-flush throttle case) to be indicative of what we are speeding up. So on this latest benchmark, its a ratio of 2.5 seal : 1.5 file write. Normalized, 1s of sealing needed .6s of net con write.

Originally this case was: 1s of sealing, needed 1.73s of net conn write. So this is an almost 3x speedup to this part of the bottleneck! (And now the unneeded buffer .Put and .Get matter to optimize out)

[cason]

Minor suggestions. Lets merge this.