Port forwarding dies after cancelling a TCP request

[ramondeklein]

I use kubectl port-forward to forward traffic to a service and sometimes the service is not reachable anymore. After some investigation it looks like the port forwarding cannot handle with cancelled TCP requests and the port-forwarding locks up and showed the following error message:

E0705 12:50:42.897359   76107 portforward.go:394] error copying from local connection to remote stream: read tcp6 [::1]:8080->[::1]:44368: read: connection reset by peer
E0705 12:50:42.897360   76107 portforward.go:381] error copying from remote stream to local connection: readfrom tcp6 [::1]:8080->[::1]:44368: write tcp6 [::1]:8080->[::1]:44368: write: broken pipe

The kubectl process keeps running, but the port-forwarding is dead. Restarting kubectl port-forward brings the service back alive.

I have created a very basic application (in Go) that serves a 32MiB file and it can be invoked using curl -o /dev/null http://localhost:8080 and it will complete the download. When running curl http://localhost:8080, it will cancel the download after the request, because it won't display binary data. This application works fine locally.

When run in Kubernetes and using kubectl port-forward server 8080:8080 it works fine when files are downloaded, but it locks up when a download is cancelled. It looks like kubectl is still accepting requests on port 8080, but doesn't seem to be able to contact the pod anymore.

I tested with both Kubernetes v1.30 (using a single node Kind cluster) and in Kubernetes v1.29 using AKS (MS Azure). A full ready-to-run reproduction is available on https://github.com/ramondeklein/k8s-portforward-bug.

[k8s-ci-robot]

This issue is currently awaiting triage.

SIG CLI takes a lead on issue triage for this repo, but any Kubernetes member can accept issues by applying the triage/accepted label.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ramondeklein]

Maybe a duplicate of containerd/containerd#9875, but my repo may add a test scenario.