Timeout on fetch cert #368
Comments
|
what version of EKS are you running? |
|
eks.7 with Kubernetes 1.14. I think everything should be up to date. |
|
Any special RBAC settings? |
|
Hmm, I haven't done any special setup other than what AWS documentation recommends |
|
Issue #317 is similar. Can you please try the troubleshooting questions presented in that issue? In the meantime I'll try to reproduce in eks. |
|
I'll see if I've missed something from that issue, but as far as I can see that was an issue with GKE in a private network. How does the curl call work? Does it depend on a proxy? And how is it authenticated? |
|
The main conversation in that thread is about GKE, but it might be a similar issue in that it all boils down to the apiserver proxy. When you run "kube proxy" locally, your client authenticates to the cluster as usual and opens port 8080 on localhost. Now you can curl and access any k8s api including all the http ports exposed by services. That's the same mechanism that kubeseal uses to talk to the controller (but in-process) |
|
Dumping some more debug: Controller is running: Component descriptions: |
|
Can you check whether you have some Network Policies that would block the api server from talking with the port 8080 of the controller? |
|
None |
|
Yeah. I'd really like to move away from this model. I'd like to put the cert into a config-map resource or a CRD which we can then access directly via the api server |
|
Config map sounds like a good approach. I can store the cert locally for now. If this can be solved in future release we can close this issue. |
|
There is another approach detailed in #282 I'll close this issue when the PR that implements the fix lands. |
|
In my case on GKE communication was also blocked on 8080 port. Allowing it from master CIDR solved it. It could be added to readme as a requirement. Pasting output of |
|
The GKE case at least is documented in https://github.com/bitnami-labs/sealed-secrets/blob/master/docs/GKE.md#private-gke-clusters This is clearly causing more pain than necessary. I don't think a general solution can assume the CLI tool can talk directly with the controller. |
|
JFYI: I'm using provider Linode and getting same error: $ kubeseal --fetch-cert -v 10 > kubeseal.pem
I0730 14:44:00.291228 23849 round_trippers.go:423] curl -k -v -XGET -H "Accept: application/x-pem-file, */*" -H "User-Agent: kubeseal/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Authorization: Bearer blabla/api/v1/namespaces/kube-system/services/http:sealed-secrets-controller:/proxy/v1/cert.pem'
I0730 14:44:30.974004 23849 round_trippers.go:443] GET https://blabla.net:443/api/v1/namespaces/kube-system/services/http:sealed-secrets-controller:/proxy/v1/cert.pem 503 Service Unavailable in 30682 milliseconds
I0730 14:44:30.974048 23849 round_trippers.go:449] Response Headers:
I0730 14:44:30.974069 23849 round_trippers.go:452] Content-Length: 191
I0730 14:44:30.974079 23849 round_trippers.go:452] Date: Fri, 30 Jul 2021 11:44:30 GMT
I0730 14:44:30.974089 23849 round_trippers.go:452] Cache-Control: no-cache, private
I0730 14:44:30.974098 23849 round_trippers.go:452] Content-Type: application/json
I0730 14:44:30.974161 23849 request.go:968] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"error trying to reach service: dial tcp 10.2.193.7:8080: i/o timeout","reason":"ServiceUnavailable","code":503}
error: cannot fetch certificate: error trying to reach service: dial tcp 10.2.193.7:8080: i/o timeoutNo network rules, no custom controller configuration, +just fresh cluster. Workaround: using |
|
Same here with Linode LKE. Spent quite some time to understand the issue. Can this https://github.com/bitnami-labs/sealed-secrets/blob/main/docs/GKE.md be bumped to the main Readme.md? |
|
Same timeout with DigitalOcean in October 2021. Kubeseal v0.16.0, K8s 1.21.3. Helm chart 1.16.1, using Flux v2.
|
|
Same with Civo |
|
Could it be, that (in my case ArgoCD) is somehow clashing with the port 8080? |
juan131
added
backlog
|
For anyone who referred to this issue, I have had the same problem with EKS. The reason is that the Cluster API server needs to call the controller. However, the request was blocked by the security group of the node where the pod is deployed. So, you need to allow the port 8080 for the entry rule of the node's security group. I hope that could help someone. |
|
Thanks @renxunsaky ! For people not familiar with manual customisation of security groups:
Here we go |
|
for anyone having trouble to obtain the public key using
Source: https://fluxcd.io/flux/flux-e2e/#fluxs-default-configuration-for-networkpolicy My solution: kubectl get secret \
--namespace flux-system \
--selector sealedsecrets.bitnami.com/sealed-secrets-key=active \
--output jsonpath='{.items[0].data.tls\.crt}' \
| base64 -d |
|
I've been having the same issue in a k8s onprem cluster where apiservers are not located in normal CNI network but in host network. I workarounded it by exposing the public certificate via an ingress (using sealed-secrets helm chart). Overriden parameters: With this, I'm able to get the public certificate: In order to use the kubeseal with this exposed certificate, you can simply use: For example: It works like a charm :) |
If you are using terraform, add this to the eks module: module "eks" {
source = "terraform-aws-modules/eks/aws"
(...)
node_security_group_additional_rules = {
# kubeseal requires this
ingress_cluster_8080 = {
description = "Cluster API to 8080"
protocol = "tcp"
from_port = 8080
to_port = 8080
type = "ingress"
source_cluster_security_group = true
}
}
} |
On initial installation on AWS I get the following timeout error:
I applied the controller at https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.9.8/controller.yaml and installed a precompiled cli from https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.9.8/kubeseal-linux-amd64
Some additional debugging:
I am able to do a port forward like so:
And then curl the cert:
The text was updated successfully, but these errors were encountered: