Add kubeadmConfigPatches and metrics-server references to docs

[rikatz]

This PR adds the following documentations:

• kubeadmConfigPatches (that can be used to put kube-proxy in IPVS mode)
• deploying metrics-server in KinD as being a known issue

[k8s-ci-robot]

Welcome @rikatz!

It looks like this is your first PR to kubernetes-sigs/kind 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/kind has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @rikatz. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: rikatz
To complete the pull request process, please assign munnerz
You can assign the PR to them by writing /assign @munnerz in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[rikatz]

/cc @BenTheElder @aojea

[BenTheElder]

/ok-to-test

[BenTheElder]

this seems substantially less than ideal, why do we need to do this?

I haven't looked at metrics server recently.

[rikatz]

Hi @BenTheElder,

It seems the CA that signs kubelet is different from the CA used by the APIServer.

Metrics-server is able to connect without any problem to apiserver and fetch all the nodes (E0407 12:22:00.252207 1 reststorage.go:135] unable to fetch node metrics for node "kind118-control-plane": no metrics known for node) but when it tries to connect to the kubelet it receiver the certificate signed by unknown authority error. It seems that the certificates from the endpoints are signed by different CA.

As an example, the following is the CA of the nodes from my cluster called kind118:
issuer=CN = kind118-control-plane-ca@1586194038

And from APIServer:
issuer=CN = kubernetes

I don't know really if this is the case with KinD. Fetching the certs with openssl I really can see they're different.

To be honest I couldn't find into KinD kubelet config where it's using a specific CA, so I'm guessing it auto generates a certificate for its secure port (10250) and that's why the metrics-server Pod doesn't know which CA is that and I'm skipping the tls validation.

[aojea]

certificates are handled by kubeadm https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/

[rikatz]

But also the certificates from the secure metrics endpoints (10250)? This is related to controller signing the certificates of each kubelet to communicate to api-server right?

[aojea]

no idea 😅 , @neolit123 may help us

[rikatz]

@BenTheElder ok, so last comment and to be practical:

• I can split this PR into two, one dealing only with kubeadm config patching docs, and other dealing with metrics-server

• About @neolit123 suggestions, I think at least to move forward we can put into the security-goose alert the referenced issue, OR we can wait until a better solution is achieved for the integration between metrics-server and kubelet, and disregard this by now.

WDYT?

[BenTheElder]

the one thing is I'd very much like to actually enable this core API in kind and I'm not inclined to do that by making it even more insecure.

whatever we document will get used widely, I feel like if metrics are important (... seems so) we can actually solve the problem

I also don't actually want people relying on kubeadm patches too heavily, they're available for power users that know what they're doing but they're a bit brittle across versions and I think most users have no idea what kubeadm config looks like etc.

[BenTheElder]

every time I find a heavily patched target I'm trying to introduce a way for kind to actually be aware that you e.g. want to enable a feature gate, and handle it intelligently (like toggling it throughout kubeadm config for all the components, being able to default certain gates in kind so our system components work, etc...)

[rikatz]

So maybe disregarding the metrics server stuff, is it a better idea to implement a way that kind understands that kube-proxy should run with IPVS instead of iptables? I can try making this happen :)

[rikatz]

About the heavy users, anyway wouldn't be good to have the patching method documented? Or is this some feature that KinD is targeting to remove later?

[rikatz]

@BenTheElder do you want me to split this PR, so we got the kubeadmPatch part and can discuss a little bit more about metrics-server and kubelet certs? :)

[BenTheElder]

yes please, let's do that!

[ereslibre]

A comment from my side, I was able to deploy the metrics-server without any specific configuration, just some added parameters to it:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: metrics-server
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        k8s-app: metrics-server
      name: metrics-server
    spec:
      containers:
      - args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-insecure-tls
        - --kubelet-preferred-address-types=InternalIP
        image: k8s.gcr.io/metrics-server-amd64:v0.3.6
        imagePullPolicy: IfNotPresent
        name: metrics-server
        ports:
        - containerPort: 4443
          name: main-port
          protocol: TCP
        resources: {}
        securityContext:
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /tmp
          name: tmp-dir
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/arch: amd64
        kubernetes.io/os: linux
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: metrics-server
      serviceAccountName: metrics-server
      terminationGracePeriodSeconds: 30
      volumes:
      - emptyDir: {}
        name: tmp-dir

In my case, --kubelet-insecure-tls and --kubelet-preferred-address-types=InternalIP made the trick for me. The first will just ignore the presented certificate by the kubelets (not containing the IP address on the SAN), and the second will use the internal IP directly instead of trying to resolve the hostname with CoreDNS (that of course cannot resolve).

Edit: while I completely get the parameters added to the metrics-server is not suitable for production, in this case (when using kind) is perfectly suitable for me, since this is not going to be anywhere near production :)

[BenTheElder]

The hostnames should resolve now, I think (in v0.8.0).

[rikatz]

I've opened PR #1595 so instead of patching the kubeadm config KinD will support this as an internal configuration :D

If this is fine, I can close this PR and maybe open an issue so we can discuss about the metrics-server stuff before putting this into code or docs

[rikatz]

Closing, as KinD now supports ipvs mode without the need of patching.