Easily Install Kubernetes Metrics Server in minikube Today!

This article explains how to install the Metrics Server in Kubernetes, (k8s) specifically using minikube, which is a single-node Kubernetes instance that you can run locally. The Kubernetes Metrics Server is an important component for monitoring and optimizing the performance of your k8s cluster. We’ll explore how to solve the “cannot validate certificate for” x509 IP SANs Error as well, which you may encounter in the installation process — we’ll take a closer look at this in the next section.

How To Solve The "cannot validate certificate for" x509 IP SANs Error in minikube Metrics Server

The following practice question for the Certified Kubernetes Application Developer (CKAD) exam came up this weekend while reviewing the observability exercises available on Dimitris-Ilias Gkanatsios’ CKAD Exercises project page:

Get CPU/memory utilization for nodes (metrics-server must be running)

Installing the Kubernetes (K8s) Metrics Server appears to be a straightforward one-liner from the command line (CLI) however I ran into problems while using minikube on both Ubuntu and OSX. Below I have the full details of the issue along with a full example of a working solution (the issue on GitHub includes the correct command however it doesn’t indicate where exactly it needs to be placed — we cover this below).

In this example, we rely on Kubectl version v1.19.4, Kubernetes version v1.17.3, and Docker version v19.03.6.

The Problem: Unexpected x509 certificate validation errors

Step One: Install the Kubernetes Metrics Server from components.yaml

According to the kubernetes-sigs / metrics-server page we should be able to install the Metrics Server with a single line (step one):

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

This results in the following, which looks promising:

creationTimestamp: “2020-11-29T22:07:21Z”
serviceaccount/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created

— however on closer inspection, we can see that the Metrics Server isn’t working. For example, executing:

kubectl top nodes

returns:

Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)

and executing:

kubectl get deployment metrics-server -n kube-system

returns:

NAME READY UP-TO-DATE AVAILABLE AGE
metrics-server 0/1 1 0 105s

— which is also wrong. I’ve included an image below with blue arrows that point to the command and several orange arrows that direct your attention to the output that shows that the Metrics Server is unavailable.

The block in orange is directly related to the issue on Github entitled “metrics-server error because it doesn’t contain any IP SANs” — buried in the conversation is one suggested solution which works and we’ll cover that next.

The Solution: Add commands to the Kubernetes Metrics Server

Step Two: Modify The components.yaml File

In order to fix this we need to add the following to the components.yaml file:

command:
- /metrics-server
- --kubelet-insecure-tls
- --kubelet-preferred-address-types=InternalIP

Below we have the complete file with notes, see lines 141 to 215, specifically as this is where the change needs to be applied.

				
					#
# https://gist.github.com/thospfuller/d0d918e0b9fdb719a34d3d355b0886bb
#
# Jump to line ~ 141 to ~ 215 for the command that pertains to this change.
#
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    k8s-app: metrics-server
    rbac.authorization.k8s.io/aggregate-to-admin: "true"
    rbac.authorization.k8s.io/aggregate-to-edit: "true"
    rbac.authorization.k8s.io/aggregate-to-view: "true"
  name: system:aggregated-metrics-reader
rules:
- apiGroups:
  - metrics.k8s.io
  resources:
  - pods
  - nodes
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    k8s-app: metrics-server
  name: system:metrics-server
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - nodes
  - nodes/stats
  - namespaces
  - configmaps
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server-auth-reader
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server:system:auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: system:metrics-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:metrics-server
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
spec:
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    k8s-app: metrics-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  strategy:
    rollingUpdate:
      maxUnavailable: 0
  template:
    metadata:
      labels:
        k8s-app: metrics-server
    spec:
      containers:
      - args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-use-node-status-port
        image: k8s.gcr.io/metrics-server/metrics-server:v0.4.1
        imagePullPolicy: IfNotPresent
        #
        # Precondition: Metrics Server installation attempted on Minikube (on Ubuntu) as per:
        #
        # https://github.com/kubernetes-sigs/metrics-server
        #
        # Specifically, the instructions suggest running the following in order to install the Metrics Server:
        #
        # kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
        #
        # and running:
        #
        # kubectl top nodes
        #
        # results in:
        #
        # "Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)"
        #
        # and running:
        #
        # kubectl get deployment metrics-server -n kube-system
        #
        # results in something that looks like:
        #
        # NAME             READY   UP-TO-DATE   AVAILABLE   AGE
        # metrics-server   1/1     1            0           6m
        #
        # -----
        #
        # We can see in the log that there's a problem:
        #
        # kubectl logs -n kube-system deploy/metrics-server
        #
        # E1129 22:08:56.736988       1 server.go:132] unable to fully scrape metrics: [unable to fully scrape metrics from node minikube: unable to fetch metrics from node minikube: Get "https://192.168.99.100:10250/stats/summary?only_cpu_and_memory=true": x509: cannot validate certificate for 192.168.99.100 because it doesn't contain any IP SANs, unable to fully scrape metrics from node m01: unable to fetch metrics from node m01: Get "https://192.168.99.100:10250/stats/summary?only_cpu_and_memory=true": x509: cannot validate certificate for 192.168.99.100 because it doesn't contain any IP SANs]
        # I1129 22:08:56.748352       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
        # I1129 22:08:56.748378       1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
        # I1129 22:08:56.748400       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
        # I1129 22:08:56.748404       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
        # I1129 22:08:56.748414       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
        # I1129 22:08:56.748417       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
        # I1129 22:08:56.749024       1 secure_serving.go:197] Serving securely on [::]:4443
        # I1129 22:08:56.749096       1 dynamic_serving_content.go:130] Starting serving-cert::/tmp/apiserver.crt::/tmp/apiserver.key
        # I1129 22:08:56.749163       1 tlsconfig.go:240] Starting DynamicServingCertificateController
        # I1129 22:08:56.848729       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
        # I1129 22:08:56.848758       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
        # I1129 22:08:56.849090       1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController
        #
        # -----
        #
        # The solution at the link below will remedy this issue:
        #
        # https://github.com/kubernetes-sigs/metrics-server/issues/196#issuecomment-451061841
        #
        # And we'll need to either modify the component.yaml file with the command below and
        # apply the file or we can edit the current configuration using:
        #
        # kubectl edit deployment -n kube-system metrics-server 
        #
        # Then execute:
        #
        # minikube stop
        # minikube start
        #
        # and try "kubectl top nodes" again and the Metrics Server should be working now.
        #
        # See also:
        # * https://github.com/kubernetes-sigs/metrics-server/issues/196
        # * https://github.com/juan-vg/metrics-server/commit/264f9c551abdeedc575d36e74da40671c51cb747
        #
        command:
        - /metrics-server
        - --kubelet-insecure-tls
        - --kubelet-preferred-address-types=InternalIP
        #
        # Changes end.
        #
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /livez
            port: https
            scheme: HTTPS
          periodSeconds: 10
        name: metrics-server
        ports:
        - containerPort: 4443
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /readyz
            port: https
            scheme: HTTPS
          periodSeconds: 10
        securityContext:
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
        volumeMounts:
        - mountPath: /tmp
          name: tmp-dir
      nodeSelector:
        kubernetes.io/os: linux
      priorityClassName: system-cluster-critical
      serviceAccountName: metrics-server
      volumes:
      - emptyDir: {}
        name: tmp-dir
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  labels:
    k8s-app: metrics-server
  name: v1beta1.metrics.k8s.io
spec:
  group: metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: metrics-server
    namespace: kube-system
  version: v1beta1
  versionPriority: 100

Step Three: Apply the components.yaml file to your local minikube metrics server deployment

Once we’ve edited the configuration directly, we can restart minikube and we should see something that looks like what we have below. Note that the green arrows point to output that indicates that the Metrics Server is now running correctly — you should see something similar to this.

Note that you can edit your components.yaml file however you’ll need to apply it once the change has been added — so either (step three):

kubectl edit deployment -n kube-system metrics-server

and add this text to the appropriate place or modify the components.yaml file and then run:

kubectl apply -f ./components.yaml

Step Four: Stop minikube

In the next step we’ll stop minikube as follows:

minikube stop

Step Five: Start minikube

Finally, we’ll start minikube as follows:

minikube start

The image below details what this should look like along with the expected output.

As the Metrics Server is now running correctly, this issue has been addressed.

Easy Kubernetes Metrics Server Install Conclusion

If you found this article to be helpful you may also like the article I wrote recently entitled “Answers to Five Kubernetes CKAD Practice Questions (2020)” where we review five questions as they pertain to the Certified Kubernetes Application Developer (CKAD) exam and include answers along with commentary and verification that the given solutions work.

Addressing problems such as the one described here can be difficult when working in isolation. One strategy which can help make this easier is pair programming, which is part of the Agile Software Development methodology.

Questions and comments are welcomed.