Tutorial: Easily Install Kubernetes Metrics Server in minikube Today!

This instructional explains how to install the Metrics Server in Kubernetes, (k8s) specifically using minikube; we’ll also explore how to solve the “cannot validate certificate for” x509 IP SANs Error as well, which appears with some frequency during the installation process.

How To Solve The "cannot validate certificate for" x509 IP SANs Error in minikube Metrics Server

The following practice question for the Certified Kubernetes Application Developer (CKAD) exam came up this weekend while reviewing the observability exercises available on Dimitris-Ilias Gkanatsios’ CKAD Exercises project page:

Get CPU/memory utilization for nodes (metrics-server must be running)

Installing the Kubernetes (K8s) Metrics Server appears to be a straightforward one-liner from the command line (CLI) however I ran into problems while using minikube on both Ubuntu and OSX.

Below I have the full details of the issue along with a full example of a working solution (the issue on GitHub includes the correct command however it doesn’t indicate where exactly it needs to be placed — we cover this below).

In this example, we rely on Kubectl version v1.19.4, Kubernetes version v1.17.3, and Docker version v19.03.6.

The Problem: Unexpected x509 certificate validation errors

Step One: Install the Kubernetes Metrics Server from components.yaml

According to the kubernetes-sigs / metrics-server page we should be able to install the Metrics Server with a single line (step one):

				
					kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

This results in the following, which looks promising:

creationTimestamp: “2020-11-29T22:07:21Z”
serviceaccount/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created

— however on closer inspection, we can see that the Metrics Server isn’t working — for example, executing:

				
					kubectl top nodes

returns:

Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)

and executing:

				
					kubectl get deployment metrics-server -n kube-system

returns:

NAME READY UP-TO-DATE AVAILABLE AGE
metrics-server 0/1 1 0 105s

— which is also wrong.

I’ve included an image below with blue arrows that point to the command and several orange arrows that direct your attention to the output that shows that the Metrics Server is unavailable.

The block in orange is directly related to the issue on Github entitled “metrics-server error because it doesn’t contain any IP SANs” — buried in the conversation is one suggested solution that works and we’ll cover that next.

The Solution: Add commands to the Kubernetes Metrics Server

Step Two: Modify The components.yaml File

In order to fix this we need to add the following command to the components.yaml file:

				
					command:
- /metrics-server
- --kubelet-insecure-tls
- --kubelet-preferred-address-types=InternalIP

Below we have the complete file with notes, see lines 141 to 215, specifically as this is where the change needs to be applied.

				
					#
# https://gist.github.com/thospfuller/d0d918e0b9fdb719a34d3d355b0886bb
#
# Jump to line ~ 141 to ~ 215 for the command that pertains to this change.
#
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    k8s-app: metrics-server
    rbac.authorization.k8s.io/aggregate-to-admin: "true"
    rbac.authorization.k8s.io/aggregate-to-edit: "true"
    rbac.authorization.k8s.io/aggregate-to-view: "true"
  name: system:aggregated-metrics-reader
rules:
- apiGroups:
  - metrics.k8s.io
  resources:
  - pods
  - nodes
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    k8s-app: metrics-server
  name: system:metrics-server
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - nodes
  - nodes/stats
  - namespaces
  - configmaps
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server-auth-reader
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server:system:auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: system:metrics-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:metrics-server
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
spec:
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    k8s-app: metrics-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  strategy:
    rollingUpdate:
      maxUnavailable: 0
  template:
    metadata:
      labels:
        k8s-app: metrics-server
    spec:
      containers:
      - args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-use-node-status-port
        image: k8s.gcr.io/metrics-server/metrics-server:v0.4.1
        imagePullPolicy: IfNotPresent
        #
        # Precondition: Metrics Server installation attempted on Minikube (on Ubuntu) as per:
        #
        # https://github.com/kubernetes-sigs/metrics-server
        #
        # Specifically, the instructions suggest running the following in order to install the Metrics Server:
        #
        # kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
        #
        # and running:
        #
        # kubectl top nodes
        #
        # results in:
        #
        # "Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)"
        #
        # and running:
        #
        # kubectl get deployment metrics-server -n kube-system
        #
        # results in something that looks like:
        #
        # NAME             READY   UP-TO-DATE   AVAILABLE   AGE
        # metrics-server   1/1     1            0           6m
        #
        # -----
        #
        # We can see in the log that there's a problem:
        #
        # kubectl logs -n kube-system deploy/metrics-server
        #
        # E1129 22:08:56.736988       1 server.go:132] unable to fully scrape metrics: [unable to fully scrape metrics from node minikube: unable to fetch metrics from node minikube: Get "https://192.168.99.100:10250/stats/summary?only_cpu_and_memory=true": x509: cannot validate certificate for 192.168.99.100 because it doesn't contain any IP SANs, unable to fully scrape metrics from node m01: unable to fetch metrics from node m01: Get "https://192.168.99.100:10250/stats/summary?only_cpu_and_memory=true": x509: cannot validate certificate for 192.168.99.100 because it doesn't contain any IP SANs]
        # I1129 22:08:56.748352       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
        # I1129 22:08:56.748378       1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
        # I1129 22:08:56.748400       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
        # I1129 22:08:56.748404       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
        # I1129 22:08:56.748414       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
        # I1129 22:08:56.748417       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
        # I1129 22:08:56.749024       1 secure_serving.go:197] Serving securely on [::]:4443
        # I1129 22:08:56.749096       1 dynamic_serving_content.go:130] Starting serving-cert::/tmp/apiserver.crt::/tmp/apiserver.key
        # I1129 22:08:56.749163       1 tlsconfig.go:240] Starting DynamicServingCertificateController
        # I1129 22:08:56.848729       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
        # I1129 22:08:56.848758       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
        # I1129 22:08:56.849090       1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController
        #
        # -----
        #
        # The solution at the link below will remedy this issue:
        #
        # https://github.com/kubernetes-sigs/metrics-server/issues/196#issuecomment-451061841
        #
        # And we'll need to either modify the component.yaml file with the command below and
        # apply the file or we can edit the current configuration using:
        #
        # kubectl edit deployment -n kube-system metrics-server 
        #
        # Then execute:
        #
        # minikube stop
        # minikube start
        #
        # and try "kubectl top nodes" again and the Metrics Server should be working now.
        #
        # See also:
        # * https://github.com/kubernetes-sigs/metrics-server/issues/196
        # * https://github.com/juan-vg/metrics-server/commit/264f9c551abdeedc575d36e74da40671c51cb747
        #
        command:
        - /metrics-server
        - --kubelet-insecure-tls
        - --kubelet-preferred-address-types=InternalIP
        #
        # Changes end.
        #
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /livez
            port: https
            scheme: HTTPS
          periodSeconds: 10
        name: metrics-server
        ports:
        - containerPort: 4443
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /readyz
            port: https
            scheme: HTTPS
          periodSeconds: 10
        securityContext:
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
        volumeMounts:
        - mountPath: /tmp
          name: tmp-dir
      nodeSelector:
        kubernetes.io/os: linux
      priorityClassName: system-cluster-critical
      serviceAccountName: metrics-server
      volumes:
      - emptyDir: {}
        name: tmp-dir
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  labels:
    k8s-app: metrics-server
  name: v1beta1.metrics.k8s.io
spec:
  group: metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: metrics-server
    namespace: kube-system
  version: v1beta1
  versionPriority: 100

Step Three: Apply the components.yaml file to your local minikube metrics server deployment

Once we’ve edited the configuration directly, we can restart minikube and we should see something that looks like what we have below.

Note that the green arrows point to output that indicates that the Metrics Server is now running correctly — you should see something similar to this.

You can edit your components.yaml file however you’ll need to apply the configuration file once the change has been added — so either (step three):

				
					kubectl edit deployment -n kube-system metrics-server

and add this text to the appropriate place

modify the components.yaml file and then run:

				
					kubectl apply -f ./components.yaml

Step Four: Stop minikube

In the next step we’ll stop minikube as follows:

				
					minikube stop

See the first blue arrow below:

Step Five: Start minikube

Finally, we’ll start minikube as follows:

				
					minikube start

See the second blue arrow below and also note that the green arrows indicate that the metrics API is now running correctly.

As the Metrics Server is now operating correctly, this issue has been resolved.

Tutorial Conclusion

If you found this guide to be helpful you may also like the article I wrote recently entitled “Answers to Five Kubernetes CKAD Practice Questions (2020)” where we review five questions as they pertain to the Certified Kubernetes Application Developer (CKAD) exam and include answers along with commentary and verification that the given solutions work.

See the Kubernetes category and the containerization category for other articles that may be of interest.

Questions and comments are welcomed.

If you’re looking to hire a Temporary CTO then I might be able to help you — schedule an appointment with me today and we’ll discuss your requirements in more detail.

Frequently Asked Questions (FAQ)

What is Metrics Server in Kubernetes?

The Metrics Server in Kubernetes (k8s) is a scalable, efficient, and Kubernetes-native solution for aggregating resource usage data, such as CPU and memory, from each node and pod in a Kubernetes cluster. The k8s Metrics Server serves as a backend for the Kubernetes Horizontal Pod Autoscaler, which automatically scales the number of pod replicas based on observed CPU and memory usage. The Metrics Server collects this data at regular intervals, processes it, and then exposes it via the Kubernetes API, making it accessible for various use cases like autoscaling or capacity planning.

Learn how to install the Kubernetes Metrics Server in minikube in five steps here.

What is the Kubernetes Metrics Server license?

The Kubernetes Metrics Server has been released under the Apache License 2.0 license. The Apache License 2.0 license is a permissive free software license written by the Apache Software Foundation that allows users considerable freedom in using the software for any purpose, including commercial and proprietary uses.