Secure and Harden Kubernetes, AKS and EKS Cluster with kube-bench, kube-hunter and CIS Benchmarks

Security is a concern that never fades away, and it doesn’t matter how recent the technology has rolled out. So we should always harden the Kubernetes cluster by using kube-bench, kube-hunter and the CIS benchmarks. The more secure our cluster is, they are less prone to downtimes and hackers.

I will go into detail with kube-bench, kube-hunter and check loopholes on a cluster. So, the topics covered in this post are:

• Container Security
• CIS benchmarks for Kubernetes
• What is kube-bench?
• What is a kube-hunter?
• Kubernetes Security Best Practices
• kube-bench on a Kubernetes Cluster
• kube-bench on Managed Kubernetes
• kube-hunter on a Kubernetes Cluster
• Conclusion

Container Security and Best Practices

A containerised platform needs certain parts to come together to work in full flow. So, to start with, we need a Container Image, an Image Registry, a Runtime Environment, an Orchestration platform and finally, a Host OS. It gets difficult to check for vulnerabilities with these many objects involved; continuous monitoring becomes hectic.

But, let’s look at some of the basic things, a.k.a best practices, that we can do to keep our environment secure:

• Use updated container images, scan and sign them for further use.
• Make your image registry private and keep monitoring it.
• App security must be at par, and the network protocol and payloads must be secure.
• Limit the Orchestration platform according to roles, and monitor the pod communication.
• The Host OS should have an access control system and, monitoring is a must.

The tool used for scanning a container image is Trivy; Prometheus and Grafana for monitoring purposes, and Cilium for the firewall.

CIS Benchmarks for Kubernetes

Center for Internet Security (CIS) lays down guidelines and benchmarks for secure software development and maintenance. In addition, provide us with the best practices for the secure configuration of a target system.

The CIS Benchmark is tied up to a particular version of a system. Hence, it is the same for K8s as well. The CIS Kubernetes Benchmark is a collection of propositions for securing a secure Kubernetes environment. It is made open-source for Kubernetes distribution.

What is kube-bench?

With more flexibility comes more vulnerabilities and, configuring  Kubernetes security is a task indeed. The Center for Internet Security (CIS) provides guidelines and benchmarks tests for securing Kubernetes and achieving a Hardening level for a K8s cluster.

Kube Bench is an open-source Go application that runs the CIS Kubernetes Benchmark and tests a K8s cluster to ensure that it meets the CIS guidelines for security.

Although there are many kinds of Kubernetes clusters, kube-bench runs as a container. Since we can test all types of clusters that exist, I will be showing you a step-by-step procedure for running a test on your Kubernetes cluster. Aquasec, the guys who created the tool, has got you covered for the on-premise cluster too.

What is a kube-hunter?

Aquasec has got the security game strong. It has built another tool that checks for all the vulnerabilities of a Kubernetes Cluster. It’s intended to boost awareness and visibility of the security controls in Kubernetes environments.

kube-hunter is an open-source tool that hunts for security issues in your Kubernetes clusters.

It proposes three options: remote scanning, network scanning, and internal scanning. Users will be able to see a list of the tests run, either in passive or active mode and set the logging level as desired.

Kubernetes Security Best Practices

We know that the best practices are not mandatory to make our cluster functional. However, there has to be a reason for these to be called the ‘best practices’, right? Well, let’s look at some of them regarding the security of Kubernetes.

• Keep Kubernetes up to date.
• Enable Kubernetes Role-Based Access Control (RBAC)
• Use Third-Party Authentication for API Server
• Make sure you’re using namespace
• Use network policies to restrict access
• Do not run as root
• Protect etcd with TLS, Firewall and Encryption
• Isolate Kubernetes Nodes
• Set up IAM access
• Have regular security reviews

So, let’s see everything in action and make our clusters secure as ever!

kube-bench on a Kubernetes Cluster

As with everything on the container platform, even the kube-bench has to run as a container. It can run inside a pod, but we’d need the host’s PID namespace. We can also define it inside a job and check for the pod that’s created from it.

Wait, wait, before we get started, it goes without saying that we need a Kubernetes Cluster up and running to proceed further.

Step 1: Running inside a container 

docker run --pid=host -v /etc:/etc:ro -v /var:/var:ro -t aquasec/kube-bench:latest --version 1.19

Step 2: Rectifying the issue

If you see any ‘Fail’  in the kube-bench test, scroll up to that section and check for the number associated with it. The next step is to download the CIS Benchmark document for your Kubernetes release from here. And search for the indexed number and follow the instructions given.

kube-bench on Managed Kubernetes Cluster [AKS, EKS, GKE]

The thing with the managed Kubernetes services such as EKS, GKE, AKS, and so on is that you can’t run the checks on the master node. Therefore, you have to follow the worker checks from the previous procedure or run a Kubernetes job to validate the environment.

So, in a managed K8s Cluster, we will deploy a job and check for the logs of the deployed kube-bench pod.

Step 1: Defining job.yaml

vim job.yaml

---
apiVersion: batch/v1
kind: Job
metadata:
  name: kube-bench
spec:
  template:
    metadata:
      labels:
        app: kube-bench
    spec:
      hostPID: true
      containers:
        - name: kube-bench
          image: aquasec/kube-bench:latest
          command: ["kube-bench"]
          volumeMounts:
            - name: var-lib-etcd
              mountPath: /var/lib/etcd
              readOnly: true
            - name: var-lib-kubelet
              mountPath: /var/lib/kubelet
              readOnly: true
            - name: var-lib-kube-scheduler
              mountPath: /var/lib/kube-scheduler
              readOnly: true
            - name: var-lib-kube-controller-manager
              mountPath: /var/lib/kube-controller-manager
              readOnly: true
            - name: etc-systemd
              mountPath: /etc/systemd
              readOnly: true
            - name: lib-systemd
              mountPath: /lib/systemd/
              readOnly: true
            - name: srv-kubernetes
              mountPath: /srv/kubernetes/
              readOnly: true
            - name: etc-kubernetes
              mountPath: /etc/kubernetes
              readOnly: true
              # /usr/local/mount-from-host/bin is mounted to access kubectl / kubelet, for auto-detecting the Kubernetes version.
              # You can omit this mount if you specify --version as part of the command.
            - name: usr-bin
              mountPath: /usr/local/mount-from-host/bin
              readOnly: true
            - name: etc-cni-netd
              mountPath: /etc/cni/net.d/
              readOnly: true
            - name: opt-cni-bin
              mountPath: /opt/cni/bin/
              readOnly: true
      restartPolicy: Never
      volumes:
        - name: var-lib-etcd
          hostPath:
            path: "/var/lib/etcd"
        - name: var-lib-kubelet
          hostPath:
            path: "/var/lib/kubelet"
        - name: var-lib-kube-scheduler
          hostPath:
            path: "/var/lib/kube-scheduler"
        - name: var-lib-kube-controller-manager
          hostPath:
            path: "/var/lib/kube-controller-manager"
        - name: etc-systemd
          hostPath:
            path: "/etc/systemd"
        - name: lib-systemd
          hostPath:
            path: "/lib/systemd"
        - name: srv-kubernetes
          hostPath:
            path: "/srv/kubernetes"
        - name: etc-kubernetes
          hostPath:
            path: "/etc/kubernetes"
        - name: usr-bin
          hostPath:
            path: "/usr/bin"
        - name: etc-cni-netd
          hostPath:
            path: "/etc/cni/net.d/"
        - name: opt-cni-bin
          hostPath:
            path: "/opt/cni/bin/"

Step 2: Create a job

We create a job and check for the pod deployed from the job in the following process.

kubectl apply -f job.yaml
kubectl get pods

Kindly copy the name of the pod for the following procedure.

Step 2: Check the result 

So, finally, to check the results, you should run the following command with the pod name you copied.

kubectl logs kube-bench-pdb6

All the managed Kubernetes clusters provide you with a certain level of security by following the CIS benchmarks by default. Hence, you don’t see any ‘FAIL’ in the result of a kube-bench test on managed K8s.

Kube Bench also provides platform-specific job manifests that you can use for your cluster, such as job-gke.yaml, job-aks.yaml, and job-eks.yaml here

kube-hunter on a Kubernetes Cluster

Running a kube-hunter test on your cluster gives you a view of what the attackers see and how can you get rid of this vulnerability.

Step 1: Install kube-hunter

pip install kube-hunter

Step 2: Run kube-hunter

You can run kube-hunter in 3 ways:

1. Remote Scanning – You can either specify your IP or DNS name of the managed Kubernetes cluster
2. Interface Scanning – It probe all the local network interfaces.
3. IP Range scanning – It scans only in the range of the given IP.

kube-hunter

The result:

Well, there is another method where you can run it as a pod. You’d have to follow the same procedure of defining a job.yaml, check for the running pod and get the logs of it. You can get more info on it from here.

Conclusion

Thank you for reading. I hope that you have run a test on your cluster by now!

Kube Bench gives us a robust way of assuring that our K8s cluster is set up correctly and appropriately secured. Kube Hunter shows us the vulnerabilities that we have to rectify. The best part is that the standards are updated frequently, and it becomes crucial that you standardise the Kube Bench test in your operating procedure.

The post Secure and Harden Kubernetes, AKS and EKS Cluster with kube-bench, kube-hunter and CIS Benchmarks appeared first on Cloud Training Program.