How to setup Kyverno on GKE (Google Kubernetes Cluster), the right way

As Kubernetes continues to gain popularity as a powerful container orchestration platform, the need for effective policy management becomes increasingly important. One such solution to address this need is Kyverno

Apr 16, 2023

Introduction

As Kubernetes continues to gain popularity as a powerful container orchestration platform, the need for effective policy management becomes increasingly important. One such solution to address this need is Kyverno, a Kubernetes-native policy management tool. In this article, we will explore what Kyverno is, its key features, and how it can help streamline policy enforcement in your Kubernetes clusters.

What is Kyverno?

Kyverno, an open-source project, is a policy engine designed specifically for Kubernetes. It enables administrators and developers to define, validate, mutate, and generate configurations for their Kubernetes resources using a simple yet flexible policy language. Kyverno works as an admission controller, which means it can enforce policies at the time of resource creation or modification, preventing non-compliant configurations from being applied to the cluster.

Key Features

Policy Enforcement
Kyverno allows you to define policies that enforce specific configurations or best practices for your Kubernetes resources. These policies can be applied on a namespace or cluster-wide level, ensuring that all resources adhere to the desired configurations.

Resources Validation
Kyverno enables validation of Kubernetes resources by allowing you to create policies that check for compliance with specific rules. These rules can include anything from requiring specific labels or annotations to ensuring resource quotas are not exceeded.

Resources Mutation
Mutation policies in Kyverno can be used to automatically modify Kubernetes resources as they are created or updated. This can be useful for injecting sidecar containers, setting default values, or adding specific labels and annotations to resources.

Resources Generator
Kyverno's generation policies allow you to automatically create additional Kubernetes resources when a specific resource is created or updated. This can be helpful in creating and managing related resources, such as ConfigMaps or Secrets, alongside the main application.

How it works

Kyverno operates as a Kubernetes admission controller, which means it intercepts API requests and processes them based on the defined policies. The policies are created using the CustomResourceDefinition (CRD) feature in Kubernetes and are written in YAML format.

Kyverno policies consist of rules and match/exclude blocks. The rules specify the actions to be taken (validate, mutate, or generate), while the match/exclude blocks determine the resources that the rule should be applied to.

When a request is made to the Kubernetes API, Kyverno evaluates the request against the defined policies. If the request violates any policy rules, Kyverno will reject the request and return an error message to the user, indicating which policy was violated and why.

Why Kyverno

Benefits

Using Kyverno provides numerous benefits for Kubernetes cluster management, streamlining policy enforcement, and ensuring consistent and compliant configurations. While it may not be mandatory for every cluster, incorporating Kyverno can significantly improve the security and reliability of your applications. Here are some key benefits of using Kyverno:

Kubernetes-native policy management: Kyverno is designed specifically for Kubernetes, making it highly compatible with Kubernetes workflows and providing seamless integration with the platform.
Simplified policy creation: Kyverno uses YAML-based policy language, which is familiar to Kubernetes users, making it easy to create, understand, and maintain policies.
Dynamic policy enforcement: Kyverno operates as an admission controller, allowing it to enforce policies at the time of resource creation or modification. This prevents non-compliant configurations from being applied to the cluster, enhancing security and consistency.
Validation: Kyverno enables you to validate Kubernetes resources against specific rules, ensuring that resources comply with organizational best practices and security policies.
Mutation: With Kyverno, you can create mutation policies that automatically modify resources during creation or updates. This feature helps in setting default values, injecting sidecar containers, or adding specific labels and annotations.
Generation: Kyverno's generation policies let you automatically create related Kubernetes resources when a specific resource is created or updated. This simplifies the management of related resources, such as ConfigMaps or Secrets.
Extensibility: Kyverno provides a wide range of sample policies and the ability to create custom policies, allowing it to cater to various use cases and environments.
Improved security and compliance: By enforcing policies, validating resources, and ensuring consistent configurations, Kyverno can help you maintain a secure and compliant Kubernetes environment.

While Kyverno offers numerous benefits, it is essential to evaluate your organization's specific needs and requirements before deciding whether to implement it in your Kubernetes cluster. For organizations with complex policy requirements or those looking to improve security and compliance, Kyverno can be a valuable addition to their Kubernetes management toolkit.

Why your cluster needs to use Kyverno

While Kyverno provides a range of benefits, it's important to note that not every Kubernetes cluster may require it. However, there are several compelling reasons why many clusters could benefit from using Kyverno:

Consistency: Kyverno ensures consistent configurations across all resources within a cluster. This can be particularly valuable in large organizations with multiple teams working on different projects, as it can help maintain a standardized environment.
Improved security: Kyverno helps enforce security best practices by validating and mutating resources according to defined policies. This can prevent misconfigurations and vulnerabilities, thereby enhancing the overall security posture of your cluster.
Compliance: For organizations subject to regulatory requirements or internal policies, Kyverno can help maintain compliance by enforcing specific rules and configurations.
Automation: Kyverno automates many tasks, such as injecting sidecar containers, setting default values, or adding labels and annotations. This reduces manual intervention, saves time, and minimizes the risk of human error.
Simplified management: Kyverno enables you to define policies in a simple, declarative manner, making it easy to create, understand, and maintain policies. This simplification can streamline Kubernetes resource management.
Adaptability: With its extensible design, Kyverno can be customized to address a variety of use cases, making it a versatile solution for different environments and organizations.

While these reasons make Kyverno a compelling choice for many Kubernetes clusters, it is essential to carefully evaluate your organization's specific needs and requirements before deciding to implement it. If your cluster has complex policy requirements, or you're looking to improve security, compliance, and resource management, Kyverno can be a valuable addition to your Kubernetes toolkit. However, if your organization's use case is simple and does not require advanced policy management, it may not be necessary to incorporate Kyverno into your cluster.

Kyverno on GKE

Kyverno can be installed on any Kubernetes cluster, from self-hosted k8s to managed k8s. This article explains how to set up Kyverno on GKE. It assumes you use Terraform as an infrastructure provisioner, but you can always adjust it when using Google Cloud Console directly.

Setup Firewall

By default, GKE restricts communication between control and worker nodes. This occurs by closing certain ports on worker nodes. In this case, we just need to open port 9443.

Before that, we need to know the control plane IP CIDR by executing this command. For example, we have a k8s cluster called my-k8s in region asia-southeast2-a.

Here is our Terraform code. We will allow the control plane to access worker nodes through port 9443.

Run and apply Terraform

~$ terraform apply -auto-approve

Install Kyverno with Helm

Kyverno can be installed with the help of Helm. First, we create a dedicated namespace for Kyverno.

~$ kubectl create namespace kyverno

Run these commands to install Kyverno into your k8s cluster, inside the namespace kyverno

~$ helm repo add kyverno https://kyverno.github.io/kyverno/
~$ helm repo update
~$ helm install kyverno kyverno/kyverno -n kyverno

Kyverno Deployment

Assume we have a namespace called production where we deploy our applications in the production environment. Since it's a production environment, we only deploy stable versions of our applications. We do not want to deploy applications with the latest tag. We can use Kyverno to add a policy that disallows the latest tag in the production namespace. Here is our cluster policy, created with Kyverno.

# firewall.tf

# for kyverno
# GKE restricts communication between control plane and worker nodes, 
# so we need to set firewall rules to allow control plane communicates with worker node through port 9443
resource "google_compute_firewall" "kyverno-ingress-firewall" {
  name = "allow-ingress-kyverno"
  
  # your k8s vpc
  network = google_compute_network.k8s.name

  allow {
    protocol = "tcp"
    ports = [ "9443" ]
  }

  # GKE control plane IP CIDR Block
  source_ranges = [
    "172.18.0.0/28"
  ]
}

Execute this configuration and we will have Kyverno as our guard that prevent any application with latest tag to be deployed in namespace production .

About 8grams

We are a small DevOps Consulting Firm that has a mission to empower businesses with modern DevOps practices and technologies, enabling them to achieve digital transformation, improve efficiency, and drive growth.

Ready to transform your IT Operations and Software Development processes? Let's join forces and create innovative solutions that drive your business forward.

Subscribe to our newsletter for cutting-edge DevOps practices, tips, and insights delivered straight to your inbox!

Subscribe Now

8grams Tech

Discussion about this post