- Dhrubajyoti Chakraborty
Validate, Clean & Secure K8s YAML files
Kubernetes workloads & configurations are most commonly defined in YAML. The configuration in k8s can sometime become complicated for a developer with the pod and container configuration files as those manifests scale in size you could easily end up overlooking a configuration option which could turn out to cost you some good bucks. One of the major challenges with YAML is to express constraints and the relationships between the manifest files.
Common questions that folks end up hitting:
What if I wish to check that all images deployed into the cluster are pulled from a trusted registry?
How can I prevent Deployments that don't have PodDisruptionBudgets from being submitted to the cluster?
Now integrating static checking allows catching errors and brings policy violations closer to the development lifecycle and since the confirmation around the validity and safety of the resource definitions is improved you can trust that production workloads are following best practices.
In this topic we'll run through some of the best practices of K8s development and also checkout tools that could help one to maintain the quality of the development lifecycle with minimal efforts. If you are a developer, devops engineer, devops specialist or simply interested in learning more about the quality assurance process around creating and managing K8s manifests then this should help you in some way or the other.
Getting some context
Kubernetes already has a powerful and segreggates process of validating resources in the form of the admission controllers, policy enforcement via tools like Open Policy Agent, Kyverno or Kubewarden, pod security policies (deprecated in 1.21) etc This tools are great but they derive from the cluster operator layer and are more aligned on a platform rather than a tool that fits neatly in a standard developer workflow. The tools and methods we learn in this article will transfer the kick-off point of a validation process to a development workflow and thus greatly improve the quality of the whole ecosystem.
The focus here is to make illegal states irrepresentable. This is difficult to do with a non-statically typed language like YAML but with the right tools it can be done.
Static Checking Categories
The ecosystem of static checking of K8s YAML files can be grouped in the following categories:
API validators — Tools in this category validate a given YAML manifest against the Kubernetes API server.
Built-in checkers — Tools in this category bundle opinionated checks for security, best practices, etc.
What do we need to validate?
Let's try to break down the validation process into categories and understand how following the same distinction can help you validate your yaml files faster than ever?
Structural validation: Maintaining the yaml file structure can be a bit mess sometimes for the developer thus validating the structure before execution can help scale down error encountering probability to some extent.
Semantic validation of K8s schema: Here the focus is given on validating whether the file is a correct K8s YAML file. This is an automated process is a bit too late in the lifecycle. An interesting tool that we'll explore in this space is kubeconform.
Pragmatic validation of the resource: This is where the validation process looks at the file from different contexts. We want to check the file and configuration for security vulnerabilities, performance issues, adherence to best practices, versioning schemes and many more.
Here I have mentioned 4 different tools that helps us validating the YAML files but there are various others in the marketplace so make sure to check them out as well.
Kubeconform is a Kubernetes manifest validation tool. YOu can add it to your CI workflow, or use it locally to validate the k8s configuration!
It is inspired by contains code from and is designed to stay close to Kubeval but with the following improvements:
- High performance: will validate & download manifests over multiple routines, caching - downloaded files in memory
- Configurable list of remote, or local schemas locations, enabling validating K8s custom resources (CRDs) and offline validation capabilities
- Uses by default a self-updating fork of the schemas registry maintained by the kubernetes-json-schema project - which guarantees up-to-date schemas for all recent versions of Kubernetes.
The fact that kubeconform has all the schemas for all the different k8s versions available—and also doesn't require Minikube setup (as kubectl does)—makes it a superior tool when comparing these capabilities to its alternatives.
Kube-score is a tool that performs static code analysis of your Kubernetes object definitions. The output is a list of recommendations of what you can improve to make your application more secure and resilient.
You can test kube-score out in the browser with the online demo (source). Kube-score checks are an excellent tool to enforce best practices but currently it does not support customization or addition of custom rules.
If you want to write custom checks to comply with your organisational policies you can use one of the next two options - config-lint or polaris.
This is essentially a command line tool (devs love it) i.e used to validate configuration files using rules specified in YAML. The configuration files can be one of several formats: Terraform, JSON, YAML, with support for K8s. There are built-in rules provided for Terraform and custom files can be used for other formats.
Now this does not comes with in-built checks for k8s manifests which means you'll have to write your own custom rules to perform any validations.
The rules are written as YAML files referred to as rulesets and have the following structure:
description: Rules for Kubernetes spec files
# list of rules
Polaris is an open source policy engine for Kubernetes that validates and remediates resource configuration. It includes 30+ built in configuration policies, as well as the ability to build custom policies with JSON Schema. When run on the command line or as a mutating webhook, it can automatically remediate issues based on policy criteria.
It can essentially be run in three different modes:
- As a dashboard: used for validating k8s resources against policy-as-code.
- As an admission controller: automatically reject or modify workloads that don't adhere to your organization's policies.
- As a command-line tool: allows you to integrate policy-as-code into the CI/CD process to test local YAML files.
Similar to kube-score, polaris identifies several test cases where the manifest falls short of recommended best practices which include:
- Pods health-check.
- Unspecification of tags in container images.
- Container run status(i.e root).
- CPU and memory requests and limits are not set.
- Checks are classified with a severity level of warning or danger.
While there are a bunch of tools to validate, score and lint k8s YAML files it becomes important to have a structural model on how you want to design and perform the checks.
The following table presents a summary of the tools:
Now the tools that I mentioned are not dependent on access to a k8s cluster and thus it becomes easier to set up by allowing you to implement gating as well as give quick feedback to pull request authors for projects.
Thanks. Share it if you find it useful.