Policy based kubernetes admission webhook part 1
A lightweight but powerful and programmable rule engine for kubernetes admission webhook called
kinitiras
powered byk-cloud-labs
.
1.Background
Admission webhooks are HTTP callbacks that receive admission requests and do something with them. You can define two types of admission webhooks, validating admission webhook
and mutating admission webhook
. Mutating admission webhooks are invoked first
, and can modify objects sent to the API server to enforce custom defaults.
After all object modifications are complete, and after the incoming object is validated by the API server, validating admission webhooks are invoked and can reject requests to enforce custom policies.
The above is the definition given by kubernetes. In short, k8s webhooks are divided into two categories validating
and mutating
for verifying and modifying k8s resource modification operations, respectively. When a user creates or modifies a resource, the apiserver
calls (http) the registered validating
and mutating
webhooks and does not actually process the modification until after the response of these webhooks.
For those who are familiar with cloud native development or k8s, webhook should be a very familiar component and should have written more or less relevant code. As a native capability of k8s, the current webhook development/use approach is very unfriendly and primitive. While many of the k8s capabilities can be achieved by modifying existing objects or CRD information (e.g., updating images, rolling upgrades, etc.), webhook requires code to be developed and registered to the apiserver
and run to achieve the purpose.
However, in daily development, most of the requirements for webhook are limited to some simple capabilities, such as verifying the fields and legality of certain objects, modifying the information of specific objects (label, annotation or modifying resource specifications, etc.), which leads to the need for development to package and release the code every time new requirements are added or modified. This makes the whole process inefficient, and there are too many webhooks registered in a cluster to make the apiserver’s response time longer and less reliable.
In order to solve these problems, we developed a new set of rule-based programmable webhooks – kinitiras, hoping to replace all the webhooks in a cluster with this component and all the requirements. We hope to use this component to replace all the webhooks in the cluster and all the requirements can be solved by this component without developing new webhooks, improving efficiency and reducing the security risks caused by multiple webhooks.
2.Ability
First, lets talk about what this webhook can do.
2.1 Validate resources
We can configure different policies for different resources to reduce the occurrence of some uncontrollable situations or to restrict some special operations, for example.
- For create update operations, some fields of the resource can be restricted (not null or with values equal to or not equal to the specified value, etc.)
- Restrict update or delete operations. For some specific resources (ns or deployment), you can prohibit deletion or secondary confirmation mechanism (deletion is allowed only if the specified annotation exists)
- Field validation
The fields and values of these checks can be the value of the current object or the value of other objects (e.g., compared with other objects such as cm
or secret
), and the data obtained by third-party http services can be checked.
2.2 Mutate resources
For modifying resources, our strategy can be configured for many different scenarios to meet different needs, such as
- Uniformly add label or annotation to resources.
- Modify the pod resource of limit or request.
- modify the affinity toleration of the resource, etc.
The modified values can be dynamic and can be obtained from other objects (e.g., writing the owner’s properties to the pod) or from third-party http services (I have similar needs myself).
3.Design
Kinitiras
is from the Greek κινητήρας (kini̱tí̱ras)
, meaning engine/motor, and the core capability of the project is also a rules-based engine that provides efficient and powerful capabilities.
3.1 Basic concept
Concept | Explain |
---|---|
validating |
Verifying the legitimacy of objects |
mutating |
Modifying objects info |
policy |
a crd object contains rules |
override policy |
policy contains mutate object rules |
validate policy |
policy contains validate object rules |
cue |
an open source program language,https://cuelang.org |
3.2 Core logic
Here is the core logic of kinitiras:
- Define the crd corresponding to validating and mutating to represent a policy, record the scope of the policy (specified resource name or label) and the execution rules (verify or modify the content)
- register a unified webhook configuration and subscribe to all modification and deletion events for resources with a specific label by default (you can customize this configuration during installation)
- when receiving a callback from apiserver, the current modified resource and the existing policy match to filter the list of hit policies
- execute policies in a sequential manner
3.3 Api definition
This project has defined three CRD:
- OverridePolicy: mutate object info(namespace level)
- ClusterOverridePolicy: mutate object info(cluster level)
- ClusterValidatePolicy: validate object info(cluster level)
Now, lets talk about core part of those CRDs.
3.3.1 Resource selector
ResourceSelector the resources will be selected.
Field | type | required | Description |
---|---|---|---|
apiVersion | string | Y | APIVersion represents the API version of the target resources. |
kind | string | Y | Kind represents the Kind of the target resources. |
namespace | string | N | Namespace of the target resource. Default is empty, which means inherit from the parent object scope. |
name | string | N | Name of the target resource. Default is empty, which means selecting all resources. |
labelSelector | Kubernetes meta/v1.LabelSelector | N | A label query over a set of resources. If name is not empty, labelSelector will be ignored. |
fieldSelector | FieldSelector | N | A field query over a set of resources. If name is not empty, fieldSelector wil be ignored. |
This structure is used to select a policy to match a resource, either by specifying a particular resource, or by specifying a label or field to take effect for a group of resources.
for example:
|
|
3.3.2 Validate rule
Defines validate rules on operations.
Field | type | required | Description |
---|---|---|---|
targetOperations | []Kubernetes admission/v1.Operation | Y | Operations is the operations the admission hook cares about - CREATE, UPDATE, DELETE, CONNECT or * for all of those operations and any future admission operations that are added. If * is present, the length of the slice must be one. |
cue | string | N | Cue represents validate rules defined with cue code. |
template | ValidateRuleTemplate | N | Template of condition which defines validate cond, and it will be rendered to CUE and store in RenderedCue field, so if there are any data added manually will be erased. |
renderedCue | string | N | RenderedCue represents validate rule defined by Template. Don’t modify the value of this field, modify Rules instead of. |
Here is the information that defines the execution logic of the validate policy.
- targetOperations: indicates the type of operation that will take effect, i.e., it can be defined to take effect only for create or delete events
- cue: This field can be filled with a cue code, which will be executed after the policy is hit, see the following Example
- template: defines a simple template that templates some common checks (no need to write a cue anymore)
- renderedCue: the template will eventually be automatically rendered into cue code and stored on this field.
cue example:
|
|
template example:
|
|
This templated approach can reduce the cost of learning cue and can satisfy most scenarios such as determining whether a field exists and doing size comparison with other fields.
3.3.3 Override rule
Overriders offers various alternatives to represent the override rules.
If more than one alternative exist, they will be applied with following order:
- RenderCue
- Cue
- Plaintext
Field | type | required | Description |
---|---|---|---|
plaintext | []PlaintextOverrider | N | Plaintext represents override rules defined with plaintext overriders. |
cue | string | N | Cue represents override rules defined with cue code. |
template | OverrideRuleTemplate | N | Template of rule which defines override rule, and it will be rendered to CUE and store in RenderedCue field, so if there are any data added manually will be erased. |
renderedCue | string | N | RenderedCue represents override rule defined by Template. Don’t modify the value of this field, modify Rules instead of. |
The core part of the Override policy is defined here, i.e. the modification of resource information policy, which means the following.
- plaintext: This is a simple modification method, just fill in the fields and values of the operation
- cue: this field can be filled with a cue code, which will be executed after the policy is hit, see the following Example
- template: defines a simple template that templates some common modifications (no need to write a cue anymore)
- renderedCue: the template will eventually be automatically rendered into cue code and stored on this field.
plaintext example:
|
|
cue example:
|
|
template example:
|
|
The template shown above is only a part of the implementation, for more usage you can check the examples on the official website, click to jump
4.Implement
The above describes how kinitiras works and the core concepts, from here on the implementation of its core capabilities are briefly described, so that it is easy to debug or understand the underlying implementation when using it.
kinitiras
, pkg
, and pidalio
. Each repo is positioned differently, where kinitiras
is a more conventional webhook project, responsible for registering itself to the apiserver and handling callbacks, and pkg
is the implementation of the core modules, such as api definitions, execution policies, and so on. And pidalio
is the transport middleware for client-go, which can intercept requests and execute policy applications on the client side. This article focuses on the core implementation of the first two repo’s.4.1 kinitiras
Repo URL: https://github.com/k-cloud-labs/kinitiras
The core logic of this project, as a webhook, is to initialize the parameters and register itself to the apiserver, and then call the unified processing method provided by pkg
in the callback function.
4.1.2 Initialize
|
|
The above code is quite conventional, only sm.setupInterrupter()
is mentioned separately. The webhook defines several CRDs as the carrier of the policy, and the policy itself needs to be verified and modified, especially after providing a template (template
), which needs to be rendered into a cue
script, so the concept of interrupter
is introduced in order to be able to verify and render the policy when it is created. Interrupter
, as the name suggests – an interceptor, is used to intercept the policy and perform checks on specific fields of the policy and rendering of the template, these logic is not quite the same as the regular object checks and modifications, so it does not go through the ordinary logic, but only through the logical part of interrupter
. And the above sm.setupInterrupter()
is used to initialize these interrupters
with the following code.
|
|
4.1.2 admission handler
Now, lets take a look to the registered handler.
Take mutating as an example:
|
|
Here, the core logic of kinitiras
is basically finished, his responsibility is to initialize, register, and direct callback requests to the implemented processing methods, which are implemented by the pkg
project.
4.2 pkg
Repo URL: https://github.com/k-cloud-labs/pkg
The pkg
contains most of the logic implementation, as well as the definition of the crd and the generated client code. Following the above, we will focus on the implementation of interrupter
and the hitting and execution of the policy.
4.2.1 interrupter
definition:
|
|
PolicyInterrupterManager
inherits from PolicyInterrupter
and adds a method to add interrupters to manage multiple interrupters, and each interrupter implements the following three methods.
- OnMutating: Called when the apiserver calls back the
/mutating
interface, mainly to render and supplement policy information - OnValidating: called when the apiserver calls back the
/validating
interface, mainly to validate policy information - OnStartUp: called during the webhook startup phase to do some initialization work (pulling cache, etc.)
The implementation of manager is different from the actual interrupter, it first identifies if the current resource is the policy crd we defined, then looks for a corresponding registered interrupter from memory and calls the corresponding method of that interrupter. The code is as follows.
|
|
4.2.2 Render
The matter of rendering has been mentioned few times before, now lets talk about it.
About background.
The project has supported the user to write cue by hand to execute complex logic in the strategy to meet different needs in the early days. However, writing cue requires some knowledge of the syntax and features of the language and there is no good mechanism to verify the legitimacy of cue scripts, which makes it difficult to get started, so we came up with the idea of abstracting some common cases into a structured template, where the user only needs to fill in the necessary parameters, and webhook itself translates the template into cue scripts.
To be able to translate structured data into cue scripts, we wrote a more complex go/tmpl
([template link](https://github.com/k-cloud-labs/pkg/tree/main/utils/templatemanager/ templates)), and then proceeded to translate it. The process is as follows.
- interrupter checks if template information is filled in
- render according to the template type (tmpl.Execute) to generate the cue script
- format and lint check the result
This process is called render
.
About implement.
Since there are more related templates and code, instead of showing them here, only the core implementation is illustrated.
- different
tmpl
s are written for different policies. Since the structure of cue execution results for validate and override policies are different, twotmpl
s are written to perform different rendering according to the policy. code link 2. - use the official go package provided by cue for formatting and lint. cue is implemented in the go language, so it is friendly to go and provides a package for formatting and linting cue scripts directly in the code to ensure that the rendered cue scripts are legal and runnable. code link
4.2.3 How to match with policies
The matching process between the current object and the policy is as follows.
- List all current policies. This is read from informer memory, and the corresponding policy list is read depending on whether it is currently validating or mutating.
- For policies that do not have a resource selector set, the policy is considered a hit by default.
- for policies with resource selector set, perform policy matching (code will be shown below.)
- Match the operation type of the hit policy with the operation type of the current object.
- Matching is done.
Resource selector matching rules.
any means no matter if it’s empty or not.
name | label selector | field selector | result |
---|---|---|---|
not empty | any | any | match name only |
empty | empty | empty | match all |
empty | not empty | empty | match labels only |
empty | empty | not empty | match fields only |
empty | not empty | not empty | match both labels and fields |
Related code:
|
|
4.2.4 How to execute policies
After the policy is hit in the previous step, the policies are sorted in a dictionary and then executed in order, while the execution process is based on the rules configured for each policy. The process is as follows (using the Override policy as an example).
- check if the template is configured and rendered, and if so, execute the cue script
The template supports referencing the current object or other objects in the cluster or even external http interface data, so you need to determine what data is referenced and prepare the data in advance before executing cue (i.e., get the object or request http for the response body)
|
|
Execute cue script:
|
|
cue example:
|
|
- Check if a custom cue script is configured, and if so, execute
|
|
- Check if the plaintext form of patch is configured, and if so, apply it directly
|
|
5.Summarize
This article introduces the webhook product launched by k-cloud-labs
, which is excellent in terms of functionality and usability. I am now working as one of the maintainers of the project to add and optimize certain features to the project, and will continue to update new capabilities and solve more problems later.
Main content.
- Introduced the background of developing the webhook and the problem it solves
- Introduced the core design ideas and api definitions
- Introduces the implementation of the core logic
For more detailed design details and use cases and installation methods, please click here to jump the official website to understand.
6.Links🔗
- Official Website: https://k-cloud-labs.github.io/kinitiras-doc/
- Github: https://github.com/k-cloud-labs