Finer permission control using K8s Admission Controller

4 min readApr 5, 2022

Situation

In our company, the whole company is using only one unified K8s cluster, and department services are separated by namespaces with different service accounts. Developers have the same set of permissions on corresponding namespace, basically only describe / logs to debug their own services. This architecture is simplest and easy to maintain for a small infra team, yet secured enough.

On the rise of DX, we want to open more permissions for our beloved devs, for example let them delete pods for restarting services / cronjobs, or create a deployment / cronjob by themselves without the need of sysadmins or intermediary like jenkins… so they will be happier at work!

Problem

There are problems arose:

With CREATE permission for deployments / cronjobs, developers can leverage to have privilege escalation and get root or doing some nasty things on the physical hosts. Or simply they create too much rubbish in the system.
With DELETE permission for pods, a naughty developer can intentionally or mistakenly drop mission-critical services running inside the namespace, for example redis, cassandra, elasticsearch…, causing chaos to the whole system.

Kubernetes RBAC just could not solve these issues. We want them to be happy, but we also dont want us to suffer.

Here comes Admission Controller for the rescue.

Admission Controller

There’s a long official document on k8s.io, therefore we wont go too deep and theoretic into it. In this blogpost we just focus on Validating Admission Controller, which validates the k8s manifest before make it applied and persisted in etcd. We dont want to make use of Mutating counterpart because the applying manifests will be different with the original ones stored in version control, making confusion for operations.

Look at the diagram below and it can explain itself a lot.

Basically when a manifest comes, Admission Controller invokes a AdmissionReview request to a webhook service (a HTTPS webserver). If the webserver return “OK” then the manifest is accepted, otherwise it will be rejected. Our job is simply to build a HTTPS webserver following format defined by Admission Controller, and we have full control of what to accept or reject according to the infomation including who requests and the manifest requested!

Having known that, we will try making ourselves a simple admission controller to reject deletion of critical services.

Admission Webhook Service

The demo will be using Python’s Flask webserver, as it’s more familiar to me. You can easily find others in the Internet using Go as well.

Firstly, we need to identify the format of AdmissionReview request and AdmissionReview response. There will be 2 functions responsible for parsing request and templating response.

def populate_ret(allowed, msg):
    return jsonify(
        {
            "apiVersion": "admission.k8s.io/v1",
            "kind": "AdmissionReview",
            "response": {
                "allowed": allowed,
                "uid": request.json["request"]["uid"],
                "status": {
                    "message": msg
                }
            }
        }
    )
---
def populate_info():
    #print(json.dumps(request.json, indent=4))    info = {}    info["namespace"] = request.json["request"]["namespace"]
    info["name"] = request.json["request"]["name"]
    ... # extract what you need
    
    return info

Secondly, after getting all the needed information we can decide to accept or reject the AdmissionReview.

# Validate DELETE on POD only
@app.route("/validate-delete", methods=["POST"])
def validate_delete():
    info = populate_info()
    allowed = True
    msg = ""    # Whitelist is allowed
    for user in WHITELIST_USER:
        if info["user"]["username"].startswith(user):
            return populate_ret(allowed, msg)    for pod in SYSTEM_ENTITY:
        if info["name"].startswith(pod):
            allowed = False
            msg = "deletion of {} prohibited".format(info["name"])
            return populate_ret(allowed, msg)    return populate_ret(allowed, msg)

The simplified code shows that we can define WHITELIST_USER — for power users who can delete everything , and SYSTEM_ENTITY — for restricted services — which can not be deleted by ordinary users. We can add more variables to apply restriction on some specific namespaces only.

The curiosity of TLS handling
If you have already known about Admission Controller, you will be curious about how we handle the TLS, as Admission Webhook Service have to be HTTPS. In our company, we’ve been already using Ingress-nginx for HA and TLS termination, for this webhook service we also offload the TLS termination to Ingress, so that our webserver will be much simpler.

It’s simple like that! Let’s test it out. I’ve removed myself from WHILELIST to test, and was trying to delete a redis pod. This is the result

Error from server: admission webhook "validate-delete" denied the request: deletion of redis-cluster-xxx is prohibited

While deleting other pod outside SYSTEM_ENTITY is still ok. Our Validating Admission Controller is working as expected!

Conclusion

K8s Admission Controller renders permission control to the very deep details. With it, we can control almost everything that’s defined in the manifest, according to our needs, basically everything that you can imagine of. For that, permissions for developers can be loosen but still under control, we’ve granted freedom under surveillance, and both developers and sysadmins can be happy at the same time!