Kubernetes HTTP Request Flow

As we learned in earlier chapters, when we run any kubectl command, the command is translated into an HTTP API request in JSON format and is sent to the API server. Then, the API server returns a response to the client, along with any requested information. The following diagram shows the API request life cycle and what happens inside the API server when it receives a request:

Figure 4.2: API server HTTP request flow

As you can see in the preceding figure, the HTTP request goes through the authentication, authorization, and admission control stages. We will take a look at each of these in the following subtopics.

Authentication

In Kubernetes, every API call needs to authenticate with the API server, regardless of whether it comes from outside the cluster, such as those made by kubectl, or a process inside the cluster, such as those made by kubelet.

When an HTTP request is sent to the API server, the API server needs to authenticate the client sending this request. The HTTP request will contain the information required for authentication, such as the username, user ID, and group. The authentication method will be determined by either the header or the certificate of the request. To deal with these different methods, the API server has different authentication plugins, such as ServiceAccount tokens, which are used to authenticate ServiceAccounts, and at least one other method to authenticate users, such as X.509 client certificates.

Note

The cluster administrator usually defines authentication plugins during cluster creation. You can learn more about the various authentication strategies and authentication plugins at https://kubernetes.io/docs/reference/access-authn-authz/authentication/.

We will take a look at the implementation of certificate-based authentication in Chapter 11, Build Your Own HA Cluster.

The API server will call those plugins one by one until one of them authenticates the request. If all of them fail, then the authentication fails. If the authentication succeeds, then the authentication phase is complete and the request proceeds to the authorization phase.

Authorization

After authentication is successful, the attributes from the HTTP request are sent to the authorization plugin to determine whether the user is permitted to perform the requested action. There are various levels of privileges that different users may have; for example, can a given user create a pod in the requested namespace? Can the user delete a Deployment? These kinds of decisions are made in the authorization phase.

Consider an example where you have two users. A user called ReadOnlyUser (just a hypothetical name) should be allowed to list pods in the default namespace only, and ClusterAdmin (another hypothetical name) should be able to perform all tasks across all namespaces:

Figure 4.3: Privileges for our two users

To understand this better, take a look at the following demonstration:

Note

We will not pe into too much detail about how to create users as this will be discussed in Chapter 13, Runtime and Network Security in Kubernetes. For this demonstration, the users, along with their permissions, are already set up, and the limitation of their privileges is demonstrated by switching contexts.

Figure 4.4: Demonstrating different user privileges

Notice, from the preceding screenshot, that the ReadOnlyUser can only list pods in the default namespace, but when trying to perform other tasks, such as deleting a pod in the default namespace or listing pods in other namespaces, the user will get a Forbidden error. This Forbidden error is returned by the authorization plugin.

kubectl provides a tool that you can call by using kubectl auth can-i to check whether an action is allowed for the current user.

Let's consider the following examples in the context of the previous demonstration. Let's say that the ReadOnlyUser runs the following commands:

kubectl auth can-i get pods --all-namespaces

kubectl auth can-i get pods -n default

The user should see the following responses:

Figure 4.5: Checking privileges for ReadOnlyUser

Now, after switching context, let's say that the ClusterAdmin user runs the following commands:

kubectl auth can-i delete pods

kubectl auth can-i get pods

kubectl auth can-i get pods --all-namespaces

The user should see the following response:

Figure 4.6: Checking privileges for ClusterAdmin

Unlike authentication phase modules, authorization modules are checked in sequence. If multiple authorization modules are configured, and if any authorizer approves or denies a request, that decision is immediately returned, and no other authorizer will be contacted.

Admission Control

After the request is authenticated and authorized, it goes to the admission control modules. These modules can modify or reject requests. If the request is only trying to perform a READ operation, it bypasses this stage; but if it is trying to create, modify, or delete, it will be sent to the admission controller plugins. Kubernetes comes with a set of predefined admission controllers, although you can define custom admission controllers as well.

These plugins may modify the incoming object, in some cases to apply system-configured defaults or even to deny the request. Like authorization modules, if any admission controller module rejects the request, then the request is dropped and it will not process further.

Some examples are as follows:

  • If we configure a custom rule that every object should have a label (which you will learn how to do in Chapter 16, Kubernetes Admission Controllers), then any request to create an object without a label will be rejected by the admission controllers.
  • When you delete a namespace, it goes to the Terminating state, where Kubernetes will try to evict all the resources in it before deleting it. So, we cannot create any new objects in this namespace. NamespaceLifecycle is what prevents that.
  • When a client tries to create a resource in a namespace that does not exist, the NamespaceExists admission controller rejects the request.

Out of the different modules included in Kubernetes, not all of the admission control modules are enabled by default, and the default modules usually change based on the Kubernetes version. Providers of cloud-based Kubernetes solutions, such as Amazon Web Services (AWS), Google, and Azure, control which plugins can be enabled by default. Cluster administrators can also decide which modules to enable or disable when initializing the API server. By using the --enable-admission-plugins flag, administrators can control which modules should be enabled other than the default ones. On the other hand, the --disable-admission-plugins flag controls which modules from the default modules should be disabled.

Note

You will learn more about admission controllers, including creating custom ones, in Chapter 16, Kubernetes Admission Controllers.

As you will recall from Chapter 2, An Overview of Kubernetes, when we created a cluster using the minikube start command, Minikube enabled several modules for us by default. Let's take a closer look at that in the next exercise in which we will not only view the different API modules enabled for us by default but also start Minikube with a custom set of modules.

Exercise 4.01: Starting Minikube with a Custom Set of Modules

In this exercise, we will take a look at how to view the different API modules enabled for our instance of Minikube, and then restart Minikube using a custom set of API modules:

  1. If Minikube is not already running on your machine, start it up by using the following command:

    minikube start

    You should see the following response:

    Figure 4.7: Starting up Minikube

  2. Now, let's see which modules are enabled by default. Use the following command:

    kubectl describe pod kube-apiserver-minikube -n kube-system | grep enable-admission-plugins

    You should see the following response:

    Figure 4.8: Default modules enabled in Minikube

    As you can observe from the preceding output, Minikube has enabled the following modules for us: NamespaceLifecycle, LimitRanger, ServiceAccount, DefaultStorageClass, DefaultTolerationSeconds, NodeRestriction, MutatingAdmissionWebhook, ValidatingAdmissionWebhook, and ResourceQuota.

    Note

    To know more about modules, please refer the following link: https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/

  3. Another way to check the modules is to view the API server manifest by running the following command:

    kubectl exec -it kube-apiserver-minikube -n kube-system -- kube-apiserver -h | grep "enable-admission-plugins" | grep -vi deprecated

    Note

    We used grep -vi deprecated because there is another flag, --admission-control, that we are discarding from the output, as this flag will be deprecated in future versions.

    kubectl has the exec command, which allows us to execute a command to our running pods. This command will execute kube-apiserver -h inside our kube-apiserver-minikube pod and return the output to our shell:

    Figure 4.9: Checking the modules enabled by default in Minikube

  4. Now, we will start Minikube with our desired configuration. Use the following command:

    minikube start --extra-config=apiserver.enable-admission-plugins="LimitRanger,NamespaceExists,NamespaceLifecycle,ResourceQuota,ServiceAccount,DefaultStorageClass,MutatingAdmissionWebhook"

    As you can see here, the minikube start command has the --extra-config configurator flag, which allows us to pass additional configurations to our cluster installation. In our case, we can use the --extra-config flag, along with --enable-admission-plugins, and specify the plugins we need to enable. Our command should produce this output:

    Figure 4.10: Restarting Minikube with a custom set of modules

  5. Now, let's compare this instance of Minikube with our earlier one. Use the following command:

    kubectl describe pod kube-apiserver-minikube -n kube-system | grep enable-admission-plugins

    You should see the following response:

Figure 4.11: Checking a custom set of modules for Minikube

If you compare the set of modules seen here to the ones in Figure 4.7, you will notice that only the specified plugins were enabled; while the DefaultTolerationSeconds, NodeRestriction, and ValidatingAdmissionWebhook modules are no longer enabled.

Note

You can revert to the default configurations in Minikube by running minikube start again.

Validation

After letting the request pass through all three stages, the API server then validates the object—that is, it checks whether the object specification, which is carried in JSON format in the response body, meets the required format and standard.

After successful validation, the API server stores the object in the etcd datastore and returns a response to the client. After that, as you learned in Chapter 2, An Overview of Kubernetes, other components, such as the scheduler and the controller manager, take over to find a suitable node and actually implement the object on your cluster.