Exploring Kubernetes from within Kubernetes... with Python - PART 1

Part 2 is available here

So my blog posts will obviously not be daily, as I start to delve into deeper more technical topics.

Today I'm looking at using Python and integrating with the Kubernetes API, in a two part post. The first part will be just introducing the basics - kind of setting the scene. Part two will be the actual Python solution.

To make this practical, I choose a topic for which there may be a number of better alternatives, but yet, it's good to gain understanding on some of these mechanisms.

I want to know one simple thing: Do I have enough physical resources (CPU and RAM) to handle all my committed deployments?

For this post, I also assume the reader to be familiar with some basics of Kubernetes. Feel free to refer to this blog entry that shows how I implement my lab environments and on which commands in this blog post will also be based.

Context

Certain services, like AWS EKS provide out-of-the-box solutions to automatically scale the number of nodes. The scaling metrics can be tweaked, but for the most part it all just happens automatically.

However... Sometimes you run Kubernetes on physical tin or in a cloud environments with some financial constraints, and you have to ensure that the deployed applications do not consume more than the available resources. This can be tricky, but one of the fundamental problems to solve upfront is to know how much capacity is available and then measure what is actually being committed. Actual usage may be lower than the total commitment, however, but for now we focus on what is committed as this serves as some upper boundary or guarantee that our users expect to be honored if their applications run on our cluster.

Background on resources

I am going to assume anyone reading this will at least understand the concepts around CPU and RAM - particularly around the physical constraints and at least some basic understanding of how application use CPU and RAM resources.

Over and above this basic understanding There are also important principles to take note of:

Principle Description
Resources are finite As mentioned, in our context of physical hardware, both CPU and RAM resources are finite - we cannot use more than what is available
Swap is not an option There is much debate around using swap within the Kubernetes ecosystem, but for the purpose of this blog we assume no swap mechanism is being used (standard for the bulk of use cases)
Deployments must impose limits This is a best practice principle. Refer to the Kubernetes documentation for the gory detail.

If we look at our Python Demonstration/Reference Application, the following section if the Kubernetes manifest is important:

        resources:
          requests:
            memory: "128Mi"
            cpu: "250m"
          limits:
            memory: "256Mi"
            cpu: "500m"

The specifics of what this means and technically how this is implemented, can be read in this section of the Kubernetes documentation.

What is important is that although you could get away with over-subscribing the CPU (within reasonable limits), memory (RAM) is far less forgiving. Without any swap available, should all your pods start to consume too much available RAM, Kubernetes will start to evict pods. The technical detail of this subject and how it is implemented is vast, but feel free to refer to the Kubernetes documentation dealing with Scheduling, Preemption and Eviction.

Setting up our playground

I use a local lab environment to run all the tests. Please have a look at this blog entry for details.

Note: Since my last Kubernetes technical post I have updated the demo application to include a memory monster :-D

The sample application repository is still available on GitHub. An image of the latest build is on Docker Hub under the 0.0.2 tag

My lab environment has the following IP addresses:

To deploy the demo app, create a test namespace and deploy the manifest:

$ kubectl create namespace test

$ kubectl apply -f https://raw.githubusercontent.com/nicc777/pyk8sdemo-app/main/kubernetes_manifests/troubleshooting-app.yaml  -n test

$ kubectl get all -n NAME                                             READY   STATUS    RESTARTS   AGE
pod/svclb-flask-demo-app-service-xhg4t           1/1     Running   0          16s
pod/svclb-flask-demo-app-service-tvzvl           1/1     Running   0          16s
pod/svclb-flask-demo-app-service-fz8mm           1/1     Running   0          16s
pod/flask-demo-app-deployment-7b967d6bc8-xnhrr   1/1     Running   0          16s
pod/flask-demo-app-deployment-7b967d6bc8-c6427   1/1     Running   0          16s

NAME                             TYPE           CLUSTER-IP      EXTERNAL-IP                          PORT(S)          AGE
service/flask-demo-app-service   LoadBalancer   10.43.164.224   10.0.50.170,10.0.50.222,10.0.50.72   8880:32670/TCP   16s

NAME                                          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/svclb-flask-demo-app-service   3         3         3       3            3           <none>          16s

NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/flask-demo-app-deployment   2/2     2            2           16s

NAME                                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/flask-demo-app-deployment-7b967d6bc8   2         2         2       16s

Scaling up

To increase the number of pods we have running, issue the following command:

kubectl scale deployment.apps/flask-demo-app-deployment --replicas=6 -n test

You should now see a total of 6 pods running

Load Testing Basics

For testing, we are going to create a little locust test suite.

In the python test application project, there is a directory called test/locust which contains our locust tests. For this test I optted not to use a load balancer, but rather reference the node endpoints from locust. This is perhaps not very portable, but it is sufficient for our tests in this scenario.

In order to use these test files, export the Node IP addresses as HOST1, HOST2 and HOST3, for example:

export HOST1=10.0.50.170
export HOST2=10.0.50.222
export HOST3=10.0.50.72

You can now run the locust test with a moderate number of users (for example 12) to see that each of the nodes are actually used:

locust -f basic_home_page_test.py -u 12 -r 3 --autostart

When you visit http://localhost:8089/ you should see the test already running.

To watch what the memory performance of the cluster is like, we can use the command:

kubectl describe node/node1 | grep memory | tail -1 && kubectl describe node/node2 | grep memory | tail -1 && kubectl describe node/node3 | grep memory | tail -1

The output should look something like this:

  memory             396Mi (4%)  682Mi (8%)
  memory             256Mi (3%)  512Mi (6%)
  memory             256Mi (3%)  512Mi (6%)

This shows the requested memory and memory limits as per our deployments. We are still well below the physical limits of our system.

Similarly, we can easily get the actual memory stats directly from the nodes with the following command:

multipass exec node1 -- free -m | grep "Mem:" && multipass exec node2 -- free -m | grep "Mem:" && multipass exec node3 -- free -m | grep "Mem:"

And the output should now look something like this:

Mem:           7957         917        4178           1        2861        6742
Mem:           7957         506        5055           1        2395        7155
Mem:           7957         477        5222           1        2256        7188

The 4th column and last column are of interest: free and available RAM.

To watch this number over time, use the following command:

watch -n3 "multipass exec node1 -- free -m | grep \"Mem:\" && multipass exec node2 -- free -m | grep \"Mem:\" && multipass exec node3 -- free -m | grep \"Mem:\""

Now let's run a basic memory intensive test:

locust -f basic_hunger_test.py -u 12 -r 3 --autostart

After a couple of minutes, my physical memory numbers looks like this:

Mem:           7957        1091        3999           1        2866        6569
Mem:           7957         659        4897           1        2400        7001
Mem:           7957         637        5058           1        2262        7027

We can see that more memory is being consumed. However, the maximum commitment is still unchanged:

  memory             396Mi (4%)  682Mi (8%)
  memory             256Mi (3%)  512Mi (6%)
  memory             256Mi (3%)  512Mi (6%)

After some time, I stopped the test with the following statistics:

 Name                                                                              # reqs      # fails  |     Avg     Min     Max  Median  |   req/s failures/s
----------------------------------------------------------------------------------------------------------------------------------------------------------------
 GET /                                                                                714     0(0.00%)  |       3       2      10       3  |    1.93    0.00
 GET /hungry                                                                          746     0(0.00%)  |      50      21    1007      33  |    2.02    0.00
----------------------------------------------------------------------------------------------------------------------------------------------------------------
 Aggregated                                                                          1460     0(0.00%)  |      27       2    1007      23  |    3.95    0.00

Response time percentiles (approximated)
 Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs
--------|--------------------------------------------------------------------------------|---------|------|------|------|------|------|------|------|------|------|------|------|
 GET      /                                                                                       3      3      3      3      4      4      7      8     11     11     11    714
 GET      /hungry                                                                                33     36     39     43     86    140    320    380   1000   1000   1000    746
--------|--------------------------------------------------------------------------------|---------|------|------|------|------|------|------|------|------|------|------|------|
 None     Aggregated                                                                             23     29     33     34     43     90    160    320    800   1000   1000   1460

Concluding part 1

In this first part I looked at some basics of Kubernetes resource management and how we can measure and test it using existing tools.

Part two, which will hopefully follow in a week or so, will look at how we create a Python based pod that monitors these statistics for us in a more summarized way across all our pods and namespaces.

Tags

kubernetes, python, resources, api, locust