CKA Simulator Kubernetes 1.31

https://killer.sh

Pre Setup

Once you've gained access to your terminal it might be wise to spend ~1 minute to setup your environment. You could set these:

Vim

The following settings will already be configured in your real exam environment in ~/.vimrc. But it can never hurt to be able to type these down:

More setup suggestions are in the tips section.

 

 

Question 1 | Contexts

 

You have access to multiple clusters from your main terminal through kubectl contexts. Write all those context names into /opt/course/1/contexts.

Next write a command to display the current context into /opt/course/1/context_default_kubectl.sh, the command should use kubectl.

Finally write a second command doing the same thing into /opt/course/1/context_default_no_kubectl.sh, but without the use of kubectl.

 

Answer:

Maybe the fastest way is just to run:

Or using jsonpath:

The content should then look like:

Next create the first command:

And the second one:

In the real exam you might need to filter and find information from bigger lists of resources, hence knowing a little jsonpath and simple bash filtering will be helpful.

The second command could also be improved to:

 

 

Question 2 | Schedule Pod on Controlplane Nodes

 

Use context: kubectl config use-context k8s-c1-H

 

Create a single Pod of image httpd:2.4.41-alpine in Namespace default. The Pod should be named pod1 and the container should be named pod1-container. This Pod should only be scheduled on controlplane nodes. Do not add new labels to any nodes.

 

Answer:

First we find the controlplane node(s) and their taints:

 

Next we create the Pod template:

Perform the necessary changes manually. Use the Kubernetes docs and search for example for tolerations and nodeSelector to find examples:

Important here to add the toleration for running on controlplane nodes, but also the nodeSelector to make sure it only runs on controlplane nodes. If we only specify a toleration the Pod can be scheduled on controlplane or worker nodes.

Now we create it:

Let's check if the pod is scheduled:

 

 

Question 3 | Scale down StatefulSet

 

Use context: kubectl config use-context k8s-c1-H

 

There are two Pods named o3db-* in Namespace project-c13. C13 Management asked you to scale the Pods down to one replica to save resources.

 

Answer:

If we check the Pods we see two replicas:

From their name it looks like these are managed by a StatefulSet. But if we're not sure we could also check for the most common resources which manage Pods:

Confirmed, we have to work with a StatefulSet. To find this out we could also look at the Pod labels:

To fulfil the task we simply run:

C13 Management is happy again.

 

 

Question 4 | Pod Ready if Service is reachable

 

Use context: kubectl config use-context k8s-c1-H

 

Do the following in Namespace default. Create a single Pod named ready-if-service-ready of image nginx:1.16.1-alpine. Configure a LivenessProbe which simply executes command true. Also configure a ReadinessProbe which does check if the url http://service-am-i-ready:80 is reachable, you can use wget -T2 -O- http://service-am-i-ready:80 for this. Start the Pod and confirm it isn't ready because of the ReadinessProbe.

Create a second Pod named am-i-ready of image nginx:1.16.1-alpine with label id: cross-server-ready. The already existing Service service-am-i-ready should now have that second Pod as endpoint.

Now the first Pod should be in ready state, confirm that.

 

Answer:

It's a bit of an anti-pattern for one Pod to check another Pod for being ready using probes, hence the normally available readinessProbe.httpGet doesn't work for absolute remote urls. Still the workaround requested in this task should show how probes and Pod<->Service communication works.

First we create the first Pod:

Next perform the necessary additions manually:

Then create the Pod:

And confirm it's in a non-ready state:

We can also check the reason for this using describe:

Now we create the second Pod:

The already existing Service service-am-i-ready should now have an Endpoint:

Which will result in our first Pod being ready, just give it a minute for the Readiness probe to check again:

Look at these Pods coworking together!

 

 

Question 5 | Kubectl sorting

 

Use context: kubectl config use-context k8s-c1-H

 

There are various Pods in all namespaces. Write a command into /opt/course/5/find_pods.sh which lists all Pods sorted by their AGE (metadata.creationTimestamp).

Write a second command into /opt/course/5/find_pods_uid.sh which lists all Pods sorted by field metadata.uid. Use kubectl sorting for both commands.

 

Answer:

A good resources here (and for many other things) is the kubectl-cheat-sheet. You can reach it fast when searching for "cheat sheet" in the Kubernetes docs.

And to execute:

For the second command:

And to execute:

 

 

Question 6 | Storage, PV, PVC, Pod volume

 

Use context: kubectl config use-context k8s-c1-H

 

Create a new PersistentVolume named safari-pv. It should have a capacity of 2Gi, accessMode ReadWriteOnce, hostPath /Volumes/Data and no storageClassName defined.

Next create a new PersistentVolumeClaim in Namespace project-tiger named safari-pvc . It should request 2Gi storage, accessMode ReadWriteOnce and should not define a storageClassName. The PVC should bound to the PV correctly.

Finally create a new Deployment safari in Namespace project-tiger which mounts that volume at /tmp/safari-data. The Pods of that Deployment should be of image httpd:2.4.41-alpine.

 

Answer

Find an example from https://kubernetes.io/docs and alter it:

Then create it:

Next the PersistentVolumeClaim:

Find an example from https://kubernetes.io/docs and alter it:

Then create:

And check that both have the status Bound:

Next we create a Deployment and mount that volume:

Alter the yaml to mount the volume:

We can confirm it's mounting correctly:

 

 

Question 7 | Node and Pod Resource Usage

 

Use context: kubectl config use-context k8s-c1-H

 

The metrics-server has been installed in the cluster. Your college would like to know the kubectl commands to:

  1. show Nodes resource usage

  2. show Pods and their containers resource usage

Please write the commands into /opt/course/7/node.sh and /opt/course/7/pod.sh.

 

Answer:

The command we need to use here is top:

We see that the metrics server provides information about resource usage:

We create the first file:

For the second file we might need to check the docs again:

With this we can finish this task:

 

 

Question 8 | Get Controlplane Information

 

Use context: kubectl config use-context k8s-c1-H

 

Ssh into the controlplane node with ssh cluster1-controlplane1. Check how the controlplane components kubelet, kube-apiserver, kube-scheduler, kube-controller-manager and etcd are started/installed on the controlplane node.

Also find out the name of the DNS application and how it's started/installed in the cluster.

Write your findings into file /opt/course/8/controlplane-components.txt. The file should be structured like:

Choices of [TYPE] are: not-installed, process, static-pod, pod

 

Answer:

We could start by finding processes of the requested components, especially the kubelet at first:

We can see which components are controlled via systemd looking at /usr/lib/systemd directory:

This shows kubelet is controlled via systemd, but no other service named kube nor etcd. It seems that this cluster has been setup using kubeadm, so we check in the default manifests directory:

(The kubelet could also have a different manifests directory specified via parameter --pod-manifest-path in it's systemd startup config)

This means the main 4 controlplane services are setup as static Pods. Actually, let's check all Pods running on in the kube-system Namespace on the controlplane node:

There we see the 4 static pods, with -cluster1-controlplane1 as suffix.

We also see that the dns application seems to be coredns, but how is it controlled?

Seems like coredns is controlled via a Deployment. We combine our findings in the requested file:

You should be comfortable investigating a running cluster, know different methods on how a cluster and its services can be setup and be able to troubleshoot and find error sources.

 

 

Question 9 | Kill Scheduler, Manual Scheduling

 

Use context: kubectl config use-context k8s-c2-AC

 

Ssh into the controlplane node with ssh cluster2-controlplane1. Temporarily stop the kube-scheduler, this means in a way that you can start it again afterwards.

Create a single Pod named manual-schedule of image httpd:2.4-alpine, confirm it's created but not scheduled on any node.

Now you're the scheduler and have all its power, manually schedule that Pod on node cluster2-controlplane1. Make sure it's running.

Start the kube-scheduler again and confirm it's running correctly by creating a second Pod named manual-schedule2 of image httpd:2.4-alpine and check if it's running on cluster2-node1.

 

Answer:
Stop the Scheduler

First we find the controlplane node:

Then we connect and check if the scheduler is running:

Kill the Scheduler (temporarily):

And it should be stopped:

 

Create a Pod

Now we create the Pod:

And confirm it has no node assigned:

 

Manually schedule the Pod

Let's play the scheduler now:

The only thing a scheduler does, is that it sets the nodeName for a Pod declaration. How it finds the correct node to schedule on, that's a very much complicated matter and takes many variables into account.

As we cannot kubectl apply or kubectl edit , in this case we need to delete and create or replace:

How does it look?

It looks like our Pod is running on the controlplane now as requested, although no tolerations were specified. Only the scheduler takes tains/tolerations/affinity into account when finding the correct node name. That's why it's still possible to assign Pods manually directly to a controlplane node and skip the scheduler.

 

Start the scheduler again

Checks it's running:

Schedule a second test Pod:

Back to normal.

 

 

Question 10 | RBAC ServiceAccount Role RoleBinding

 

Use context: kubectl config use-context k8s-c1-H

 

Create a new ServiceAccount processor in Namespace project-hamster. Create a Role and RoleBinding, both named processor as well. These should allow the new SA to only create Secrets and ConfigMaps in that Namespace.

 

Answer:
Let's talk a little about RBAC resources

A ClusterRole|Role defines a set of permissions and where it is available, in the whole cluster or just a single Namespace.

A ClusterRoleBinding|RoleBinding connects a set of permissions with an account and defines where it is applied, in the whole cluster or just a single Namespace.

Because of this there are 4 different RBAC combinations and 3 valid ones:

  1. Role + RoleBinding (available in single Namespace, applied in single Namespace)

  2. ClusterRole + ClusterRoleBinding (available cluster-wide, applied cluster-wide)

  3. ClusterRole + RoleBinding (available cluster-wide, applied in single Namespace)

  4. Role + ClusterRoleBinding (NOT POSSIBLE: available in single Namespace, applied cluster-wide)

To the solution

We first create the ServiceAccount:

Then for the Role:

So we execute:

Which will create a Role like:

Now we bind the Role to the ServiceAccount:

So we create it:

This will create a RoleBinding like:

To test our RBAC setup we can use kubectl auth can-i:

Like this:

Done.

 

 

Question 11 | DaemonSet on all Nodes

 

Use context: kubectl config use-context k8s-c1-H

 

Use Namespace project-tiger for the following. Create a DaemonSet named ds-important with image httpd:2.4-alpine and labels id=ds-important and uuid=18426a0b-5f59-4e10-923f-c0e078e82462. The Pods it creates should request 10 millicore cpu and 10 mebibyte memory. The Pods of that DaemonSet should run on all nodes, also controlplanes.

 

Answer:

As of now we aren't able to create a DaemonSet directly using kubectl, so we create a Deployment and just change it up:

(Sure you could also search for a DaemonSet example yaml in the Kubernetes docs and alter it.)

 

Then we adjust the yaml to:

It was requested that the DaemonSet runs on all nodes, so we need to specify the toleration for this.

Let's confirm:

 

 

Question 12 | Deployment on all Nodes

 

Use context: kubectl config use-context k8s-c1-H

 

Implement the following in Namespace project-tiger:

 

ℹ️ Because there are two worker nodes and the Deployment has three replicas the result should be that the third Pod won't be scheduled. In a way it simulates the behaviour of a DaemonSet, but using a Deployment and a fixed number of replicas.

 

 

Answer:

There are two possible ways, one using podAntiAffinity and one using topologySpreadConstraint.

 

PodAntiAffinity

The idea here is that we create a "Inter-pod anti-affinity" which allows us to say a Pod should only be scheduled on a node where another Pod of a specific label (here the same label) is not already running.

Let's begin by creating the Deployment template:

Then change the yaml to:

Specify a topologyKey, which is a pre-populated Kubernetes label, you can find this by describing a node.

 

TopologySpreadConstraints

We can achieve the same with topologySpreadConstraints. Best to try out and play with both.

 

Apply and Run

Let's run it:

Then we check the Deployment status where it shows 2/3 ready count:

And running the following we see one Pod on each worker node and one not scheduled.

If we kubectl describe the Pod deploy-important-58db9db6fc-lnxdb it will show us the reason for not scheduling is our implemented podAntiAffinity ruling:

Or our topologySpreadConstraints:

 

 

Question 13 | Multi Containers and Pod shared Volume

 

Use context: kubectl config use-context k8s-c1-H

 

Create a Pod named multi-container-playground in Namespace default with three containers, named c1, c2 and c3. There should be a volume attached to that Pod and mounted into every container, but the volume shouldn't be persisted or shared with other Pods.

Container c1 should be of image nginx:1.17.6-alpine and have the name of the node where its Pod is running available as environment variable MY_NODE_NAME.

Container c2 should be of image busybox:1.31.1 and write the output of the date command every second in the shared volume into file date.log. You can use while true; do date >> /your/vol/path/date.log; sleep 1; done for this.

Container c3 should be of image busybox:1.31.1 and constantly send the content of file date.log from the shared volume to stdout. You can use tail -f /your/vol/path/date.log for this.

Check the logs of container c3 to confirm correct setup.

 

Answer:

First we create the Pod template:

And add the other containers and the commands they should execute:

Oh boy, lot's of requested things. We check if everything is good with the Pod:

Good, then we check if container c1 has the requested node name as env variable:

And finally we check the logging:

 

 

Question 14 | Find out Cluster Information

 

Use context: kubectl config use-context k8s-c1-H

 

You're ask to find out following information about the cluster k8s-c1-H:

  1. How many controlplane nodes are available?

  2. How many worker nodes are available?

  3. What is the Service CIDR?

  4. Which Networking (or CNI Plugin) is configured and where is its config file?

  5. Which suffix will static pods have that run on cluster1-node1?

Write your answers into file /opt/course/14/cluster-info, structured like this:

 

Answer:
How many controlplane and worker nodes are available?

We see one controlplane and two workers.

 

What is the Service CIDR?

 

Which Networking (or CNI Plugin) is configured and where is its config file?

By default the kubelet looks into /etc/cni/net.d to discover the CNI plugins. This will be the same on every controlplane and worker nodes.

 

Which suffix will static pods have that run on cluster1-node1?

The suffix is the node hostname with a leading hyphen. It used to be -static in earlier Kubernetes versions.

 

Result

The resulting /opt/course/14/cluster-info could look like:

 

 

Question 15 | Cluster Event Logging

 

Use context: kubectl config use-context k8s-c2-AC

 

Write a command into /opt/course/15/cluster_events.sh which shows the latest events in the whole cluster, ordered by time (metadata.creationTimestamp). Use kubectl for it.

Now delete the kube-proxy Pod running on node cluster2-node1 and write the events this caused into /opt/course/15/pod_kill.log.

Finally kill the containerd container of the kube-proxy Pod on node cluster2-node1 and write the events into /opt/course/15/container_kill.log.

Do you notice differences in the events both actions caused?

 

Answer:

Now we delete the kube-proxy Pod:

Now check the events:

Write the events the killing caused into /opt/course/15/pod_kill.log:

Finally we will try to provoke events by killing the container belonging to the container of the kube-proxy Pod:

We killed the main container (1e020b43c4423), but also noticed that a new container (0ae4245707910) was directly created. Thanks Kubernetes!

Now we see if this caused events again and we write those into the second file:

Comparing the events we see that when we deleted the whole Pod there were more things to be done, hence more events. For example was the DaemonSet in the game to re-create the missing Pod. Where when we manually killed the main container of the Pod, the Pod would still exist but only its container needed to be re-created, hence less events.

 

 

Question 16 | Namespaces and Api Resources

 

Use context: kubectl config use-context k8s-c1-H

 

Write the names of all namespaced Kubernetes resources (like Pod, Secret, ConfigMap...) into /opt/course/16/resources.txt.

Find the project-* Namespace with the highest number of Roles defined in it and write its name and amount of Roles into /opt/course/16/crowded-namespace.txt.

 

Answer:
Namespace and Namespaces Resources

Now we can get a list of all resources like:

Which results in the file:

 

Namespace with most Roles

Finally we write the name and amount into the file:

 

 

Question 17 | Find Container of Pod and check info

 

Use context: kubectl config use-context k8s-c1-H

 

In Namespace project-tiger create a Pod named tigers-reunite of image httpd:2.4.41-alpine with labels pod=container and container=pod. Find out on which node the Pod is scheduled. Ssh into that node and find the containerd container belonging to that Pod.

Using command crictl:

  1. Write the ID of the container and the info.runtimeType into /opt/course/17/pod-container.txt

  2. Write the logs of the container into /opt/course/17/pod-container.log

 

Answer:

First we create the Pod:

Next we find out the node it's scheduled on:

Then we ssh into that node and and check the container info:

Then we fill the requested file (on the main terminal):

Finally we write the container logs in the second file:

The &> in above's command redirects both the standard output and standard error.

You could also simply run crictl logs on the node and copy the content manually, if it's not a lot. The file should look like:

 

 

Question 18 | Fix Kubelet

 

Use context: kubectl config use-context k8s-c3-CCC

 

There seems to be an issue with the kubelet not running on cluster3-node1. Fix it and confirm that cluster has node cluster3-node1 available in Ready state afterwards. You should be able to schedule a Pod on cluster3-node1 afterwards.

Write the reason of the issue into /opt/course/18/reason.txt.

 

Answer:

The procedure on tasks like these should be to check if the kubelet is running, if not start it, then check its logs and correct errors if there are some.

Always helpful to check if other clusters already have some of the components defined and running, so you can copy and use existing config files. Though in this case it might not need to be necessary.

Check node status:

First we check if the kubelet is running:

Nope, so we check if it's configured using systemd as service:

Yes, it's configured as a service with config at /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf, but we see it's inactive. Let's try to start it:

We see it's trying to execute /usr/local/bin/kubelet with some parameters defined in its service config file. A good way to find errors and get more logs is to run the command manually (usually also with its parameters).

Another way would be to see the extended logging of a service like using journalctl -u kubelet.

Well, there we have it, wrong path specified. Correct the path in file /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf and run:

Also the node should be available for the api server, give it a bit of time though:

Finally we write the reason into the file:

 

 

Question 19 | Create Secret and mount into Pod

 

ℹ️ This task can only be solved if questions 18 or 20 have been successfully implemented and the k8s-c3-CCC cluster has a functioning worker node.

 

Use context: kubectl config use-context k8s-c3-CCC

 

Do the following in a new Namespace secret. Create a Pod named secret-pod of image busybox:1.31.1 which should keep running for some time.

There is an existing Secret located at /opt/course/19/secret1.yaml, create it in the Namespace secret and mount it readonly into the Pod at /tmp/secret1.

Create a new Secret in Namespace secret called secret2 which should contain user=user1 and pass=1234. These entries should be available inside the Pod's container as environment variables APP_USER and APP_PASS.

Confirm everything is working.

 

Answer

First we create the Namespace and the requested Secrets in it:

We need to adjust the Namespace for that Secret:

Next we create the second Secret:

Now we create the Pod template:

Then make the necessary changes:

It might not be necessary in current K8s versions to specify the readOnly: true because it's the default setting anyways.

And execute:

Finally we check if all is correct:

All is good.

 

 

Question 20 | Update Kubernetes Version and join cluster

 

Use context: kubectl config use-context k8s-c3-CCC

 

Your coworker said node cluster3-node2 is running an older Kubernetes version and is not even part of the cluster. Update Kubernetes on that node to the exact version that's running on cluster3-controlplane1. Then add this node to the cluster. Use kubeadm for this.

 

Answer:
Upgrade Kubernetes to cluster3-controlplane1 version

Search in the docs for kubeadm upgrade: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade

Controlplane node seems to be running Kubernetes 1.31.1. Node cluster3-node1 might not yet be Ready or part of cluster depending on the completion of a previous task. But this task is about node cluster3-node2 so we can continue anyways:

Above we can see that kubeadm is already installed in the wanted version, so we don't need to install it. Hence we can run:

This is usually the proper command to upgrade a node. But this error means that this node was never even initialised, so nothing to update here. This will be done later using kubeadm join. For now we can continue with kubelet and kubectl:

Now we're up to date with kubeadm, kubectl and kubelet. Restart the kubelet:

These errors occur because we still need to run kubeadm join to join the node into the cluster. Let's do this in the next step.

 

Add cluster3-node2 to cluster

First we log into the controlplane1 and generate a new TLS bootstrap token, also printing out the join command:

We see the expiration of 23h for our token, we could adjust this by passing the ttl argument.

Next we connect again to cluster3-node2 and simply execute the join command:

If you have troubles with kubeadm join you might need to run kubeadm reset before.

This looks great though for us. Finally we head back to the main terminal and check the node status:

Give it a bit of time till the node is ready.

We see cluster3-node2 is now available and up to date.

 

 

Question 21 | Create a Static Pod and Service

 

Use context: kubectl config use-context k8s-c3-CCC

 

Create a Static Pod named my-static-pod in Namespace default on cluster3-controlplane1. It should be of image nginx:1.16-alpine and have resource requests for 10m CPU and 20Mi memory.

Then create a NodePort Service named static-pod-service which exposes that static Pod on port 80 and check if it has Endpoints and if it's reachable through the cluster3-controlplane1 internal IP address. You can connect to the internal node IPs from your main terminal.

 

Answer:

Then edit the my-static-pod.yaml to add the requested resource requests:

And make sure it's running:

Now we expose that static Pod:

This would generate a Service like:

Then run and test:

Looking good.

 

 

Question 22 | Check how long certificates are valid

 

Use context: kubectl config use-context k8s-c2-AC

 

Check how long the kube-apiserver server certificate is valid on cluster2-controlplane1. Do this with openssl or cfssl. Write the expiration date into /opt/course/22/expiration.

Also run the correct kubeadm command to list the expiration dates and confirm both methods show the same date.

Write the correct kubeadm command that would renew the apiserver server certificate into /opt/course/22/kubeadm-renew-certs.sh.

 

Answer:

First let's find that certificate:

Next we use openssl to find out the expiration date:

There we have it, so we write it in the required location on our main terminal:

And we use the feature from kubeadm to get the expiration too:

Looking good. And finally we write the command that would renew the kube-apiserver certificate into the requested location:

 

 

Question 23 | Kubelet client/server cert info

 

Use context: kubectl config use-context k8s-c2-AC

 

Node cluster2-node1 has been added to the cluster using kubeadm and TLS bootstrapping.

Find the "Issuer" and "Extended Key Usage" values of the cluster2-node1:

  1. kubelet client certificate, the one used for outgoing connections to the kube-apiserver.

  2. kubelet server certificate, the one used for incoming connections from the kube-apiserver.

Write the information into file /opt/course/23/certificate-info.txt.

Compare the "Issuer" and "Extended Key Usage" fields of both certificates and make sense of these.

 

Answer:

First we check the kubelet client certificate:

Next we check the kubelet server certificate:

We see that the server certificate was generated on the worker node itself and the client certificate was issued by the Kubernetes api. The "Extended Key Usage" also shows if it's for client or server authentication.

 

 

Question 24 | NetworkPolicy

 

Use context: kubectl config use-context k8s-c1-H

 

There was a security incident where an intruder was able to access the whole cluster from a single hacked backend Pod.

To prevent this create a NetworkPolicy called np-backend in Namespace project-snake. It should allow the backend-* Pods only to:

Use the app label of Pods in your policy.

After implementation, connections from backend-* Pods to vault-* Pods on port 3333 should for example no longer work.

 

Answer:

First we look at the existing Pods and their labels:

We test the current connection situation and see nothing is restricted:

Now we create the NP by copying and changing an example from the K8s Docs:

The NP above has two rules with two conditions each, it can be read as:

 

Wrong example

Now let's shortly look at a wrong example:

The NP above has one rule with two conditions and two condition-entries each, it can be read as:

Using this NP it would still be possible for backend-* Pods to connect to db2-* Pods on port 1111 for example which should be forbidden.

 

Create NetworkPolicy

We create the correct NP:

And test again:

Also helpful to use kubectl describe on the NP to see how k8s has interpreted the policy.

Great, looking more secure. Task done.

 

 

Question 25 | Etcd Snapshot Save and Restore

 

Use context: kubectl config use-context k8s-c3-CCC

 

Make a backup of etcd running on cluster3-controlplane1 and save it on the controlplane node at /tmp/etcd-backup.db.

Then create any kind of Pod in the cluster.

Finally restore the backup, confirm the cluster is still working and that the created Pod is no longer with us.

 

Answer:
Etcd Backup

First we log into the controlplane and try to create a snapshop of etcd:

But it fails or hangs because we need to authenticate ourselves. For the necessary information we can check the etc manifest:

We only check the etcd.yaml for necessary information we don't change it.

But we also know that the api-server is connecting to etcd, so we can check how its manifest is configured:

We use the authentication information and pass it to etcdctl:

 

ℹ️ Don't use snapshot status because it can alter the snapshot file and render it invalid.

 

Etcd restore

Now create a Pod in the cluster and wait for it to be running:

 

ℹ️ If you didn't solve questions 18 or 20 and cluster3 doesn't have a ready worker node then the created pod might stay in a Pending state. This is still ok for this task.

 

Next we stop all controlplane components:

Now we restore the snapshot into a specific directory:

We could specify another host to make the backup from by using etcdctl --endpoints http://IP, but here we just use the default value which is: http://127.0.0.1:2379,http://127.0.0.1:4001.

The restored files are located at the new folder /var/lib/etcd-backup, now we have to tell etcd to use that directory:

Now we move all controlplane yaml again into the manifest directory. Give it some time (up to several minutes) for etcd to restart and for the api-server to be reachable again:

Then we check again for the Pod:

Awesome, backup and restore worked as our pod is gone.

 

 

Extra Question 1 | Find Pods first to be terminated

Use context: kubectl config use-context k8s-c1-H

 

Check all available Pods in the Namespace project-c13 and find the names of those that would probably be terminated first if the nodes run out of resources (cpu or memory) to schedule all Pods. Write the Pod names into /opt/course/e1/pods-not-stable.txt.

 

Answer:

When available cpu or memory resources on the nodes reach their limit, Kubernetes will look for Pods that are using more resources than they requested. These will be the first candidates for termination. If some Pods containers have no resource requests/limits set, then by default those are considered to use more than requested.

Kubernetes assigns Quality of Service classes to Pods based on the defined resources and limits, read more here: https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod

Hence we should look for Pods without resource requests defined, we can do this with a manual approach:

Or we do:

We see that the Pods of Deployment c13-3cc-runner-heavy don't have any resources requests specified. Hence our answer would be:

To automate this process you could use jsonpath like this:

This lists all Pod names and their requests/limits, hence we see the three Pods without those defined.

Or we look for the Quality of Service classes:

Here we see three with BestEffort, which Pods get that don't have any memory or cpu limits or requests defined.

A good practice is to always set resource requests and limits. If you don't know the values your containers should have you can find this out using metric tools like Prometheus. You can also use kubectl top pod or even kubectl exec into the container and use top and similar tools.

 

 

Extra Question 2 | Curl Manually Contact API

Use context: kubectl config use-context k8s-c1-H

 

There is an existing ServiceAccount secret-reader in Namespace project-hamster. Create a Pod of image curlimages/curl:7.65.3 named tmp-api-contact which uses this ServiceAccount. Make sure the container keeps running.

Exec into the Pod and use curl to access the Kubernetes Api of that cluster manually, listing all available secrets. You can ignore insecure https connection. Write the command(s) for this into file /opt/course/e4/list-secrets.sh.

 

Answer:

https://kubernetes.io/docs/tasks/run-application/access-api-from-pod

It's important to understand how the Kubernetes API works. For this it helps connecting to the api manually, for example using curl. You can find information fast by search in the Kubernetes docs for "curl api" for example.

First we create our Pod:

Add the service account name and Namespace:

Then run and exec into:

Once on the container we can try to connect to the api using curl, the api is usually available via the Service named kubernetes in Namespace default (You should know how dns resolution works across Namespaces.). Else we can find the endpoint IP via environment variables running env.

So now we can do:

The last command shows 403 forbidden, this is because we are not passing any authorisation information with us. The Kubernetes Api Server thinks we are connecting as system:anonymous. We want to change this and connect using the Pods ServiceAccount named secret-reader.

We find the the token in the mounted folder at /var/run/secrets/kubernetes.io/serviceaccount, so we do:

Now we're able to list all Secrets, registering as the ServiceAccount secret-reader under which our Pod is running.

To use encrypted https connection we can run:

For troubleshooting we could also check if the ServiceAccount is actually able to list Secrets using:

Finally write the commands into the requested location: