Best Kubernetes Admin Job Interview Questions | SPOTO

A Custom Resource Definition (CRD) allows you to extend the Kubernetes API with your own custom resources. CRDs enable users to create and manage custom objects that represent application-specific configurations or other resources not included in the standard Kubernetes distribution. Once defined, these custom resources can be managed using the same Kubernetes tools and API.

An Init Container is a special type of container that runs first, doing setup tasks like loading configurations or checking if services are ready. It completes its tasks before the main application container starts running.

Let's you inject configuration data into the application. ConfigMap in Kubernetes is a flexible and scalable way to manage configuration data for your applications, allowing you to separate configuration from application code, make dynamic updates to configuration, and easily consume configuration data within your Kubernetes cluster. apiVersion: v1 kind: ConfigMap metadata: name: example-configmap data: database.properties: | database.url=jdbc:mysql://localhost:3306/mydb database.username=admin database.password=secretpassword app.properties: | app.debug=true app.max_connections=100 #Create a ConfigMap from Literal: kubectl create configmap --from-literal== --from-literal== #Create a ConfigMap from File: kubectl create configmap --from-file= #Create a ConfigMap from Directory: kubectl create configmap --from-file= #View ConfigMaps: kubectl get configmaps #View Details of a ConfigMap: kubectl describe configmap #Edit a ConfigMap: kubectl edit configmap #Delete a ConfigMap: kubectl delete configmap #Apply Changes to a ConfigMap from File: kubectl apply -f #Get ConfigMap Data: kubectl get configmap -o yaml #Use ConfigMap in a Pod: volumes: - name: configMap: name: #As Environment Variables: envFrom: - configMapRef: name: #As a file spec: containers: - name: mycontainer image: myimage volumeMounts: - name: config-volume mountPath: /etc/config volumes: - name: config-volume configMap: name: example-configmap

Kubernetes cluster backup and restore involves backing up and restoring the configuration and data stored in the Kubernetes objects, such as pods, services, and deployments.

Lets you link your cluster into the cloud provider's API, separating out the components that interact with that cloud platform from components that just interact with your cluster.

A container runtime is responsible for starting and stopping containers on a node. Examples include Docker, containerd, and CRI-O.

You'd check Pod labels, Service selectors, endpoints, and network policies.

Kubernetes facilitates gradual, controlled updates of applications, reducing the risk of downtime or failure during deployment.

Annotations in Kubernetes are key-value pairs used to attach arbitrary non-identifying metadata to objects. Unlike labels, annotations are not used to identify and select objects. Instead, they provide a mechanism for storing additional information that can be used by various tools and libraries. Annotations can store: - Build information: Such as build numbers or source code versions. - Configuration details: That might affect how an application behaves. - Arbitrary data: Required by external systems or tools. Example usage in a Pod configuration: metadata: annotations: buildVersion: "1.0.3" checksum/config: "a1234"

etcd is a distributed, consistent database based on the Raft consensus algorithm. It enables high availability and fault tolerance by replicating data across multiple nodes. Kubernetes components such as the API server interact with etcd to read and persist cluster state changes. For example, when a user deploys a new pod, the API server stores the pod specification in etcd, making it accessible to other control plane components. Because etcd contains critical cluster data, it must be secured, backed up regularly, and monitored for performance and health to maintain Kubernetes reliability.

Recreate Deployment stops old instances before deploying new ones, which is suitable for deployments involving major alterations such as substantial database changes.

We can troubleshoot a Pod by checking its logs using the kubectl logs command, describing the Pod using the kubectl describe command, and examining events related to the Pod using the kubectl get events command.

Scaling in Kubernetes can be performed manually by changing the number of replicas in a Deployment, or automatically using Horizontal Pod Autoscalers which adjust the number of replicas based on CPU usage or other select metrics.

Achieving high availability in Kubernetes: By running multiple instances of applications across various nodes, automatically replacing failed pods, and distributing load across healthy nodes.

Kubernetes supports upgrading one minor version at a time, so this is two steps: v1.28 → v1.29, then v1.29 → v1.30. For kubeadm-based clusters: # For each step: # 1. Review release notes for deprecations # 2. Backup etcd ETCDCTL_API=3 etcdctl snapshot save backup.db # 3. Plan and upgrade control plane kubeadm upgrade plan kubeadm upgrade apply v1.29.x # 4. Upgrade worker nodes one by one kubectl drain --ignore-daemonsets --delete-emptydir-data # ... upgrade kubelet and kubectl on the node ... kubectl uncordon # 5. Validate workloads after each step kubectl get pods --all-namespaces For managed clusters (EKS, GKE, AKS), the control plane upgrade is handled by the provider, but node upgrades and workload validation follow a similar pattern

Kube-scheduler job assigns pods to nodes based on resource availability and other scheduling requirements, ensuring optimal placement and load balancing.

The kubelet is the primary node agent responsible for: Pod Lifecycle Management → Container Runtime Interface → Node Resource Reporting → Volume Management When kubelet restarts frequently, it indicates fundamental node health issues that can cascade into cluster-wide problems. Systematic Troubleshooting Approach 1. Incident Impact Assessment Before diving into root cause analysis, understand the blast radius: - How many nodes are affected? - Are workloads being disrupted? - Is this a single-node or cluster-wide issue? 2. Resource Pressure Analysis Node-level resource pressure is the most common cause of kubelet instability: Memory Pressure Indicators: - OOM killer events in system logs - High memory utilization on the node - Pods being evicted due to memory pressure Disk Pressure Indicators: - High disk utilization on root filesystem - Container image storage issues - Log file accumulation CPU Pressure (Less Common): - High CPU utilization affecting system processes - Process starvation issues Diagnostic Strategy 1. System-Level Investigation Check fundamental system health: - Overall resource utilization (CPU, memory, disk, network) - System service status (container runtime, networking) - Kernel messages and hardware issues 2. kubelet-Specific Analysis # Examine kubelet service status and recent restarts systemctl status kubelet journalctl -u kubelet --since "1 hour ago" Key log patterns to look for: - "Out of memory" errors - "No space left on device" errors - Container runtime communication failures - API server connectivity issues 3. Container Runtime Health kubelet depends heavily on the container runtime: - containerd/Docker daemon health - Runtime socket connectivity - Image pull and container creation capabilities Common Root Causes and Solutions 1. Resource Exhaustion Memory Issues: - System processes consuming excessive memory - Memory leaks in running containers - Insufficient node memory for kubelet operation Resolution approach: - Implement proper resource requests/limits on pods - Configure kubelet memory reservation - Set up node-level monitoring and alerting 2. Storage Problems Container Image Accumulation: Images not being garbage collected properly, filling up disk space. Log File Growth: Application logs growing without rotation, consuming disk space. Resolution strategy: - Configure automatic image garbage collection - Implement log rotation policies - Monitor disk usage and set up alerting 3. Network Connectivity Issues API Server Communication: kubelet losing connectivity to the API server due to network issues. DNS Resolution Problems: Node unable to resolve cluster DNS names. Prevention and Monitoring Strategy 1. Node Health Monitoring Implement comprehensive node monitoring covering: - System resource utilization - kubelet health and restart frequency - Container runtime health - Network connectivity to control plane 2. Proactive Maintenance Regular Health Checks: - Automated node health validation - Preventive maintenance windows - Capacity planning and resource monitoring Graceful Node Management: - Node draining procedures for maintenance - Automated node replacement for persistent issues - Blue-green node group strategies 3. Cluster-Level Resilience Workload Distribution: - Anti-affinity rules to distribute critical workloads - Pod disruption budgets to prevent service interruption - Multiple availability zones for node placement Automatic Recovery: - Node auto-replacement through cluster autoscaler - Health check-based node cycling - Workload migration during node issues Node Lifecycle Management 1. Node Replacement Strategy When nodes consistently exhibit problems: - Cordon and drain the problematic node - Analyze the node for patterns before termination - Replace with fresh node infrastructure - Monitor the replacement for similar issues 2. Capacity Planning Regular assessment of: - Node resource utilization trends - Workload growth patterns - Peak usage planning and auto-scaling thresholds 3. Compliance and Security - Regular security patching schedules - Configuration drift detection - Compliance validation and remediation

Etcd is an open-source key value store that manages the crucial data of a distributed system. It comes under the non-relational database and stores the information in the form of values and keys. It works as a leader election incase of tolerance or partition machine failure. Due to its simple interface, anyone can read or write values with standard HTTP tools.

ConfigMaps hold non-sensitive config; Secrets store sensitive data like credentials.

Kubernetes assigns DNS names to services, making them accessible via predictable hostnames.

ArgoCD is a tool classified under GitOps that provides continuous deployment automation for Kubernetes. It watches a Git repository for changes, then mirrors them into the cluster so that deployments can always be synchronized with the repository.

Upgrading a Kubernetes cluster involves several steps to ensure minimal downtime and consistency across the cluster. The process typically starts with upgrading the control plane components. The API server is upgraded first, followed by the controller manager and scheduler. Once the control plane is upgraded, the etcd database is updated. After the control plane is stable, the nodes are upgraded. This can be done by draining a node (which safely evicts all pods), upgrading the kubelet and kube-proxy, and then uncordoning the node to bring it back into the cluster. This process is repeated for all nodes. Kubernetes also provides tools like kubeadm to simplify and automate the upgrade process.

Security is a key aspect, and every administrator should follow best practices to secure a Kubernetes cluster: - Enable RBAC: Restrict user access using Roles and RoleBindings. - Pod security admission: Use admission controllers to enforce the Pod security standards. - Enforce NetworkPolicies: Restrict Pod communication, as by default, all Pods can communicate with each other. - Rotate API tokens and certificates regularly. - Use secrets management: Use tools like Vault, AWS Secrets Manager, etc.

A Kubernetes Service is an abstraction that defines a logical set of Pods and a policy by which to access them. Services enable communication between different components of an application, providing a stable IP address and DNS name for accessing Pods. They abstract away the complexity of managing Pod IP addresses, which can change over time, and provide load balancing across the Pods.

Highlight your understanding of Kubelet (node lifecycle) and how to systematically troubleshoot node issues with these steps: - Check the node status - Describe the node for events/logs - Check Kubelet logs for errors - Verify system resource availability - Restart Kubelet - Remove the node from the cluster if it's unresponsive - Scale the cluster up if needed. Cluster Autoscaler may replace the failed node if it's a cloud-based cluster.

A DaemonSet ensures that all (or some) Nodes run a copy of a Pod. It's typically used for cluster-wide services like log collection agents or node monitoring agents.

kubectl rollout undo deployment kubectl rollout history deployment - Ensure probes and rollout settings are tuned to catch issues early.

We can interact with the Kubernetes cluster using the kubectl tool. We can use kubectl to deploy applications, manage the cluster, and monitor the pods. We can also use the Kubernetes Dashboard to interact with the cluster. The dashboard allows users to create deployments, view pod logs, monitor the usage of the nodes, etc.

Kube-schedulers: It is an in-built scheduler of this tool that assigns nodes to cluster. This procedure is done on the basis of resources and constraints. In this process, it ranks the nodes and binds them with the best suitable pods. It can also make copies of pods and distribute them among various nodes to achieve improved scalability.

Kubernetes handles networking through a flat network model that allows all Pods to communicate with each other, regardless of the node they are on. This is achieved through: - Cluster IP: Provides a stable internal IP for accessing services within the cluster. - NodePort: Exposes a service on a static port on each node's IP, allowing external access. - LoadBalancer: Creates an external load balancer for a service, usually provided by a cloud provider. - Network Policies: Define rules for how Pods communicate with each other and other network endpoints.

Kubernetes Namespaces group a collection of resources, such as Pods or StatefulSets, inside your Kubernetes cluster. They're used to implement Kubernetes multi-tenancy by logically isolating resources belonging to different apps, teams, or environments. For instance, you could create staging and production namespaces to deploy two instances of an application, each with its own customizations. Namespaces also integrate with the Kubernetes RBAC system, letting you precisely control who can access each Namespace.

Kubernetes Deployments support rolling updates to avoid downtime. You can perform a rolling update by editing a Deployment or explicitly setting its image to a new version using: kubectl set image deployment/my-deployment nginx=nginx:1.21 You can then check the Deployment status: kubectl rollout status deployment my-deployment If you want to roll back to the previous version, you can run: kubectl rollout undo deployment my-deployment

Kubernetes logging is the process of collecting and analyzing the logs generated by the applications and services running in a Kubernetes cluster.

The Kubernetes (K8s) was designed and developed by Google. It can be defined as an open-source orchestration tool. In other words, Kubernetes is an open-source container manager. It helps the user to organize, scale, and automate many large containers. As it is developed by Google it doesn't have to prove its excellence. In layman's terms, we can call Kubernetes a solution for the management of multiple containers.

Kubernetes Secrets are base64-encoded by default, not encrypted. A production-grade approach involves multiple layers. Enable encryption at rest using an EncryptionConfiguration: apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration resources: - resources: - secrets providers: - aescbc: keys: - name: key1 secret: - identity: {} Use an external secrets manager (e.g., Vault, AWS Secrets Manager, GCP Secret Manager) as the source of truth. Prefer mounting Secrets as files rather than environment variables. While values are not shown directly, environment variables are easier to expose through debugging or process inspection.

The characteristics of Kubernetes such as: - Automates various manual processes: Kubernetes can take full control of the server and host the container. - Interacts with various groups of containers: Kubernetes can manage more clusters at the same time - Provides additional services: In addition to container management, Kubernetes offers security, networking, and storage services - Self-monitoring: Kubernetes checks the health of nodes and containers constantly - Horizontal scaling: Kubernetes allows you to scale out the resources vertically as well as horizontally more easily and quickly. - Storage orchestration: You can add a storage system of your choice in Kubernetes to run any of the applications. - Automates rollouts and rollbacks: If any change is required in your application or anything goes wrong, then Kubernetes helps you to achieve automatic rollbacks. - Container balancing: Kubernetes calculates the best location for the containers and places it in the right area to achieve load balancing. - Run everywhere: Kubernetes is an open-source tool and you can take advantage of any cloud infrastructure such as on-premises, hybrid, or public and helps you to move workloads to anywhere you need.

Kubernetes Jobs and CronJobs are used to run batch tasks and scheduled tasks, respectively. - Jobs: Run a Pod to completion. They ensure that a specified number of Pods successfully terminate. - CronJobs: Schedule Jobs to run at specific times, similar to cron jobs in Unix. Example Job configuration: apiVersion: batch/v1 kind: Job metadata: name: myjob spec: template: spec: containers: - name: mycontainer image: busybox command: ["sh", "-c", "echo Hello Kubernetes! && sleep 30"] restartPolicy: OnFailure Example CronJob configuration: apiVersion: batch/v1 kind: CronJob metadata: name: mycronjob spec: schedule: "*/5 * * * *" jobTemplate: spec: template: spec: containers: - name: mycontainer image: busybox command: ["sh", "-c", "date; echo Hello from the Kubernetes cluster"] restartPolicy: OnFailure Applying the Job and CronJob configurations: kubectl apply -f myjob.yaml kubectl apply -f mycronjob.yaml

A StatefulSet is used for stateful applications that require stable network identities, persistent storage, and ordered deployment. Unlike Deployments, StatefulSets ensure that: - Pods have stable and unique network identities, where Pods are named like pod-0 ,pod-1 etc. - Pods are created, updated, and deleted in order. - Each Pod retains persistent storage across restarts. Persistent storage is defined as part of the StatefulSet YAML definition.

The deployment is either using an immutable image digest or relying on imagePullPolicy: IfNotPresent. Existing nodes cache images locally and won't pull newer versions unless forced.

The kubelet is a service agent which controls and maintains group pf pods by checking pod specification using Kubernetes. The kubelet runs on each node and allows to communicate between a master node and a slave node.

The Kubernetes Horizontal Pod Autoscaler (HPA) automatically scales the number of Pods in a Deployment, ReplicaSet, or StatefulSet based on observed CPU utilization or other custom metrics. HPA ensures that applications can handle varying workloads without manual intervention. - CPU-based scaling: Uses CPU utilization as the default metric for scaling. - Custom metrics: Can be configured to scale based on other metrics, such as memory usage or application-specific metrics. - Scaling policies: Define the behavior for scaling up or down. Example configuration for an HPA: apiVersion: autoscaling/v1 kind: HorizontalPodAutoscaler metadata: name: myapp-hpa spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: myapp minReplicas: 1 maxReplicas: 10 targetCPUUtilizationPercentage: 50 To create and apply the HPA: kubectl apply -f myapp-hpa.yaml Check the HPA status: kubectl get hpa

Kubernetes master operates the nodes and containers. A number of containers are present in a particular pod based on requirement and configuration. These pods can be deployed via user interface or command-line interface. Further, these nodes are allocated on nodes based on the resource requirements. At last, the Kube-apiserver ensures that there is a proper communication between nodes and master components.

94% of students found the test questions almost the same.

- Imperative: Directly manage resources via commands. Quick fixes are possible, but it's harder to maintain state consistency. - Declarative: Define the desired state, and let Kubernetes handle changes. It's more reproducible and easier to audit but can be complex to debug if the state drifts.

When you deploy directly on a host, the application depends on that server's setup. If servers are configured individually, the application may behave differently. Scaling means provisioning and configuring additional servers. In container deployment, the application and its dependencies are packaged into a single unit. It runs the same way everywhere. Scaling means running more containers, not building more servers.

A Service is an abstraction that exposes a set of Pods as a network service, enabling communication between different parts of an application within a Kubernetes cluster.

Here's a specification for a persistent volume for “visualcv-main”: apiVersion: v2.01 kind: PersistentVolume metadata: name: visualcv-main spec: capacity: storage: 5Gi volumeMode: Filesystem accessModes:

Node status includes information about available resources (CPU, memory), node's health, running containers, and network configurations.

Consider these: - Check the Ingress status - Describe the Ingress resource - Verify the Ingress Controller is running. Restart it if it's down. - Check service backend mapping. - Inspect the Ingress logs - Manually test routing with curl

To instantly retrieve a list of all running, pending, or failing Pods within your currently active namespace, execute the following command: kubectl get pods This command returns a clean table displaying the Pod's NAME, its READY status (e.g., 1/1 containers running), its current STATUS (Running, CrashLoopBackOff), the number of RESTARTS, and its AGE. If you need detailed operational insights—such as the IP address assigned to each Pod or the specific Node hosting it—simply append the wide output flag: kubectl get pods -o wide.

In Kubernetes, Labels are key-value pairs attached to resources, and Selectors are the tools used to manage and interact with these labeled resources. - Efficient Grouping: Labels group resources, simplifying management and configuration. - Decoupling: Resources are decoupled from consuming services through selectors, making management flexible and robust. - Queries: Selectors filter resources by matching label sets, enabling focused actions. Kubernetes resources, such as Pods or Services, are associated with labels to indicate attributes. A spec section can include labels during resource definition. Example labels: metadata: labels: environment: prod tier: frontend Select: - All resources without any specific labels are significant when not tagged. - Specific Resources: Indicate labels to MATCH (AND condition) for resource retrieval. Example, to find all "frontend" Pods in the "prod" environment: matchLabels: environment: prod tier: frontend

To scale the application, I would follow these steps: - Identify the bottleneck: Analyze resource utilization, including CPU, memory, and network, to determine the limiting factor. - Horizontal scaling: If the bottleneck is CPU or memory, I would scale the application horizontally by increasing the number of replicas using a Horizontal Pod Autoscaler (HPA). - Vertical scaling: If the bottleneck is resource-specific, I would vertically scale the application by upgrading the resources allocated to each pod. - Monitor and validate: Monitor the application's performance after scaling to ensure the desired scalability is achieved without impacting stability.

Pod affinity and anti-affinity rules allow you to specify how pods should be placed relative to other pods. Affinity attracts pods to nodes with specific labels, while anti-affinity ensures pods are not co-located.

Kubernetes ConfigMaps and Secrets both let you supply key-value configuration data to your Pods. ConfigMaps are designed to store plain-text values that require no special treatment. Secrets are a dedicated solution for sensitive values such as passwords, API keys, and certificates that must be kept secure. They can be encrypted when stored at rest, reducing the risk of exposure.

A service mesh manages service-to-service communication, providing: - Traffic management (load balancing, canary deployments). - Security (mTLS encryption between services). - Observability (tracing, metrics, logging) All these features are included in the infrastructure layer, so no code changes are required. Popular Kubernetes service mesh solutions include: Istio, Linkerd and Consul.

Kubernetes is designed to be extensible, allowing developers to write and register custom controllers or operators that can manage any object.

Kubernetes ensures high availability through mechanisms like replication controllers, which maintain a specified number of pod replicas, and by deploying these replicas across multiple nodes. Additionally, features like automatic rescheduling and self-healing help Kubernetes adapt to node failures, ensuring continuous availability of applications.

Role-based access control (RBAC) is the technique by which one creates Kubernetes Roles and RoleBindings. A Kubernetes Role is an abstraction that defines a relationship between a Kubernetes resource and action — for example, “can create a pod” where pod is the Kubernetes resource and create is the action. A RoleBinding maps a particular user or service to a particular Role.

Start by moving from manual releases to containerized applications managed by Deployments. That brings consistency and makes updates predictable. Introduce CI/CD, rolling updates, and health checks. Add resource limits and autoscaling so the platform handles growth without manual intervention. Back it with monitoring and access controls. Lastly, have a system that scales with the business.

A DaemonSet ensures that all nodes run a copy of a Pod, while a Deployment manages the deployment of application replicas across the cluster.

Taints mark nodes to reject Pods. Tolerations let specific Pods override those taints. # Taint a node — no Pod can schedule here unless it tolerates this kubectl taint nodes workload=gpu:NoSchedule A Pod that tolerates the taint: apiVersion: v1 kind: Pod metadata: name: ml-training spec: tolerations: - key: "workload" operator: "Equal" value: "gpu" effect: "NoSchedule" containers: - name: trainer image: ml-training:latest Common use case: you have GPU nodes that should only run machine learning workloads. Taint the GPU nodes, add tolerations to the ML Pods, and other workloads are automatically scheduled elsewhere.

Network Policies in Kubernetes are used to control the traffic flow between Pods and other network endpoints. They define rules for ingress (incoming) and egress (outgoing) traffic at the Pod level. Network Policies use selectors to specify which Pods the rules apply to and support a variety of criteria such as namespace, labels, and ports. Example Network Policy: apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-web spec: podSelector: matchLabels: role: web policyTypes: - Ingress - Egress ingress: - from: - podSelector: matchLabels: role: frontend egress: - to: - podSelector: matchLabels: role: database ports: - protocol: TCP port: 3306

Kubernetes custom resource definition is a way to extend the Kubernetes API with custom resources and APIs that are specific to a particular application or framework.

Kubernetes is an open-source container orchestration platform that automates the deployment, scaling, and management of containerized applications. It was originally developed by Google and is now maintained by the Cloud Native Computing Foundation (CNCF). Kubernetes provides a robust set of APIs for managing clusters of containers, making it easier to manage and scale applications in a consistent and reliable manner.

Admission controllers sit in the request path between the API server receiving a request and the resource being persisted to etcd. They intercept requests to create, update, or delete resources and can either modify or reject them before anything is written. There are two types: - Mutating admission controllers can modify the incoming request. For example, automatically injecting a sidecar container into every pod, adding default resource limits, or attaching labels that your organization requires. - Validating admission controllers check whether a request meets certain criteria and reject it if it does not. For example, blocking any pod that tries to run as root, or ensuring every Deployment has a specific annotation. Mutating controllers always run first, then validating controllers run against the final version of the resource. This ordering matters because validation should check what will actually be persisted, including any changes made by mutation. Kubernetes ships with several built-in admission controllers, but you can also define your own policies using tools like OPA/Gatekeeper or Kyverno, which let you write custom validation and mutation rules without building a webhook from scratch.

#List Pods with Labels: kubectl get pods --show-lables #Add or Update Labels to a Pod: kubectl label pod = #Remove a Label from a Pod: kubectl label pod - #List Pods with a Specific Label: kubectl get pod -l = #List Pods with Label Selectors: kubectl get pods --selector= #Update Labels for Multiple Pods using Selector: kubectl label pod -l = #Label Nodes: kubectl node = #List Nodes with Labels: kubectl get nodes --show-labels #Remove a Label from a Node: kubectl label node - #Label Namespaces: kubectl label namespace = #List Namespaces with Labels: kubectl get namespaces --show-labels #Remove a Label from a Namespace: kubectl label namespace -

CrashLoopBackOff indicates Kubernetes is giving up on restarting a pod because it keeps crashing immediately after startup. The 'no logs' aspect makes this particularly challenging because the usual debugging approach (checking logs) isn't helpful. Systematic Debugging Approach 1. Understand the Pod Lifecycle Pod Creation → Image Pull → Container Start → Application Init → Running ↓ (Crash happens here - before logging starts) When there are no logs, the crash occurs during the container initialization phase, before the application even starts logging. 2. Event-Driven Investigation The most valuable information comes from Kubernetes events, not application logs. Events tell you what Kubernetes observed during the pod lifecycle: kubectl describe pod Key sections to analyze in the output: - Conditions: Shows readiness and liveness probe failures - Events: The timeline of what happened (image pulls, volume mounts, container starts) - Last State: Information about the previous crash 3. Common Root Cause Categories Resource Constraints - Memory limits too low → OOMKilled - CPU limits preventing startup - Insufficient ephemeral storage Configuration Issues - Missing environment variables required for startup - Incorrect volume mounts or permissions - Wrong working directory or user context Image Problems - Wrong entrypoint or command - Missing dependencies in the container image - Architecture mismatch (arm64 vs amd64) Health Check Conflicts - Liveness probe killing pods too aggressively - Readiness probe misconfiguration - Startup probe timeout too short Problem-Solving Strategy Phase 1: Gather Intelligence - Check previous container logs: kubectl logs --previous - Examine pod events and conditions - Compare working vs non-working environments - Review recent changes to deployments or configurations Phase 2: Isolation Testing - Temporarily disable health checks to see if pod stays running - Override container command with a simple sleep to test image viability - Test with minimal resource limits to identify constraint issues Phase 3: Progressive Debugging - Add debug containers or init containers to inspect the environment - Use interactive sessions to manually test startup commands - Implement verbose logging during the startup sequence Architecture Perspective The key insight is understanding that Kubernetes manages the container lifecycle through multiple layers: Kubernetes Scheduler → kubelet → Container Runtime → Your Application ↓ (Failures can happen at any layer) Each layer has different failure modes and debugging approaches. CrashLoopBackOff specifically indicates the container runtime successfully started the container, but the process inside exited unexpectedly.

- Label GPU nodes: kubectl label nodes gpu=true - Taint GPU nodes: kubectl taint nodes dedicated=gpu:NoSchedule - Add toleration and node affinity to ML workload Pod specs.

Kubernetes pod readiness probe is used to determine if a pod is ready to serve traffic.

Kubernetes uses resource quotas to allocate and restrict resource consumption for individual namespaces, preventing resource contention.

Kubernetes offers different types of services for flexible & reliable communication between pods. Let's explore each. - ClusterIP: Role: Internal communication within the cluster. - How it Works: Pods communicate with the Service's fixed IP and port. Service distributes traffic among pods based on defined selector. - Practical Use: Ideal for back-end services where direct access from external sources isn't necessary. - NodePort: Role: Exposes the Service on a static port on each node of the cluster. - How it Works: In addition to the ClusterIP functionality, it also opens a specific port on each node. This allows external traffic to reach the service through the node's IP address and the chosen port. - Practical Use: Useful for reaching the service from outside the cluster during development and testing phases. - LoadBalancer: Role: Makes the Service externally accessible through a cloud provider's load balancer. - How it Works: In addition to NodePort, this service provides a cloud load balancer's IP address, which then distributes traffic across Service Pods. - Practical Use: Especially useful in cloud settings for a public-facing, production deployment. - ExternalName: Role: Maps the Service to an external service using a DNS name. - How it Works: When the Service is accessed, Kubernetes points it directly to the external (DNS) name provided. - Practical Use: Useful if you want environments to have a uniform way of accessing external services and don't want to use environment-specific methods like altering /etc/hosts. - Headless: Role: Disables the cluster IP, allowing direct access to the Pods selected by the Service. - How it Works: This type of Service doesn't allocate a virtual IP (ClusterIP) to the Service, letting the clients directly access the Pods. It returns the individual Pod IPs. - Practical Use: Useful for specialized use cases when direct access to Pod IPs is necessary.

RBAC (Role-Based Access Control) is how Kubernetes controls who can do what within a cluster. This is a common interview question because it touches on security, multi-tenancy, and operational maturity. The interviewer wants to see that you understand not just the concept but how the pieces fit together in practice. A good place to start is from the perspective of checking permissions. The command kubectl auth can-i lets you verify whether a given user or service account has permission to perform a specific action. For example: kubectl auth can-i create deployments --namespace production This returns a simple yes or no. It is a useful debugging tool when someone reports they cannot perform an action, and it is also a good way to audit what access exists in the cluster. You can also impersonate other users to check their permissions: kubectl auth can-i get secrets --as system:serviceaccount:default:my-app The permissions that can-i checks against are defined through four key RBAC resources: A practical pattern is to define a ClusterRole once (for example, a "read-only" role that can list and get most resources) and then bind it at the namespace level using a RoleBinding where needed. This avoids duplicating Role definitions across namespaces. Service account tokens are how pods authenticate to the API server. Every pod in Kubernetes runs under a service account, and by default it gets the default service account in its namespace. That default account typically has very limited permissions. Since Kubernetes 1.24, service account tokens are no longer automatically mounted as long-lived secrets. Instead, Kubernetes uses bound, time-limited tokens projected into the pod. This is worth mentioning because it shows awareness of how token management has tightened over recent versions.

Kubernetes is used for the automation of the manual operations that are involved in the deployment, management, and scaling of containerized applications. It keeps track of the ones that are deployed into the cloud, restarts orphaned ones, shuts down the unused, and automatically provides resources such as storage, memory, and CPU when required.

Securing a Kubernetes cluster requires a defense-in-depth approach. Highly recommended measures include: - Implement RBAC: Enforce Role-Based Access Control to follow the principle of least privilege for users and service accounts. - Network Policies: Restrict East-West traffic by defining strict ingress/egress rules between Pods (Default Deny). - Pod Security Standards: Prevent Pods from running as the root user or mounting sensitive host file systems. - Audit Logging: Enable API server audit logs to track all kubectl interactions. - Image Scanning: Continuously scan container images for CVEs before deploying them.

Kubernetes assigns every pod to one of three QoS classes that determine eviction priority: Guaranteed (Never evicted unless exceeding limits) ↓ Burstable (Evicted when node under pressure) ↓ BestEffort (First to be evicted) QoS Class Assignment Logic Guaranteed QoS: All containers have identical CPU and memory requests and limits. resources: requests: memory: "1Gi" cpu: "500m" limits: memory: "1Gi" # Same as requests cpu: "500m" # Same as requests Burstable QoS: At least one container has resource requests or limits, but not identical requests/limits. BestEffort QoS: No resource requests or limits specified. Eviction Process Understanding Node Pressure Detection: The kubelet monitors node resources and triggers eviction when thresholds are exceeded: Resource Monitoring → Threshold Detection → Pod Selection → Graceful Termination Eviction Criteria Priority: - QoS class (BestEffort → Burstable → Guaranteed) - Resource usage relative to requests - Pod priority and preemption policy - Creation time (older pods evicted first within same priority) Prevention Strategies 1. Implement Priority Classes apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: critical-priority value: 1000000 # Higher value = higher priority globalDefault: false description: "Critical system components" Application to critical workloads: spec: priorityClassName: critical-priority # Protects from preemption containers: - name: critical-app resources: requests: memory: "512Mi" cpu: "250m" limits: memory: "1Gi" # Allows some burst capacity cpu: "500m" Key concepts: priorityClassName: Links pod to priority class Higher priority pods preempt lower priority ones Priority affects both scheduling and eviction decisions 2. Pod Disruption Budgets apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: critical-app-pdb spec: minAvailable: 2 # Always keep 2 pods running selector: matchLabels: app: critical-app PDB protects against: - Voluntary disruptions (node maintenance, upgrades) - Involuntary disruptions (hardware failures, kernel panics) - Eviction due to resource pressure 3. Resource Reservation Strategy Node-Level Reservations: Configure kubelet to reserve resources for system processes: --system-reserved=cpu=200m,memory=250Mi --kube-reserved=cpu=200m,memory=250Mi --eviction-hard=memory.available<500Mi Cluster-Level Planning: - Maintain spare capacity across the cluster - Implement cluster autoscaling with appropriate buffer - Plan for peak usage scenarios Advanced Resource Management 1. Vertical Pod Autoscaling (VPA) Automatically adjusts resource requests based on actual usage: apiVersion: autoscaling.k8s.io/v1 kind: VerticalPodAutoscaler metadata: name: critical-app-vpa spec: targetRef: apiVersion: apps/v1 kind: Deployment name: critical-app updatePolicy: updateMode: "Auto" # Automatically apply recommendations VPA Benefits: - Right-sizes resource requests based on actual usage - Reduces resource waste and improves cluster efficiency - Prevents over-provisioning that leads to QoS class issues 2. Horizontal Pod Autoscaling (HPA) Scales the number of pod replicas based on metrics: apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: critical-app-hpa spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: critical-app minReplicas: 3 maxReplicas: 20 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 70 # Scale up when CPU > 70% Monitoring and Alerting Strategy 1. Resource Usage Monitoring Track key metrics: - Pod resource utilization vs requests/limits - Node resource availability and pressure - QoS class distribution across the cluster - Eviction events and patterns 2. Predictive Alerting Set up alerts for: - Node resource pressure approaching eviction thresholds - Critical pods running with insufficient resource guarantees - Cluster capacity approaching limits - Unusual eviction patterns 3. Capacity Planning Regular analysis of: - Resource utilization trends - Growth patterns and seasonal variations - Cost optimization opportunities - Performance impact of resource constraints Best Practices for Production 1. Defense in Depth - Multiple layers of protection (QoS, priority, PDB) - Redundancy across availability zones - Automated recovery mechanisms 2. Progressive Resource Allocation - Start with conservative resource requests - Use monitoring data to optimize over time - Implement gradual scaling policies 3. Testing and Validation - Chaos engineering to test eviction scenarios - Load testing to validate resource planning - Regular disaster recovery exercises

When a node is tainted, the pods don't get scheduled by default, however, if we have to still schedule a pod to a tainted node we can start applying tolerations to the pod spec. Apply a taint to a node: kubectl taint nodes node1 key=value:NoSchedul Apply toleration to a pod: spec: tolerations: - key: "key" operator: "Equal" value: "value" effect: "NoSchedule"

Think of network policies as firewalls that operate at the pod level. Unlike traditional firewalls that work with IP addresses, Kubernetes network policies use label selectors and namespace selectors. Default: All traffic allowed (if no policies exist) ↓ Policy Applied: Default deny + explicit allow rules ↓ Traffic Flow: Evaluated against all applicable policies Understanding Policy Application Logic 1. Policy Selection: Network policies apply to pods based on label selectors. A pod can be affected by multiple policies simultaneously. 2. Traffic Direction: Policies can control: - Ingress: Traffic coming into pods - Egress: Traffic leaving pods - Both: Comprehensive traffic control 3. Rule Evaluation: Traffic is allowed if it matches ANY allow rule in ANY applicable policy. There's no concept of deny rules – policies work on an allow-list basis. Cross-Namespace Communication Design Architecture Pattern: Frontend Namespace → Backend Namespace → Database Namespace (Web Apps) (APIs) (Stateful Services) Security Zones Approach: - Public Zone (Frontend): Accepts traffic from internet - Internal Zone (Backend): Only accepts traffic from frontend - Data Zone (Database): Only accepts traffic from backend Policy Design Strategy 1. Start with Default Deny Create a baseline policy that denies all traffic, then explicitly allow what's needed: # This creates a default deny policy for the namespace apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-all namespace: backend spec: podSelector: {} # Applies to all pods policyTypes: - Ingress - Egress The empty podSelector: {} means this policy applies to all pods in the namespace. The policyTypes list specifies that both incoming and outgoing traffic are controlled. 2. Layer Specific Allow Rules # Allow frontend to access backend apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-frontend-to-backend namespace: backend spec: podSelector: matchLabels: tier: api # Only applies to API pods policyTypes: - Ingress ingress: - from: - namespaceSelector: matchLabels: name: frontend # Allow from frontend namespace podSelector: matchLabels: tier: web # Only from web tier pods ports: - protocol: TCP port: 8080 # Only on API port Key elements explained: podSelector: Defines which pods this policy protects namespaceSelector: Allows traffic from specific namespaces podSelector (in ingress): Further restricts to specific pods within allowed namespaces ports: Limits to specific ports and protocols Debugging Network Policy Issues 1. Understanding Policy Overlap When multiple policies affect the same pod, they combine using OR logic. Debug by: - Listing all policies affecting a pod - Understanding how rules combine - Testing with progressive policy removal 2. Traffic Flow Testing Systematic testing approach: Pod A (source) → Pod B (destination) ↓ Check: Does Pod A have egress rules allowing this traffic? ↓ Check: Does Pod B have ingress rules allowing this traffic? ↓ Both must allow for traffic to flow 3. Common Gotchas - DNS Resolution: Pods need egress access to kube-dns - Service Discovery: Traffic to services still goes through network policies - Load Balancer Traffic: External load balancer traffic might bypass policies Advanced Patterns Microsegmentation Strategy: Create fine-grained security zones based on application tiers, trust levels, and data sensitivity. Dynamic Policy Management: Use labels to dynamically include/exclude pods from security groups as they're deployed. Compliance Integration: Design policies that map to regulatory requirements (PCI-DSS zones, HIPAA boundaries, etc.).