Configuring and Managing Kubernetes Storage and Scheduling

Table of Contents

• 1. Implement Kubernetes Persistent Storage Concepts and Management
  • 1.1 Why Persistent Storage Is Critical
  • 1.2 PersistentVolume (PV) and PersistentVolumeClaim (PVC)
  • 1.3 Demo: Mounting Persistent Storage in a Pod
  • 1.4 StorageClasses and Dynamic Provisioning
  • 1.5 Demo: Creating a StorageClass and PVC for Dynamic Provisioning
• 2. Implement Advanced Pod Scheduling Techniques
  • 2.1 Taints and Tolerations
  • 2.2 Affinity and Anti-Affinity
  • 2.3 Resource Requests and Limits
  • 2.4 Demo: Enforcing Node Isolation
  • 2.5 Demo: Balanced and Efficient Scheduling
• 3. Kubernetes Pod Scheduling Mechanisms
  • 3.1 How the Kubernetes Scheduler Works
  • 3.2 Pod Disruption Budgets (PDB)
  • 3.3 Topology Spread Constraints
  • 3.4 Demo: Protecting Availability with PDBs
  • 3.5 Demo: Topology Spread Constraints
• 4. Reference Tables
• 5. Summary and Key Points

1. Implement Kubernetes Persistent Storage Concepts and Management

1.1 Why Persistent Storage Is Critical

By default, Kubernetes containers are ephemeral: any data stored in their filesystem disappears with every restart, rescheduling, or pod replacement. This behavior is intentional — it keeps containers lightweight and portable — but it poses a major problem for stateful applications.

Real-world scenario at Globomantics:
Steve, a DevOps engineer, containerized databases, analytics tools, and APIs. After every new deployment, data disappears — logs wiped, uploaded files lost. The cause: the default ephemeral storage of containers.

Workloads that require persistent storage:

• Databases (PostgreSQL, MySQL, MongoDB)
• Centralized log systems
• File-sharing applications
• Stateful analytics tools

Workloads that do not need persistent storage:

• Stateless web frontends
• Stateless APIs
• On-the-fly processing microservices

flowchart TD
    A[Pod starts] --> B{Data in\ncontainer filesystem?}
    B -- No --> C[Clean start]
    B -- Yes --> D[Pod stopped / rescheduled]
    D --> E[Data lost ❌]
    E --> F[Need for persistent storage]
    F --> G[PersistentVolume PV]
    G --> H[PersistentVolumeClaim PVC]
    H --> I[Pod mounted with volume]
    I --> J[Data persisted ✅]

1.2 PersistentVolume (PV) and PersistentVolumeClaim (PVC)

Kubernetes abstracts storage into two complementary objects:

PV / PVC Architecture:

graph LR
    subgraph Infrastructure
        DISK[💾 Physical disk\nhostPath / NFS / Cloud]
    end

    subgraph Kubernetes Cluster
        PV[PersistentVolume\nCapacity: 1Gi\nAccessMode: RWO\nReclaimPolicy: Retain]
        PVC[PersistentVolumeClaim\nStorage: 1Gi\nAccessMode: RWO]
        POD[Pod\nvolumeMounts: /data]
    end

    DISK --> PV
    PV <-->|Binding| PVC
    PVC --> POD

Available Access Modes:

Reclaim Policies:

1.3 Demo: Mounting Persistent Storage in a Pod

Step 1 — Define a PersistentVolume:

# pv-demo.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-demo
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: /mnt/data
  persistentVolumeReclaimPolicy: Retain

Step 2 — Create a PersistentVolumeClaim:

# pvc-demo.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-demo
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

Important: The accessMode and storage size of the PVC must match those of the PV for binding to occur.

Step 3 — Create a Pod using the PVC:

# pod-demo.yaml
apiVersion: v1
kind: Pod
metadata:
  name: pod-demo
spec:
  containers:
    - name: busybox
      image: busybox
      command: ["sleep", "3600"]
      volumeMounts:
        - name: storage
          mountPath: /data
  volumes:
    - name: storage
      persistentVolumeClaim:
        claimName: pvc-demo

Deployment and verification commands:

# Apply the resources
kubectl apply -f pv-demo.yaml
kubectl apply -f pvc-demo.yaml
kubectl apply -f pod-demo.yaml

# Verify the PVC is Bound
kubectl get pvc

# Check the pod
kubectl get pods

# Write data to the volume
kubectl exec pod-demo -- sh -c "echo 'persistent data' > /data/test.txt"

# Verify persistence
kubectl exec pod-demo -- cat /data/test.txt

PV/PVC binding lifecycle:

stateDiagram-v2
    [*] --> Available : PV created
    Available --> Bound : Matching PVC created
    Bound --> Released : PVC deleted
    Released --> Available : ReclaimPolicy = Recycle
    Released --> [*] : ReclaimPolicy = Delete
    Released --> Released : ReclaimPolicy = Retain\n(manual intervention)

1.4 StorageClasses and Dynamic Provisioning

Manual provisioning of PVs quickly becomes a bottleneck as the cluster grows. StorageClasses enable dynamic provisioning: Kubernetes automatically creates a PV on demand.

Key concepts:

Architecture with Dynamic Provisioning:

sequenceDiagram
    participant Dev as Developer
    participant PVC as PersistentVolumeClaim
    participant SC as StorageClass
    participant Prov as Provisioner
    participant PV as PersistentVolume
    participant Pod as Pod

    Dev->>PVC: Creates PVC with storageClassName
    PVC->>SC: References the StorageClass
    SC->>Prov: Requests volume creation
    Prov->>PV: Creates PV automatically
    PV->>PVC: Automatic binding
    Dev->>Pod: Deploys Pod with the PVC
    PVC->>Pod: Volume mounted in container

Example StorageClass for AWS EBS:

# storageclass-fast.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-storage
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp3
  iopsPerGB: "10"
  encrypted: "true"
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

StorageClass without dynamic provisioner (local environment):

# storageclass-demo.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: demo-storage
provisioner: kubernetes.io/no-provisioner
reclaimPolicy: Delete
volumeBindingMode: Immediate

1.5 Demo: Creating a StorageClass and PVC for Dynamic Provisioning

Step 1 — Apply the StorageClass:

kubectl apply -f storageclass.yaml

# List available StorageClasses
kubectl get sc

Step 2 — Create a PVC referencing the StorageClass:

# pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: demo-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: demo-storage
  resources:
    requests:
      storage: 500Mi

kubectl apply -f pvc.yaml
kubectl get pvc  # Verify status = Bound

Step 3 — Pod using this PVC:

# pod-storage-demo.yaml
apiVersion: v1
kind: Pod
metadata:
  name: storage-demo-pod
spec:
  containers:
    - name: app
      image: busybox
      command: ["sleep", "3600"]
      volumeMounts:
        - name: data-volume
          mountPath: /app/data
  volumes:
    - name: data-volume
      persistentVolumeClaim:
        claimName: demo-pvc

kubectl apply -f pod-storage-demo.yaml
kubectl get pods
kubectl exec storage-demo-pod -- df -h /app/data

2. Implement Advanced Pod Scheduling Techniques

2.1 Taints and Tolerations

In a real cluster, nodes have different capabilities. Without scheduling rules, Kubernetes places pods wherever there is space, creating imbalances.

Taints: mark a node to repel certain pods.
Tolerations: allow a pod to tolerate a taint and be scheduled on the marked node.

Taint Effects:

Example Taint:

# Apply a taint to a node
kubectl taint nodes desktop-worker dedicated=analytics:NoSchedule

# Verify the taint
kubectl describe node desktop-worker | grep Taints

# Remove a taint (append the minus sign)
kubectl taint nodes desktop-worker dedicated=analytics:NoSchedule-

Pod with matching Toleration:

# analytics-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: analytics-pod
spec:
  tolerations:
    - key: "dedicated"
      operator: "Equal"
      value: "analytics"
      effect: "NoSchedule"
  nodeSelector:
    dedicated: analytics
  containers:
    - name: analytics
      image: busybox
      command: ["sleep", "3600"]

Note: The toleration alone allows the pod to be scheduled on the tainted node, but does not force it there. nodeSelector is needed to ensure the pod lands only on that node.

Taints / Tolerations Flow:

flowchart TD
    subgraph Node["Node: desktop-worker"]
        T[Taint:\ndedicated=analytics:NoSchedule]
    end

    P1[Pod A\nNo toleration] --"❌ Blocked"--> Node
    P2[Pod B\nToleration:\ndedicated=analytics] --"✅ Allowed"--> Node
    P3[Pod C\nToleration:\notherKey=otherValue] --"❌ Blocked"--> Node

2.2 Affinity and Anti-Affinity

Where taints/tolerations repel pods, affinities attract them to nodes or co-locate them with other pods.

Affinity types:

Constraint modes:

Node Affinity — example:

# analytics-affinity-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: analytics-affinity-pod
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: role
                operator: In
                values:
                  - analytics
  containers:
    - name: analytics
      image: busybox
      command: ["sleep", "3600"]
      resources:
        requests:
          memory: "128Mi"
          cpu: "250m"
        limits:
          memory: "256Mi"
          cpu: "500m"

Pod Anti-Affinity — high availability:

# webapp-anti-affinity.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
spec:
  replicas: 3
  selector:
    matchLabels:
      app: webapp
  template:
    metadata:
      labels:
        app: webapp
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - webapp
              topologyKey: kubernetes.io/hostname
      containers:
        - name: webapp
          image: nginx

Overview of scheduling mechanisms:

graph TD
    subgraph "Placement Mechanisms"
        direction TB
        NS[nodeSelector\nSimple label matching]
        NA[nodeAffinity\nComplex rules\non node labels]
        PA[podAffinity\nPod co-location]
        PAA[podAntiAffinity\nPod separation]
        TT[Taints & Tolerations\nNode isolation]
    end

    subgraph "Outcomes"
        PERF[Optimized performance]
        HA[High availability]
        ISO[Workload isolation]
    end

    NS --> PERF
    NA --> PERF
    PA --> PERF
    PAA --> HA
    TT --> ISO

2.3 Resource Requests and Limits

Resource requests and limits allow the scheduler to make informed decisions about pod placement.

# Example of requests and limits
containers:
  - name: app
    image: myapp:latest
    resources:
      requests:
        memory: "256Mi"
        cpu: "250m"
      limits:
        memory: "512Mi"
        cpu: "500m"

Best practice: Always define requests and limits. Without requests, Kubernetes assumes the pod consumes nearly zero resources, which can lead to node over-packing (resource starvation).

2.4 Demo: Enforcing Node Isolation

# 1. List nodes
kubectl get nodes

# 2. Apply a taint on desktop-worker (reserved for analytics)
kubectl taint nodes desktop-worker dedicated=analytics:NoSchedule

# 3. Verify the taint
kubectl describe node desktop-worker | grep Taints

# 4. Label the node
kubectl label node desktop-worker dedicated=analytics

# analytics-pod.yaml — Pod with toleration + nodeSelector
apiVersion: v1
kind: Pod
metadata:
  name: analytics-pod
spec:
  tolerations:
    - key: "dedicated"
      operator: "Equal"
      value: "analytics"
      effect: "NoSchedule"
  nodeSelector:
    dedicated: analytics
  containers:
    - name: analytics
      image: busybox
      command: ["sleep", "3600"]

# 5. Deploy and verify
kubectl apply -f analytics-pod.yaml
kubectl get pods -o wide  # Verify the pod is on desktop-worker

2.5 Demo: Balanced and Efficient Scheduling

# 1. Label nodes by role
kubectl label node desktop-worker role=analytics
kubectl label node desktop-worker2 role=web

# 2. Verify labels
kubectl get nodes --show-labels

# analytics-affinity-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: analytics-affinity-pod
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: role
                operator: In
                values:
                  - analytics
  containers:
    - name: analytics
      image: busybox
      command: ["sleep", "3600"]
      resources:
        requests:
          memory: "128Mi"
          cpu: "250m"
        limits:
          memory: "256Mi"
          cpu: "500m"

kubectl apply -f analytics-affinity-pod.yaml
kubectl get pods -o wide
kubectl describe pod analytics-affinity-pod | grep -A5 "Node-Selectors\|Tolerations\|Affinity"

3. Kubernetes Pod Scheduling Mechanisms

3.1 How the Kubernetes Scheduler Works

The Kubernetes Scheduler is the control plane component responsible for assigning each new pod to an appropriate node. Each pending pod goes through a two-phase cycle:

flowchart LR
    subgraph "Phase 1: Filtering"
        F1[Sufficient resources?]
        F2[Required labels present?]
        F3[Taints tolerated?]
        F4[Affinity rules satisfied?]
    end

    subgraph "Phase 2: Scoring"
        S1[LeastRequestedPriority]
        S2[BalancedResourceAllocation]
        S3[NodeAffinityPriority]
        S4[ImageLocalityPriority]
    end

    POD[Pending Pod] --> F1
    F1 --> F2 --> F3 --> F4
    F4 -->|Eligible nodes| S1
    S1 --> S2 --> S3 --> S4
    S4 -->|Winning node| ASSIGN[Pod assigned\nto optimal node]

Phase 1 — Filtering (elimination filters):

• Does the node have enough CPU/memory for the requests?
• Are the labels required by nodeSelector or nodeAffinity present?
• Does the node have taints the pod cannot tolerate?
• Are podAffinity / podAntiAffinity rules satisfied?
• Is the pod in the required zone/region?

Phase 2 — Scoring (ranking eligible nodes):

• LeastRequestedPriority: favors nodes with the most free resources
• BalancedResourceAllocation: balances CPU and memory
• NodeAffinityPriority: favors nodes that match preferred affinities
• ImageLocalityPriority: favors nodes that already have the image cached

3.2 Pod Disruption Budgets (PDB)

A Pod Disruption Budget protects application availability during voluntary disruptions (draining a node, updates, maintenance). It guarantees a minimum number of pods remains available at all times.

Types of disruptions:

PDB Parameters:

# webapp-pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: webapp-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: webapp

minAvailable: 2 and maxUnavailable: 1 are equivalent for a 3-replica deployment.

Verification commands:

kubectl get pdb
kubectl describe pdb webapp-pdb
# OUTPUT: Allowed Disruptions: 1  (with 3 pods and minAvailable: 2)

3.3 Topology Spread Constraints

Topology Spread Constraints ensure even distribution of pods across topological domains (nodes, availability zones, regions). They prevent single points of failure.

Key parameters:

# topology-spread-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: topology-demo
spec:
  replicas: 4
  selector:
    matchLabels:
      app: topology-demo
  template:
    metadata:
      labels:
        app: topology-demo
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: topology-demo
      containers:
        - name: app
          image: busybox
          command: ["sleep", "3600"]

Distribution example with maxSkew: 1 across 2 nodes and 4 pods:

Node desktop-worker  : [Pod1, Pod2]    → 2 pods
Node desktop-worker2 : [Pod3, Pod4]    → 2 pods
Skew = |2 - 2| = 0 ✅ (≤ maxSkew: 1)

Multi-zone distribution (high availability in production):

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: webapp
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app: webapp

3.4 Demo: Protecting Availability with PDBs

# web-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
spec:
  replicas: 3
  selector:
    matchLabels:
      app: webapp
  template:
    metadata:
      labels:
        app: webapp
    spec:
      containers:
        - name: webapp
          image: busybox
          command: ["sleep", "3600"]

kubectl apply -f web-deployment.yaml
kubectl get pods -l app=webapp  # 3 pods running

# webapp-pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: webapp-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: webapp

kubectl apply -f webapp-pdb.yaml
kubectl get pdb
# NAME        MIN AVAILABLE  MAX UNAVAILABLE  ALLOWED DISRUPTIONS  AGE
# webapp-pdb  2              N/A              1                     10s

# Attempt a drain — Kubernetes respects the PDB
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

Kubernetes will never drain more than one pod at a time as long as minAvailable: 2 is not violated.

3.5 Demo: Topology Spread Constraints

# 1. List available nodes
kubectl get nodes
# desktop-control-plane, desktop-worker, desktop-worker2

# topology-demo.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: topology-demo
spec:
  replicas: 4
  selector:
    matchLabels:
      app: topology-demo
  template:
    metadata:
      labels:
        app: topology-demo
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: topology-demo
      containers:
        - name: app
          image: busybox
          command: ["sleep", "3600"]

kubectl apply -f topology-demo.yaml

# Verify the distribution
kubectl get pods -o wide -l app=topology-demo
# 2 pods on desktop-worker, 2 pods on desktop-worker2

# Describe a pod to see constraints
kubectl describe pod <pod-name> | grep -A10 "Topology Spread Constraints"

4. Reference Tables

Kubernetes Storage Objects

Kubernetes Scheduling Objects

Scheduling Fields in the Pod Spec

Essential kubectl Commands — Storage

# PersistentVolumes
kubectl get pv
kubectl describe pv <name>
kubectl delete pv <name>

# PersistentVolumeClaims
kubectl get pvc [-n <namespace>]
kubectl describe pvc <name>

# StorageClasses
kubectl get sc
kubectl describe sc <name>

Essential kubectl Commands — Scheduling

# Nodes and labels
kubectl get nodes --show-labels
kubectl label node <name> <key>=<value>
kubectl label node <name> <key>-              # Remove a label

# Taints
kubectl taint nodes <name> <key>=<value>:<effect>
kubectl taint nodes <name> <key>=<value>:<effect>-  # Remove a taint

# Cordon / Drain
kubectl cordon <node>           # Mark node as non-schedulable
kubectl uncordon <node>         # Re-enable scheduling
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data

# PDB
kubectl get pdb
kubectl describe pdb <name>

5. Summary and Key Points

Module 1 — Persistent Storage

• Containers are ephemeral by default — all data disappears on restart
• PersistentVolumes (PV) represent physical storage; PersistentVolumeClaims (PVC) request it
• PV/PVC binding happens automatically if accessModes and size match
• StorageClasses enable dynamic provisioning: no more manual PV creation
• The reclaimPolicy determines what happens to data when the PVC is deleted (Retain / Delete)

Module 2 — Advanced Scheduling

• Taints mark a node to repel pods; tolerations allow certain pods to bypass them
• Node affinities attract pods to nodes with certain labels (hard or soft)
• Pod anti-affinity spreads replicas across different nodes for high availability
• Resource requests are used by the scheduler for node selection; limits protect other workloads

Module 3 — Scheduling Mechanisms

• The scheduler operates in two phases: filtering (elimination) then scoring (ranking)
• Pod Disruption Budgets (PDB) protect applications from voluntary disruptions
• Topology Spread Constraints distribute pods evenly across nodes/zones
• maxSkew: 1 ensures no node has more than one extra pod compared to others

Full Architecture — Storage + Scheduling

graph TB
    subgraph "Control Plane"
        SCHED[Kubernetes Scheduler]
        API[API Server]
    end

    subgraph "Storage Layer"
        SC[StorageClass\nfast-storage]
        PV[PersistentVolume\n10Gi / RWO]
        PVC[PersistentVolumeClaim\n10Gi / RWO]
    end

    subgraph "Worker Nodes"
        subgraph "Node 1 — role=analytics"
            T1[Taint: dedicated=analytics:NoSchedule]
            P1[Pod: analytics-app\n+ Toleration\n+ PVC mounted at /data]
        end
        subgraph "Node 2 — role=web"
            P2[Pod: webapp-replica-1]
        end
        subgraph "Node 3 — role=web"
            P3[Pod: webapp-replica-2]
        end
    end

    subgraph "Resilience"
        PDB[PodDisruptionBudget\nminAvailable: 1]
        TSC[TopologySpreadConstraint\nmaxSkew: 1]
    end

    API --> SCHED
    SCHED -->|Evaluate taints/affinity/resources| P1
    SCHED -->|Anti-affinity spread| P2
    SCHED -->|Anti-affinity spread| P3
    SC --> PV
    PVC --> PV
    P1 --> PVC
    PDB --> P2
    PDB --> P3
    TSC --> P2
    TSC --> P3

Search Terms

configuring · managing · kubernetes · storage · scheduling · containers · pod · persistent · commands · constraints · dynamic · essential · implement · kubectl · mechanisms · objects · provisioning · pvc · spread · topology