Resilient Infrastructure: Security and Troubleshooting

In modern networks, resilience is what separates a minor disruption from a full-scale outage. This course explores how to design and operate infrastructure that remains secure, available, and recoverable under stress, focusing on building resilient security from the foundations up. Many outages and security incidents are not caused by sophisticated attackers, but by weak defaults, misconfigurations, and poorly secured devices. It covers how to harden infrastructure, reduce the attack surface, and apply layered security controls that protect systems before problems occur — while also assuming that attacks and failures will happen. Security is not only about preventing breaches, but also about maintaining availability: defending against denial-of-service attacks, securing Layer 2 environments, protecting critical services, and moving toward zero-trust approaches. Finally, resilience means being able to recover quickly, which requires a structured troubleshooting process along with backup and rollback practices that allow services to be restored safely.

Table of Contents

• Module 1: Building a Secure and Resilient Network Foundation
  • 1.1 Security, Availability, and Resilience as Core Design Goals
  • 1.2 Demo: Hardening a Router with Cisco IOS
  • 1.3 Security Hygiene and Attack Surface Reduction
  • 1.4 Demo: Standard vs. Extended Access Control Lists
  • 1.5 Building an Access Control Strategy
  • 1.6 Demo: Centralized Authentication with RADIUS and TACACS+
  • 1.7 Identity as the Control Plane
  • 1.8 Demo: Layered Firewall Defense with pfSense and Suricata
• Module 2: Attacks, Failures, and Defensive Visibility
  • 2.1 Distinguishing Attacks from Misconfigurations
  • 2.2 Demo: Simulating Denial-of-Service Attacks
  • 2.3 Availability as a Security Objective
  • 2.4 Demo: ARP Poisoning and VLAN Hopping
  • 2.5 Why Layer 2 Is Still Dangerous
  • 2.6 Demo: DNS Spoofing and DHCP Misconfiguration
  • 2.7 Critical Services as Single Points of Failure
  • 2.8 Demo: Zero Trust Architecture in Practice
• Module 3: Troubleshooting, Recovery, and Operational Resilience
  • 3.1 The Six-Step Troubleshooting Methodology
  • 3.2 Demo: Applying the Six-Step Process to a Name Resolution Failure
  • 3.3 Demo: CLI Tools and Packet Analysis
  • 3.4 Reading the Network Like a Story
  • 3.5 Demo: Troubleshooting OSPF Routing Adjacencies
  • 3.6 Demo: VLAN Segmentation and Inter-VLAN Routing Issues
  • 3.7 Demo: NAT and PAT Configuration
  • 3.8 Demo: Security Policy Logs and Alerting
  • 3.9 Backup and Recovery of Device Configurations
• Summary

Module 1: Building a Secure and Resilient Network Foundation

1.1 Security, Availability, and Resilience as Core Design Goals

When designing infrastructure, engineers need to think about three distinct goals:

In reality, many outages are self-inflicted, caused by small mistakes in configuration or routine operational changes rather than by attackers:

• Misconfigurations are the most common cause of outages. A simple error in a device setting can block traffic, disable a service, or expose a system unexpectedly.
• Expired certificates cause secure connections to suddenly fail when they are not renewed in time.
• Bad ACL or firewall rules — a rule that is too restrictive may block legitimate traffic.
• Routing mistakes send traffic to the wrong destination or create loops.
• Accidental service shutdowns during maintenance or troubleshooting take critical systems offline.

Misconfigurations are particularly dangerous because a single configuration change can immediately impact multiple parts of a system at once. Modifying a firewall rule might unintentionally expose a sensitive service to the internet or block traffic that critical applications depend on. Infrastructure components are interconnected, so incorrect settings can create a surprisingly large blast radius. Engineers responding under pressure to incidents, outages, or urgent business requests also increases the chance of human error.

Attackers often succeed because the underlying foundations are weak. Many attacks begin by exploiting common, preventable weaknesses:

• Devices or services still using default usernames and passwords, allowing attackers to gain access through publicly known defaults.
• Services unnecessarily exposed to the network or internet, providing additional entry points.
• Flat networks that let attackers move laterally across the environment.
• Weak access controls that let users or systems access resources they don’t need.
• Poor segmentation, lack of monitoring, and the absence of a recovery plan.

Because any infrastructure should assume failure will happen at some point, resilience rests on three pillars:

flowchart LR
    A[Prevention] --> B[Detection] --> C[Recovery]
    A -->|"Strong configurations &<br/>security controls reduce<br/>likelihood of incidents"| A1[Reduced Attack Surface]
    B -->|"Monitoring, logging,<br/>and alerting identify<br/>problems & suspicious activity"| B1[Early Warning]
    C -->|"Backups & procedures<br/>restore services efficiently"| C1[Fast Restoration]

mindmap
  root((Resilient<br/>Infrastructure))
    Security
      Strong authentication
      Network protections
      Prevent compromise
    Availability
      Services reachable
      Business continuity
      No disruption
    Resilience
      Assume failure happens
      Continue operating
      Recover quickly
    Common Root Causes
      Misconfigurations
      Expired certificates
      Bad ACL/firewall rules
      Routing mistakes
      Accidental shutdowns

1.2 Demo: Hardening a Router with Cisco IOS

Device hardening means reducing the attack surface of a system by disabling unnecessary features, enforcing strong authentication, and enabling monitoring. The walkthrough below hardens a Cisco IOS router step by step.

Step 1 — Verify device status. Start by checking system information such as OS version, hardware platform, and uptime. Admins often start here to confirm the device type and software version during security reviews.

show version

Step 2 — Assign a hostname. Naming devices properly helps administrators identify them quickly when managing many systems across the network.

configure terminal
hostname C2700-R1
end

Step 3 — Disable unused interfaces. Check interface status, then shut down anything not currently needed. Open interfaces can be abused if left active.

show ip interface brief
configure terminal
interface GigabitEthernet2/0
shutdown
end

Step 4 — Create a named local admin account. Using a named account instead of shared credentials means activity can be tied to specific users. The secret keyword stores the password hashed.

configure terminal
username admin privilege 15 secret StrongPassword123!

Step 5 — Configure secure remote management with SSH. Set a domain name, generate RSA keys, enable SSHv2, and restrict the VTY lines to SSH only with local authentication — replacing insecure protocols such as Telnet.

configure terminal
ip domain-name pluralsight.com
crypto key generate rsa modulus 2048
ip ssh version 2
line vty 0 4
transport input ssh
login local
end

Step 6 — Restrict who can reach the management plane. Even with SSH enabled, access should be limited to trusted systems. Create an ACL that allows only the management network and apply it to the VTY lines.

configure terminal
access-list 10 permit 192.168.1.0 0.0.0.255
line vty 0 4
access-class 10 in
end

Step 7 — Secure the console port. Console access usually requires physical access, but it should still be protected against unauthorized configuration changes.

configure terminal
line console 0
password StrongConsolePass!
login
end

Step 8 — Disable unnecessary services. Turning off HTTP/HTTPS management, CDP, and small TCP/UDP servers reduces potential attack vectors.

conf t
no ip http server
no ip http secure-server
no cdp run
no service tcp-small-servers
no service udp-small-servers

Step 9 — Configure centralized logging. Logging is critical for detecting suspicious activity and supporting incident investigations. Send logs to a syslog server, enable timestamps, and log both successful and failed login attempts.

logging host 192.168.1.100
logging trap informational
service timestamps log datetime msec

login on-success log
login on-failure log

Step 10 — Remove unused routing protocols. If a routing protocol is not needed, it should not be running, since eliminating unused protocols reduces exposure to routing attacks.

no router rip

Step 11 — Encrypt stored passwords and save the configuration.

service password-encryption
end
copy running-config startup-config

Step 12 — Configure SNMPv3. Older SNMP versions transmit information unencrypted; Version 3 supports authentication and privacy through SHA and AES.

configure terminal
snmp-server group ADMIN v3 auth
snmp-server user admin ADMIN v3 auth sha StrongPass123! priv aes 128 StrongPrivPass!

Step 13 — Disable DNS lookups for mistyped commands, avoiding unnecessary network traffic and delays.

no ip domain-lookup

Step 14 — Configure NTP so the router’s clock stays synchronized. Accurate time is important for reliable logging and correlating security events across systems.

ntp server 192.168.1.50 prefer

At the end of this walkthrough, the router has secure management access, restricted remote administration, disabled unnecessary services, centralized logging, and secure monitoring — foundational steps that significantly improve the security posture of network infrastructure devices.

flowchart TD
    A[Start: Unhardened Router] --> B[Verify version/status]
    B --> C[Set hostname]
    C --> D[Disable unused interfaces]
    D --> E[Create named admin account]
    E --> F[Enable SSHv2, disable Telnet]
    F --> G[Restrict VTY access via ACL]
    G --> H[Secure console port]
    H --> I[Disable HTTP/CDP/small servers]
    I --> J[Enable centralized syslog + timestamps]
    J --> K[Remove unused routing protocols]
    K --> L[Enable password encryption]
    L --> M[Configure SNMPv3]
    M --> N[Disable DNS lookup]
    N --> O[Configure NTP]
    O --> P[Hardened, Monitored Router]

1.3 Security Hygiene and Attack Surface Reduction

Security hygiene and attack surface reduction are some of the most effective ways to prevent incidents. Simple habits — regularly patching systems, removing unused services, and changing default credentials — prevent many common attacks by reducing opportunities for attackers to gain an initial foothold. In many cases, breaches and outages begin with a simple exposure: a forgotten service left open to the internet, or an unused account with elevated privileges.

An attack surface refers to all the possible points where an attacker could attempt to interact with or compromise a system:

• Network services that are running and accessible
• Management interfaces used to control devices or systems
• Credentials that provide authentication and access
• Configuration rules such as firewall policies

If any of these elements are poorly configured, unnecessary, or exposed, they can become an entry point for attackers. The larger the attack surface, the more opportunities there are for failure, misconfiguration, or abuse.

The following diagram illustrates a network environment with multiple exposed access points. The organization connects to the internet through the ISP — the first major exposure point, representing the boundary between the external world and the internal network. Any services exposed here (web portals, remote access systems) must be carefully secured. Traffic then flows through the core switch to the data center, which hosts core services like applications, storage, and internal platforms and becomes a high-value target. Multiple departments (HR, executives, finance, engineering) each represent another potential exposure point — every connection increases the attack surface, which is why segmentation, access control, and monitoring matter.

flowchart LR
    INET((Internet)) --- ISP[ISP Boundary]
    ISP --- SW[Core / Global Switch]
    SW --- DC[(Data Center:<br/>Apps, Storage, Platforms)]
    SW --- HR[HR Department]
    SW --- EXEC[Executives]
    SW --- FIN[Finance]
    SW --- ENG[Engineering]

    style ISP fill:#f96,stroke:#333
    style DC fill:#f66,stroke:#333

Many systems ship with default configurations that prioritize convenience over security:

• Default credentials left unchanged allow attackers to gain access through guessing.
• Unused services left running by default create additional entry points even when not needed.
• Overly permissive firewall/ACL rules (“allow any”) permit traffic from anywhere to reach internal systems.
• Management interfaces left open to large parts of the network — or the internet — provide powerful admin access that is very attractive to attackers.

Good security hygiene is a key part of building resilient infrastructure, not just protection from attackers:

Security hygiene is not a one-time deployment activity — it must be continuous.

1.4 Demo: Standard vs. Extended Access Control Lists

This demo uses a simple routed topology with two routers connecting three networks:

flowchart LR
    PC1["PC1<br/>192.168.10.10/24"] --- R1["Router 1<br/>192.168.10.1"]
    R1 ---|"10.0.0.0/30"| R2["Router 2<br/>10.0.0.2"]
    R2 --- PC2["PC2<br/>192.168.20.0/24"]
    R2 --- PC3["PC3<br/>192.168.30.0/24"]

• PC1 (192.168.10.10) sends traffic to its default gateway, R1 (192.168.10.1).
• R1 forwards traffic to R2 (10.0.0.2).
• R2 connects to two additional networks: PC2 on 192.168.20.0/24 and PC3 on 192.168.30.0/24.
• Traffic from PC1 travels R1 → R2, and from there can reach either PC2 or PC3 depending on the destination network.

Standard ACL — filters by source address only. On R2, create access list 1 that denies traffic from a specific host and permits everything else. ACLs are processed top-to-bottom, evaluation stops at the first match, and there is an implicit deny at the end of every ACL — which is why an explicit permit any is needed to allow other hosts.

access-list 1 deny 192.168.10.10
access-list 1 permit any

conf t
interface f1/0
ip access-group 1 in
end

Because the standard ACL denies traffic from 192.168.10.10 specifically, traffic from PC1 is blocked at R2, while traffic from PC2 and PC3 is still permitted by the permit any statement.

Extended ACL — filters by protocol, source, destination, and port. Extended ACLs use number ranges 100–199 (or 2000–2699) and allow much more granular control:

conf t
access-list 100 deny tcp host 192.168.10.10 192.168.20.0 0.0.0.255 eq 80
access-list 100 permit ip any any

This example is an extended ACL because it specifies the protocol (tcp), the source host, the destination network, and the destination port (80/HTTP).

In real-world environments, extended ACLs are often preferred because they allow admins to apply more targeted security policies while minimizing unintended traffic disruption.

flowchart TD
    A[Packet arrives at interface] --> B{ACL applied?}
    B -->|No| Z[Permit - default behavior]
    B -->|Yes| C[Evaluate rule 1 top-down]
    C --> D{Match?}
    D -->|Yes| E[Apply action: permit/deny]
    D -->|No| F{More rules?}
    F -->|Yes| C
    F -->|No| G[Implicit deny all]

1.5 Building an Access Control Strategy

Access control is often thought of as simply blocking bad traffic, but it plays a much larger role: deciding who or what is allowed to communicate, under what conditions, and at which points in the infrastructure. Placing controls too loosely may expose critical services; placing them incorrectly can disrupt legitimate traffic and cause outages. Engineers need a clear mental model of where controls belong — at the network edge, between internal segments, or close to critical resources.

Least privilege is a core concept that applies to networks just as much as to user accounts: allow only what’s necessary for systems and users to perform their intended functions, and nothing more.

• Applies to devices, which should only communicate with systems they actually need.
• Applies to users, who should only access the resources required for their role.
• Applies to network paths and services, ensuring traffic is only permitted where specifically required.

A common way to enforce this is a default-deny mindset: instead of allowing all traffic and blocking specific threats, block everything by default and only allow explicitly approved communication. This significantly reduces the potential impact if a system is compromised.

ACLs are a foundational tool for fast, simple filtering based on IP addresses, ports, and protocols — ideal for basic segmentation, limiting exposure, and enforcing simple policies. They are simple and predictable, providing a clear, rule-based approach to traffic control that is easy to audit and maintain.

ACLs have important limitations, though: they do not understand who the user is, provide limited visibility into what traffic actually contains, and can become difficult to manage as networks grow. This is where firewalls take over.

Simply put: ACLs control traffic, while firewalls understand traffic.

Effective security also requires balancing enforcement and visibility:

• Enforcement controls behavior — blocking unwanted traffic, restricting access, and enforcing policy through ACLs, firewalls, and other mechanisms.
• Visibility is understanding what is actually happening in the network — logs, alerts, and traffic patterns that provide insight into activity and potential security events.

Without visibility, troubleshooting incidents becomes very difficult. Without enforcement, even the best monitoring will not stop attacks. Effective security requires both.

flowchart LR
    subgraph Enforcement
        ACL[ACLs]
        FW[Firewalls]
        POL[Policy Controls]
    end
    subgraph Visibility
        LOG[Logs]
        ALERT[Alerts]
        TRAFFIC[Traffic Patterns]
    end
    Enforcement -->|blocks/allows| Traffic((Network Traffic))
    Traffic -->|generates| Visibility
    Visibility -->|informs tuning of| Enforcement

1.6 Demo: Centralized Authentication with RADIUS and TACACS+

The lab topology places a RADIUS server and router R1 (192.168.100.1) inside the 192.168.100.0/24 internal network, with R1 acting as the gateway to external/cloud networks. An administrator connects via SSH from outside; the router forwards authentication requests to the RADIUS server, which validates credentials before granting access — combining centralized authentication with secure remote management.

flowchart LR
    ADMIN["Admin (external)"] -->|SSH| R1["Router R1<br/>192.168.100.1"]
    R1 <-->|RADIUS 1812/1813<br/>TACACS+ TCP/49| AAA["AAA Server<br/>192.168.100.10"]
    R1 --- LAN[192.168.100.0/24]

Both a RADIUS server and a TACACS+ server run at 192.168.100.10 in this lab. The router already has a basic configuration (hostname, management interface, SSH).

Step 1 — Enable the AAA subsystem. Without this, the device relies only on traditional login/enable passwords.

enable
configure terminal
aaa new-model

Step 2 — Create a local backup account. If both RADIUS and TACACS+ become unreachable, the admin can still log in locally and recover the system.

username backup privilege 15 secret StrongBackupPass123!

Step 3 — Define the RADIUS server and group. RADIUS uses port 1812 for authentication and 1813 for accounting. The shared secret must match exactly what is configured on the RADIUS server. Server groups let multiple servers be referenced together (and support redundancy).

radius server RADIUS-SERVER
 address ipv4 192.168.100.10 auth-port 1812 acct-port 1813
 key SharedRadiusSecret123

aaa group server radius RADIUS-GROUP
 server name RADIUS-SERVER

Step 4 — Define the TACACS+ server and group. TACACS+ typically operates over TCP port 49 and provides strong separation between authentication, authorization, and accounting.

tacacs server TACACS-SERVER
 address ipv4 192.168.100.10
 key SharedTacacsSecret123

aaa group server tacacs+ TACACS-GROUP
 server name TACACS-SERVER

Step 5 — Configure the login authentication method list. The router attempts RADIUS first, then TACACS+, then the local database as a final fallback. This layered approach ensures both centralized authentication and local emergency access.

aaa authentication login default group RADIUS-GROUP group TACACS-GROUP local

Step 6 — Configure exec authorization. The RADIUS/TACACS+ server can return attributes such as privilege levels, letting admins control permissions centrally.

aaa authorization exec default group RADIUS-GROUP group TACACS-GROUP local

Step 7 — Enable accounting. Accounting records session start/stop information for auditing, compliance, and security monitoring.

aaa accounting exec default start-stop group RADIUS-GROUP group TACACS-GROUP

Step 8 — Apply AAA to VTY and console lines, and save the configuration.

line vty 0 4
 login authentication default
line console 0
 login authentication default
exit
copy running-config startup-config

At this point, the router authenticates users via RADIUS first, then TACACS+, and finally the local backup account — providing centralized security management with reliable fallback access.

sequenceDiagram
    participant Admin
    participant Router as Router R1
    participant RADIUS
    participant TACACS as TACACS+
    participant Local as Local DB

    Admin->>Router: SSH login attempt
    Router->>RADIUS: Authenticate (port 1812)
    alt RADIUS reachable & valid
        RADIUS-->>Router: Accept + attributes
    else RADIUS unreachable
        Router->>TACACS: Authenticate (TCP/49)
        alt TACACS+ reachable & valid
            TACACS-->>Router: Accept + attributes
        else TACACS+ unreachable
            Router->>Local: Authenticate against local DB
            Local-->>Router: Accept (backup account)
        end
    end
    Router-->>Admin: Access granted, session logged (accounting)

1.7 Identity as the Control Plane

Traditionally, network security was built around location: inside the network meant trusted, outside meant untrusted. Modern environments no longer work that way — users access systems from many locations (remote work, cloud platforms, mobile devices, third-party networks), so location alone is no longer a reliable indicator of trust. Today, identity has become the real control plane: access decisions are based on authenticated users, devices, and roles, applying controls according to permissions and context rather than network boundaries. This is the shift from perimeter-based models toward identity-driven, zero-trust architectures.

When identity is used as the foundation for access control, organizations can apply:

• Role-based access — permissions based on job function rather than network location.
• Least privilege at the user level — each user gets only the access they actually need, especially important for admin access.
• Precise auditing — tying privileges to specific identities and roles makes administrative actions safer and easier to audit.

Auditing and forensics are far more effective when actions can be tied to identity. The first question in any investigation is “who did what and when?” — when logs are tied to identities, investigators can build accurate incident timelines and quickly focus on the accounts involved, instead of searching through ambiguous IP-based data that may be shared through NAT or reused over time.

Identity-based security also reduces the impact of human error:

• Role-based access limits the potential damage of a mistake, since users only have permissions required for their role.
• Temporary/just-in-time access provides elevated privileges only when needed, automatically removing them afterward, reducing long-term risk of unused or forgotten privileges.
• Central authentication simplifies revocation — disabling one account in one place, instead of updating multiple systems individually, when a user leaves.

flowchart TD
    A[Perimeter-Based Trust<br/>Location = Trust] -->|Users now remote/mobile/cloud| B[Identity-Driven / Zero-Trust]
    B --> C[Authenticate the user]
    B --> D[Authenticate the device]
    B --> E[Evaluate role & context]
    C & D & E --> F[Access Decision]
    F --> G[Role-Based Access]
    F --> H[Least Privilege]
    F --> I[Auditable, Identity-Tagged Logs]

1.8 Demo: Layered Firewall Defense with pfSense and Suricata

This demo uses pfSense to configure basic firewall rules, then extends the setup with intrusion detection and prevention using Suricata.

Basic firewall rules on the WAN interface:

After adding the rules, clicking Apply Changes activates the updated firewall policy. Returning to Firewall → Rules, rule counters increase over time as traffic reaches the firewall and is processed by the configured rules — hovering over a counter highlights the specific rule handling that traffic. The States table shows source IP, destination IP, and connection state (for example, ESTABLISHED for an active tracked session).

Adding intrusion detection (IDS) with Suricata:

1. In pfSense, navigate to Package Manager and install Suricata (an open-source IDS that analyzes traffic and generates alerts on suspicious activity).
2. From another terminal, run an Nmap port scan against the server to generate test traffic:

nmap -sS <target-ip>

3. In pfSense’s Suricata Alerts tab, multiple alerts appear, indicating detection of behavior consistent with network scanning or potential intrusion attempts. Without IDS monitoring, it would be far more difficult to know tools such as Nmap were probing the network.

Enabling intrusion prevention (IPS) mode:

1. On the WAN interface settings in Suricata, enable IPS mode. IPS mode uses inline inspection — Suricata sits between the network interface and the OS, so it can not only detect suspicious traffic but actively block it.
2. Save the configuration, then switch to the LAN interface settings and add custom detection rules (one rule per line) designed to detect and drop traffic generated by tools such as PuTTY and curl.
3. Generate additional test traffic and check the Suricata Alerts page for the LAN interface — the custom rules trigger and alerts are generated, confirming Suricata detected the activity and applied the configured rules.

flowchart TB
    subgraph pfSense Firewall
        direction TB
        R1["Rule: Allow TCP/80"]
        R2["Rule: Allow TCP/443"]
        R3["Rule: Allow ICMP"]
        R4["Rule: Block TCP/22"]
    end
    WAN((WAN Traffic)) --> pfSense
    pfSense --> IDS[Suricata IDS: Detect & Alert]
    IDS -->|IPS Mode enabled| IPS[Suricata IPS: Inline Block]
    IPS --> LAN[(Protected LAN)]

This demo shows three layered capabilities working together: basic firewall filtering, intrusion detection (Suricata monitors and alerts on suspicious activity), and intrusion prevention (Suricata operates inline to actively block malicious traffic before it reaches the system) — a layered approach to network security monitoring and protection.

Module 2: Attacks, Failures, and Defensive Visibility

2.1 Distinguishing Attacks from Misconfigurations

In modern networks, disruptions come from many sources, including both malicious attacks and simple misconfigurations. Not every outage is caused by an attacker — often the root cause is an operational mistake or an incorrect configuration change. At the same time, not every attack is dramatic or obvious; some attackers are subtle, and their activity can look very similar to normal system issues. Engineers need the ability to distinguish real security signals from everyday operational noise.

From a user’s perspective, most problems look similar regardless of cause: “the internet is down,” extremely slow applications, timeouts, authentication failures, or intermittent connectivity. From an engineer’s perspective, technical indicators appear instead: packet loss, high latency, CPU spikes on devices under unusual load, routing instability, or DNS resolution failures.

Visibility is the deciding factor when troubleshooting network or security issues:

• Logs show when something happened, which system was involved, and sometimes which user/process triggered it.
• Packet captures show the actual traffic moving across the network — communication patterns, abnormal behavior, and how systems interact.
• Monitoring tools show trends over time — rising latency, error rates, or unusual activity spikes.

Without visibility, guessing replaces diagnosing, recovery takes longer, and engineers lack the evidence needed for confident decisions. Systems should be designed with observation in mind from the start: reliable logging, critical alerting, and clear health/performance indicators. Resilience requires observation before you can even see it.

flowchart TD
    Symptom[Reported Symptom] --> Q{Sudden config change<br/>or deployment?}
    Q -->|Yes, correlates in time| MISCONFIG[Likely Misconfiguration]
    Q -->|No obvious change| Q2{Unusual traffic/ARP/DNS<br/>patterns present?}
    Q2 -->|Yes| ATTACK[Possible Attack]
    Q2 -->|No| INVESTIGATE[Investigate further:<br/>logs, packet captures, monitoring]
    MISCONFIG --> ROLLBACK[Roll back / correct config]
    ATTACK --> RESPOND[Security investigation & containment]
    INVESTIGATE --> MISCONFIG
    INVESTIGATE --> ATTACK

2.2 Demo: Simulating Denial-of-Service Attacks

This demo simulates a denial-of-service attack from an attacker machine against a web server, observing how different traffic types impact system performance and availability.

Step 1 — ICMP flood via ping:

ping -f <target-ip>

The -f (flood) option sends ICMP echo requests as fast as possible instead of normal spaced-out pings, consuming network bandwidth and putting pressure on the target’s network stack.

Step 2 — Repeated HTTP requests via curl (application-layer):

while true; do curl -s -o /dev/null http://<target-ip>/; done

Unlike the ping flood, this is application-layer traffic. It forces the web server to process each request, which can consume CPU and memory over time.

Step 3 — More aggressive ICMP flood using hping3:

hping3 --icmp --flood <target-ip>

hping3 provides more flexibility and higher packet rates than standard ping, increasing the intensity of the network-level flood.

Step 4 — UDP flood targeting port 80:

hping3 --udp -p 80 --flood <target-ip>

Even though web servers typically expect TCP on port 80, the sheer volume of UDP traffic still consumes bandwidth and processing resources.

At this point, checking the web server with htop shows relatively normal, stable resource usage — the server can still handle this load.

Step 5 — TCP SYN flood (targeted attack):

hping3 -S -p 80 --flood <target-ip>

SYN floods are particularly effective because they exploit the TCP three-way handshake, forcing the server to allocate resources for connections that are never fully established. Checking htop on the server now shows a dramatic CPU increase and heavy resource strain from a large number of half-open connections. Stopping the attack shows resource utilization drop again, confirming how closely server performance is tied to incoming traffic volume and type.

Step 6 — Legitimate high-volume load with Apache Bench:

ab -n 10000 -c 200 http://<target-ip>/

This sends 10,000 HTTP requests with 200 concurrent users — a high but realistic load scenario. Even without malicious intent, high traffic can stress a system that is not properly scaled or optimized, causing slow responses and dropped connections.

Step 7 — Combined attack: run the ICMP flood and the Apache Bench benchmark simultaneously. The benchmarking results now show failures, including connection reset by peer — the server can no longer reliably handle incoming requests. This is what real users experience during a DoS condition: failed connections, slow load times, or complete inability to access the service.

sequenceDiagram
    participant Attacker
    participant Server
    Attacker->>Server: ping -f (ICMP flood)
    Attacker->>Server: curl loop (HTTP flood)
    Attacker->>Server: hping3 --icmp --flood
    Attacker->>Server: hping3 --udp -p 80 --flood
    Note over Server: htop shows stable load so far
    Attacker->>Server: hping3 -S -p 80 --flood (SYN flood)
    Note over Server: CPU spikes, half-open connections pile up
    Attacker--xServer: Stop attack
    Note over Server: Resource usage drops
    Attacker->>Server: ab -n 10000 -c 200 (legit load)
    Attacker->>Server: ICMP flood + ab simultaneously
    Server-->>Attacker: connection reset by peer (service degraded)

This demonstrates why resilience strategies such as rate limiting, load balancing, and traffic filtering are critical in production environments.

2.3 Availability as a Security Objective

Security is often associated with protecting confidential data, but confidentiality is only one part of the picture — availability is just as important. If users cannot access systems when needed, those systems have effectively failed regardless of how secure the data might be.

The three core CIA principles:

Organizations often focus heavily on confidentiality, sometimes overlooking availability and reliability — but a secure system must balance all three.

Factors that threaten availability:

• Distributed denial-of-service (DDoS) attacks — large traffic volumes overwhelm a system.
• Resource exhaustion — systems run out of CPU, memory, storage, or network capacity.
• Misconfigurations — a simple mistake during deployment or maintenance disrupts services.
• Architectural issues — a single point of failure can take an entire service offline.
• Poor capacity planning — systems fail when demand exceeds what the infrastructure was designed for.

Controls that protect availability:

DoS attacks remain highly relevant because they are relatively easy to perform — attackers can use widely available tools or compromised devices to overwhelm a target without needing deep access to the environment. The goal is simply to prevent legitimate users from accessing the system, directly targeting business continuity and potentially causing reputation damage and financial loss. DoS attacks are also frequently used as a distraction while attackers attempt other exploits or unauthorized access elsewhere in the environment.

Availability requires deliberate design: assume systems will experience high traffic (legitimate or malicious) and build infrastructure to handle it; limit abuse with rate limiting and request validation; contain problems so they don’t cascade; and monitor capacity to detect stress early. Resilience means systems continue functioning even under pressure or unexpected demand.

mindmap
  root((Availability<br/>Threats))
    DDoS
      Volumetric floods
      Application-layer floods
    Resource Exhaustion
      CPU
      Memory
      Storage
      Network capacity
    Misconfiguration
      Deployment mistakes
      Maintenance errors
    Architecture
      Single points of failure
      Poor capacity planning

2.4 Demo: ARP Poisoning and VLAN Hopping

ARP poisoning man-in-the-middle attack:

1. On the attacker machine, run ifconfig to identify the attacker’s interface, IP, and MAC address — establishing position on the network before launching an interception attack.
2. Open Ettercap (a common MITM tool) and scan for hosts, actively probing the local subnet to build a list of potential targets.
3. Select Target 1 as the victim machine, and Target 2 as 192.168.11.254 (the default gateway).
4. Launch ARP poisoning MITM in Ettercap. The attacker sends forged ARP messages to both the victim and the gateway, tricking each into believing the attacker’s MAC address corresponds to the other party’s IP — so traffic that should flow directly between victim and gateway is now routed through the attacker.
5. Access a test page that prompts for username/password over basic authentication.
6. On the target machine, Wireshark shows unusual ARP activity: repeated ARP replies that don’t correspond to normal behavior — a strong indicator of ARP poisoning in progress. From the victim’s perspective, everything appears normal.
7. Back on the attacker machine, Ettercap shows the captured username and password — because the attacker sits in the middle of the communication, they can intercept and read the transmitted data.

sequenceDiagram
    participant Victim
    participant Attacker
    participant Gateway as Gateway (192.168.11.254)

    Attacker->>Victim: Forged ARP reply: "I am the Gateway"
    Attacker->>Gateway: Forged ARP reply: "I am the Victim"
    Victim->>Attacker: Traffic intended for Gateway (unknowingly)
    Attacker->>Gateway: Relayed traffic
    Gateway-->>Attacker: Response
    Attacker-->>Victim: Relayed response
    Note over Attacker: Captures credentials in transit (basic auth)

VLAN hopping via double tagging:

1. On a target machine, identify interface details with ifconfig (IP and MAC for ens33), confirming which interface to monitor.
2. Start capturing ICMP traffic with tcpdump:

tcpdump -i ens33 icmp

3. From an attacker machine (Kali Linux), run a prepared script (vlan_hop_basic) that crafts a custom Ethernet frame with two VLAN tags: an outer tag representing the native VLAN, and an inner tag representing the target VLAN. In misconfigured networks, switches may strip the outer tag when forwarding, leaving the inner tag to determine final delivery. The frame encapsulates a standard ICMP echo request, making it look like a normal ping but with manipulated VLAN tagging.
4. Run the script to send a small number of crafted packets onto the network.
5. Back on the target machine, tcpdump shows ICMP request/reply traffic arriving from a source that should not normally have access to this VLAN — the packet crossed VLAN boundaries via double tagging. The outer tag was stripped by the switch, and the inner tag delivered the packet into the target’s VLAN.

flowchart LR
    A[Attacker crafts frame] --> B["Outer tag = Native VLAN<br/>Inner tag = Target VLAN"]
    B --> C[Switch strips outer/native tag]
    C --> D["Frame delivered using<br/>inner tag = Target VLAN"]
    D --> E[Packet crosses VLAN boundary]

In a well-configured network, disabling unnecessary trunking, avoiding use of the native VLAN for user traffic, and enforcing strict VLAN policies would prevent this type of behavior. This highlights how Layer 2 misconfigurations can break network segmentation and allow unintended communication between isolated networks.

2.5 Why Layer 2 Is Still Dangerous

Many engineers focus heavily on protecting the network perimeter with firewalls and external defenses. While important, this creates a false sense of security — internal protections are often weaker, and traffic is more trusted by default. If an attacker gains internal access, they may move laterally, intercept traffic, or manipulate network behavior. The biggest risks often exist inside the boundary, not just at the edge.

Traditional network security assumed that a strong perimeter firewall would protect against external threats, and anything inside the boundary could generally be trusted. Modern environments have shown this assumption no longer holds: endpoints can be compromised through malware or phishing, insider threats can cause harm intentionally or unintentionally, and rogue/unauthorized devices can appear on internal networks. Trust based purely on network location is outdated.

At Layer 2, several design assumptions create risk:

• ARP was designed with a basic trust assumption — devices generally accept ARP responses without verifying authenticity, making address-mapping manipulation possible.
• Broadcast domains mean devices communicate by sending to many systems in the same segment, typically without built-in authentication verifying device identity.
• Within a VLAN, traffic flows relatively freely between connected systems. While efficient, this also lets attackers who gain access move laterally or intercept traffic.

Layer 2 was designed to enable connectivity and performance, not security.

North-south vs. east-west traffic:

Once an attacker gains internal access, east-west traffic becomes extremely important — attackers move laterally from system to system seeking higher privileges or more valuable data. In flat networks where many systems communicate freely, lateral movement is much easier, and a single compromised device can lead to widespread damage.

Segmentation controls, from coarse to granular:

flowchart TD
    A[Flat Network<br/>No Segmentation] --> B[VLAN Segmentation]
    B --> C[Inter-VLAN Access Controls]
    C --> D[Internal Firewalls<br/>Stronger Inspection Between Zones]
    D --> E[Micro-Segmentation<br/>Per-Workload/Application Policies]
    style A fill:#f66
    style E fill:#6f6

• VLAN segmentation places different types of systems into separate VLANs, reducing unnecessary communication.
• Inter-VLAN access controls regulate how traffic moves between segments, allowing only required services.
• Internal firewalls enforce stronger inspection and policy between network zones.
• Micro-segmentation applies security policies at a much smaller level — sometimes down to individual workloads or applications — restricting communication precisely to contain incidents.

Modern network design should assume internal compromise is possible, reduce unnecessary internal access, control internal trust, and monitor east-west traffic, since unusual communication between internal systems is often an early stage of lateral movement.

2.6 Demo: DNS Spoofing and DHCP Misconfiguration

DNS spoofing:

The lab uses dnsmasq as a lightweight DNS server that listens on 127.0.0.2 (so it does not interfere with the system’s default DNS service):

# DNS spoof lab: fake DNS server for demo only
# Listen on 127.0.0.2 so we don't override system DNS
listen-address=127.0.0.2
bind-interfaces
port=53

# Spoof: www.bank.com resolves to our fake web server (127.0.0.1)
address=/www.bank.com/127.0.0.1

# Optional: spoof more domains to the same fake server
# address=/bank.com/127.0.0.1
# address=/secure.bank.com/127.0.0.1

# Don't read /etc/resolv.conf for upstream (we're authoritative for our entries only)
no-resolv
# For any other names, forward to real DNS (optional; comment out for "only spoof" behavior)
server=8.8.8.8
server=8.8.4.4

The system resolver (/etc/resolv.conf) is pointed at the local spoofing server so that queries are sent there and the spoofing takes effect. A startup script runs dnsmasq in the foreground using the custom config:

#!/bin/bash
# Run dnsmasq with our spoof config (listens on 127.0.0.2:53)
cd "$(dirname "$0")"
if ! command -v dnsmasq &>/dev/null; then
  echo "Install dnsmasq: sudo apt install dnsmasq"
  exit 1
fi
echo "Starting DNS spoof server on 127.0.0.2 (www.bank.com -> 127.0.0.1)"
echo "Press Ctrl+C to stop."
sudo dnsmasq -C dnsmasq-spoof.conf --no-daemon

A companion script serves a fake banking page over HTTP on 127.0.0.1:80:

#!/bin/bash
# Serve the fake bank page on 127.0.0.1:80 (requires sudo for port 80)
cd "$(dirname "$0")"
echo "Serving fake bank page at http://127.0.0.1/"
echo "Press Ctrl+C to stop."
if command -v python3 &>/dev/null; then
  sudo python3 -m http.server 80 --bind 127.0.0.1 --directory .
else
  sudo python -m http.server 80 --bind 127.0.0.1 --directory .
fi

With both components running, opening a browser to bank.com displays the fake site hosted locally instead of the legitimate banking website — the domain name appears correct in the address bar, but the content is fully controlled by the attacker. In a real-world scenario, this technique could harvest credentials or deliver malicious content, which is why DNS security controls, HTTPS validation, and user awareness are critical defenses.

sequenceDiagram
    participant Victim
    participant SpoofedDNS as Spoofed DNS (127.0.0.2)
    participant FakeWeb as Fake Web Server (127.0.0.1)
    participant RealBank as Real bank.com

    Victim->>SpoofedDNS: Query: www.bank.com
    SpoofedDNS-->>Victim: A record: 127.0.0.1 (spoofed!)
    Victim->>FakeWeb: HTTP GET / (thinks it's the real bank)
    FakeWeb-->>Victim: Fake login page
    Note over Victim: Enters credentials, unaware
    Note over RealBank: Never contacted

DHCP misconfiguration:

Reviewing the DHCP server configuration reveals a rogue route pointing to 192.168.224.205 — not part of the normal network design, representing a malicious or accidental configuration change. DHCP is not just responsible for assigning IP addresses; it can also distribute additional network parameters, including routes.

After restarting the DHCP service to apply the change, a client is forced to request a fresh configuration:

# On the client
dhclient -r    # release
dhclient       # renew
ip route show  # confirm the rogue route was delivered

The client’s routing table now shows the rogue route, confirming the misconfiguration took effect via DHCP rather than manual configuration. Testing connectivity to an external resource (e.g., a public DNS server) fails because traffic is directed incorrectly by the rogue route instead of following a valid path to the internet.

flowchart TD
    A[DHCP Server Config Modified<br/>Rogue Route Added] --> B[DHCP Service Restarted]
    B --> C[Client releases/renews IP]
    C --> D[Client receives rogue route via DHCP]
    D --> E[Client attempts external connectivity]
    E --> F[Traffic sent to wrong destination]
    F --> G[Connectivity Fails]

This demonstrates how a simple DHCP misconfiguration can disrupt normal network operations — even small infrastructure-level changes can significantly impact availability and security.

2.7 Critical Services as Single Points of Failure

Some services operate quietly in the background and are almost invisible during normal operations — but when they fail, the impact is immediate and widespread. Even relatively small services can trigger large outages because many other systems depend on them.

What makes these failures difficult to diagnose is that symptoms often appear unrelated at first — different teams may see different problems across multiple systems, when the root cause is frequently a single, quiet, widely-depended-upon service.

Protecting critical infrastructure services:

• Remove unnecessary exposure — DNS, DHCP, and NTP should only be accessible to systems that actually need them.
• Tightly restrict management access — only authorized admins should modify configs, with access controlled and monitored.
• Build in redundancy — running multiple servers ensures continuity if one fails; separate services across different systems rather than a single server.
• Strong access controls and segmentation — ensure only trusted systems can interact with these critical services.

Detecting problems early:

• Continuously monitor service availability and response times.
• Configure alerting for abnormal behavior (traffic spikes, repeated failures, unusual activity).
• Track log errors to reveal configuration issues or emerging failures.
• Regularly validate replication and synchronization so backup/distributed services remain consistent and ready to take over.

Reducing dependency risk:

• Identify hidden dependencies — quiet services or components that many applications rely on without teams always being fully aware of them.
• Avoid single points of failure by building in redundancy so another system can take over.

mindmap
  root((Critical<br/>Services))
    DNS
      Name resolution
      Failure: "internet is down"
    DHCP
      IP assignment
      Failure: new devices can't connect
    NTP
      Time sync
      Failure: auth issues, bad log timestamps
    Protection
      Reduce exposure
      Restrict management
      Redundancy
      Segmentation
    Detection
      Availability monitoring
      Alerting
      Log analysis
      Replication validation

True resilience comes from combining several practices together: hardening reduces the risk of compromise, redundancy ensures continuity during failures, and monitoring detects issues early. In many environments, the most important systems are not the most visible ones — resilience depends on protecting the quiet components everything else relies on.

2.8 Demo: Zero Trust Architecture in Practice

This demonstration is not a full enterprise implementation, but clearly illustrates the core zero-trust ideas: per-request verification, identity-based access, device posture checks, and — most importantly — no implicit trust.

Architecture: a user browser sends a request to a reverse proxy. Before the request reaches the protected application, two checks are enforced — identity verification and a device posture check. Only if both pass is access granted.

flowchart LR
    Browser -->|HTTP request| Proxy["NGINX Reverse Proxy :8080"]
    Proxy -->|1: Basic Auth| Identity{Identity Verified?}
    Identity -->|No| Deny1[401 Unauthorized]
    Identity -->|Yes| Posture{"2: auth_request<br/>Device Posture OK?"}
    Posture -->|No: 403| Deny2[403 Forbidden]
    Posture -->|Yes: 200| App["Protected App :9000"]

Step 1 — Install components: NGINX, Apache utilities (for basic-auth password files), and curl.

Step 2 — Create a simple protected web application representing a sensitive internal service (conceptually an admin panel, database interface, or internal API) that should never be directly exposed without controls. It is served with a simple Python HTTP server on port 9000 and has no built-in authentication — it relies entirely on the reverse proxy to protect it.

python3 -m http.server 9000

Step 3 — Configure NGINX as the zero-trust enforcement point. This reverse-proxy configuration is where enforcement begins — two checks (identity, then device posture) run before any request is forwarded to the application:

server {
    listen 8080;

    location / {
        # 1. Identity check
        auth_basic "Zero Trust Login";
        auth_basic_user_file /etc/nginx/.htpasswd;

        # 2. Device posture check (must return 200 to allow)
        auth_request /devicecheck;

        proxy_pass http://127.0.0.1:9000;
    }

    # Internal endpoint: only Nginx calls this
    location = /devicecheck {
        internal;
        proxy_pass http://127.0.0.1:7000/devicecheck;
        proxy_pass_request_body off;
        proxy_set_header Content-Length "";
        proxy_set_header X-Original-URI $request_uri;
    }
}

The application itself is never trusted to enforce access — all decisions are made before the request reaches it, which is the core of zero trust.

Step 4 — Create a user to simulate identity (in a real environment this would integrate with an identity provider such as Active Directory, OAuth, or SSO; basic authentication is sufficient here to demonstrate the concept):

htpasswd -c /etc/nginx/.htpasswd admin

Opening http://localhost:8080 immediately prompts for credentials — identity verification happening at the proxy layer, on every request, with no implicit trust even for a local service.

Step 5 — Implement the device posture check. A script checks whether the host firewall is active:

#!/bin/bash

if ufw status | grep -q active
then
  exit 0
else
  exit 1
fi

This is exposed as an HTTP endpoint via a small Flask server listening on 127.0.0.1:7000, which NGINX calls before allowing access:

#!/usr/bin/env python3
"""
Zero Trust Demo: Device Posture Check Server
Returns 200 if UFW firewall is active, 403 otherwise.
Run: python3 posture_server.py
Listens on 127.0.0.1:7000
"""

from flask import Flask, Response
import subprocess

app = Flask(__name__)

def firewall_active():
    """Check if UFW firewall is enabled (status: active)."""
    try:
        out = subprocess.check_output(["ufw", "status"], text=True, timeout=2)
        return "Status: active" in out
    except (subprocess.CalledProcessError, FileNotFoundError, subprocess.TimeoutExpired):
        return False

@app.route("/devicecheck", methods=["GET"])
def device_check():
    if firewall_active():
        return Response("OK", status=200)
    return Response("Device posture failed: firewall not active", status=403)

if __name__ == "__main__":
    app.run(host="127.0.0.1", port=7000)

If the firewall is active, the endpoint returns HTTP 200; if not, HTTP 403 — meaning access is no longer based only on identity, but also on the condition of the device.

Step 6 — Test the behavior:

1. Disable the firewall (sudo ufw disable), refresh the browser: access is denied with a 403 Forbidden, even with valid credentials, because the device no longer meets the required security posture.
2. Re-enable the firewall (sudo ufw enable), refresh again: access is restored.

Access is not granted based on location or network — every request is evaluated dynamically: identity is verified, device posture is checked, and only then is access allowed. There is no implicit trust, only continuous verification.

sequenceDiagram
    participant Browser
    participant Nginx as NGINX (Zero Trust Enforcement)
    participant Posture as Posture Server :7000
    participant App as Protected App :9000

    Browser->>Nginx: GET / (no credentials)
    Nginx-->>Browser: 401 Unauthorized
    Browser->>Nginx: GET / (with credentials)
    Nginx->>Posture: auth_request /devicecheck
    alt Firewall active
        Posture-->>Nginx: 200 OK
        Nginx->>App: proxy_pass request
        App-->>Nginx: Response
        Nginx-->>Browser: 200 OK (content)
    else Firewall inactive
        Posture-->>Nginx: 403 Forbidden
        Nginx-->>Browser: 403 Forbidden (access denied)
    end

Module 3: Troubleshooting, Recovery, and Operational Resilience

3.1 The Six-Step Troubleshooting Methodology

Network failures are a normal part of operations — no environment runs perfectly all the time. The difference between panic and progress is having a structured approach. When engineers follow a clear process, they move step-by-step toward the root cause instead of reacting randomly, staying calm, gathering the right information, and resolving problems faster.

One common mistake during outages is jumping straight into solutions without first understanding the problem. Making random changes can create more risk — when multiple things change at the same time, it becomes much harder to identify the real cause, and guessing can even make the outage worse.

The six steps:

flowchart TD
    S1["1. Identify the problem<br/>(gather info, understand user impact, define scope)"] --> S2
    S2["2. Establish a theory of<br/>probable cause"] --> S3
    S3["3. Test the theory"] --> S3D{Confirmed?}
    S3D -->|No| S2
    S3D -->|Yes| S4["4. Establish a plan of action<br/>(minimize disruption)"]
    S4 --> S5["5. Implement the solution<br/>and monitor results"]
    S5 --> S6["6. Verify full resolution<br/>and document the fix"]

1. Identify the problem — gather information, understand what users are experiencing, and define the scope of the issue.
2. Establish a theory of probable cause — based on available evidence, form a reasonable explanation.
3. Test the theory — determine whether it actually explains the symptoms; if not, return to step 2.
4. Establish a plan of action to resolve the issue while minimizing disruption.
5. Implement the solution carefully and monitor the system to confirm the fix works as expected.
6. Verify the problem is fully resolved and document the fix so the team can learn from the incident and respond faster in the future.

Mapping symptoms to network layers helps narrow down where an issue is occurring:

Engineers must also remain security aware while troubleshooting — not every failure is a technical fault. Sudden traffic floods could indicate a DDoS attack rather than normal usage; unexpected ARP behavior might suggest ARP spoofing or MITM activity; DNS anomalies (unusual queries or incorrect responses) could indicate manipulation or redirection. Because these symptoms can look like normal operational issues, engineers should always consider both possibilities — misconfiguration/failure and potential malicious activity — without assuming a single cause too quickly.

3.2 Demo: Applying the Six-Step Process to a Name Resolution Failure

This walkthrough diagnoses a connectivity issue to a web server named websrv, systematically applying the six-step process.

Step 1 — Identify the problem, starting at the application layer. Check whether the service is responding, since this is fast to test and helps quickly determine if the issue is even network-related:

curl -I http://websrv

The -I flag fetches only HTTP headers, making the command fast and easy to interpret. A healthy response looks like HTTP/1.1 200 OK; in this case, the request fails.

Step 2 — Establish a theory: isolate variables by bypassing the hostname. Connect directly using the server’s IP address to determine whether the issue is DNS/name-resolution related:

SERVER_IP="172.16.130.128"
curl -I http://$SERVER_IP

If this succeeds and returns valid headers, it’s an important clue: the web server is up, the service is responding correctly, and the network path is functional — narrowing the problem down to how the hostname websrv is being resolved.

Step 3 — Test the theory: check name resolution explicitly.

getent hosts websrv

If this returns nothing, it confirms the system cannot resolve websrv to an IP address — strong evidence of a DNS/local name-resolution issue. To rule out any underlying network problem first, validate basic connectivity:

ping -c 2 websrv
ping -c 2 $SERVER_IP

Pinging the hostname likely fails (expected, since name resolution isn’t working), while pinging the IP directly succeeds — confirming the network path works and there are no routing issues, reinforcing that the problem is name resolution, not connectivity.

Step 4/5 — Plan and implement a fix. A quick local fix is a static entry in the hosts file (not a scalable/permanent solution like proper DNS, but extremely useful for testing, temporary fixes, or isolated environments):

sudo sed -i.bak '/[[:space:]]websrv$/d' /etc/hosts
echo "$SERVER_IP websrv" | sudo tee -a /etc/hosts

Step 6 — Verify the resolution.

getent hosts websrv
curl -I http://websrv

Name resolution should now return the IP address, and the application should now return a valid HTTP 200 OK response.

sequenceDiagram
    participant Eng as Engineer
    participant App as curl (app layer)
    participant DNS as Name Resolution
    participant Net as Network (ping)
    participant Hosts as /etc/hosts

    Eng->>App: curl -I http://websrv
    App-->>Eng: FAILS
    Eng->>App: curl -I http://$SERVER_IP
    App-->>Eng: 200 OK (server & network fine)
    Eng->>DNS: getent hosts websrv
    DNS-->>Eng: (empty - resolution fails)
    Eng->>Net: ping websrv / ping $SERVER_IP
    Net-->>Eng: hostname fails, IP succeeds
    Eng->>Hosts: Add static websrv entry
    Eng->>DNS: getent hosts websrv (re-check)
    DNS-->>Eng: Returns IP
    Eng->>App: curl -I http://websrv (re-check)
    App-->>Eng: 200 OK

By following a structured approach — starting at the application layer, isolating DNS from connectivity, validating network reachability, identifying root cause, and verifying the fix — this method avoids guesswork and ensures each step provides meaningful information. Over time, this systematic thinking becomes second nature and significantly speeds up troubleshooting.

3.3 Demo: CLI Tools and Packet Analysis

This demo uses ping and traceroute to check reachability and understand traffic paths, then validates the findings in Wireshark across Layers 2–4.

Ping sends ICMP echo request messages to a destination; a reachable host responds with ICMP echo reply messages.

ping -c 4 -n <SERVER_IP>

Two key things to look for: whether replies are received, and the round-trip time (RTT) for each response — RTT indicates latency between source and destination.

Traceroute maps the path packets take to a destination by manipulating the TTL (time to live) value in the IP header. TTL starts low and increases with each set of probes; when a packet’s TTL reaches zero, the router handling it discards the packet and sends back an ICMP time-exceeded message. Each response reveals one hop along the path.

traceroute -n -m 6 -q 1 <SERVER_IP>

The result is a hop-by-hop list of routers between source and destination — verifiable in Wireshark by observing the ICMP time-exceeded messages from intermediate routers.

Correlating CLI output with packet-level detail:

As you observe multiple packets in Wireshark, each router decrements the TTL by one; note that Ethernet headers only reflect the local hop (MAC addresses change every segment), unlike IP addresses which stay consistent end to end. With traceroute, you’ll see both outgoing probe packets and the returning ICMP time-exceeded messages — these responses are what make hop discovery possible.

Wireshark display filters:

icmp
icmp or udp

Start with icmp to isolate ICMP traffic, then expand to icmp or udp to include traceroute probes if UDP is being used. Wireshark provides detailed packet-level evidence across Layers 2 through 4.

sequenceDiagram
    participant Host
    participant R1 as Hop 1
    participant R2 as Hop 2
    participant Dest as Destination

    Host->>R1: Probe, TTL=1
    R1-->>Host: ICMP Time Exceeded
    Host->>R2: Probe, TTL=2
    R2-->>Host: ICMP Time Exceeded
    Host->>Dest: Probe, TTL=3
    Dest-->>Host: ICMP Echo Reply / Port Unreachable
    Note over Host,Dest: Each hop revealed one TTL-expiry at a time

3.4 Reading the Network Like a Story

Modern networks constantly generate signals — every device, service, and connection produces logs, metrics, and traffic patterns reflecting what’s happening inside the environment. Troubleshooting is not only about running commands or checking configurations; it’s also about interpreting those signals. Experienced engineers learn to “read the network” almost like a story — small details form patterns that reveal the underlying cause of an issue.

Establishing a baseline is important: in a healthy environment, most systems operate in predictable patterns. Regular DNS queries, stable latency/packet flow without sudden spikes, and consistent authentication activity (e.g., logins during normal working hours) are examples of “normal.” Monitoring tools continuously collect metrics and logs to define this baseline, so that once established, any significant deviation quickly stands out.

Distinguishing failure indicators from attack indicators:

Recognizing these differences helps engineers determine whether they are dealing with a routine failure or a potential security incident.

Sources of evidence:

• Logs capture detailed records: authentication attempts (who tried to access, success/denial), firewall logs (which connections were allowed/blocked), and configuration changes (what was modified and by whom). Combined, they create a timeline of activity, allowing reconstruction of events and root cause identification, and providing accountability.
• Packet captures provide one of the most accurate views of what’s actually happening — protocol behavior in detail, traffic patterns (spikes, unexpected connections, abnormal communication), and communication failures (dropped packets, retransmissions, incomplete handshakes).

flowchart LR
    subgraph Evidence Sources
        LOGS[Logs: auth, firewall, config changes]
        PCAP[Packet Captures]
        MON[Monitoring Dashboards]
    end
    LOGS --> TIMELINE[Reconstructed Timeline]
    PCAP --> TIMELINE
    MON --> BASELINE[Established Baseline]
    BASELINE --> DEVIATION{Deviation Detected?}
    DEVIATION -->|Failure pattern| FAIL[Operational Issue]
    DEVIATION -->|Attack pattern| SEC[Security Incident]
    TIMELINE --> FAIL
    TIMELINE --> SEC

3.5 Demo: Troubleshooting OSPF Routing Adjacencies

On each machine (e.g., serverb and desktop), run the following setup checks.

Step 1 — Confirm the interface is up and correctly configured:

ip -br addr
ip link show dev ens160

Step 2 — Configure OSPF on FRR (using vtysh):

sudo vtysh <<'EOF'
configure terminal
router ospf
 ospf router-id DESKTOP_ROUTER_ID
 network OSPFSUBNET area 0
exit
interface IF_OSPF
 ip ospf hello-interval 2
 ip ospf dead-interval 8
exit
end
write memory
EOF

Step 3 — Verify the OSPF neighbor relationship with a detailed view from the FRR routing stack:

show ip ospf neighbor detail

In a healthy state, the output shows one OSPF neighbor with router ID 1.1.1.1, reachable at 172.16.130.130, while the local interface ens160 has IP 192.16.130.129 — confirming the two devices are directly connected and communicating over the same subnet.

Key fields to examine:

A State: Full, elected DR/BDR, an actively resetting dead timer, and an empty exchange list together mean: OSPF peering is up, stable, and fully converged — no missing neighbors, no instability, no synchronization issues.

Simulating a failure and recovery:

ping -c 3 172.16.130.129

# On serverb, bring the interface down
sudo ip link set dev ens160 down

# From the other system, check neighbor status
show ip ospf neighbor detail
# -> no detailed output: OSPF adjacency dropped

# Re-enable the interface
sudo ip link set dev ens160 up

# After a short moment, re-check
show ip ospf neighbor detail
# -> adjacency re-established, network reconverged

stateDiagram-v2
    [*] --> Down
    Down --> Init: Hello received
    Init --> TwoWay: Bidirectional Hello
    TwoWay --> ExStart: DR/BDR election
    ExStart --> Exchange: Master/Slave negotiated
    Exchange --> Loading: DBD exchange
    Loading --> Full: LSAs synchronized
    Full --> Down: Interface down / dead timer expires

3.6 Demo: VLAN Segmentation and Inter-VLAN Routing Issues

In this lab, VLAN 10 is assigned to serverf and VLAN 20 to serverg, both in the 172.16.130.0/24 range but logically isolated via VLAN tagging. Routing between the VLANs is handled by serverg, acting as a router and providing gateway address 172.16.130.1 for VLAN 10.

Baseline connectivity check:

ping -c 3 172.16.130.132

This should succeed, confirming inter-VLAN routing works end to end.

Verify the routing decision — because both VLANs share the same subnet, a host-specific route was added; without it, Linux would assume the destination is directly reachable on the local network:

ip route get 172.16.130.132

The output should show traffic going via 172.16.130.1 on interface ens160.10, confirming traffic is correctly sent to the VLAN 10 gateway rather than attempting direct Layer 2 communication.

Verify Layer 2 isolation — checking ARP/neighbor resolution on serverf shows it resolves the MAC address of its own gateway on VLAN 10, not the destination host directly — confirming VLAN 10 and VLAN 20 remain isolated at Layer 2, requiring routing between them:

ip neigh

Misconfiguration #1 — incorrect next hop. Point the route at an unused IP that doesn’t exist on the network:

sudo ip route replace 172.16.130.132/32 via 172.16.130.88 dev ens160.10
ping -c 3 172.16.130.132
# Fails: cannot resolve next hop, no valid neighbor
sudo ip route replace 172.16.130.132/32 via 172.16.130.1 dev ens160.10
# Restore correct route -> connectivity restored

Misconfiguration #2 — missing IP address on the VLAN 20 interface. Remove the IP address that responds on that network (VLAN tagging itself is unaffected):

sudo ip addr flush dev ens160.20
ping -c 3 172.16.130.132
# Fails: destination host address no longer exists
sudo ip addr add 172.16.130.2/24 dev ens160.20
# Restore address -> connectivity restored

Misconfiguration #3 — ICMP filtering on the router. Since the destination is local to serverg, IP forwarding is not the factor — the traffic is handled by the INPUT chain, not FORWARD. Block ICMP specifically:

sudo iptables -I INPUT 1 -d 172.16.130.132 -p icmp -j DROP
ping -c 3 172.16.130.132
# Fails even though routing and VLAN config are correct — router explicitly drops ICMP
sudo iptables -D INPUT -d 172.16.130.132 -p icmp -j DROP
# Remove rule -> connectivity restored

flowchart TD
    Start[Baseline: Ping Fails] --> Check1{Correct next-hop<br/>route configured?}
    Check1 -->|No| Fix1[Fix route: correct gateway IP]
    Check1 -->|Yes| Check2{Destination interface<br/>has valid IP address?}
    Check2 -->|No| Fix2[Restore IP address on VLAN interface]
    Check2 -->|Yes| Check3{ICMP blocked by<br/>iptables INPUT chain?}
    Check3 -->|Yes| Fix3[Remove blocking iptables rule]
    Check3 -->|No| Other[Investigate further:<br/>ARP, VLAN tagging, physical layer]
    Fix1 --> Verify[Re-test connectivity]
    Fix2 --> Verify
    Fix3 --> Verify

Each misconfiguration affects connectivity in a different way, even though the underlying VLAN design remains the same — an incorrect next hop, a missing IP address on the VLAN interface, and ICMP filtering on the router.

3.7 Demo: NAT and PAT Configuration

NAT (Network Address Translation) allows multiple devices on a private network to share a single public IP address when accessing the internet — used everywhere from home routers to enterprise networks and cloud systems. PAT (Port Address Translation) is a type of NAT that also translates port numbers, allowing many devices behind a router to access the internet simultaneously using a single public IP.

Step 1 — Reset and disable NAT/forwarding, confirming the private network cannot reach the internet without it:

sudo sysctl -w net.ipv4.ip_forward=0
sudo iptables -t nat -F
sudo iptables -F

Step 2 — Test connectivity without NAT from a private namespace simulating an internal machine:

ping -c 2 8.8.8.8

With forwarding disabled and no NAT rules, this ping fails — demonstrating the private network cannot yet reach the internet.

Step 3 — Enable IP forwarding, allowing the system to act as a router:

sudo sysctl -w net.ipv4.ip_forward=1

Step 4 — Add forwarding rules between the private interface (veth-nat0) and the external interface (ens160) — one rule allows return traffic for established connections, the other allows outbound traffic:

sudo iptables -I FORWARD 1 -i ens160 -o veth-nat0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
sudo iptables -I FORWARD 1 -i veth-nat0 -o ens160 -j ACCEPT

Step 5 — Configure NAT with MASQUERADE — the key step for PAT, rewriting the source IP of outgoing packets to match the public interface and tracking port numbers so multiple internal clients can share one public IP:

sudo iptables -t nat -A POSTROUTING -o ens160 -j MASQUERADE

Step 6 — Verify and observe traffic:

curl -I http://example.com
sudo timeout 10 tcpdump -n -i ens160 host 8.8.8.8 &
sudo ip netns exec natlab-priv curl -I http://example.com

The request now succeeds, proving NAT/PAT is working — tcpdump on the external interface confirms the translated traffic leaving/returning via the public interface.

A companion single-host lab script automates the full private-namespace + NAT setup:

#!/usr/bin/env bash
# Single-host NAT/PAT lab: one network namespace ("private") + host ("router").

set -euo pipefail

NETNS_NAME="natlab-priv"
VETH_HOST="veth-nat0"   # host end (iptables PRIVATE_IF)
VETH_NS="veth-nat1"     # inside namespace
# Lab-only subnet — must not overlap your LAN (change if needed)
HOST_VETH_IP="192.168.200.1/24"
NS_IP="192.168.200.10/24"
NS_GW="192.168.200.1"

PACKAGES=(iptables tcpdump curl iproute2)

if [[ "${EUID}" -ne 0 ]]; then
  echo "Run with sudo: sudo $0" >&2
  exit 1
fi

# Interface with default route to the Internet (your "public" / WAN side)
PUBLIC_IF="${PUBLIC_IF:-}"
if [[ -z "$PUBLIC_IF" ]]; then
  PUBLIC_IF="$(ip -4 route show default 2>/dev/null | awk '{print $5}' | head -n1)"
fi
if [[ -z "$PUBLIC_IF" ]] || ! ip link show "$PUBLIC_IF" &>/dev/null; then
  echo "Could not detect PUBLIC_IF. Set it explicitly, e.g.:  sudo PUBLIC_IF=ens192 $0" >&2
  exit 1
fi

export DEBIAN_FRONTEND=noninteractive
apt-get update -qq
apt-get install -y -qq "${PACKAGES[@]}"

# Tear down any previous run
ip netns del "$NETNS_NAME" 2>/dev/null || true
ip link del "$VETH_HOST" 2>/dev/null || true

ip netns add "$NETNS_NAME"
ip link add "$VETH_HOST" type veth peer name "$VETH_NS"
ip link set "$VETH_NS" netns "$NETNS_NAME"

ip addr add "$HOST_VETH_IP" dev "$VETH_HOST"
ip link set "$VETH_HOST" up

ip netns exec "$NETNS_NAME" ip addr add "$NS_IP" dev "$VETH_NS"
ip netns exec "$NETNS_NAME" ip link set lo up
ip netns exec "$NETNS_NAME" ip link set "$VETH_NS" up
ip netns exec "$NETNS_NAME" ip route add default via "$NS_GW"

# DNS for curl to hostnames from inside the namespace (Ubuntu)
mkdir -p /etc/netns/"$NETNS_NAME"
echo "nameserver 8.8.8.8" > /etc/netns/"$NETNS_NAME"/resolv.conf

sysctl -w net.ipv4.ip_forward=1

# Baseline NAT + forwarding for the lab (demo may flush and re-add)
iptables -t nat -F
iptables -F
iptables -P FORWARD ACCEPT 2>/dev/null || true
iptables -I FORWARD 1 -i "$PUBLIC_IF" -o "$VETH_HOST" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -I FORWARD 1 -i "$VETH_HOST" -o "$PUBLIC_IF" -j ACCEPT
iptables -t nat -A POSTROUTING -o "$PUBLIC_IF" -j MASQUERADE

flowchart LR
    subgraph Private["Private Namespace (natlab-priv)"]
        PC["Client 192.168.200.10"]
    end
    subgraph Router["Host acting as Router"]
        VETH["veth-nat0<br/>192.168.200.1"]
        MASQ["iptables MASQUERADE"]
        PUB["Public IF (ens160)"]
    end
    PC -->|"default route via 192.168.200.1"| VETH
    VETH --> MASQ
    MASQ -->|"Source IP/Port rewritten"| PUB
    PUB --> INET((Internet))

In summary: without NAT, private IP addresses cannot reach the internet. Enabling IP forwarding allows routing between interfaces; iptables FORWARD rules control which traffic is permitted; MASQUERADE enables PAT by translating private IPs/ports to the public IP; and tools like tcpdump help verify what’s happening. This is foundational knowledge for working with cloud infrastructure, containers, and network security.

3.8 Demo: Security Policy Logs and Alerting

This demo walks through a compact security-monitoring workflow that simulates SSH attack activity, inspects logs, and applies a simple policy-based alert — a miniature SIEM flow.

Step 1 — Generate a small burst of test events using a script that creates realistic authentication-style log noise from multiple simulated sources (via Linux network namespaces), acting as a lab traffic injector rather than waiting for a real attack:

#!/usr/bin/env bash
set -euo pipefail

# multi_source_log_generator.sh
# Single-host multi-source SSH failure generator using Linux network namespaces.
#
# What it does:
# - Creates a bridge (br-demo) on the host
# - Creates N namespaces, each with a unique source IP
# - From each namespace, runs failed SSH attempts to the host bridge IP
# - SSH logs then show events as if from different sources
#
# Usage:
#   sudo ./multi_source_log_generator.sh up
#   sudo ./multi_source_log_generator.sh generate [attempts_per_source]
#   sudo ./multi_source_log_generator.sh down
#   sudo ./multi_source_log_generator.sh demo [attempts_per_source]

BRIDGE_NAME="br-demo"
SUBNET_CIDR="10.200.1.0/24"
HOST_BRIDGE_IP="10.200.1.1/24"
HOST_TARGET_IP="10.200.1.1"
NS_PREFIX="ns"
SOURCE_COUNT=3
BASE_IP_OCTET=11
DEFAULT_ATTEMPTS=3
INVALID_USER="doesnotexist"

setup() {
  echo "Setting up ${SOURCE_COUNT} source namespaces on ${SUBNET_CIDR}..."
  ensure_bridge
  for i in $(seq 1 "${SOURCE_COUNT}"); do
    create_ns "$i"
  done
}

generate() {
  local attempts="${1:-$DEFAULT_ATTEMPTS}"
  for i in $(seq 1 "${SOURCE_COUNT}"); do
    ns="$(namespace_name "$i")"
    for _ in $(seq 1 "${attempts}"); do
      ip netns exec "${ns}" ssh \
        -o BatchMode=yes -o ConnectTimeout=2 \
        -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
        "${INVALID_USER}@${HOST_TARGET_IP}" 'exit' >/dev/null 2>&1 || true
      sleep 1
    done
  done
  echo "Check evidence:"
  echo "  sudo journalctl -u ssh -t sshd --since \"5 min ago\" --no-pager | tail -n 80"
}

case "${1:-}" in
  up) setup ;;
  generate) generate "${2:-$DEFAULT_ATTEMPTS}" ;;
  down) teardown ;;
  demo) demo "${2:-$DEFAULT_ATTEMPTS}" ;;
esac

Step 2 — Inspect recent SSH daemon logs from systemd-journal, narrowed to the SSH service and the sshd tag, over a short window so failed password attempts, invalid users, or unusual repetition patterns can be spotted immediately:

journalctl -u ssh -t sshd --since "2 min ago" --no-pager | tail -n 40

Step 3 — Widen the time window for trend context. A 2-minute view is great for recency; a 5-minute view helps answer whether this is isolated noise or sustained suspicious behavior:

journalctl -u ssh -t sshd --since "5 min ago" --no-pager

Step 4 — Apply a policy script. The script normalizes SSH failure lines, extracts source IPs, counts events, and evaluates thresholds. It supports two modes: per_source (a single IP must exceed the threshold) and aggregate (all failures in the window are counted together). If the threshold is met, it prints a clear alert plus supporting evidence lines for explainability:

#!/usr/bin/env bash
set -euo pipefail

# policy_ssh_alert.sh
# Purpose (SIEM-style demo):
# - Normalize: extract source IPs from SSH authentication failures in a time window
# - Correlate: count failures per IP
# - Detect: alert when the count meets a policy threshold
# - Evidence: print the underlying matching log lines for explainability
#
# Usage:
#   sudo ./policy_ssh_alert.sh [window_minutes] [threshold] [mode]
# Modes:
#   per_source (default): threshold applies to a single source IP
#   aggregate: threshold applies to total matching events in the window

WINDOW_MIN="${1:-2}"
THRESHOLD="${2:-5}"
MODE="${3:-per_source}"

LOG_JOURNAL_ARGS=(-u ssh -t sshd --since "${WINDOW_MIN} min ago" --no-pager)

# Normalize: extract source IPs from raw evidence lines
mapfile -t ips < <(
  journalctl "${LOG_JOURNAL_ARGS[@]}" \
  | awk '
    /Failed password|Invalid user/ {
      if (match($0, / from ([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)/, a)) print a[1]
    }'
)

total_count="${#ips[@]}"
top_ip=""; top_count=0
while IFS= read -r line; do
  count="$(awk '{print $1}' <<<"$line")"; ip="$(awk '{print $2}' <<<"$line")"
  if [ "$count" -gt "$top_count" ]; then top_count="$count"; top_ip="$ip"; fi
done < <(printf "%s\n" "${ips[@]}" | sort | uniq -c | sort -nr)

if [ "${MODE}" = "per_source" ]; then
  if [ "$top_count" -ge "$THRESHOLD" ]; then
    echo "ALERT (policy): potential SSH brute-force"
    echo "  mode=${MODE} window=${WINDOW_MIN}m threshold=${THRESHOLD}"
    echo "  evidence_ip=${top_ip} event_count=${top_count}"
  else
    echo "No alert: mode=${MODE} top_ip=${top_ip:-none} event_count=${top_count} < threshold=${THRESHOLD}"
  fi
else
  if [ "$total_count" -ge "$THRESHOLD" ]; then
    echo "ALERT (policy): aggregate SSH auth failures in window"
    echo "  mode=${MODE} window=${WINDOW_MIN}m threshold=${THRESHOLD}"
    echo "  total_event_count=${total_count} top_ip=${top_ip} top_ip_count=${top_count}"
  else
    echo "No alert: mode=${MODE} total_event_count=${total_count} < threshold=${THRESHOLD}"
  fi
fi

Step 5 — Generate a larger event set to test alert behavior under stronger signal conditions and verify thresholds are not overly sensitive.

Step 6 — Run in aggregate mode with a moderate threshold — last 5 minutes, alert if total matching SSH failures are 8 or more. Aggregate mode is useful for spotting distributed low-and-slow attempts, where each source alone might stay below the per-IP threshold but total activity is still risky:

sudo ./policy_ssh_alert.sh 5 8 aggregate

Step 7 — Run a stricter aggregate policy — a 10-minute window with threshold 5. The longer window can expose slower attack patterns; the low threshold increases sensitivity. This is where tuning matters: balancing detection coverage against alert fatigue.

sudo ./policy_ssh_alert.sh 10 5 aggregate

Step 8 — Run without an explicit mode (defaults to per_source), checking whether any single IP crosses five events in the last two minutes — useful for direct brute-force signatures from one origin:

sudo ./policy_ssh_alert.sh

If triggered, the alert and the exact matching log lines tied to the offending source are shown.

flowchart LR
    A[Generate SSH auth events<br/>multi_source_log_generator.sh] --> B[journalctl -u ssh -t sshd]
    B --> C[policy_ssh_alert.sh]
    C --> D{Mode?}
    D -->|per_source| E{Single IP >= threshold?}
    D -->|aggregate| F{Total events >= threshold?}
    E -->|Yes| ALERT1[ALERT: brute-force from IP]
    F -->|Yes| ALERT2[ALERT: aggregate SSH failures]
    E -->|No| NOALERT1[No alert]
    F -->|No| NOALERT2[No alert]
    ALERT1 --> EVID[Print evidence log lines]
    ALERT2 --> EVID

This sequence demonstrates a practical mini-SIEM flow: generate data → inspect logs → apply policy in different modes → validate with evidence. The key is not only alerting, but explainable alerting — every detection is backed by log lines that make triage faster.

3.9 Backup and Recovery of Device Configurations

Network devices rely heavily on configuration files to operate correctly — they define how devices handle traffic, communicate with other systems, and enforce security and routing policies. Because of this, even a small config change can significantly impact the entire network. If a device fails, becomes corrupted, or is misconfigured, recovery depends on having reliable backups. With a good backup, engineers can quickly restore a device to a known working state and minimize downtime.

Without backups, recovery is much more difficult — engineers may have to rebuild configurations from memory or documentation, which is slow and risky, and important settings can easily be missed or implemented incorrectly. Common failure examples include a router misconfiguration that breaks routing, a firewall rule mistake that blocks legitimate traffic, or an accidental deletion of configuration settings.

Three core backup operations:

Recovery requires more than a single backup — it requires maintaining multiple, usable versions so teams can restore the most appropriate configuration depending on when the issue occurred.

Designing a reliable backup process:

• Automate configuration backups whenever possible, reducing the risk of human error and ensuring recent configurations are always available.
• Store backup copies off-device — if a device fails completely or is compromised, an external copy ensures the configuration can still be restored.
• Maintain a version history so engineers can roll back to a known-good configuration from a specific point in time, especially useful if a recent change caused an issue.
• Protect backup files themselves — access control should limit who can view or modify backups, preventing unauthorized changes or deletion.

flowchart LR
    A[Working Configuration] -->|Automated, scheduled| B[Save / Backup]
    B --> C[(Off-device Storage<br/>Version History)]
    D[Failure / Misconfiguration Detected] --> E{Recent change<br/>caused it?}
    E -->|Yes| F[Rollback to previous version]
    E -->|No, device failed| G[Restore known-good backup]
    F --> H[Verify Service Restored]
    G --> H
    C -.provides versions to.-> F
    C -.provides versions to.-> G

Operational resilience starts with preparation:

• Quick restoration of services limits operational impact, keeping downtime and disruption to a minimum.
• Clear documentation and well-defined procedures guide engineers confidently and efficiently through recovery.
• Tracking changes and version histories helps engineers know exactly what was modified and when, making troubleshooting faster and more accurate.
• Regularly practicing restoration procedures allows teams to respond confidently during real incidents.

In short: the more prepared you are, the faster you can recover, minimizing downtime and operational impact.

Summary

Building resilient infrastructure means treating security, availability, and resilience as inseparable design goals rather than competing priorities. Most outages trace back to weak foundations — default credentials, unnecessary exposure, flat networks, and unhardened devices — not sophisticated attackers. The course’s core principles:

• Harden before you monitor. Reducing the attack surface (disabling unused services/interfaces, enforcing named accounts and SSH-only management, restricting management-plane access with ACLs, centralizing AAA via RADIUS/TACACS+) removes the easy wins attackers rely on.
• Identity beats IP. As users and devices move across networks, clouds, and VPNs, identity — not source IP — is the stable, auditable control plane for access decisions, logging, and forensics.
• Layer your defenses. ACLs, stateful firewalls, IDS/IPS (Suricata), and zero-trust reverse-proxy enforcement (identity + device posture) each add a layer that the others don’t cover alone.
• Assume Layer 2 and internal traffic are not automatically trustworthy. ARP spoofing, VLAN hopping, and unrestricted east-west traffic are common lateral-movement paths; segmentation and micro-segmentation contain the blast radius.
• Availability is a security objective, not just an operations concern — rate limiting, redundancy, load balancing, and segmentation defend against both DoS attacks and simple resource exhaustion.
• Critical background services (DNS, DHCP, NTP) are single points of failure hiding in plain sight — protect, replicate, and monitor them deliberately.
• Troubleshoot with a structured, six-step method, mapping symptoms to network layers, and stay security-aware — a “failure” and an “attack” can look identical from the user’s perspective.
• Visibility (logs, packet captures, monitoring) and enforcement (ACLs, firewalls, policy) are both required — neither alone is sufficient.
• Resilience requires rehearsed recovery: automated, versioned, access-controlled configuration backups with practiced restore/rollback procedures turn a potential outage into a brief, well-understood interruption.

Troubleshooting Quick-Reference Checklist

Search Terms

resilient · infrastructure · security · troubleshooting · networking · fundamentals · systems · attacks · control · access · availability · failure · network · recovery · resilience · routing · six-step · vlan