AWS Storage Fundamentals

Table of Contents

1. Module 1 — Understanding Block Storage
   • What is Block Storage?
   • Block Storage Use Cases
   • EBS Snapshots and Data Lifecycle Manager
   • S3 Versioning
2. Module 2 — Exploring Elastic File System
   • Understanding EFS
   • Demo: Creating a First EFS Volume
   • Demo: Mounting EFS on an EC2 Instance
3. Module 3 — Delving into Object Storage
   • Introduction to Object Storage
   • S3 Tiering Options
   • S3 Intelligent-Tiering in Detail
   • S3 Express One Zone
   • Glacier: Retrieval Options
   • S3 Pricing
4. Module 4 — Configuring Simple Storage Service
   • S3 Static Website Hosting
   • Demo: Implementing an S3 Static Website
   • S3 Lifecycle Rules
   • Demo: Configuring S3 Bucket Replication
5. Complementary Services
   • AWS Backup
   • AWS Storage Gateway
6. boto3 Examples — Python SDK
7. AWS Storage Services Comparison
8. Summary and Key Points

Module 1 — Understanding Block Storage

What is Block Storage?

Block storage is the most familiar type of storage for a system administrator, as it corresponds to how a traditional hard drive works. In AWS, the corresponding service is EBS (Elastic Block Storage).

EBS volumes are presented from arrays managed by AWS and exposed to EC2 instances as virtual disks. Multiple EBS volumes can be attached to the same EC2 instance, and in some cases an io1/io2 volume can be shared between multiple instances via EBS Multi-Attach.

How Block Storage Works

flowchart LR
    subgraph Client
        User[User\nmodifies a file]
    end
    subgraph Storage
        B1[Block 1]
        B2[Block 2 updated]
        B3[Block 3]
        B4[Block 4 updated]
        B5[Block 5]
    end
    User -- "Initial save\n(all blocks)" --> B1
    User -- "Update\n(only modified blocks)" --> B2
    User -- "Update\n(only modified blocks)" --> B4

Key principle: On an update, only the modified blocks are rewritten — not the entire file. This is the main advantage of block storage for I/O performance.

EBS Architecture with EC2

flowchart TB
    subgraph AZ_A["Availability Zone A"]
        EC2[Primary EC2 Instance]
        EBS1[EBS Volume\ngp3 — OS/Boot]
        EBS2[EBS Volume\nio2 — Database]
        EC2 --- EBS1
        EC2 --- EBS2
    end
    subgraph AZ_B["Availability Zone B"]
        EC2B[Secondary EC2 Instance\nio2 Multi-Attach]
    end
    subgraph Replication
        R["Automatic replication\nwithin the AZ"]
    end
    EBS1 -.-> R
    EBS2 -.-> R
    EBS2 -. "EBS Multi-Attach\n(io1/io2 only)" .-> EC2B

Key EBS Characteristics

Block Storage Use Cases

EBS Volume Types

flowchart TB
    EBS[EBS Volume Types]
    SSD[SSD-based\nFor transactional workloads]
    HDD[HDD-based\nFor throughput-intensive workloads]
    EBS --> SSD
    EBS --> HDD
    SSD --> io2BE["io2 Block Express\n256,000 IOPS — 4,000 MB/s\nSub-ms latency — 99.999%"]
    SSD --> io1["io1\n64,000 IOPS — 1,000 MB/s\n99.9% durability"]
    SSD --> GP3["gp3 RECOMMENDED\n80,000 IOPS — 2,000 MB/s\n20% cheaper than gp2"]
    SSD --> GP2["gp2 legacy\n16,000 IOPS — 250 MB/s\nAvoid for new projects"]
    HDD --> ST1["st1 Throughput Optimized\n500 MB/s — Big Data"]
    HDD --> SC1["sc1 Cold HDD\n250 MB/s — Low-cost archiving"]

Complete EBS Volume Comparison Table (official AWS data)

Note gp3: Includes 3,000 IOPS and 125 MB/s of baseline for free, regardless of size. Additional IOPS cost $0.005/IOPS-month.

Note io2 Block Express: Average latency below 500 microseconds for 16 KiB I/Os. Available only on Nitro-based EC2 instances. Ratio up to 1,000 IOPS/GB.

gp3 vs gp2 Advantage — IOPS / Size Decoupling

With gp2, IOPS are directly tied to volume size (3 IOPS/GB, min 100). For 10,000 IOPS, you need a 3,333 GB volume at $0.10/GB = $333/month.

With gp3, IOPS and throughput are configured independently of size:

• 100 GB at $0.08 = $8 + 7,000 additional IOPS × $0.005 = $35 → $43/month total for the same 10,000 IOPS

AWS CLI Commands — EBS

# Create a gp3 EBS volume with custom IOPS and throughput
aws ec2 create-volume \
  --volume-type gp3 \
  --size 100 \
  --availability-zone us-east-1a \
  --iops 6000 \
  --throughput 500 \
  --tag-specifications 'ResourceType=volume,Tags=[{Key=Name,Value=db-data}]'

# Create a high-performance io2 volume
aws ec2 create-volume \
  --volume-type io2 \
  --size 500 \
  --availability-zone us-east-1a \
  --iops 50000

# List available EBS volumes
aws ec2 describe-volumes \
  --filters "Name=status,Values=available"

# Attach a volume to an EC2 instance
aws ec2 attach-volume \
  --volume-id vol-0123456789abcdef0 \
  --instance-id i-0123456789abcdef0 \
  --device /dev/sdf

# Modify an existing volume in-place (e.g. migrate gp2 -> gp3 with no downtime)
aws ec2 modify-volume \
  --volume-id vol-0123456789abcdef0 \
  --volume-type gp3 \
  --iops 3000 \
  --throughput 125

# Enable EBS encryption by default for the region
aws ec2 enable-ebs-encryption-by-default \
  --region us-east-1

EBS Snapshots and Data Lifecycle Manager

How Snapshots Work

EBS snapshots are incremental backups stored in S3 managed by AWS (transparent to the user):

flowchart LR
    EBS[EBS Volume\n100 GB used]
    S3["(AWS internal S3\nSnapshot storage)"]
    EBS -- "Snapshot 1\n100 GB captured (initial)" --> S3
    EBS -- "Snapshot 2\n10 GB modified only" --> S3
    EBS -- "Snapshot 3\n5 GB modified only" --> S3
    S3 -- "Full restore\nto new volume\n(any AZ/region)" --> EBS2[New EBS Volume]

Important snapshot facts:

• First snapshot: captures all used blocks
• Subsequent snapshots: capture only the modified blocks since the last snapshot
• Can create volumes in other regions (DR)
• Can be shared between AWS accounts or made public
• Compressed storage to minimize costs

# Create a snapshot with tags
aws ec2 create-snapshot \
  --volume-id vol-0123456789abcdef0 \
  --description "Snapshot before migration" \
  --tag-specifications 'ResourceType=snapshot,Tags=[{Key=Name,Value=pre-migration}]'

# Copy a snapshot to another region (DR)
aws ec2 copy-snapshot \
  --source-region us-east-1 \
  --source-snapshot-id snap-0123456789abcdef0 \
  --destination-region eu-west-1 \
  --description "DR Copy London"

# Create a volume from a snapshot
aws ec2 create-volume \
  --snapshot-id snap-0123456789abcdef0 \
  --availability-zone us-east-1b \
  --volume-type gp3

Amazon Data Lifecycle Manager (DLM)

DLM automates the creation, retention, and deletion of EBS snapshots, based on resource tags:

# DLM policy — daily snapshot with 7-day retention
aws dlm create-lifecycle-policy \
  --description "Daily EBS Snapshots" \
  --state ENABLED \
  --execution-role-arn arn:aws:iam::123456789012:role/AWSDataLifecycleManagerDefaultRole \
  --policy-details '{
    "PolicyType": "EBS_SNAPSHOT_MANAGEMENT",
    "ResourceTypes": ["VOLUME"],
    "TargetTags": [{"Key": "Backup", "Value": "true"}],
    "Schedules": [{
      "Name": "Daily",
      "CreateRule": {
        "Interval": 24,
        "IntervalUnit": "HOURS",
        "Times": ["03:00"]
      },
      "RetainRule": {"Count": 7},
      "CopyTags": true
    }]
  }'

S3 Versioning

S3 versioning preserves multiple versions of the same object in a bucket. Once enabled, it can only be suspended (not disabled).

flowchart TB
    subgraph bucket["S3 Bucket with versioning enabled"]
        v1["report.pdf — Version AAA\n(first upload)"]
        v2["report.pdf — Version BBB\n(modification)"]
        v3["report.pdf — Version CCC\n(latest)"]
    end
    upload1["Initial upload"] --> v1
    upload2["Re-upload modification"] --> v2
    upload3["Re-upload correction"] --> v3
    v3 -- "GET without version ID\nalways returns the latest" --> latest[Latest version]
    v1 -- "GET ?versionId=AAA\nreturns the historical version" --> old[Old version]

# Enable versioning
aws s3api put-bucket-versioning \
  --bucket my-bucket \
  --versioning-configuration Status=Enabled

# List versions of an object
aws s3api list-object-versions \
  --bucket my-bucket \
  --prefix report.pdf

# Retrieve a specific version
aws s3api get-object \
  --bucket my-bucket \
  --key report.pdf \
  --version-id AAA123xyz \
  report-v1.pdf

Versioning use cases:

• Protection against accidental deletions or overwrites
• Mandatory prerequisite for S3 Bucket Replication
• Audit trail for object changes
• Combined with lifecycle rules to manage old versions (NoncurrentVersionExpiration)

Module 2 — Exploring Elastic File System

Understanding EFS

EFS (Elastic File System) is a serverless file storage service provided by AWS. Unlike EBS (attached to a single instance), EFS can be simultaneously mounted by thousands of Linux instances in the same region.

EBS vs EFS vs FSx Comparison

Note: For Windows file storage, AWS offers Amazon FSx for Windows File Server (out of scope for this course).

Regional EFS Architecture

flowchart TB
    subgraph Region["AWS Region (e.g. us-east-1)"]
        subgraph AZ_A["Availability Zone A"]
            EC2_A[EC2 Instance A]
            MT_A[Mount Target A\nENI with private IP]
        end
        subgraph AZ_B["Availability Zone B"]
            EC2_B[EC2 Instance B]
            MT_B[Mount Target B\nENI with private IP]
        end
        subgraph AZ_C["Availability Zone C"]
            EC2_C[EC2 Instance C]
            MT_C[Mount Target C\nENI with private IP]
        end
        EFS["(EFS Volume\nData replicated\nacross 3 AZs\nAutomatic scaling)"]
    end
    EC2_A -- "NFS port 2049" --> MT_A --> EFS
    EC2_B -- "NFS port 2049" --> MT_B --> EFS
    EC2_C -- "NFS port 2049" --> MT_C --> EFS

Important rule: For a regional EFS, create a mount target in each AZ where you have instances. For a One Zone EFS, a single mount target in the same AZ is sufficient.

EFS Configuration Options

Throughput modes:

EFS Storage classes:

EFS automatically moves files via lifecycle management rules configured on the volume. You cannot manually choose the class for a file (unlike S3).

Redundancy options:

Demo: Creating a First EFS Volume

Prerequisites

1. Two Amazon Linux 2023 EC2 instances in two different AZs
2. A webservers security group with SSH (port 22) allowed
3. Note the IPv4 CIDR of your default VPC (e.g. 172.31.0.0/16)

Creation Steps

1. Create the EC2 instances:

# Instance in AZ-A
aws ec2 run-instances \
  --image-id ami-0c02fb55956c7d316 \
  --instance-type t2.micro \
  --subnet-id subnet-xxxx1a \
  --security-group-ids sg-xxxxxxxx \
  --associate-public-ip-address \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=web-aza}]'

# Instance in AZ-B
aws ec2 run-instances \
  --image-id ami-0c02fb55956c7d316 \
  --instance-type t2.micro \
  --subnet-id subnet-xxxx1b \
  --security-group-ids sg-xxxxxxxx \
  --associate-public-ip-address \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=web-azb}]'

2. Allow NFS (port 2049) from the VPC in the security group:

aws ec2 authorize-security-group-ingress \
  --group-id sg-xxxxxxxx \
  --protocol tcp \
  --port 2049 \
  --cidr 172.31.0.0/16

3. Create the EFS volume:

EFS_ID=$(aws efs create-file-system \
  --performance-mode generalPurpose \
  --throughput-mode elastic \
  --encrypted \
  --tags Key=Name,Value=my-first-efs \
  --query 'FileSystemId' --output text)

echo "EFS ID: $EFS_ID"

# Create mount targets in each subnet
aws efs create-mount-target \
  --file-system-id $EFS_ID \
  --subnet-id subnet-xxxx1a \
  --security-groups sg-xxxxxxxx

aws efs create-mount-target \
  --file-system-id $EFS_ID \
  --subnet-id subnet-xxxx1b \
  --security-groups sg-xxxxxxxx

Demo: Mounting EFS on an EC2 Instance

Installing EFS Utilities

# On EC2-A AND EC2-B via EC2 Instance Connect
sudo yum install -y amazon-efs-utils

Mounting the EFS Volume

# Create the mount directory
sudo mkdir -p /mnt/efs

# Mount with TLS (recommended in-transit encryption)
sudo mount -t efs -o tls fs-0123456789abcdef0:/ /mnt/efs

# Verify the mount
df -h /mnt/efs

Persistent Mount via fstab

echo "fs-0123456789abcdef0:/ /mnt/efs efs defaults,_netdev,tls 0 0" | sudo tee -a /etc/fstab

File Sharing Test Between Instances

# On EC2-A: create a file
echo "Hello from EC2-A in AZ-A" | sudo tee /mnt/efs/test.txt

# On EC2-B: read the same file (available immediately)
cat /mnt/efs/test.txt
# Output: Hello from EC2-A in AZ-A

# On EC2-B: append content
echo "Reply from EC2-B in AZ-B" | sudo tee -a /mnt/efs/test.txt

This test demonstrates the shared and synchronous nature of EFS: both instances see the same files in real time.

Configure EFS Lifecycle Rules

# Move to IA after 30 days, to Archive after 90 days
aws efs put-lifecycle-configuration \
  --file-system-id fs-0123456789abcdef0 \
  --lifecycle-policies \
    TransitionToIA=AFTER_30_DAYS \
    TransitionToArchive=AFTER_90_DAYS \
    TransitionToPrimaryStorageClass=AFTER_1_ACCESS

Module 3 — Delving into Object Storage

Introduction to Object Storage

Object storage is fundamentally different from block storage and file storage. In AWS, the primary object storage service is S3 (Simple Storage Service).

Comparison of the Three Storage Types

flowchart TB
    subgraph Block["Block Storage EBS"]
        F1["File split into blocks\n(low-level structure)"]
        U1["Update: only\nmodified blocks\nVery efficient for DBs"]
    end
    subgraph File["File Storage EFS"]
        F2["Files in\na folder hierarchy\n(POSIX NFS system)"]
        U2["Update: standard\nfile operations"]
    end
    subgraph Object["Object Storage S3"]
        O1["Object = flat file\nstored in a bucket\n(no real hierarchy)"]
        U3["Update: the ENTIRE\nfile is rewritten\n(immutable object)"]
    end

Key differences with S3:

• Objects stored as flat files with a unique key
• On an update, the entire file is rewritten (unlike block storage)
• Accessible via HTTP/HTTPS URLs and REST APIs
• Each object has associated metadata (system + user)

Why Use S3?

S3 Tiering Options

Unlike EFS (automatic class management), with S3 you choose the storage class at upload time, or configure lifecycle rules to automate transitions.

Decision Tree — Which Storage Class to Choose?

flowchart TB
    Start[What type of data access?]

    Start -->|"Frequent daily access"| Standard[S3 Standard\nGeneral purpose]
    Start -->|"Unknown or variable access"| IT[S3 Intelligent-Tiering\nAutomatic optimization]
    Start -->|"Max performance ms latency"| EZ[S3 Express One Zone\nSingle-digit ms ML/Analytics]
    Start -->|"Rare monthly access"| IA{Need\nmulti-AZ resilience?}
    Start -->|"Archive a few times per year"| Archive{Acceptable\nretrieval delay?}

    IA -->|"Yes"| StandardIA[S3 Standard-IA\nMilliseconds]
    IA -->|"No recreatable data"| OneZoneIA[S3 One Zone-IA\n20% cheaper]

    Archive -->|"Milliseconds"| GIR[Glacier Instant Retrieval]
    Archive -->|"Minutes to hours"| GFR[Glacier Flexible Retrieval]
    Archive -->|"12-48 hours ok"| GDA[Glacier Deep Archive]

    style Standard fill:#2ecc71,color:#000
    style IT fill:#3498db,color:#fff
    style EZ fill:#9b59b6,color:#fff
    style StandardIA fill:#e67e22,color:#000
    style OneZoneIA fill:#f39c12,color:#000
    style GIR fill:#95a5a6,color:#000
    style GFR fill:#7f8c8d,color:#000
    style GDA fill:#2c3e50,color:#fff

Complete S3 Storage Class Comparison Table

One Zone: susceptible to the loss of an entire AZ.

S3 Intelligent-Tiering in Detail

S3 Intelligent-Tiering monitors access patterns and automatically moves objects to the most cost-effective tier, with no performance impact and no retrieval fees.

flowchart LR
    IT["S3 Intelligent-Tiering\n(continuous monitoring)"]

    IT --> FA["Frequent Access Tier\nSame cost as S3 Standard\nAccess within last 30 days"]
    IT --> IA_IT["Infrequent Access Tier\n40% cheaper\n30 days without access"]
    IT --> AIA["Archive Instant Access Tier\n68% cheaper\n90 days without access"]
    IT -.->|"Optional\nexplicit configuration"| AA["Archive Access Tier\nGlacier Flexible Retrieval\n180 days without access"]
    IT -.->|"Optional\nexplicit configuration"| DAA["Deep Archive Access Tier\n95% cheaper vs Standard\n= Glacier Deep Archive"]

    IA_IT -- "Access detected\nautomatically moves up" --> FA
    AIA -- "Access detected\nautomatically moves up" --> FA
    AA -- "RestoreObject required" --> FA
    DAA -- "RestoreObject required" --> FA

Intelligent-Tiering billing rules:

• Monitoring fee per monitored object per month (very low)
• Objects < 128 KB: always billed at Frequent Access rate (no monitoring fee)
• No retrieval fee for automatic tiers (Frequent, Infrequent, Archive Instant)
• Archive Access and Deep Archive Access tiers require explicit restoration

# Upload directly to Intelligent-Tiering
aws s3 cp my-file.dat s3://my-bucket/ \
  --storage-class INTELLIGENT_TIERING

# Configure optional archive tiers
aws s3api put-bucket-intelligent-tiering-configuration \
  --bucket my-bucket \
  --id my-it-config \
  --intelligent-tiering-configuration '{
    "Id": "my-it-config",
    "Status": "Enabled",
    "Tierings": [
      {"AccessTier": "ARCHIVE_ACCESS", "Days": 180},
      {"AccessTier": "DEEP_ARCHIVE_ACCESS", "Days": 730}
    ]
  }'

S3 Express One Zone

S3 Express One Zone is a high-performance storage class for the most latency-demanding workloads:

• Single-digit milliseconds latency — up to 10x faster than S3 Standard
• Request cost reduced by up to 80% vs S3 Standard
• Data in a “directory” bucket type (different from general purpose bucket)
• Scales to 2 million requests per second
• AWS integrations: SageMaker Model Training, Athena, EMR, Glue Data Catalog

Limitation: No lifecycle transitions from S3 Express One Zone.

Glacier: Retrieval Options

Detailed Comparison of Retrieval Options

# Restore from Glacier Flexible Retrieval — Expedited (1-5 min)
aws s3api restore-object \
  --bucket my-glacier-bucket \
  --key archive/report-2020.pdf \
  --restore-request '{"Days": 7, "GlacierJobParameters": {"Tier": "Expedited"}}'

# Restore in Bulk (free, 5-12h)
aws s3api restore-object \
  --bucket my-glacier-bucket \
  --key archive/report-2020.pdf \
  --restore-request '{"Days": 3, "GlacierJobParameters": {"Tier": "Bulk"}}'

# Check restoration status
aws s3api head-object \
  --bucket my-glacier-bucket \
  --key archive/report-2020.pdf \
  --query 'Restore'

S3 Pricing

The main cost factors to consider when designing an S3 architecture:

mindmap
  root((S3 Costs))
    Storage Class
      Standard
      Standard-IA
      One Zone-IA
      Intelligent-Tiering
      Glacier
    Data Transfer
      IN free
      OUT billed
      Inter-region billed
    Requests
      PUT COPY POST LIST
      GET SELECT DELETE
    Data Retrieval
      Standard free
      IA billed per GB
      Glacier billed per GB
    Replication
      Source and destination storage
      Inter-region transfer
    Intelligent-Tiering
      Monitoring fee per object

Key Pricing Points

• Data Transfer IN (to S3): always free
• Data Transfer OUT (from S3 to Internet): $0.09/GB for the first TB, decreasing after
• Requests: each API operation (PUT, GET, LIST, etc.) is billed separately
• Data Retrieval: accessing data in Standard-IA or Glacier generates additional per-GB fees
• Replication: storage in both source and destination buckets is billed, plus inter-region transfer
• Minimum duration: if you delete a Standard-IA object (30-day minimum), you are still billed 30 days

Approximate Storage Prices (us-east-1, USD)

Module 4 — Configuring Simple Storage Service

S3 Static Website Hosting

S3 Static Website Hosting allows hosting static websites directly from an S3 bucket, without EC2 or any compute service.

S3 Static Website Architecture

flowchart LR
    User[User\nInternet]
    R53[Route 53\nCustom domain]
    CF[CloudFront CDN\nHTTPS plus ACM certificate]
    S3[S3 Bucket\nStatic Website\nHTTP only]

    User -- "HTTP" --> S3
    User -- "HTTPS recommended" --> CF
    CF -- "HTTP internal" --> S3
    R53 -- "DNS" --> CF

Characteristics and Limitations

Demo: Implementing an S3 Static Website

Bucket policy for public read access (example_bucket_policy.json)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PublicReadGetObject",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::YOUR-BUCKET-NAME/*"
    }
  ]
}

Replace YOUR-BUCKET-NAME with your bucket name before applying this policy.

Complete Configuration Steps

1. Create the bucket and disable Block Public Access:

BUCKET_NAME="myfirsts3staticwebsite-$(openssl rand -hex 4)"

aws s3api create-bucket --bucket $BUCKET_NAME --region us-east-1

aws s3api put-public-access-block \
  --bucket $BUCKET_NAME \
  --public-access-block-configuration \
  "BlockPublicAcls=false,IgnorePublicAcls=false,BlockPublicPolicy=false,RestrictPublicBuckets=false"

2. Enable Static Website Hosting and upload files:

aws s3 website s3://$BUCKET_NAME/ \
  --index-document index.html \
  --error-document error.html

aws s3 cp index.html s3://$BUCKET_NAME/

3. Apply the bucket policy:

sed "s/YOUR-BUCKET-NAME/$BUCKET_NAME/g" example_bucket_policy.json > policy-applied.json

aws s3api put-bucket-policy \
  --bucket $BUCKET_NAME \
  --policy file://policy-applied.json

4. Website URL:

http://<BUCKET_NAME>.s3-website-us-east-1.amazonaws.com

S3 Lifecycle Rules

S3 Lifecycle Rules automate the movement of objects between storage classes and/or their deletion based on their age.

Types of Actions

Illustrated Lifecycle (course example)

gantt
    title S3 Object Lifecycle — Course Example
    dateFormat  X
    axisFormat  Day %s

    section S3 Tiers
    S3 Standard                  :active, s1, 0, 30
    S3 Standard-IA               :active, s2, 30, 60
    Glacier Instant Retrieval    :active, s3, 60, 150
    Expires and deleted          :done,   s4, 150, 151

1. Day 0: Object created → S3 Standard
2. Day 30: Automatic transition → S3 Standard-IA
3. Day 60: Automatic transition → Glacier Instant Retrieval
4. Day 150 (90 days in Glacier): Expiration — object deleted

Complete JSON Configuration of a Lifecycle Policy

{
  "Rules": [
    {
      "ID": "ArchiveAndExpireLogs",
      "Status": "Enabled",
      "Filter": {"Prefix": "logs/"},
      "Transitions": [
        {"Days": 30, "StorageClass": "STANDARD_IA"},
        {"Days": 60, "StorageClass": "GLACIER_IR"},
        {"Days": 365, "StorageClass": "DEEP_ARCHIVE"}
      ],
      "Expiration": {"Days": 2555},
      "NoncurrentVersionTransitions": [
        {"NoncurrentDays": 30, "StorageClass": "STANDARD_IA"}
      ],
      "NoncurrentVersionExpiration": {"NoncurrentDays": 90}
    }
  ]
}

# Apply the configuration
aws s3api put-bucket-lifecycle-configuration \
  --bucket my-bucket \
  --lifecycle-configuration file://lifecycle.json

# Verify the configuration
aws s3api get-bucket-lifecycle-configuration --bucket my-bucket

Storage Class Constraints — Billing Considerations

Warning: Deletion before the minimum duration = you are still billed for the full minimum period.

Valid Transition Order (one-way — colder only)

flowchart LR
    STD[S3 Standard] --> SIT[Intelligent-Tiering]
    STD --> SIA[Standard-IA]
    STD --> OZIA[One Zone-IA]
    STD --> GIR[Glacier Instant]
    STD --> GFR[Glacier Flexible]
    STD --> GDA[Glacier Deep Archive]
    SIA --> GIR
    SIA --> GFR
    SIA --> GDA
    GIR --> GFR
    GIR --> GDA
    GFR --> GDA

Demo: Configuring S3 Bucket Replication

S3 Bucket Replication automatically copies new objects from a source bucket to a destination bucket, typically in another region (CRR — Cross-Region Replication).

Cross-Region Replication Architecture

flowchart LR
    subgraph Source["Source Region us-east-1 N. Virginia"]
        SB[S3 Source Bucket\nnvir-source-bucket-xxx\nVersioning enabled]
    end
    subgraph Dest["Destination Region eu-west-2 London"]
        DB[S3 Destination Bucket\nlon-dest-bucket-xxx\nVersioning enabled]
    end
    IAM[IAM Role\nS3 ReplicateObject\nReplicateDelete] -.-> SB
    SB -- "Automatic replication\nnew objects only" --> DB

Prerequisites: Versioning enabled on both buckets + IAM Role with replication permissions.

Configuration Steps

1. Create buckets and enable versioning:

# Source bucket
aws s3api create-bucket --bucket nvir-source-bucket-abc123 --region us-east-1

# Destination bucket
aws s3api create-bucket \
  --bucket lon-dest-bucket-abc123 \
  --region eu-west-2 \
  --create-bucket-configuration LocationConstraint=eu-west-2

# Versioning on both buckets
for bucket in nvir-source-bucket-abc123 lon-dest-bucket-abc123; do
  aws s3api put-bucket-versioning \
    --bucket $bucket \
    --versioning-configuration Status=Enabled
done

2. JSON replication rule:

{
  "Role": "arn:aws:iam::123456789012:role/s3-replication-role",
  "Rules": [
    {
      "ID": "ReplicateAll",
      "Status": "Enabled",
      "Filter": {"Prefix": ""},
      "Destination": {
        "Bucket": "arn:aws:s3:::lon-dest-bucket-abc123",
        "StorageClass": "STANDARD"
      },
      "DeleteMarkerReplication": {"Status": "Enabled"}
    }
  ]
}

aws s3api put-bucket-replication \
  --bucket nvir-source-bucket-abc123 \
  --replication-configuration file://replication.json

3. Test replication:

for i in 1 2 3; do
  echo "Hello, this is file $i" > file${i}.txt
  aws s3 cp file${i}.txt s3://nvir-source-bucket-abc123/
done

# Check after a few seconds
aws s3 ls s3://lon-dest-bucket-abc123/ --region eu-west-2

Important points:

• Existing objects are not automatically replicated → use S3 Batch Operations
• Useful for DR, geographic compliance, latency reduction for distant users
• Possible to replicate to multiple destinations

Complementary Services

AWS Backup

AWS Backup is a fully managed service that centralizes and automates data protection across all AWS services and hybrid workloads.

AWS Backup Architecture

flowchart TB
    subgraph Services["Services protected by AWS Backup"]
        EBS2[EBS Volumes]
        EFS2[EFS]
        FSx[Amazon FSx]
        S3B[S3 Buckets]
        RDS[RDS / Aurora]
        DDB[DynamoDB]
        EC2I[EC2 Instances]
        SGW[Storage Gateway]
        VMW[VMware on-premises]
    end

    subgraph Backup["AWS Backup"]
        BP[Backup Plans\nscheduled policies]
        BV[Backup Vault\nKMS encrypted storage]
        BAM[Backup Audit Manager\ncompliance and reporting]
        VL[Vault Lock WORM\nbackup immutability]
    end

    Services --> BP --> BV
    BV --> BAM
    BV --> VL
    BV -- "Cross-Region Copy\nfor DR" --> BV2[Vault Region B]
    BV -- "Cross-Account Copy\nmaximum isolation" --> BV3[Vault Account B]

AWS Backup Key Features

AWS CLI Backup Plan Example

# Create a backup plan with cross-region copy
aws backup create-backup-plan \
  --backup-plan '{
    "BackupPlanName": "DailyBackupPlan",
    "Rules": [
      {
        "RuleName": "DailyBackups",
        "TargetBackupVaultName": "Default",
        "ScheduleExpression": "cron(0 3 * * ? *)",
        "StartWindowMinutes": 60,
        "CompletionWindowMinutes": 180,
        "Lifecycle": {
          "MoveToColdStorageAfterDays": 30,
          "DeleteAfterDays": 365
        },
        "CopyActions": [
          {
            "DestinationBackupVaultArn": "arn:aws:backup:eu-west-1:123456789012:backup-vault:DR-Vault",
            "Lifecycle": {"DeleteAfterDays": 365}
          }
        ]
      }
    ]
  }'

# Assign resources by tag
aws backup create-backup-selection \
  --backup-plan-id <plan-id> \
  --backup-selection '{
    "SelectionName": "TaggedResources",
    "IamRoleArn": "arn:aws:iam::123456789012:role/AWSBackupDefaultServiceRole",
    "ListOfTags": [
      {"ConditionType": "STRINGEQUALS", "ConditionKey": "Backup", "ConditionValue": "true"}
    ]
  }'

AWS Storage Gateway

AWS Storage Gateway is a hybrid storage service that connects on-premises environments to AWS, allowing existing applications to access cloud storage without modification.

The Four Gateway Types

flowchart TB
    SG[AWS Storage Gateway\nHybrid on-premises to cloud service]

    SG --> S3FG["S3 File Gateway\nNFS and SMB protocols\nNative S3 object backend\nUse case: file backup,\nML and Analytics workloads"]
    SG --> FSxFG["FSx File Gateway\nSMB protocol\nFSx for Windows backend\nUse case: Active Directory,\nlocal Windows file shares"]
    SG --> TG["Tape Gateway\niSCSI-VTL protocol\nS3 and Glacier backend\nUse case: replacement\nof physical tapes"]
    SG --> VG["Volume Gateway\niSCSI protocol\nEBS Snapshots backend\nUse case: on-premises\nserver backup and DR"]

Gateway Types Comparison

Volume Gateway — Two Modes

Storage Gateway Deployment

AWS Storage Gateway can be deployed as:

• On-premises VM (VMware ESXi, Microsoft Hyper-V, Linux KVM)
• Hardware appliance from AWS (for environments without virtualization)
• AMI in Amazon EC2 (cloud extension to other services)
• VM in VMware Cloud on AWS

boto3 Examples — Python SDK

S3 Operations with boto3

import boto3
from botocore.exceptions import ClientError

s3 = boto3.client('s3', region_name='us-east-1')

def create_bucket(bucket_name: str, region: str = 'us-east-1') -> bool:
    """Create an S3 bucket in the specified region."""
    try:
        if region == 'us-east-1':
            s3.create_bucket(Bucket=bucket_name)
        else:
            s3.create_bucket(
                Bucket=bucket_name,
                CreateBucketConfiguration={'LocationConstraint': region}
            )
        return True
    except ClientError as e:
        print(f"Error creating bucket: {e}")
        return False

def upload_with_storage_class(
    local_file: str,
    bucket: str,
    key: str,
    storage_class: str = 'STANDARD'
):
    """
    storage_class : STANDARD | INTELLIGENT_TIERING | STANDARD_IA |
                   ONEZONE_IA | GLACIER | GLACIER_IR | DEEP_ARCHIVE
    """
    s3.upload_file(
        local_file,
        bucket,
        key,
        ExtraArgs={'StorageClass': storage_class}
    )

def enable_versioning(bucket: str):
    """Enable versioning on a bucket."""
    s3.put_bucket_versioning(
        Bucket=bucket,
        VersioningConfiguration={'Status': 'Enabled'}
    )

def list_object_versions(bucket: str, key: str):
    """List all versions of an object."""
    response = s3.list_object_versions(Bucket=bucket, Prefix=key)
    for v in response.get('Versions', []):
        print(f"VersionId: {v['VersionId']} | Date: {v['LastModified']} | "
              f"Size: {v['Size']} bytes | Latest: {v['IsLatest']}")

def set_lifecycle_policy(bucket: str):
    """Apply a complete lifecycle policy."""
    config = {
        'Rules': [
            {
                'ID': 'TransitionAndExpire',
                'Status': 'Enabled',
                'Filter': {'Prefix': ''},
                'Transitions': [
                    {'Days': 30, 'StorageClass': 'STANDARD_IA'},
                    {'Days': 90, 'StorageClass': 'GLACIER_IR'},
                    {'Days': 365, 'StorageClass': 'DEEP_ARCHIVE'},
                ],
                'Expiration': {'Days': 2555},
                'NoncurrentVersionExpiration': {'NoncurrentDays': 90}
            }
        ]
    }
    s3.put_bucket_lifecycle_configuration(
        Bucket=bucket,
        LifecycleConfiguration=config
    )

def configure_replication(
    source_bucket: str,
    dest_bucket: str,
    role_arn: str,
    account_id: str
):
    """Configure cross-region replication (CRR)."""
    config = {
        'Role': role_arn,
        'Rules': [
            {
                'ID': 'ReplicateAll',
                'Status': 'Enabled',
                'Filter': {'Prefix': ''},
                'Destination': {
                    'Bucket': f'arn:aws:s3:::{dest_bucket}',
                    'Account': account_id,
                    'StorageClass': 'STANDARD'
                },
                'DeleteMarkerReplication': {'Status': 'Enabled'}
            }
        ]
    }
    s3.put_bucket_replication(
        Bucket=source_bucket,
        ReplicationConfiguration=config
    )

def restore_glacier_object(
    bucket: str,
    key: str,
    days: int = 7,
    tier: str = 'Standard'
):
    """
    Restore an object from Glacier.
    tier : Expedited (1-5 min) | Standard (3-5h) | Bulk (5-12h free)
    """
    s3.restore_object(
        Bucket=bucket,
        Key=key,
        RestoreRequest={
            'Days': days,
            'GlacierJobParameters': {'Tier': tier}
        }
    )

def generate_presigned_url(bucket: str, key: str, expiry_seconds: int = 3600) -> str:
    """Generate a presigned URL for secure object sharing."""
    return s3.generate_presigned_url(
        'get_object',
        Params={'Bucket': bucket, 'Key': key},
        ExpiresIn=expiry_seconds
    )

EBS Operations with boto3

import boto3

ec2 = boto3.client('ec2', region_name='us-east-1')

def create_gp3_volume(
    size_gb: int,
    az: str,
    iops: int = 3000,
    throughput: int = 125,
    encrypted: bool = True
) -> str:
    """Create a gp3 EBS volume with custom parameters."""
    response = ec2.create_volume(
        VolumeType='gp3',
        Size=size_gb,
        AvailabilityZone=az,
        Iops=iops,
        Throughput=throughput,
        Encrypted=encrypted,
        TagSpecifications=[{
            'ResourceType': 'volume',
            'Tags': [{'Key': 'Name', 'Value': f'gp3-{size_gb}GB'}]
        }]
    )
    return response['VolumeId']

def create_snapshot_and_wait(volume_id: str, description: str) -> str:
    """Create a snapshot and wait for completion."""
    response = ec2.create_snapshot(
        VolumeId=volume_id,
        Description=description
    )
    snapshot_id = response['SnapshotId']
    ec2.get_waiter('snapshot_completed').wait(SnapshotIds=[snapshot_id])
    return snapshot_id

def upgrade_to_gp3(volume_id: str, iops: int = 3000, throughput: int = 125):
    """Modify an existing volume to gp3 in-place (zero downtime)."""
    ec2.modify_volume(
        VolumeId=volume_id,
        VolumeType='gp3',
        Iops=iops,
        Throughput=throughput
    )

AWS Storage Services Comparison

Decision Tree

flowchart TB
    Question[Which AWS storage service to choose?]
    Q1{Data type?}
    Q2{Shared between\nmultiple instances?}
    Q3{OS in use?}
    Q4{Access type?}
    Q5{Performance\ncritical?}

    EBS_GP3[EBS gp3\nBoot apps mid-size DBs]
    EBS_IO2[EBS io2 Block Express\nMission-critical DBs\nSAP HANA Oracle]
    EFS_REG[EFS Regional\nProduction HA]
    EFS_OZ[EFS One Zone\nDev Test cost-effective]
    FSX[Amazon FSx\nWindows AD]
    S3_STD[S3 Standard\nFrequent access]
    S3_IT[S3 Intelligent-Tiering\nVariable access]
    S3_GLACIER[S3 Glacier\nLong-term archives]

    Question --> Q1
    Q1 -- "Disk for EC2\nor database" --> Q5
    Q1 -- "Shared files\nbetween servers" --> Q2
    Q1 -- "Objects backups\nassets data lake" --> Q4

    Q5 -- "Mission-critical\n> 80,000 IOPS" --> EBS_IO2
    Q5 -- "General purpose" --> EBS_GP3

    Q2 -- "Yes" --> Q3
    Q2 -- "No" --> EBS_GP3
    Q3 -- "Linux" --> EFS_REG
    Q3 -- "Windows" --> FSX

    Q4 -- "Frequent" --> S3_STD
    Q4 -- "Variable or unknown" --> S3_IT
    Q4 -- "Rare or archiving" --> S3_GLACIER

    style EBS_GP3 fill:#ff9900,color:#000
    style EBS_IO2 fill:#e67e22,color:#fff
    style EFS_REG fill:#3498db,color:#fff
    style EFS_OZ fill:#2980b9,color:#fff
    style FSX fill:#9b59b6,color:#fff
    style S3_STD fill:#2ecc71,color:#000
    style S3_IT fill:#27ae60,color:#fff
    style S3_GLACIER fill:#95a5a6,color:#000

Complete Summary Table

Summary and Key Points

Module 1 — Block Storage (EBS)

• EBS is block storage attached to EC2 instances, presented from AWS arrays
• gp3 is preferred over gp2: up to 20% cheaper, IOPS (max 80,000) and throughput (max 2,000 MB/s) configurable independently of size
• io2 Block Express offers 99.999% durability, sub-ms latency, and up to 256,000 IOPS — for SAP HANA, Oracle, SQL Server mission-critical
• EBS snapshots are incremental, stored in AWS internal S3, copyable between regions
• EBS multi-attach allows multiple EC2 instances to share an io1/io2 volume (same AZ)
• Use Amazon Data Lifecycle Manager to automate snapshot creation and retention

Module 2 — File Storage (EFS)

• EFS is serverless file storage for Linux only (NFS v4 protocol)
• Supports thousands of simultaneous EC2 instances from a single region
• Regional EFS replicates data across 3+ AZs; One Zone reduces costs by ~47%
• Storage classes (Standard / IA / Archive) are managed automatically via lifecycle management rules
• Elastic throughput mode (recommended) automatically adjusts performance on demand
• For Windows, use Amazon FSx for Windows File Server (out of scope for this course)

Module 3 — Object Storage (S3)

• S3 is infinitely scalable object storage with 11 nines durability (99.999999999%)
• 8 storage classes: Standard, Intelligent-Tiering, Express One Zone, Standard-IA, One Zone-IA, Glacier Instant, Glacier Flexible, Glacier Deep Archive
• Intelligent-Tiering automatically moves objects to the optimal tier — savings up to 95% vs Standard for rarely accessed data
• S3 Express One Zone offers single-digit ms latency for ultra-latency-sensitive ML/Analytics workloads
• Data Transfer IN to S3 is always free; respect minimum storage durations to avoid unexpected charges

Module 4 — Configuring S3

• S3 Static Website Hosting hosts static sites without EC2 (HTTP only — CloudFront for HTTPS)
• S3 Versioning preserves object history — mandatory prerequisite for replication
• S3 Lifecycle Rules automate transitions and expiration — respect minimum storage durations
• S3 Bucket Replication requires versioning enabled on both buckets — applies only to new objects
• Combine lifecycle rules + versioning to effectively manage the costs of old versions

Complementary Services

• AWS Backup centralizes data protection for all AWS services — WORM (Vault Lock), cross-region, cross-account, regulatory compliance (SEC 17a-4, HIPAA, PCI DSS)
• AWS Storage Gateway connects on-premises to AWS: S3 File Gateway (NFS/SMB → S3), FSx File Gateway (SMB → FSx), Tape Gateway (VTL → Glacier), Volume Gateway (iSCSI → EBS Snapshots)

Sources: aws.amazon.com/s3/storage-classes | aws.amazon.com/ebs/volume-types | aws.amazon.com/storagegateway | aws.amazon.com/backup

Search Terms

aws · storage · fundamentals · core · services · amazon · web · efs · ebs · comparison · lifecycle · architecture · block · gateway · types · volume · backup · configuration · options · boto3 · class · configuring · data · object