AWS: Storage, Databases, ML and Big Data

Table of Contents

1. Module 1: Amazon Relational Database Service (RDS)

   • Reviewing Relational Databases
   • Amazon RDS Overview
   • Amazon RDS Networking
   • RDS Backups and Maintenance
   • Amazon RDS Multi-AZ and High Availability
   • Offloading Read Traffic with Read Replicas
   • Amazon RDS Authentication
   • Amazon RDS Encryption
   • Cost Optimization for Amazon RDS
   • RDS Custom
   • RDS Proxy
   • Module 1 Summary and Exam Tips

2. Module 2: Amazon Aurora

   • Amazon Aurora Overview
   • Amazon Aurora Storage
   • Replicas in Amazon Aurora
   • High Availability and Scaling
   • Backing up Amazon Aurora
   • Aurora Serverless
   • Amazon Aurora Global Databases
   • Amazon Aurora Machine Learning
   • Module 2 Summary and Exam Tips

3. Module 3: Amazon DynamoDB

   • Amazon DynamoDB Overview
   • DynamoDB High Availability and Monitoring
   • DynamoDB Capacity Modes
   • DynamoDB Security
   • DynamoDB Global Tables
   • DynamoDB Streams
   • DynamoDB Accelerator (DAX)
   • DynamoDB Items Time to Live (TTL)
   • Amazon DynamoDB Backups
   • Module 3 Summary and Exam Tips

4. Module 4: Other AWS Databases

   • Amazon DocumentDB
   • Amazon Neptune
   • Amazon Keyspaces (for Apache Cassandra)
   • Amazon Quantum Ledger Database (QLDB)
   • Amazon Timestream
   • Module 4 Summary and Exam Tips

5. Module 5: Other AWS Storage, Transfer, and Migration Services

   • AWS Snow Family
   • AWS Storage Gateway - File Gateways
   • AWS Storage Gateway - Volume Gateways
   • AWS Storage Gateway - Tape Gateways
   • AWS Lake Formation
   • AWS Transfer Family
   • AWS DataSync
   • AWS Backup
   • AWS Application and Server Migration Services
   • AWS Database Migration Service (DMS)
   • Database Schema Conversions
   • Module 5 Summary and Exam Tips

6. Module 6: Big Data and Analytics Services

   • Amazon Redshift
   • Redshift High Availability, Snapshots, and Performance
   • Amazon Redshift Spectrum
   • Amazon Elastic MapReduce (EMR)
   • AWS Glue
   • Amazon Athena
   • Amazon QuickSight
   • Amazon OpenSearch Service
   • Module 6 Summary and Exam Tips

7. Module 7: Machine Learning Services

   • Amazon SageMaker
   • Amazon Rekognition
   • Amazon Polly
   • Amazon Translate
   • Amazon Lex
   • Amazon Connect
   • Amazon Comprehend
   • Amazon Forecast
   • Amazon Kendra
   • Amazon Textract
   • Amazon Personalize
   • Amazon Transcribe
   • Amazon Fraud Detector
   • Module 7 Summary and Exam Tips

Module 1: Amazon Relational Database Service (RDS)

Reviewing Relational Databases

A relational database, in the simplest terms, is a type of database that stores different related data points in the form of a table. Tables are meant to store logically-connected data (related data). In those tables you have:

• Rows — each row is considered an item
• Columns — each column is considered a field

Real-world Use Cases for Relational Databases

• Financial systems — banking transactions, accounting software, payment processing
• E-commerce platforms — product catalogs, managing customer orders, inventory
• Inventory management — stock-level information, supplier information, order tracking
• Human resource management — employee records, payroll systems
• Reservation systems — hotel bookings, airline ticketing
• Telecommunications — ISP accounts, billing transactions

Relational databases are by far the most common type of database in use today.

OLTP vs OLAP

For the exam, you must understand two types of processing:

Exam Tip: OLTP workloads — use Amazon RDS. OLAP workloads — use Amazon Redshift. These are not interchangeable.

Most commercial RDBMS software uses Structured Query Language (SQL) to access data.

Amazon RDS Overview

Amazon RDS (Relational Database Service) is an AWS cloud service for setting up, operating, and scaling relational databases in the cloud.

Supported Database Engines

• Microsoft SQL Server
• MySQL
• PostgreSQL
• MariaDB
• Oracle
• Db2
• Amazon Aurora (covered separately in Module 2)

Key Benefits

• Managed service — handles backups, software patching, automatic failover detection and recovery
• Automated backup scheduling or manual backup snapshots
• High availability and optimal read performance
• Access control via IAM (who can manage resources), VPC Security Groups and NACLs (network access), and encryption at rest

DB Instances

When you start an RDS database, you provision a DB instance — an isolated database environment in your account. You can host one or more databases on a single DB instance. You interact with these via standard database tools, but you do not get direct access to the underlying server (unlike EC2).

Instance Classes

Similar to EC2, DB instances offer:

• General purpose — development and testing
• Memory optimized — memory-intensive workloads
• Compute optimized — CPU-intensive workloads
• Burstable performance — smaller or variable workloads

Instance syntax example: db.m5.large (note the db. prefix distinguishes it from EC2 instances)

Storage Types

Amazon RDS Networking

VPC Deployment

RDS instances are created within a VPC. Best practice is to use custom VPCs (never the default VPC for production workloads).

DB Subnet Groups

A DB subnet group is a set of VPC subnets dedicated to your database instances only. Best practice is to isolate database instances within their own subnet tier.

graph TB
    Internet((Internet)) --> IGW[Internet Gateway]
    IGW --> PublicTier[Public Web Tier<br/>Subnets]
    PublicTier --> AppTier[Private App Tier<br/>Subnets]
    AppTier --> DBTier[Private Database Tier<br/>Subnets - RDS]
    
    style DBTier fill:#f96,stroke:#333
    style AppTier fill:#9cf,stroke:#333
    style PublicTier fill:#9f9,stroke:#333

Security Controls

• VPC Security Groups — virtual firewall attached to the network interfaces of RDS instances
• Public access option — can be enabled but should never be used in real architectures
• No SSH/RDP access — RDS is a managed service; you cannot SSH into the underlying instance

Exam Tip: You can reference other security groups within your inbound rules (e.g., reference your App Layer security group in your database security group). This avoids manually tracking IP addresses.

Network Architecture (Three-Tier)

A three-tier architecture provides better network segmentation, security, and access control:

1. Public web tier — internet-facing
2. Private app tier — application servers
3. Private database tier — RDS instances (most restricted)

RDS Backups and Maintenance

Automated Backups

• You specify a backup window when creating instances; RDS takes a volume snapshot during this window
• The first snapshot is a full copy; subsequent snapshots are incremental (capture only changes)
• Retention period: 0 to 35 days
  • 0 = disables automated backups entirely
  • 1–35 = number of days to retain backups
• Backups allow point-in-time recovery to any moment within the retention period
• Backups are stored in Amazon S3 (you have no access to the underlying S3 storage)
• You can enable cross-region replication of backups for disaster recovery and compliance

Exam Tip: Cross-region replication for automated backups is not supported for Multi-AZ DB clusters.

Manual Backups (Snapshots)

• You can trigger backups whenever you want
• Retention is indefinite — you pay for them until you manually delete them
• Useful when you need to keep a backup longer than 35 days

Backup Restoration

• Restoring creates a brand new database instance — you cannot restore to an existing instance
• Lazy loading — RDS loads data in the background; if you access data not yet loaded, it downloads from S3 on demand
• You can use a different storage type when restoring (e.g., restore from gp3 to io1)
• MySQL users can restore on-premises backups that are sent to S3

Exam Pro Tip: You pay for RDS storage even when your RDS instance is stopped. You are billed for volume storage and backup storage at all times.

Demo: SQL Commands for RDS (Multi-AZ)

-- Use the database
USE pluralsight;

-- Create a customers table
CREATE TABLE customers (
    id INT PRIMARY KEY,
    first_name VARCHAR(50) NOT NULL,
    last_name VARCHAR(50) NOT NULL,
    email VARCHAR(100) UNIQUE NOT NULL,
    phone_number VARCHAR(20),
    subscription_start_date DATE NOT NULL,
    subscription_end_date DATE,
    subscription_type VARCHAR(7) NOT NULL,
    last_login TIMESTAMP
);

-- Insert sample data
INSERT INTO customers (id, first_name, last_name, email, phone_number,
    subscription_start_date, subscription_end_date, subscription_type, last_login)
VALUES 
(001, 'Alice', 'Johnson', 'alice.johnson@example.com', '555-456-7890',
    '2023-01-15', '2024-01-15', 'Pro', '2023-09-10 08:15:00'),
(002, 'Bob', 'Smith', 'bob.smith@example.com', '555-654-3210',
    '2022-12-01', '2023-12-01', 'Basic', '2023-09-11 10:30:00');

-- Query with filter
SELECT * FROM customers WHERE id > 003;

-- Update a record
UPDATE customers
SET email = 'new.email@example.com'
WHERE email = 'alice.johnson@example.com';

Amazon RDS Multi-AZ and High Availability

Overview

When you use Multi-AZ deployments, RDS creates an exact copy of your database in another Availability Zone — called a standby replica. AWS automatically handles data replication between the primary and standby instances.

Important: Multi-AZ deployments are for high availability and disaster recovery, NOT for scaling.

Two Deployment Models

graph LR
    subgraph "Instance Deployment"
        C1[Client] --> P1[Primary Instance<br/>AZ-A]
        P1 -->|Synchronous Replication| S1[Standby Instance<br/>AZ-B]
        S1 -.->|Cannot use directly| X[X]
    end
    
    subgraph "Cluster Deployment"
        C2[Client] --> W[Writer Instance<br/>AZ-A]
        C2 --> R1[Reader Instance<br/>AZ-B]
        C2 --> R2[Reader Instance<br/>AZ-C]
        W -->|Semi-sync replication| R1
        W -->|Semi-sync replication| R2
    end

Key Facts

• Failover time: approximately 60–120 seconds (much slower than Aurora)
• DNS endpoint: does NOT change during failover — RDS automatically updates the backend resource
• Supported engines: SQL Server, MySQL, PostgreSQL, MariaDB, Oracle, and others

Exam Tip: During a Multi-AZ failover, you still reference the same RDS endpoint/address. RDS remaps the backend resource automatically via CNAME records.

Offloading Read Traffic with Read Replicas

A Read Replica is a read-only copy of your database instance, meant to reduce load on the primary/writer database instance.

Key Concepts

• Perfect for read-heavy workloads
• Can be created in the same AZ, cross-AZ, or cross-region
• Prerequisite: Automatic backups must be enabled
• Each replica gets its own DNS endpoint
• Can be promoted to become an independent, standalone database (replication stops at that point)
• Uses asynchronous replication (not synchronous like Multi-AZ standby)

Architecture

graph LR
    App[Application] -->|Read + Write| Primary[Primary DB Instance<br/>AZ-A]
    BIServer[BI / Reporting<br/>Server] -->|Read Only| RR[Read Replica<br/>AZ-A or AZ-B]
    Primary -->|Async Replication| RR

Instance vs Cluster Read Replicas

• Instance deployment: Read Replicas are separate from the standby; standby is failover-only
• Cluster deployment: Two reader instances come built-in; you can add additional Read Replicas for further read scaling

Exam Tips

• Read Replicas are for scaling reads, not for disaster recovery
• Automatic backups must be enabled to use Read Replicas
• Cross-region Read Replicas incur data transfer charges
• When you promote a Read Replica, it becomes a fully independent database — replication stops

Amazon RDS Authentication

RDS supports three primary authentication options:

AWS Secrets Manager Integration

RDS integrates with AWS Secrets Manager to manage admin user passwords securely:

• Eliminates hard-coded credentials in application code
• Credentials are encrypted at rest using a KMS key
• Default rotation: every 7 days (configurable)
• Managed programmatically via the Secrets Manager ARN

# Retrieve the password from Secrets Manager programmatically
admin_password=$(aws secretsmanager get-secret-value \
    --secret-id "your-secret-id" \
    --query SecretString \
    --output text | jq -r '.password')

# Connect using the retrieved password
PGPASSWORD=$admin_password psql -h YOUR_RDS_ENDPOINT \
    -U postgres \
    -d your_database

Exam Tip: If you need secure admin credential rotation and storage for RDS, think Secrets Manager.

Amazon RDS Encryption

In-Transit Encryption

• TLS is enabled by default using AWS root certificates
• These certificates are broadly trusted worldwide

Encryption at Rest

• Uses KMS keys — you can specify a custom key or use the default service key
• Must be enabled at creation time — cannot be added after the fact
• Once a KMS key is selected, it cannot be changed; you must create a new DB instance with a different key
• All logs, backups, and snapshots are also encrypted when the DB instance is encrypted
• Read Replicas must be encrypted if the primary is encrypted (cannot mix)

Encrypting an Unencrypted Database

Process:

1. Take a database snapshot
2. Restore the snapshot to a new database instance
3. Enable encryption during the restoration

Transparent Data Encryption (TDE)

For Oracle and Microsoft SQL Server engines, you can use Transparent Data Encryption (TDE) as an additional encryption method.

Exam Tip: If you see Oracle or SQL Server + encryption, think TDE.

Cost Optimization for Amazon RDS

What You Pay For

• Instance hours — compute running time
• Cross-region replication data transfer charges
• Database storage — charged regardless of instance activity (even when stopped)
• Enhanced Monitoring — optional feature with additional cost

Pricing Options

Exam Tips:

• Use Reserved Instances for long-term cost savings
• There are no Spot Instances for RDS — only On-Demand and Reserved

RDS Custom

RDS Custom provides additional operating system and database customization beyond standard RDS.

• Only available for Oracle and Microsoft SQL Server engines
• Allows installation of custom OS patches
• Enables non-RDS-native database features
• Grants access to the underlying instance via SSH or Session Manager
• Does NOT support MySQL, PostgreSQL, or other engine types

RDS Proxy

RDS Proxy is a fully managed, highly available proxy service for Amazon RDS. Its primary function is to allow applications to pool and share their database connections.

Benefits

• Reduces memory and CPU overhead from opening new database connections
• Handles unpredictable traffic surges by reusing connections
• Serverless, natively highly available, and auto-scaling — all managed for you
• No application code changes required in most cases
• Supports creating a read-only endpoint for Amazon Aurora

graph LR
    L1[Lambda Function 1] --> Proxy[RDS Proxy<br/>Connection Pool]
    L2[Lambda Function 2] --> Proxy
    L3[Lambda Function 3] --> Proxy
    EC2[EC2 Instance] --> Proxy
    Proxy --> RDS[RDS DB Instance]

Important Constraints

• Must be accessed within a VPC (no public accessibility)

Exam Tip: Lambda + RDS = Think RDS Proxy. Lambda functions spin up and down constantly. Instead of creating a new database connection each time, RDS Proxy pools those connections. This is the canonical exam scenario.

Module 1 Summary and Exam Tips

Relational Databases and RDS

• RDS is ideal for OLTP workloads
• Supports most popular engines: SQL Server, MySQL, PostgreSQL, MariaDB, Oracle, Db2
• Managed service: handles backups, patching, failover

Backups

• Automated backups: 1–35 days retention; 0 disables them
• Manual snapshots: stored indefinitely
• Restore always creates a new instance
• Use DB Subnet Groups to isolate database resources in private subnets

Access Control

• Authentication methods: IAM, Kerberos, Basic password
• Network controls: VPC Security Groups and NACLs

Multi-AZ

• Instance deployment: 1 standby, failover only, synchronous replication
• Cluster deployment: 2 readable standbys, semi-synchronous replication
• Multi-AZ is for HA and DR — not for scaling
• Failover is automatic; DNS endpoint automatically updated

Read Replicas

• Read-only copies for scaling read operations
• Same AZ, cross-AZ, or cross-region
• Automatic backups must be enabled
• Asynchronous replication
• Can be promoted to standalone DBs

Secrets Manager

• Manages and rotates RDS admin passwords
• Encrypted via KMS; access controlled via IAM
• Default rotation: 7 days

RDS Custom

• Oracle and SQL Server only
• OS-level access via SSH/Session Manager

RDS Proxy

• Connection pooling for Lambda and other bursty clients
• VPC-only access
• No application changes needed

Module 2: Amazon Aurora

Amazon Aurora Overview

Amazon Aurora is an AWS fully managed, relational database engine that is compatible with MySQL and PostgreSQL workloads. Aurora is an engine type within the RDS service — it is managed and leveraged via RDS.

How Aurora Differs from Standard RDS

• Built using AWS proprietary technology (not open source)
• Cloud-optimized performance:
  • Up to 5x better performance than MySQL on RDS
  • Up to 3x better performance than PostgreSQL on RDS
• Costs approximately 20% more than traditional RDS
• Supports up to 96 vCPUs and 768 GB of memory

Aurora Cluster Architecture

Aurora databases are deployed as Aurora database clusters consisting of:

• One or more database instances
• A shared cluster volume — a virtual database storage volume spanning multiple AZs

Cluster Endpoints:

Exam Tips:

• Aurora only supports MySQL and PostgreSQL compatible workloads
• Performance is 5x MySQL, 3x PostgreSQL compared to standard RDS
• It is more expensive (~20% more) than standard RDS

Amazon Aurora Storage

Key Storage Characteristics

• Auto-scaling high-performance storage system
• Minimum: 10 GB
• Maximum: 128 TB
• Performance is not affected by storage growth — even at maximum size

Data Redundancy

• Two copies of data in each Availability Zone
• Spread across a minimum of three AZs
• Result: minimum 6 copies of your data at all times

Storage Fault Tolerance

• Storage is self-healing: disks are continuously scanned for errors and repaired automatically

Exam Tip: Aurora maintains a minimum of 6 copies of your data across 3 AZs.

Replicas in Amazon Aurora

Aurora Replicas are instances that connect to the same storage volume as the primary instance but support read-only operations.

• Up to 15 Replicas per cluster, per region
• Serve as automatic failover targets when configured in the cluster
• Support cross-region replication for global read scaling
• Replication is synchronous in milliseconds — extremely fast

graph TB
    subgraph "Aurora Cluster"
        Writer[Primary / Writer<br/>Instance AZ-A] 
        Reader1[Reader Replica<br/>AZ-B]
        Reader2[Reader Replica<br/>AZ-C]
        Reader3[Reader Replica<br/>AZ-A]
    end
    
    subgraph "Shared Cluster Volume"
        CV[Auto-scaling, Self-healing<br/>Replicated across 3 AZs]
    end
    
    Writer --> CV
    Reader1 --> CV
    Reader2 --> CV
    Reader3 --> CV

High Availability and Scaling

Aurora Auto Scaling

• Dynamically adjusts the number of Replicas in your cluster
• Primary instances are automatically replicated across AZs to Replicas

Failover and Promotion

• Automatic failover to a Replica within 30 seconds or less
• Manual Replica promotion is also supported
• Promoting a Replica is faster than recreating a new primary instance

Replica Priority Tiers

When automatic failover occurs, RDS promotes Replicas based on priority:

• Priority range: 0 to 15 (0 = highest priority, 15 = lowest)
• RDS promotes the Replica with the highest priority (lowest number)
• If two Replicas share the same priority (promotion tier), RDS promotes the largest one
• If two Replicas share the same priority AND the same size, RDS randomly selects one

Exam Tip: Priority 0 is highest; priority 15 is lowest. Smaller numbers = higher priority.

If no Replicas are present, the primary instance is automatically recreated in the same AZ if possible.

Backing up Amazon Aurora

Automated Backups

• Always enabled by default — cannot be disabled
• Continuous and incremental
• Retention period: 1 to 35 days (minimum is 1 day; there is no 0-day option like in RDS)
• One free day is always provided
• Stored in S3 (no access to underlying storage)

Exam Tip: For retention beyond 35 days, you must take a manual snapshot.

Backtracking

Backtracking allows you to rewind your Aurora cluster to a specific point in time without restoring from a backup.

• Not a replacement for backups
• Maximum backtrack window: 72 hours
• Causes a brief disruption — pause/stop applications before backtracking
• Available for MySQL-compatible Aurora databases only

Aurora Cloning

Cloning creates a copy of your production cluster quickly and efficiently:

• Set up test environments with real data without risk of corruption
• Run workload-intensive operations on a cloned database
• Create development/testing copies with identical data to production

Demo: Aurora Database Infrastructure as Code

The following CloudFormation template creates the VPC infrastructure for an Aurora demo environment:

AWSTemplateFormatVersion: "2010-09-09"
Description: Creates a custom VPC with 3 tiers of subnets and an EC2 IAM instance profile for SSM.
Resources:
  VPC:
    Type: "AWS::EC2::VPC"
    Properties:
      CidrBlock: "10.0.0.0/16"
      EnableDnsHostnames: true
      EnableDnsSupport: true
      Tags:
        - Key: "Name"
          Value: "CUSTOM-VPC"

  # Database subnets (10.0.20.0/24 through 10.0.22.0/24)
  DatabaseSubnet1:
    Type: "AWS::EC2::Subnet"
    Properties:
      CidrBlock: "10.0.20.0/24"
      MapPublicIpOnLaunch: false
      VpcId: !Ref VPC

  EC2SSMRole:
    Type: "AWS::IAM::Role"
    Properties:
      RoleName: "EC2SSMRole"
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Principal:
              Service: "ec2.amazonaws.com"
            Action: "sts:AssumeRole"
      ManagedPolicyArns:
        - "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
      Policies:
        - PolicyName: "AdditionalPermissions"
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: "Allow"
                Action:
                  - "secretsmanager:Get*"
                  - "secretsmanager:List*"
                  - "secretsmanager:Describe*"
                Resource: "*"

Aurora Database Commands

# Install PostgreSQL client on Amazon Linux
sudo dnf install postgresql15.x86_64 -y

# Retrieve password from Secrets Manager
admin_password=$(aws secretsmanager get-secret-value \
    --secret-id "your-secret-id" \
    --query SecretString \
    --output text | jq -r '.password')

# Connect to Aurora writer endpoint
PGPASSWORD=$admin_password psql -h YOUR_AURORA_WRITER_ENDPOINT \
    -U postgres \
    -d pluralsight

# SQL: Create employees table
CREATE TABLE employees (
    employee_id SERIAL PRIMARY KEY,
    first_name VARCHAR(50),
    last_name VARCHAR(50),
    email VARCHAR(100),
    hire_date DATE
);

# SQL: Insert sample data
INSERT INTO employees (first_name, last_name, email, hire_date)
VALUES
    ('John', 'Doe', 'john.doe@example.com', '2023-10-01'),
    ('Jane', 'Smith', 'jane.smith@example.com', '2023-08-15'),
    ('Alice', 'Brown', 'alice.brown@example.com', '2023-09-30');

# SQL: Query with filter
SELECT * FROM employees WHERE hire_date > '2023-09-01';

# Connect to Aurora reader endpoint (read-only)
PGPASSWORD=$admin_password psql -h YOUR_AURORA_READER_ENDPOINT \
    -U postgres \
    -d pluralsight

# This will FAIL on the reader endpoint — read-only transaction
CREATE TABLE fail (
    employee_id SERIAL PRIMARY KEY,
    first_name VARCHAR(50)
);

Aurora Serverless

Amazon Aurora Serverless is an on-demand, auto-scaling configuration option for Amazon Aurora, designed for infrequent, intermittent, or unpredictable/spiky workloads.

Key Benefits

• Scales based on actual demand — no capacity planning required
• Can scale down to near-zero when not in use, then scale up on demand
• More cost-effective for unknown or variable workloads

Aurora Capacity Units (ACUs)

• ACU (Aurora Capacity Unit) = the unit of measure for Aurora Serverless
• Each ACU provides 2 GiB of memory plus corresponding compute and networking
• Minimum: 0.5 ACUs (1 GiB memory)
• Maximum: 128 ACUs (256 GiB memory)

You set a minimum and maximum ACU range, and Aurora Serverless scales within that range automatically. You are billed for consumed capacity units.

Exam Tip: Aurora Serverless is recommended for development, initial testing, and workloads where capacity requirements are unknown or unpredictable.

Amazon Aurora Global Databases

An Aurora Global Database spans multiple AWS Regions and is the recommended DR solution for Aurora.

graph TB
    Primary[Primary Region<br/>Read + Write]
    Secondary1[Secondary Region 1<br/>Read Only]
    Secondary2[Secondary Region 2<br/>Read Only]
    Secondary3[Secondary Region 3<br/>Read Only]
    
    Primary -->|Replication < 1 second| Secondary1
    Primary -->|Replication < 1 second| Secondary2
    Primary -->|Replication < 1 second| Secondary3
    
    Secondary1 --> R1[Up to 16 Read Replicas]
    Secondary2 --> R2[Up to 16 Read Replicas]

Key Facts

• 1 primary Region (read and write)
• Up to 5 secondary Regions (read-only)
• Each secondary Region can have up to 16 Read Replicas
• Replication latency: typically less than 1 second between regions
• RTO: less than 1 minute
• RPO: approximately 5 seconds

Operations

Exam Tip: If you need an RTO of less than 1 minute at global scale, think Aurora Global Databases.

Amazon Aurora Machine Learning

Aurora Machine Learning integrates Aurora with AWS ML services to allow predictions from within your database — no ML experience required.

Supported services:

• Amazon Bedrock — fully managed foundation models
• Amazon Comprehend — managed NLP for extracting insights and sentiments
• Amazon SageMaker — fully managed ML service for data scientists

Use cases:

• Fraud detection within applications
• Customized marketing and ads for customers
• Product recommendations based on database data

Module 2 Summary and Exam Tips

Aurora as a Service

• Fully managed relational database engine (AWS proprietary technology) on RDS
• Compatible with MySQL and PostgreSQL only
• 5x MySQL performance, 3x PostgreSQL performance vs. standard RDS
• ~20% more expensive than standard RDS

Storage

• Auto-scales from 10 GB to 128 TB
• 6 copies of data across 3 AZs
• Automatic recovery if a healthy AZ is available

Clusters

• Up to 15 Read Replicas per cluster per Region
• Shared cluster volume across AZs
• Cluster, reader, custom, and instance endpoints

High Availability

• Auto scaling adjusts number of Replicas dynamically
• Failover in 30 seconds or less
• Priority tiers: 0 (highest) to 15 (lowest); larger instance wins tie-breaks
• Promoting a Replica is faster than recreating a primary

Backups

• Always enabled; 1–35 days retention (minimum 1 day)
• Stored in S3; no direct access to underlying storage
• Backtracking (MySQL only): up to 72 hours, not a backup replacement

Aurora Serverless

• For unknown/spiky workloads
• ACUs: minimum 0.5, maximum 128

Aurora Global Databases

• 1 primary + up to 5 secondary Regions
• Up to 16 Read Replicas per secondary Region
• Replication < 1 second; RTO < 1 minute; RPO ~5 seconds

Module 3: Amazon DynamoDB

Amazon DynamoDB Overview

Amazon DynamoDB is a fast and flexible NoSQL database service offering consistent, single-digit millisecond latency at any scale. It is the fully managed, serverless, purpose-built database supporting document and key-value data models.

Exam Tip: If you see NoSQL, unstructured schema, or key-value requirements, immediately think DynamoDB.

Key Capabilities

• Scales automatically for massive workloads
• Supports millions of requests per second
• Supports hundreds of terabytes of storage
• Single-digit millisecond performance at any scale

Use Cases

• Mobile apps and web applications (unstructured data)
• Online gaming (scoreboards, user records)
• Financial trading applications

Core Concepts

Limits:

• Tables can have unlimited items
• Each item can be up to 400 KB in size

Schema Data Types

Primary Key Examples

Single Primary Key:

Table: Products
Partition Key: product_id (unique per product)
Other attributes: product_name, price, quantity

Composite Primary Key:

Table: Orders
Partition Key: customer_id
Sort Key: order_id
Combined, these two attributes form a unique key

DynamoDB High Availability and Monitoring

Resilience

• Automatically replicates data across 3 distinct Availability Zones in the region
• 99.99% SLA for availability
• Zero-downtime maintenance — AWS handles all underlying infrastructure

Monitoring

• Amazon CloudWatch — DynamoDB automatically monitors tables and reports metrics
• AWS CloudTrail — captures and logs all API calls and events for your DynamoDB resources

Exam Tip: DynamoDB tables are Regional resources — you choose the Region, not individual AZs.

DynamoDB Capacity Modes

Provisioned Mode

• You manually define read and write capacity
• Pay for provisioned capacity units even if not used
• Risk of throttling if you under-provision
• Ideal when you know your capacity requirements

On-Demand Mode

• Pay per request for reads and writes
• Scales automatically — no capacity planning required
• Best for unknown or spiky workloads
• Can become expensive under constant, high-volume load

Exam Tip: You can switch between capacity modes once every 24 hours per table.

Consistency Models

WCU Calculation: 1 WCU per second for items up to 1 KB. A 2 KB item requires 2 WCUs.

RCU Calculation Reference:

Exam Tip: For items over 4 KB, round up to the nearest multiple of 4 KB. A 5 KB item requires 2 RCUs (next multiple of 4 is 8 KB → 2 units).

API Calls That Consume Capacity

Read operations: GetItem, BatchGetItem, Scan, Query

Write operations: PutItem, UpdateItem, DeleteItem, BatchWriteItem

Scans vs Queries

Best Practice: Use Queries over Scans whenever possible.

DynamoDB Auto Scaling

Uses AWS Application Auto Scaling to dynamically adjust provisioned throughput capacity. You set a minimum, maximum, and target utilization — DynamoDB scales accordingly.

DynamoDB Security

DynamoDB is a publicly accessible service (like S3 or IAM). Security controls include:

• IAM policies on users and roles to control access and authorization
• Resource-based policies on tables and indexes (attached directly to the resource)
• Encryption at rest by default using KMS keys
• VPC Endpoints (Gateway type — free) to keep traffic within the AWS network

Encryption at Rest Options

DynamoDB Resource-Based Policy Example

{
    "Id": "AllowedIPOnly",
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowIPmix",
            "Effect": "Allow",
            "Principal": "arn:aws:iam::111111111111:user/cloud_user",
            "Action": "dynamodb:*",
            "Resource": "*",
            "Condition": {
                "IpAddress": {
                    "aws:SourceIp": ["1.1.1.1/32"]
                }
            }
        }
    ]
}

KMS Key Policy for DynamoDB Encryption

{
    "Version": "2012-10-17",
    "Id": "dynamodb",
    "Statement": [
        {
            "Sid": "Allow access through Amazon DynamoDB for all principals in the account",
            "Effect": "Allow",
            "Principal": {"AWS": "*"},
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:CreateGrant",
                "kms:DescribeKey"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {"kms:CallerAccount": "ACCOUNT_ID"},
                "StringLike": {"kms:ViaService": "dynamodb.*.amazonaws.com"}
            }
        }
    ]
}

DynamoDB Global Tables

A DynamoDB Global Table deploys a multi-Region, multi-master database.

• Provides Active-Active implementation — read AND write to any regional table
• Replication latency: less than 1 second
• DynamoDB creates identical tables in all chosen Regions and automatically propagates changes
• Changes (PutItems, DeleteItems, etc.) are replicated globally in real-time

Prerequisite: DynamoDB Streams must be enabled (Streams are the underlying mechanism for Global Tables).

graph LR
    App1[App in US-East] -->|Read/Write| T1[Table: us-east-1]
    App2[App in EU-West] -->|Read/Write| T2[Table: eu-west-1]
    App3[App in AP-Southeast] -->|Read/Write| T3[Table: ap-southeast-1]
    T1 <-->|<1 second replication| T2
    T2 <-->|<1 second replication| T3
    T1 <-->|<1 second replication| T3

DynamoDB Streams

DynamoDB Streams capture a time-ordered sequence of item-level modifications for your table.

• Records are created for all creates, updates, and deletes
• Records are stored for up to 24 hours
• Written in near-real time (not immediate/real-time)
• Records appear exactly once and in the correct sequence
• A group of Stream records forms a Shard

StreamViewType Options

Common Architecture: DynamoDB Streams + Lambda

graph LR
    EC2[EC2 / Application] -->|PutItem/UpdateItem/DeleteItem| Table[DynamoDB Table]
    Table -->|Stream Record| Stream[DynamoDB Stream]
    Stream -->|Near-Real-Time Trigger| Lambda[Lambda Function]
    Lambda -->|Archive Old Image| TargetTable[Target DynamoDB Table]
    Lambda -->|Log New Image| CWLogs[CloudWatch Logs]

Important: DynamoDB Streams are the fundamental mechanism enabling Global Tables replication.

Demo: Lambda Function for DynamoDB Streams

import os
import json
import boto3
from datetime import datetime

# Initialize the DynamoDB client
dynamodb = boto3.resource("dynamodb")

# Table name is read from the environment variable
TARGET_TABLE_NAME = os.environ["TARGET_TABLE_NAME"]
target_table = dynamodb.Table(TARGET_TABLE_NAME)

def lambda_handler(event, context):
    print(f"Here is the event: {event}")

    for record in event["Records"]:
        # Archive old images for MODIFY and REMOVE events
        if record["eventName"] in ["MODIFY", "REMOVE"]:
            old_image = record["dynamodb"].get("OldImage")
            if old_image:
                deserialized_old_image = {
                    k: list(v.values())[0] for k, v in old_image.items()
                }
                deserialized_old_image["RecordedAt"] = datetime.utcnow().isoformat()
                try:
                    target_table.put_item(Item=deserialized_old_image)
                    print(f"Successfully wrote old image to {TARGET_TABLE_NAME}: "
                          f"{json.dumps(deserialized_old_image)}")
                except Exception as e:
                    print(f"Failed to write old image to {TARGET_TABLE_NAME}: {e}")

        # Log new images for INSERT and MODIFY events
        if record["eventName"] in ["INSERT", "MODIFY"]:
            new_image = record["dynamodb"].get("NewImage")
            if new_image:
                deserialized_new_image = {
                    k: list(v.values())[0] for k, v in new_image.items()
                }
                print(f"New image logged: {json.dumps(deserialized_new_image)}")

    return "Stream processing complete"

Demo: DynamoDB Table Interaction Scripts

Put a Single Item

#!/bin/bash
TABLE_NAME="CustomerOrders"
ORDER_ID="fc03c8ca-a5fe-4825-9e7a-0d5bd853bb7b"
CUSTOMER_NAME="John Doe"
ORDER_DATE="2023-10-18"
ORDER_TOTAL="199.99"
ITEMS=("PS5" "XBOX" "Controller")

ITEMS_JSON=$(printf '"%s",' "${ITEMS[@]}")
ITEMS_JSON="[${ITEMS_JSON%,}]"

aws dynamodb put-item \
    --table-name "$TABLE_NAME" \
    --item '{
        "OrderId": {"S": "'"$ORDER_ID"'"},
        "CustomerName": {"S": "'"$CUSTOMER_NAME"'"},
        "OrderDate": {"S": "'"$ORDER_DATE"'"},
        "OrderTotal": {"N": "'"$ORDER_TOTAL"'"},
        "Items": {"SS": '"$ITEMS_JSON"'}
    }' \
    --return-consumed-capacity TOTAL

Import Multiple Items from CSV

#!/bin/bash
TABLE_NAME="CustomerOrders"
CSV_FILE="orders.csv"

while IFS=',' read -r orderId customerName orderDate orderTotal items; do
    if [[ "$orderId" == "OrderId" || -z "$orderId" ]]; then
        continue
    fi

    items_array=(${items//,/ })
    items_json=$(printf "%s", "${items_array[@]}")
    items_json="[${items_json%,}]"

    aws dynamodb put-item \
        --table-name "$TABLE_NAME" \
        --item '{
            "OrderId": {"N": "'"$orderId"'"},
            "CustomerName": {"S": "'"$customerName"'"},
            "OrderDate": {"S": "'"$orderDate"'"},
            "OrderTotal": {"N": "'"$orderTotal"'"},
            "Items": {"SS": '"$items_json"'}
        }'
done < "$CSV_FILE"

DynamoDB Accelerator (DAX)

DynamoDB Accelerator (DAX) is a fully managed, highly available, in-memory caching system specifically for DynamoDB.

• Speeds up read requests by up to 10x compared to reading directly from tables
• Reduces request times from milliseconds to microseconds
• Compatible with existing DynamoDB API calls — no application code changes required
• Writes first go to the DynamoDB table, then the DAX cluster (write-through caching)

Read Behavior

graph LR
    EC2[EC2 Application] --> DAX[DAX Cluster<br/>Cache Nodes]
    Lambda[Lambda Function] --> DAX
    DAX -->|Cache Hit| EC2
    DAX -->|Cache Miss| Table[DynamoDB Table]
    Table -->|Write-through| DAX

Exam Tip: If a scenario requires improved performance efficiency of a DynamoDB table without reconfiguring the application, the answer is DAX.

DynamoDB Items Time to Live (TTL)

DynamoDB TTL allows you to define a per-item expiration timestamp to automatically delete items that are no longer needed.

• Items are deleted within a few days of the TTL expiration (not immediately)
• Deletion does not consume any write capacity units (WCUs)
• TTL is defined in epoch time (Unix timestamp)
• Common pattern: calculate TTL by adding duration to createdAt or updatedAt timestamps

Example TTL Table Structure:

Table: WebAppUserSessions
Columns: session_id, user_id, created_at, updated_at, ttl (epoch timestamp)

Exam Tip: TTL is a cost-saving feature — automatically removes expired items without consuming capacity units.

Amazon DynamoDB Backups

Backup Methods

1. DynamoDB On-Demand Backups — manual backups triggered on demand
2. Point-in-Time Recovery (PITR) — continuous, fully managed backups
3. AWS Backup — centralized backup management (covered in Module 5)

Key Backup Facts

• No impact on table performance or availability
• All backups are encrypted via KMS
• Backups can be listed and restored from within the DynamoDB service
• Billed based on size and duration of backup retention

Point-in-Time Recovery (PITR)

• Up to 35 days of recovery points
• Per-second granularity for restoration
• Does not affect table performance or latency
• Billed while enabled

Exam Tip: For recovering from accidental deletes or overwrites, use DynamoDB PITR.

Import/Export

Import:

• Supports CSV, JSON, or Amazon Ion formats
• No impact on write capacity
• Creates a brand new table (not importing to an existing table)
• Supports cross-account imports
• API calls tracked in CloudTrail

Export:

• Requires PITR to be enabled
• No impact on read capacity
• Useful for analytics (Amazon Athena), ETL workloads, and auditing
• Supports DynamoDB JSON or Ion formats; can be full or incremental

Module 3 Summary and Exam Tips

DynamoDB Basics

• Go-to choice for NoSQL and flexible-schema workloads
• Items up to 400 KB; unlimited items per table
• Attributes are flexible — add them any time
• Replicated across 3 AZs automatically; regional resource
• Publicly accessible endpoint — use IAM for least-privilege access

Capacity Modes

• Provisioned: manual WCU/RCU; risk of throttling if under-provisioned
• On-Demand: pay-per-request; best for unknown/spiky workloads
• Switch between modes once per 24 hours

Capacity Units

• WCU: 1 write/second for items up to 1 KB
• RCU: 1 strongly consistent read/second OR 2 eventually consistent reads/second for 4 KB items
• Use Queries over Scans whenever possible

Global Tables

• Active-Active, multi-Region, multi-master
• Requires DynamoDB Streams to be enabled
• Replication < 1 second

DynamoDB Streams

• Time-ordered sequence of item-level changes
• Stored for up to 24 hours
• Commonly used to trigger Lambda functions
• Foundation for Global Tables replication

DAX

• In-memory caching; up to 10x performance improvement
• Microsecond latency
• No application code changes needed

Backups and TTL

• PITR: per-second granularity, up to 35 days
• TTL: automatic item deletion, no WCU consumption
• Import creates a new table; export requires PITR enabled

Module 4: Other AWS Databases

Amazon DocumentDB

Amazon DocumentDB is a fast, reliable, fully managed database service for hosting MongoDB workloads. It stores data as JSON formatted documents.

Exam Tip: If you see MongoDB, think Amazon DocumentDB.

Key Features

• Uses the same code, drivers, and tools as official MongoDB databases
• Storage volumes automatically grow as needed (up to 128 TiB, in 10 GB increments)
• Infrastructure similar to Amazon Aurora

Deployment Options

• Instance-based clusters
• Elastic clusters: supports millions of reads and writes per second
• Up to 15 Replica instances for read operations
• Must be deployed in a VPC
• Supports point-in-time recovery and encryption via KMS

Architecture

graph TB
    subgraph "DocumentDB Cluster in us-east-1"
        CE[Cluster Endpoint<br/>Read + Write] --> P[Primary Instance AZ-A]
        RE[Reader Endpoint] --> R1[Replica AZ-B]
        RE --> R2[Replica AZ-C]
        P --> CV[Cluster Volume<br/>Auto-scaling, 3 AZs]
        R1 --> CV
        R2 --> CV
    end

Exam Scenario: Migrating MongoDB workloads from on-premises to AWS → Amazon DocumentDB.

Amazon Neptune

Amazon Neptune is a fast, reliable, fully managed graph database service for applications that work with highly connected data.

What is a Graph Database?

Stores nodes and relationships instead of tables and documents. Common real-world uses:

• Social networks (Instagram, Facebook)
• Recommendation engines
• Knowledge graphs (e.g., Wikipedia)
• Fraud detection platforms

Key Features

• Optimized for storing billions of relationships
• Query graph data with millisecond latency
• Supports popular query languages:
  • Apache TinkerPop (Gremlin)
  • openCypher (Neo4j)
  • SPARQL
• High availability via Read Replicas and automatic replication across 3 AZs
• Point-in-time recovery and continuous backups to S3

Use Cases

• Building identity graphs (social networks, ad targeting)
• Security graphs for IT infrastructure analysis
• Real-time fraud pattern detection in financial transactions

Neptune Streams

• Logs every change in the graph exactly when it occurs, in exact order
• No duplicates
• Accessible via HTTP RESTful API
• Use cases: auto-notifications on data changes, streaming data to S3/OpenSearch/ElastiCache

Exam Tips

• See SPARQL, Gremlin, or openCypher → think Neptune
• Social media profiles and user relationship data → Neptune
• Large dataset with multiple levels of connections → Neptune

Amazon Keyspaces (for Apache Cassandra)

Amazon Keyspaces is a scalable, highly available, managed service for Apache Cassandra-compatible databases.

What is Cassandra?

An open-source, distributed NoSQL database used for big data solutions. Famous user: Netflix (for certain backend services).

Key Features

• Serverless — pay only for what you use
• Tables automatically scale up and down
• Thousands of requests per second with virtually unlimited throughput and storage
• Fully managed: handles provisioning, patching, server management, and software operation

Exam Tips

• CQL (Cassandra Query Language) → think Amazon Keyspaces
• Migrating big data Cassandra clusters to AWS → Keyspaces
• Applications with low latency, open-source requirements, or IoT workloads → consider Keyspaces

Amazon Quantum Ledger Database (QLDB)

Amazon QLDB is a fully managed ledger database service providing a transparent, immutable, and cryptographically verifiable transaction log.

What is a Ledger Database?

• Immutable — you cannot update or delete records; changes add new records, creating a chain of transactions
• Transparent — all changes are visible
• Cryptographically verifiable — the integrity of the transaction history can be mathematically verified
• Centralized authority — one source of truth

Key Differentiator: QLDB has a centralized component; Amazon Managed Blockchain has decentralized components.

Use Cases

• Financial transactions — complete, accurate records of credit/debit transactions
• Supply-chain systems — track history of manufactured goods from creation through sale
• Claims histories — insurance claim tracking with cryptographic data integrity
• Digital records — centralized employee records (payroll, benefits)

Key Facts

• 2–3x better performance than other blockchain frameworks
• Uses PartiQL (SQL-style queries)
• Immutable audit trail — new records are appended, never overwritten

Amazon Timestream

Amazon Timestream is a purpose-built, serverless, fully managed database service for time-series data and LiveAnalytics.

What is Time-Series Data?

Data points logged over a series of time that allow you to track changes and trends. Examples:

• Hourly temperature readings from IoT weather stations
• Server metrics sent to Prometheus
• Stock prices, trading volumes, market indicators

Key Features

• Stores trillions of time-series data points per day
• Built-in analytics functions
• Recent data kept in memory (fast access)
• Historical data moved to cost-optimized storage tier

Service Integrations

• AWS IoT Core — IoT sensor data ingestion
• Amazon MSK — managed Kafka
• Amazon Kinesis — real-time/near-real-time streaming
• Amazon QuickSight — data visualization and dashboards
• Grafana / Prometheus — monitoring and visualization
• Amazon SageMaker — machine learning models

Use Cases

• IoT sensor data (weather tracking, agriculture)
• Application Performance Monitoring (APM) — response times, error rates
• Financial trading and market data analysis

Exam Tip: Large-scale time-series data storage and analysis → Amazon Timestream.

Module 4 Summary and Exam Tips

Module 5: Other AWS Storage, Transfer, and Migration Services

AWS Snow Family

The AWS Snow Family is a set of secure physical appliances for migrating large-scale datasets into and out of AWS, as well as edge computing in remote locations.

graph TB
    subgraph "Snow Family Devices"
        SC[Snowcone<br/>Small, portable]
        SB[Snowball Edge<br/>Larger, more capable]
    end
    
    SC -->|Offline: Ship to AWS| S3[Amazon S3]
    SC -->|Online: DataSync| S3
    SB -->|Offline: Ship to AWS| S3

AWS Snowcone

A small, portable, secure device for edge computing and smaller data transfers.

Transfer methods:

• Offline — load data, ship device to AWS; AWS imports data to S3
• Online — use DataSync over a network connection

AWS Snowball Edge

The “jack-of-all-trades” edge computing device for larger workloads.

Process:

1. Request device → device delivered
2. Install required software on local machines
3. Load data onto device
4. Ship device back to AWS via regional carrier
5. AWS imports data to S3
6. Data is securely wiped from device

Exam Tips

• Large amounts of data (petabytes) + limited bandwidth + time constraint → Snow Family
• Snowcone = smaller datasets + space-constrained locations
• Snowball Edge = larger datasets (50+ TB/petabytes)

AWS Storage Gateway - File Gateways

AWS Storage Gateway is a hybrid cloud storage service that merges on-premises resources with cloud storage.

Amazon S3 File Gateway

Allows NFS and SMB (Samba) connections to S3 buckets from on-premises servers.

• Most recently accessed data is cached locally for immediate access
• Supports multipart uploads and byte-range downloads for efficiency
• Works with S3 lifecycle policies to transition data to Glacier tiers

graph LR
    OnPrem[On-Premises Server] -->|NFS or SMB| GW[File Gateway Appliance<br/>VMware/Hardware/EC2]
    GW -->|HTTPS over VPN/Direct Connect/Internet| S3[Amazon S3]
    S3 --> LC[Lifecycle Policies → Glacier]

Supported protocols: NFS, SMB (Samba)

Supported S3 classes: Standard, Standard-IA, Intelligent-Tiering, and more

Amazon FSx File Gateway

Provides low-latency performance for Amazon FSx for Windows File Server with local caching.

• Windows-native SMB capabilities
• NTFS file systems
• Active Directory integration
• Data deduplication

AWS Storage Gateway - Volume Gateways

Volume Gateways provide S3-backed storage mounted via iSCSI protocol.

• Data is backed up by EBS snapshots
• Uses iSCSI connection (not NFS or SMB)
• Snapshots are incremental after the first full snapshot

Two Volume Types

Memory aid:
- Stored = Store everything locally (backed up to S3)
- Cached = Cache the hot data; rest lives in S3

Transfer: Data goes over HTTPS to AWS (via VPN, Direct Connect, or internet); stored as EBS snapshots in S3.

Snapshots can be restored locally or to an EC2 instance.

Exam Tip: iSCSI protocol → Volume Gateway. NFS/SMB protocol → File Gateway.

AWS Storage Gateway - Tape Gateways

Tape Gateway replaces physical tape backups with S3-backed virtual tape storage using a Virtual Tape Library (VTL).

• Backup applications see virtual tape drives and media changers via iSCSI — they don’t know they’re virtual
• Data stored in S3 Glacier Flexible Retrieval or S3 Glacier Deep Archive
• Compatible with existing tape-based backup workflows

Components:

• Virtual Tape Library (VTL) — collection of virtual tapes
• Media Changer — acts like the robot arm moving physical tapes in a library
• Virtual Tape Drive — performs I/O operations on virtual tapes

AWS Lake Formation

AWS Lake Formation is a fully managed service for building, securing, and managing data lakes hosted in Amazon S3.

What is a Data Lake?

A centralized repository where you store and process large amounts of structured, semi-structured, and unstructured data in its raw form.

Key Difference:

• Data Lake — raw data in native format; messy; used to feed a data warehouse
• Data Warehouse — cleaned, processed, structured data; organized; used for OLAP queries

How Lake Formation Works

Uses AWS Glue on the backend for ETL operations.

Process:

1. Identify existing data stores (S3, databases, etc.)
2. Move data into the data lake
3. Crawl the migrated data (discover schema, types, relationships)
4. Catalog the data
5. Prepare data for analytics or data warehouse storage
6. Grant users/services secure, self-service access

Workflows (Blueprints)

Blueprints are complex, multi-job ETL activities that create Glue crawlers, jobs, and triggers. They can be run on-demand or scheduled.

Example: Ingest data from an Aurora MySQL database to your S3 data lake, then apply column-level authorization to restrict access.

Access Controls

• Fine-grained access control for users and applications
• Row-level and column-level security
• Tag-based access control — define permissions based on resource tags

Service Integrations

Exam Tips

• Centralize permissions/access controls for data lakes → Lake Formation
• Column-level security for QuickSight dashboards → Lake Formation
• Joining data lake data with operational database data → Lake Formation

AWS Transfer Family

AWS Transfer Family is a fully managed service for transferring files into and out of Amazon S3 or Amazon EFS using standard file transfer protocols.

Supported Protocols

• SFTP — Secure File Transfer Protocol
• FTPS — File Transfer Protocol over SSL
• FTP — File Transfer Protocol (VPC only)
• AS2 — Applicability Statement 2

Exam Tip: FTP endpoints can only be used within a VPC — not publicly accessible.

Key Features

• AWS manages infrastructure, scaling, and high availability
• Pricing: per endpoint per hour + data transfer charges
• Backend storage: Amazon S3 or Amazon EFS
• Endpoint assigned an IAM service role for backend storage access

Authentication Options

Exam Scenario

Uploading a PDF to S3 via SFTP that triggers a Lambda function when the object is created — perfect use of Transfer Family.

AWS DataSync

AWS DataSync migrates large amounts of data between external storage systems and AWS cloud storage.

Agent-Based vs Agentless

Supported Destinations

• Amazon S3 (all storage classes)
• Amazon EFS
• Amazon FSx (Lustre, OpenZFS, NetApp ONTAP, Windows)

Key Features

• Schedule replication tasks: hourly, daily, or weekly
• Copy file metadata
• Exclude specific files
• Limit bandwidth consumption
• Maintain existing file permissions
• Data integrity verification after transfer
• Up to 10 Gbps per task

Architecture Scenarios

graph TB
    subgraph "On-Premises → AWS"
        OP[On-Prem Server] -->|NFS/SMB/HDFS| Agent[DataSync Agent<br/>VM on-prem]
        Agent -->|TLS over VPN/Direct Connect/Internet| AWS[Amazon S3 / EFS / FSx]
    end
    
    subgraph "AWS → AWS (No Agent)"
        S3a[S3 Bucket Account A] --> S3b[S3 Bucket Account B]
    end
    
    subgraph "Edge Computing"
        Edge[Snowball Edge / Snowcone] --> AgentE[DataSync Agent]
        AgentE --> AWS2[Amazon S3]
    end

Exam Scenarios

1. Transferring large amounts of data between NFS or SMB file systems → DataSync
2. Verifying data integrity of a large dataset transferred over Direct Connect → DataSync

Key Distinction from Storage Gateway: DataSync is for syncing/migrating data between sources. Storage Gateway is for hybrid storage — offloading storage to the cloud with local caching.

AWS Backup

AWS Backup is a fully managed service for centralizing and automating data protection and backups across AWS services.

• Eliminates custom scripts and manual processing
• Supports cross-Region and cross-account backups
• Supports backups from other clouds and on-premises VMs

Supported AWS Services

EC2, EBS, RDS, Aurora, DynamoDB, EFS, FSx, S3, Storage Gateway, and more.

Key Concepts

• Backups are encrypted
• Supports Tag-based resource selection (only back up resources with specific tags)
• Supports PITR (Point-in-Time Recovery) for supported resources (DynamoDB, RDS, S3, etc.)

Backup Schedule Settings

• Frequency: Every hour, 12 hours, daily, weekly, or monthly; also manual trigger
• Backup Window: Start time (UTC) + duration in hours
• Complete Within: Maximum time allowed for backup to finish
• Retention Period: 1 day to 100 years; unspecified = indefinite

Lifecycle Management

• Backups start in warm storage (fast access, higher cost)
• Transition to cold storage (lower cost) after a configurable number of days

Backup Vault Lock

Optional feature enforcing a WORM (Write Once Read Many) data model.

Each vault can have exactly one vault lock.

Exam Scenarios

• Retain DynamoDB/Aurora backups for years efficiently → AWS Backup
• Back up EC2 and EBS volumes to another Region for DR → AWS Backup
• Nightly EBS volume backups with tag-based filtering → AWS Backup

AWS Application and Server Migration Services

The 6 R’s of Migration

AWS Application Discovery Service

Collects usage and configuration data about on-premises servers to plan migrations. Does not perform the migration itself.

Discovery Methods:

Integrates with AWS Migration Hub for centralized migration tracking.

AWS Application Migration Service (MGN)

Automated lift-and-shift (Rehosting) solution. Replaces the old Server Migration Service (SMS) and CloudEndure Migration.

• Replicates source servers into AWS using native cloud services
• Supports staging resources for verification before cutover
• RTO: minutes (depends on OS boot time)
• RPO: sub-second range

AWS Database Migration Service (DMS)

AWS DMS enables discovering, converting schemas, and fully managing migrations of data stores to the cloud.

Key Feature: You can keep the source database online during migration.

Migration Types

Replication Task Types

Source Endpoints

• On-premises or EC2 databases (MySQL, PostgreSQL, SQL Server, MongoDB, Oracle, Db2)
• Third-party managed databases (Azure, GCP)
• All Amazon RDS engine types
• Amazon S3
• Amazon DocumentDB

Target Endpoints

All source options plus: Redshift, Redshift Serverless, DynamoDB, OpenSearch Service, ElastiCache, Kinesis Data Streams, Apache Kafka, Amazon Neptune

Database Schema Conversions

Two tools for heterogeneous migrations:

DMS Schema Conversion (Web-Based)

• Web-based tool within the DMS console
• Assesses complexity of migration
• Converts schemas and objects automatically
• Marks unconvertible objects for manual review
• Less functionality than the downloadable tool

AWS Schema Conversion Tool (SCT)

• Downloadable, installable application
• Converts schemas for OLTP, OLAP, and non-relational databases
• More functionality than the web-based tool

Only used for heterogeneous migrations — you do NOT need it for same-engine migrations.

Conversion Examples

Exam Tip: RDS is not an engine type — it is the platform for running engines. SCT converts between engine types, not between RDS and something else.

Module 5 Summary and Exam Tips

Snow Family

• Snowcone: small, portable, 8–14 TB; offline or online (DataSync)
• Snowball Edge: large datasets (petabytes); Storage, Compute, Compute+GPU options
• 50+ TB with time constraint + bandwidth limits → Snowball Edge

Storage Gateways

• S3 File Gateway: NFS/SMB → S3 buckets; caches recent data locally
• FSx File Gateway: SMB → FSx for Windows; Windows-native features + AD
• Tape Gateway: replaces physical tapes; uses iSCSI; data in Glacier
• Volume Gateway: iSCSI protocol; Cached (S3 primary) vs Stored (local primary + S3 backup)

Data Lake vs Data Warehouse

• Data Lake: raw, mixed-format data; messy; stored in S3
• Data Warehouse: clean, structured, processed data; tables and dimensions

Lake Formation

• Centralized data lake governance and security
• Uses AWS Glue for ETL on the backend
• Column-level and row-level security for controlled data access

AWS Backup

• Centralized, automated backup management
• Cross-region and cross-account backups
• Tag-based resource selection
• WORM via vault locks: Compliance mode (strictest) vs Governance mode (testable)
• Warm storage → Cold storage lifecycle transitions

Migration Services

• Application Discovery: collect data for planning (no actual migration)
• MGN: automated lift-and-shift; RTO = minutes; RPO = sub-second
• DMS: migrate databases while keeping source online; Full Load/Full+CDC/CDC Only
• SCT: schema conversion for heterogeneous migrations only

Module 6: Big Data and Analytics Services

Amazon Redshift

Amazon Redshift is a fully managed, petabyte-scale data warehouse service for OLAP workloads.

The Three V’s of Big Data

Key Characteristics

• Based on PostgreSQL — but used for OLAP/analytics, not OLTP
• Connect using standard SQL clients and BI tools via JDBC/ODBC
• Up to 10x performance of other data warehouse solutions
• Uses column-based (columnar) storage — columns stored together, not rows
• Uses a massive parallel query engine (MPP) for high performance

Fun fact: Named “Redshift” as a play on getting people to leave Oracle databases.

Column-Oriented vs Row-Oriented Storage

Deployment Modes

Cluster Architecture

graph TB
    Client[SQL Client<br/>via JDBC/ODBC] --> Leader[Leader Node<br/>Receives, parses, plans queries]
    Leader --> CN1[Compute Node 1]
    Leader --> CN2[Compute Node 2]
    Leader --> CN3[Compute Node 3]
    CN1 -->|Results| Leader
    CN2 -->|Results| Leader
    CN3 -->|Results| Leader
    Leader -->|Final Results| Client

Exam Tip: Use Reserved Instances with provisioned clusters for cost savings.

Redshift High Availability, Snapshots, and Performance

High Availability

• Supports Multi-AZ deployments (spans 2 AZs) — newer feature
• Single-AZ is still the default for provisioned clusters
• Conversion between Single-AZ and Multi-AZ is supported

Snapshots

• Incremental, point-in-time captures stored in S3 (no direct access)
• Automated or manual
• Can be restored to a different cluster type (e.g., serverless → provisioned)
• Can be configured for automatic cross-region copying (best DR method)

Default automated snapshot schedule: every 8 hours OR every 5 GB of data changes per node — whichever comes first.

Snapshot contents: number of nodes, node types, admin username, all databases.

Performance Best Practices

• Load data using large batch inserts for best performance
• Split compressed files into multiple files to maximize MPP benefits

Data Ingestion Architectures

Architecture 1: Kinesis Data Firehose → Redshift

graph LR
    Apps[Applications<br/>clickstream, web, mobile] -->|Real-time| KDS[Kinesis Data Stream]
    KDS --> KDF[Kinesis Data Firehose<br/>auto-scaling, optional transform]
    KDF -->|Direct integration| RS[Amazon Redshift]

Architecture 2: Kinesis Data Firehose → S3 → Redshift

graph LR
    Apps[Applications] --> KDS[Kinesis Data Stream]
    KDS --> KDF[Kinesis Data Firehose]
    KDF --> S3[Amazon S3<br/>Data Lake]
    S3 -->|S3 COPY command + MPP| RS[Amazon Redshift]

Architecture 3: Amazon S3 → Redshift (Direct)

-- S3 COPY command example
COPY orders FROM 's3://your-bucket/data/'
IAM_ROLE 'arn:aws:iam::ACCOUNT_ID:role/RedshiftS3Role';

Enhanced VPC Routing

When enabled, forces all COPY and UNLOAD traffic through your VPC (not the public internet). Allows use of security groups, NACLs, firewalls, and VPC endpoints.

Exam Tip: Keep traffic private → Enable enhanced VPC routing. Also, maximize MPP by splitting data into multiple files.

Amazon Redshift Spectrum

Redshift Spectrum allows you to query data directly in Amazon S3 without loading it into your cluster first.

• Requires an existing Redshift cluster
• Offloads query processing to a managed Spectrum layer (up to thousands of Spectrum instances)
• Tables defined in Spectrum are registered in an external data catalog (e.g., AWS Glue)
• Data updated in S3 is immediately available for queries from any Redshift cluster

graph LR
    Client[SQL Client] --> RS[Your Redshift Cluster]
    RS -->|Query offloaded| Spectrum[Redshift Spectrum Layer<br/>1000s of managed instances]
    Spectrum -->|Read directly| S3[Amazon S3]
    S3 -->|Results| Spectrum
    Spectrum -->|Results| RS
    RS -->|Results| Client

Exam Tip: Query S3 without loading data into Redshift cluster → Redshift Spectrum.

Amazon Elastic MapReduce (EMR)

Amazon EMR is an AWS-managed big data platform for processing vast amounts of data using open-source tools.

ETL Review

Open-Source Tools Supported

If you see any of the following, consider EMR in your solution:

• Apache Spark — in-memory big data processing
• Apache Hive — SQL-like queries on Hadoop
• HBase — NoSQL on Hadoop
• Apache Flink — real-time stream processing
• Apache Hudi — data lake management
• Presto — distributed SQL query engine

Exam Tip: Large data + Hadoop clusters → Amazon EMR.

EMR Storage Types

Node Types

Purchase Options

Cluster Types

• Long-running: permanent cluster; use Reserved or On-Demand instances
• Transient: temporary cluster; spins up for a job, then shuts down; best for cost savings

Use Cases

• Large-scale ETL workloads
• Web indexing (clickstream data, web logs)
• Large-scale genomics projects

AWS Glue

AWS Glue is a serverless data integration service for discovering, preparing, combining, and transforming data.

• Connect to up to 70 different data sources
• Manage data in one centralized Data Catalog

Core Components

graph LR
    Sources[Data Sources<br/>S3, JDBC, RDS, etc.] -->|Crawl| Crawlers[Glue Crawlers]
    Crawlers -->|Write metadata| Catalog[Glue Data Catalog<br/>Tables + Databases]
    Catalog -->|Define ETL| Jobs[Glue ETL Jobs<br/>Triggered/Scheduled]
    Jobs -->|Transform & Load| Targets[S3 / Redshift / etc.]
    Targets --> Athena[Amazon Athena<br/>Ad-hoc SQL queries]

Architecture Options

Option 1: Scheduled Crawler + Scheduled Job

1. User uploads CSV/JSON to S3 source bucket
2. Scheduled Glue crawler crawls the bucket and discovers new data
3. Scheduled Glue job performs ETL (e.g., convert CSV to Parquet)
4. Transformed data loaded to target S3 bucket
5. Amazon Athena queries the transformed data

Option 2: Event-Driven (S3 Event → Lambda → Glue Job)

1. Object created event triggers S3 event notification
2. Lambda function invoked → triggers Glue ETL job
3. ETL executes and loads data to target

Exam Tip: EventBridge can replace Lambda in this architecture.

Additional Features

Glue is used behind the scenes by: Redshift Spectrum, EMR, Amazon Athena, and open-source Hive metastore clients.

Amazon Athena

Amazon Athena is a serverless, interactive query service for analyzing data directly in Amazon S3 using SQL.

• Based on open-source Presto engine
• No servers to manage
• Pay per query (per TB of data scanned)

Supported Data Formats

Exam Tip: See Parquet + queries → immediately think Athena.

Use Cases

• Log analysis: CloudTrail logs, VPC Flow Logs, load balancer logs, CloudFront logs
• Business intelligence: ad-hoc reports, sales data analysis, behavioral analytics
• Data lake queries: raw or structured data in supported formats

Best Practices

• Use columnar formats (ORC, Parquet) for optimal performance
• Convert data formats using AWS Glue before querying
• Use data partitioning with specific prefixes for faster queries

Partitioning Examples:

Time-based: s3://bucket/AWSLogs/year=2024/month=01/day=15/hour=08/ Category-based: s3://bucket/AWSLogs/region=us-east-1/category=vpc/

The more specific the prefix, the faster the query.

Demo: Querying VPC Flow Logs with Athena

-- Step 1: Create database
CREATE DATABASE vpc_flow_logs_db;

-- Step 2: Create external table with Hive-compatible partitions
CREATE EXTERNAL TABLE IF NOT EXISTS `vpc_flow_logs` (
  version int,
  account_id string,
  interface_id string,
  srcaddr string,
  dstaddr string,
  srcport int,
  dstport int,
  protocol bigint,
  packets bigint,
  bytes bigint,
  start bigint,
  `end` bigint,
  action string,
  log_status string
)
PARTITIONED BY (
  `aws-account-id` string,
  `aws-service` string,
  `aws-region` string,
  `year` string,
  `month` string,
  `day` string,
  `hour` string
)
ROW FORMAT SERDE
  'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
STORED AS INPUTFORMAT
  'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'
OUTPUTFORMAT
  'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
LOCATION
  's3://BUCKET_NAME/AWSLogs/'
TBLPROPERTIES (
  'EXTERNAL'='true',
  'skip.header.line.count'='1'
);

-- Step 3: Repair table to load partition metadata
MSCK REPAIR TABLE `vpc_flow_logs`;

-- Step 4: Query data
SELECT * FROM vpc_flow_logs LIMIT 10;

-- Step 5: Filter by destination port
SELECT * FROM vpc_flow_logs WHERE dstport = 8080 LIMIT 10;

-- Cleanup (destructive!)
DROP TABLE vpc_flow_logs;
DROP DATABASE vpc_flow_logs_db;

Key Insight: Dropping the Athena table/database does NOT delete the underlying S3 data. Athena only queries data — it does not modify or delete files in S3.

Athena Federated Queries

Use when data is stored in sources other than Amazon S3.

• Uses data source connectors (Lambda functions) to connect to custom data sources
• Query relational, non-relational, object, and custom data sources

Prebuilt connectors: CloudWatch Logs, DynamoDB, DocumentDB, Amazon RDS, any JDBC-compliant database

Custom connectors: Build using the Athena Query Federation SDK

Amazon QuickSight

Amazon QuickSight is a fully managed, serverless business intelligence data visualization service powered by machine learning.

• Creates interactive dashboards shared with specific users
• Integrates with RDS, Aurora, Athena, S3, and many more

Key Concepts

Users, Groups, and Dashboards

• QuickSight users and groups only exist in QuickSight — not IAM entities
• Enterprise edition allows group creation
• Dashboards store configurations and filters; shareable with users/groups
• Sharing a dashboard exposes the underlying data — control carefully

Enterprise Edition Extras

• Machine learning to uncover insights, trends, and forecast business metrics
• Column-level security (CLS)
• User groups

Exam Tips

• “Business Intelligence” → QuickSight
• Visualize Athena query results → QuickSight
• Need to show KPIs in dashboards → QuickSight
• SPICE = fast in-memory calculations
• Enterprise → ML, CLS, user groups

Amazon OpenSearch Service

Amazon OpenSearch Service is a managed service for running search and analytics engine clusters. It is the successor to Amazon ElasticSearch Service.

• Search all fields within data (not limited to primary keys or indexes)
• Built-in Kibana-style dashboard for visualization

Security

• Access control via IAM or Amazon Cognito
• Encryption at rest using KMS; TLS in transit
• VPC PrivateLink for private-only access

Domain

A domain is AWS’s equivalent of an OpenSearch cluster — it has specific settings, instance types, counts, and storage allocation.

Access Policies (3 Types)

1. Resource-based (Domain Access Policy) — attached to the domain resource
2. Identity-based (IAM Policy) — attached to users and roles
3. IP-based Policy — restricts access by IP address or CIDR block

Ingestion Architectures

Architecture 1: Kinesis Data Firehose → Lambda (transform) → OpenSearch

graph LR
    KDF[Kinesis Data Firehose] --> Lambda[Lambda<br/>Data Transform]
    Lambda --> OS[OpenSearch Service]

Architecture 2: Kinesis Data Streams → Lambda → OpenSearch

• OpenSearch does NOT support direct ingestion from Kinesis Data Streams
• Use Lambda in between

Architecture 3: DynamoDB Streams → Lambda → OpenSearch

graph LR
    DDB[DynamoDB Table] -->|Streams| Lambda[Lambda Function]
    Lambda --> OS[OpenSearch Service]
    APIGW[API Gateway] -->|Search items| OS
    APIGW -->|Get/Update/Delete| DDB

Architecture 4: CloudWatch Logs → OpenSearch

• Set up a subscription filter on a CloudWatch Log Group
• AWS manages a Lambda function behind the scenes for ingestion
• Or route through Kinesis Data Firehose for more scalability

OpenSearch Serverless

On-demand, auto-scaling for infrequent, intermittent, or spiky workloads. Pay for consumed capacity units.

Module 6 Summary and Exam Tips

Amazon Redshift

• Petabyte-scale data warehouse; OLAP workloads; PostgreSQL-based
• Column-oriented storage + MPP = high analytics performance
• Single-AZ (default) or Multi-AZ; snapshot cross-region for DR
• Large batch inserts for best performance
• Reserved Instances for cost savings on provisioned clusters
• Kinesis Firehose → Redshift; S3 COPY command; Enhanced VPC Routing for private traffic

Redshift Spectrum

• Query S3 directly without loading into cluster
• Uses managed Spectrum instances; requires an existing Redshift cluster

Amazon EMR

• Managed big data platform for Hadoop, Spark, Hive, etc.
• Primary nodes (manage cluster), Core nodes (store + process), Task nodes (process only, transient)
• HDFS for intermediate data; EMRFS for S3 I/O; local FS for temporary caching
• Mix On-Demand/Reserved for Primary+Core; Spot for Task nodes
• Long-running or transient clusters

AWS Glue

• Serverless ETL and data integration
• Crawlers → Data Catalog → ETL Jobs → Targets
• Job bookmarks prevent reprocessing
• Powers Redshift Spectrum, EMR, Athena behind the scenes

Amazon Athena

• Serverless SQL queries on S3
• Supports CSV, JSON, ORC, Avro, Parquet (Parquet recommended)
• Partition data by time or category for performance
• Federated queries via Lambda connectors for non-S3 sources
• Dropping tables does NOT delete S3 data

Amazon QuickSight

• Business intelligence dashboards and reports
• SPICE for fast in-memory calculations
• Enterprise: ML insights, column-level security, user groups
• Pay per session (reader role)

Amazon OpenSearch

• Search all fields in data (not just indexed fields)
• Successor to ElasticSearch
• Ingestion: Kinesis Firehose (near-real-time), Kinesis Streams + Lambda (real-time), CloudWatch subscription filters
• Serverless for unpredictable workloads

Module 7: Machine Learning Services

Amazon SageMaker

Amazon SageMaker is the fully managed machine learning service for data scientists, engineers, and developers to build, train, and deploy machine learning models.

• Store and share data without provisioning servers
• Simplifies ML workflows: building, training, deploying, and using models

Key Concepts

Exam Tip: You can host and deploy your own custom ML models via a SageMaker endpoint.

Automation Features

SageMaker Canvas (formerly Autopilot)

• No-code/low-code ML predictions
• Chat with LLMs, use ready-to-use models, or build custom ones
• Predict customer churn, identify objects or text

SageMaker JumpStart

• Pretrained, open-source models to get started quickly
• Can be fine-tuned and incrementally trained

Cost Savings

SageMaker runs on EC2 instances — use SageMaker Savings Plans for cost savings (similar to EC2 Savings Plans).

Exam Scenario

Requirement: Build and train custom ML models for an online shopping platform, visualize complex scenarios, detect trends, and share with business analysts via BI dashboards.

Solution: Amazon SageMaker (ML models) + Amazon QuickSight (BI dashboards)

Note: SageMaker generally requires some ML experience to use effectively. Canvas provides low-code options, but most features target practitioners.

Amazon Rekognition

Amazon Rekognition is AWS’s computer vision service that automates recognition of pictures and videos using deep learning and neural networks.

Capabilities

• Object/label detection — identifies objects (cars, phones, hats, etc.)
• Face recognition — match faces, compare faces, detect celebrities
• Facial analysis — emotions, accessories (glasses), eye position, age range
• Text detection — reads text within images
• Streaming video event detection — real-time analysis of video streams

Confidence Scores

Confidence scores indicate the likelihood that a detection is correct. Higher score = greater confidence. You can set a minimum confidence threshold to flag items for review.

Content Moderation Architectures

Architecture 1: Static Images in S3

graph LR
    Users[Users Upload Images] --> S3[Amazon S3]
    S3 -->|Batch image analysis| Rekog[Amazon Rekognition<br/>Image API]
    Rekog -->|Low confidence score| A2I[Amazon Augmented AI<br/>Human Review]
    Rekog -->|Confidence score result| Results[Flag / Allow Content]

Architecture 2: Live Video Streaming

graph LR
    Camera[Live Video Source] --> KVS[Kinesis Video Streams]
    KVS -->|Synchronous analysis| Rekog[Amazon Rekognition<br/>Video API]
    Rekog -->|Metadata + confidence scores| DDB[DynamoDB / RDS]

Exam Tips

• Content moderation (prevent unwanted content) without training your own model → Rekognition
• Face detection for access control (door locks, security cameras) → Rekognition
• Real-time person/object detection in video (Ring-style alerts) → Rekognition + Kinesis Video Streams

Amazon Polly

Amazon Polly converts text to speech — enabling applications to talk to users using lifelike, natural-sounding voices.

Voice Engine Types

Use Cases

• Mobile apps that read news aloud
• eLearning platforms reading lessons out loud
• Accessibility for visually impaired users

Document Types

• Plain text — simple text input
• SSML (Speech Synthesis Markup Language) — HTML-like tags for controlling pronunciation, breaks, emphasis, volume, rate

SSML Examples

<!-- Add a pause (break) -->
<speak>
    Hey there! <break time="1s"/>Thank you for taking this course.
</speak>

<!-- Add emphasis -->
<speak>
    I <emphasis level="strong">really</emphasis> hope it is useful in your journey.
</speak>

Pronunciation Lexicons

Customize how specific words are pronounced. Uploaded as XML files to Polly (regional resource).

<?xml version="1.0" encoding="UTF-8"?>
<lexicon version="1.0"
         xmlns="http://www.w3.org/2005/01/pronunciation-lexicon"
         alphabet="ipa"
         xml:lang="en-US">

    <lexeme>
        <grapheme>GCP</grapheme>
        <alias>Google Cloud Platform</alias>
    </lexeme>

    <lexeme>
        <grapheme>AWS</grapheme>
        <alias>Amazon Web Services</alias>
    </lexeme>

</lexicon>

This lexicon makes Polly say “Amazon Web Services” whenever it reads “AWS”, and “Google Cloud Platform” for “GCP”.

Amazon Translate

Amazon Translate is an advanced machine learning service for automating language translations within applications.

• Translates unstructured text documents or builds real-time multilingual applications
• Uses deep learning and neural networks

Use Cases

• Multilingual user experiences (global apps)
• Translating company blogs, emails, customer service chats
• In-game live chat translation for global player bases

Translation Methods

1. Direct API call or console input — pass text, get translation back
2. Document upload — supports .txt, .html, .doc formats; upload, set source/target language, download translated file

Exam Scenario

Transcribing customer service calls in multiple languages and translating them to English for sentiment analysis → Amazon Translate + Amazon Transcribe + Amazon Comprehend.

Key Distinction: Translate translates between languages. It does not capture audio or extract text — you must pass in the text.

Amazon Lex

Amazon Lex builds conversational interfaces (chatbots) via text or voice. It powers Amazon Alexa on the backend.

• Uses Natural Language Understanding (NLU) to comprehend and interpret spoken/typed language (determines intent)
• Uses Automatic Speech Recognition (ASR) to convert spoken language to text

Use Cases

• Customer service chatbots for e-commerce
• Virtual agents and voice assistants

Exam Tip: “Chatbot” or “conversational interface” without ML experience → Amazon Lex.

Amazon Connect

Amazon Connect is an AI-powered contact center in the cloud for scaling customer experience.

• Receives customer service calls and routes them
• Integrates with CRM software (Zendesk, ServiceNow)
• Proactively engages customers (notices, email campaigns, rating requests)

Exam Tip: Connect is commonly used with Amazon Lex for customer service flows. Lex handles the conversational bot; Connect handles the contact center workflow.

Common Architecture: Customer calls → Connect → Lex bot handles initial interaction → Routes to human agent or automated action

Amazon Comprehend

Amazon Comprehend uses Natural Language Processing (NLP) to understand the meaning and sentiment within textual data.

Key Phrases to Watch For: NLP, sentiment analysis, key phrases, language detection

Capabilities

• Detect whether text is positive, negative, or neutral
• Identify key phrases and specific terminology
• Automate understanding of customer feedback at scale

Use Cases

• Analyze customer service call quality (positive/negative experiences)
• Index product reviews and detect sentiment (good/bad products)
• Automate tagging and categorization of documents

Amazon Comprehend Medical

Specifically detects and returns medical information from unstructured clinical text (doctor notes, discharge summaries, test results).

• Uses NLP to detect medical conditions, medications, dosages
• Specializes in identifying Protected Health Information (PHI)
• Supports asynchronous (documents in S3) and synchronous (real-time API) operations

Exam Scenario: Hospital uploads PDF reports to S3; use Comprehend Medical to detect PHI before sending reports to patients.

Amazon Forecast

Amazon Forecast uses machine learning to provide accurate time-series forecasting.

Note: Amazon Forecast is no longer available to new customers but can still appear on the exam.

• Automatically selects the best ML algorithm for your data
• Reduces forecasting time from weeks to hours
• Much more accurate than manual forecasting methods
• Used by amazon.com for Prime Day sales forecasting

Concepts

• Datasets — store input data and train predictors
• Predictors — ML models trained on your time-series data

Use Cases

• Predicting product demand in retail stores
• Agricultural planning from IoT sensor weather data
• Financial market data analysis (prices, trading volumes, market indicators)
• Resource planning using historical values

Exam Tip: Time-series data + forecasting future results → Amazon Forecast.

Amazon Kendra

Amazon Kendra is an intelligent search service powered by machine learning and NLP for enterprise search applications.

• Bridges information silos (S3, file servers, websites, databases)
• Provides semantic and contextual understanding (not just keyword matching)
• Supports descriptive questions, keyword searches, and fact lookups

Key Features

• Incremental learning — improves ranking algorithm based on user feedback
• Relevance tuning — adjust importance of specific fields/attributes in search results (manual or automatic)

Use Cases

• Accelerate R&D by centralizing research papers from disparate sources
• Minimize regulatory/compliance risk by tracking regulatory updates (HIPAA, ISO, PCI)
• Improve customer interactions with better answers
• Increase employee productivity by making internal knowledge easily searchable

Amazon Textract

Amazon Textract uses machine learning (including OCR — Optical Character Recognition) to automatically extract typed text, handwritten text, and data from scanned documents.

• Extracts information from PDFs, JPEGs, and other document formats
• Works with forms, tables, and structured documents

Use Cases

• Financial services: process invoices, receipts, loan applications
• Public sector: process IDs, driver’s licenses, passports; autofill declaration forms
• Healthcare: extract text from medical records and forms

Exam Scenario

Hospital uploads health documents (JPEG/PDF) to S3 → Textract extracts text → Send to Comprehend Medical to detect and redact PHI.

Amazon Personalize

Amazon Personalize is a fully managed ML service for real-time personalized recommendations based on your own user data.

• Generates recommendations from interaction data (clicks, purchases, views)
• Supports bulk data (S3) and real-time API calls
• Used by amazon.com for product recommendations

Use Cases

• Product recommendations on e-commerce sites
• Video recommendations on streaming platforms (like YouTube’s “For You” section)
• Personalized search results

Key Principle: Recommendations are made based on interaction data — what users click, buy, watch, etc.

Amazon Transcribe

Amazon Transcribe converts speech to text — it is the opposite of Polly.

• Standalone transcription service or integrated into existing applications
• Supports batch transcriptions (upload to S3) and real-time streaming transcriptions

Key Features

• Automatic language identification — no need to specify the language
• Custom vocabulary filters — modify how words appear in the output
• Toxic speech detection — for moderating social media or gaming platforms
• Redaction — masks or removes Personally Identifiable Information (PII) from transcription output
• Subtitle generation — automatically create subtitles for videos

Key Distinction: Transcribe = speech-to-text. Polly = text-to-speech.

Exam Scenario

Financial company stores voice call audio in S3 → Transcribe captures the text from audio → Remove PII (names, addresses) using Transcribe redaction.

Amazon Fraud Detector

Amazon Fraud Detector is a fully managed fraud detection service for automating detection of potentially fraudulent activities.

What it Detects

• Unauthorized financial transactions
• Fake/bot user account creation
• Free trial account abuse

Exam Tip: Need a highly customizable fraud detection ML model built on your own data → Amazon Fraud Detector.

Module 7 Summary and Exam Tips

SageMaker

• One-stop shop for all ML workflows: build, train, deploy
• Use Canvas for no-code/low-code ML predictions
• JumpStart for pretrained models
• Deploy models to endpoints for real-time inference
• SageMaker Savings Plans for cost savings

Rekognition, Textract, Transcribe

• Rekognition: computer vision — images and videos; content moderation, face analysis, object detection
• Textract: extract text from documents using OCR — handwritten or typed; forms and tables
• Transcribe: speech-to-text; subtitles; PII redaction; batch or real-time

Polly, Lex, Connect

• Polly: text-to-speech; SSML for control; lexicons for pronunciation; AWS Blog uses Polly
• Lex: conversational chatbots and voice assistants; powers Alexa; no ML experience needed
• Connect: AI-powered contact center; integrates with CRM; commonly used with Lex

Translate and Comprehend

• Translate: language translation; multilingual user experiences; documents and real-time API
• Comprehend: NLP, sentiment analysis; positive/negative; Comprehend Medical for PHI detection

Forecast, Kendra, Personalize, Fraud Detector

• Forecast: time-series forecasting; auto-selects ML algorithm; no ML experience needed
• Kendra: intelligent enterprise search; semantic understanding; bridges data silos
• Personalize: real-time recommendations from user interaction data; amazon.com product recs
• Fraud Detector: fraud detection for transactions, bot accounts, and trial abuse

Quick Reference: Service Selection Guide

Database Service Selector

Storage and Migration Selector

Analytics Service Selector

Machine Learning Service Selector

End of Study Guide

Search Terms

aws · storage · databases · ml · data · core · services · amazon · web · exam · dynamodb · aurora · cases · rds · database · service · types · features · architecture · migration · options · backup · concepts · encryption