Module 1 – Azure Blob Storage
Overview
- Object storage service for unstructured data.
- Hierarchy: Storage Account → Container → Blob.
- Unique URL for each blob:
https://<account>.blob.core.windows.net/<container>/<blob>.
Blob Types
| Type | Description | Use Case |
|---|---|---|
| Block Blob | Optimized for sequential upload (blocks) | Text files, images, videos, backups |
| Page Blob | Optimized for random access, max 8 TB | VHDs (Azure VM disks) |
| Append Blob | Optimized for appending data | Application logs, audit journals |
Access Tiers
| Tier | Storage Cost | Access Cost | Use Case |
|---|---|---|---|
| Hot | High | Very fast, low cost | Active data, frequently accessed |
| Cool | Medium | Medium | Infrequent data, 30-day minimum |
| Cold | Low | Slower | Rarely accessed data, 90-day min |
| Archive | Very low | Very slow (hours) + high retrieval cost | Long-term backups, compliance |
Lifecycle Management: automate tier transitions.
{
"rules": [{
"name": "MoveToArchive",
"type": "Lifecycle",
"definition": {
"filters": {"blobTypes": ["blockBlob"]},
"actions": {
"baseBlob": {
"tierToArchive": {"daysAfterModificationGreaterThan": 90}
}
}
}
}]
}
Module 2 – Redundancy (Durability)
| Option | Copies | Location | Durability SLA |
|---|---|---|---|
| LRS (Locally Redundant Storage) | 3 | Same datacenter | 99.999999999% (11 nines) |
| ZRS (Zone-Redundant Storage) | 3 | 3 different zones, same region | 99.9999999999% (12 nines) |
| GRS (Geo-Redundant Storage) | 6 | LRS + async replication to a second region | 99.99999999999999% (16 nines) |
| GZRS (Geo-Zone-Redundant Storage) | 6 | ZRS + replication to a second region | 16 nines |
| RA-GRS | 6 | GRS + read access from secondary region | |
| RA-GZRS | 6 | GZRS + read access from secondary region |
Selection Rules
Non-critical dev/test data → LRS (cheapest)
Single-region production apps → ZRS
Disaster Recovery required → GRS or GZRS
Module 3 – Azure Files
Characteristics
- Cloud file shares accessible via SMB 3.0 (Windows, Linux, macOS) or NFS 4.1 (Linux).
- Mountable as a network drive (
Z:on Windows,/mnt/shareon Linux). - Accessible from on-premises and Azure.
Azure File Sync
Windows File Server On-Premises
↕ Azure File Sync Agent
Azure Files (cloud)
- Bidirectional replication.
- Cloud tiering: infrequently accessed files are stored only in Azure (frees local space).
- Disaster recovery: restore the share from Azure if the local server goes down.
Use Cases
- Replace on-premises file servers.
- Share configuration files between Azure VMs.
- Legacy applications that require SMB file shares.
Module 4 – Storage Encryption
Server-Side Encryption (SSE)
Methods:
| Method | Description |
|---|---|
| Microsoft-managed keys | Default, automatic, transparent |
| Customer-managed keys (CMK) | Your key in Azure Key Vault, control over rotation/revocation |
| Customer-provided keys | Key provided with each request (Azure Blob only) |
Azure Portal → Storage Account → Security + Networking → Encryption
→ Encryption type: Customer-managed keys
→ Key Vault: [select vault]
→ Key: [select key]
VM Disk Encryption
- Azure Disk Encryption (ADE): uses BitLocker (Windows) or DM-Crypt (Linux).
- Integrated with Azure Key Vault for key management.
Module 5 – Azure Backup
Concepts
- Managed backup service (PaaS) integrated with Azure.
- Recovery Services Vault: container that stores backup data.
- Supports: Azure VMs, SQL Server in VMs, Azure Files, SAP HANA, blobs, databases.
Features
| Feature | Description |
|---|---|
| Application-consistent snapshots | Application data consistency (not just OS) |
| Instant restore | Restore from local snapshots (before vault transfer) |
| Flexible schedule | Daily, weekly, monthly, yearly |
| Soft delete | Deleted data retained 14 days before permanent deletion |
| Cross-region restore | Restore to another region in case of disaster |
Configure a VM Backup
Azure Portal → Recovery Services Vault → Backup
→ Backup goal: Azure / Virtual machine
→ Create backup policy:
- Frequency: Daily at 2:00 AM
- Retention: 30 days daily, 12 months monthly
→ Select VMs → Enable Backup
Summary
| Concept | Key Point |
|---|---|
| Block Blob | General-purpose files (images, videos, documents) |
| Page Blob | VM disks (VHDs) |
| Append Blob | Application logs |
| Hot/Cool/Archive | Increasing cost tiers / decreasing access speed |
| LRS/ZRS/GRS | More redundancy = more resilience = higher cost |
| Azure Files | SMB/NFS shares mountable as network drives |
| Azure File Sync | Sync on-premises ↔ Azure Files |
| Customer-managed keys | Full encryption control via Key Vault |
| Recovery Services Vault | Central hub for Azure Backup |
| Instant restore | Local snapshots for fast restore |
Module 6 – Storage Service Types (Comparative Overview)
Azure Storage groups several distinct services under a single account. Each service is optimized for a specific use case.
| Service | Protocol / API | Data Type | Typical Use Case |
|---|---|---|---|
| Azure Blob Storage | REST HTTP/S | Unstructured data (binary/text) | Images, videos, backups, logs, ML datasets |
| Azure Files | SMB 3.x / NFS 4.1 | Hierarchical files | Enterprise file shares, file server migration |
| Azure Queue Storage | REST HTTP/S | Messages (max 64 KB each) | Component decoupling, async processing, IoT |
| Azure Table Storage | REST / OData | Semi-structured data (NoSQL key-value) | Catalogs, metadata, session data |
| Azure Disk Storage | Block (internal iSCSI) | Managed disks for VMs | OS disks, data disks, high-performance apps |
Diagram — Storage Account Hierarchy
graph TD
SA[Storage Account]
SA --> BLOB[Blob Storage\nContainers → Blobs]
SA --> FILES[Azure Files\nFile Shares → Directories → Files]
SA --> QUEUE[Queue Storage\nQueues → Messages]
SA --> TABLE[Table Storage\nTables → Entities]
SA --> DISK[Disk Storage\nManaged Disks - external to account]
Note: Azure Disk Storage is a separate service — managed disks do not live in a standard storage account; they have their own resource group and API.
Module 7 – Azure Blob Storage — Deep Dive
7.1 Structure and Containers
The Blob hierarchy is: Storage Account → Container → Blob.
- A container is analogous to a root folder.
- Container names must be lowercase, between 3 and 63 characters.
- Blob URL:
https://<account>.blob.core.windows.net/<container>/<path/blob>
Container public access levels:
| Level | Behavior |
|---|---|
| Private (default) | Access only via key or SAS token |
| Blob | Anonymous read of individual blobs (no listing) |
| Container | Anonymous read + listing of container blobs |
7.2 Access Tiers — Details and Relative Costs
| Tier | Storage Cost | Read Cost | Access Latency | Min Duration | Use Case |
|---|---|---|---|---|---|
| Hot | $$$ | $ | Milliseconds | None | Active data, websites, streaming |
| Cool | $$ | $$ | Milliseconds | 30 days | Short-term backups, infrequent data |
| Cold | $ | $$$ | Milliseconds | 90 days | Intermediate archives, regulatory data |
| Archive | ¢ | $$$$ | 1–15 hours (rehydration) | 180 days | Legal compliance, long-term backup |
The Archive tier requires rehydration before access: either
Standard(up to 15 h) orHigh Priority(under 1 h, higher cost).
7.3 Lifecycle Management Policies
JSON policy that automates blob tier transitions or deletion:
{
"rules": [
{
"name": "MoveCoolThenArchive",
"enabled": true,
"type": "Lifecycle",
"definition": {
"filters": {
"blobTypes": ["blockBlob"],
"prefixMatch": ["logs/", "backups/"]
},
"actions": {
"baseBlob": {
"tierToCool": { "daysAfterModificationGreaterThan": 30 },
"tierToArchive": { "daysAfterModificationGreaterThan": 90 },
"delete": { "daysAfterModificationGreaterThan": 365 }
},
"snapshot": {
"delete": { "daysAfterCreationGreaterThan": 90 }
}
}
}
}
]
}
7.4 Immutability (WORM — Write Once Read Many)
Immutable storage protects data from modification or deletion:
| Policy Type | Description |
|---|---|
| Time-based retention | Data locked for a defined duration (e.g., 7 years) |
| Legal hold | Lock without fixed duration, until explicitly released |
Use cases: regulatory compliance (SEC 17a-4, FINRA, CFTC), medical data, audit logs.
Container → Policies → Immutability policy
→ Time-based: 2555 days (7 years)
→ Lock policy (irreversible once locked)
7.5 Versioning, Soft Delete, and Change Feed
| Feature | Description | Activation |
|---|---|---|
| Blob Versioning | Automatically creates a version on each modification | Storage Account → Data protection |
| Soft Delete | Deleted blobs retained N days (1–365) | Storage Account → Data protection |
| Change Feed | Immutable log of all blob modifications | Storage Account → Data protection |
| Point-in-time restore | Restore the state of all blobs to a point in time | Requires versioning + soft delete enabled |
7.6 Static Website Hosting
Azure Blob Storage can serve HTML/CSS/JS files directly:
Storage Account → Static website → Enabled
→ Index document: index.html
→ Error document: 404.html
- Generated URL:
https://<account>.z13.web.core.windows.net/ - Combined with Azure CDN for custom HTTPS and global performance.
- Minimal cost: pay only for storage + outbound bandwidth.
Module 8 – Azure Files — Deep Dive
8.1 Supported Protocols
| Protocol | Version | Compatible OS | Port |
|---|---|---|---|
| SMB | 3.0, 3.1.1 | Windows, Linux (Samba), macOS | 445 |
| NFS | 4.1 | Linux, macOS | 2049 |
| REST | — | Any HTTP client | 443 |
SMB 3.1.1 brings AES-128-GCM encryption and improved pre-authentication security.
8.2 Mounting a Share
Windows (PowerShell):
$connectTestResult = Test-NetConnection -ComputerName myaccount.file.core.windows.net -Port 445
if ($connectTestResult.TcpTestSucceeded) {
net use Z: \\myaccount.file.core.windows.net\myshare /u:AZURE\myaccount <access_key>
}
Linux (fstab):
sudo mount -t cifs //myaccount.file.core.windows.net/myshare /mnt/share \
-o vers=3.0,username=myaccount,password=<access_key>,serverino
8.3 Azure File Sync — Detailed Architecture
Azure File Sync lets you turn any Windows server into a local cache of an Azure Files share.
graph LR
subgraph On-Premises
WFS1[Windows File Server\nParis Office]
WFS2[Windows File Server\nLyon Office]
end
subgraph Azure
AF[Azure Files\nCloud Share]
SS[Storage Sync Service]
end
WFS1 -- File Sync Agent --> SS
WFS2 -- File Sync Agent --> SS
SS <--> AF
Key concepts:
| Concept | Description |
|---|---|
| Storage Sync Service | Azure coordination resource |
| Sync Group | Groups a cloud endpoint + 1 to N server endpoints |
| Cloud Endpoint | The Azure Files share in the sync group |
| Server Endpoint | Path on a registered Windows server |
| Cloud Tiering | Replaces infrequently accessed files with stubs (pointers) |
Cloud Tiering: keeps only recently accessed files locally. “Tiered” files appear normally in File Explorer but are downloaded on demand from Azure.
8.4 Integration with Active Directory Domain Services
Azure Files supports Kerberos authentication via:
- On-premises AD DS: join the storage account to the existing domain.
- Azure AD DS: managed AD service in Azure.
- Azure AD Kerberos (hybrid files): for cloud-only users.
This allows replacing on-premises file servers while preserving existing NTFS permissions and ACLs.
Module 9 – Redundancy — Deep Dive
9.1 Full Comparison Table
| Option | Copies | Distribution | Protects Against | RPO | RTO | Secondary Read |
|---|---|---|---|---|---|---|
| LRS | 3 | 1 datacenter, 1 region | Local hardware failure | N/A | Low | No |
| ZRS | 3 | 3 zones, 1 region | Datacenter / zone failure | N/A | Low | No |
| GRS | 6 | Primary LRS + secondary LRS (other region) | Regional disaster | ~15 min | Min–hours | No (except failover) |
| GZRS | 6 | Primary ZRS + secondary LRS | Zone + regional disaster | ~15 min | Min–hours | No (except failover) |
| RA-GRS | 6 | GRS + active secondary read | Regional disaster | ~15 min | Seconds (read) | Yes |
| RA-GZRS | 6 | GZRS + active secondary read | Zone + regional disaster | ~15 min | Seconds (read) | Yes |
RPO (Recovery Point Objective): maximum acceptable data loss.
RTO (Recovery Time Objective): maximum time to restore service.
9.2 Diagram — Geo-Replication
graph TD
subgraph Primary Region - East US
DC1[Datacenter A]
DC2[Datacenter B]
DC3[Datacenter C]
DC1 <--> DC2
DC2 <--> DC3
end
subgraph Secondary Region - West US
DC4[Datacenter D\nAsync Replication]
end
DC1 -.->|Async GRS Replication| DC4
9.3 Selection Rules
Dev/Test, non-critical data → LRS (cheapest)
Production, high SLA, single region → ZRS
Disaster Recovery required → GRS
DR + zone high availability → GZRS
Read during regional outage → RA-GRS or RA-GZRS
Regulatory compliance (EU data) → ZRS or GZRS in same region
9.4 Storage Account Failover
In case of primary region failure with GRS/GZRS:
- Microsoft triggers failover automatically (or you can trigger it manually).
- The DNS record for the storage account is redirected to the secondary region.
- After failover, the account becomes LRS in the former secondary region.
- Reconfigure geo-redundancy after recovery.
Module 10 – Performance Tiers
10.1 Standard vs Premium
| Criteria | Standard (HDD/economical SSD) | Premium (NVMe SSD) |
|---|---|---|
| Hardware | Magnetic HDD (some SSD) | NVMe SSD |
| Latency | Milliseconds (variable) | Sub-millisecond (< 1 ms) |
| IOPS | Limited | Very high |
| Storage price | Low | High |
| Billing | Consumed capacity | Provisioned capacity |
10.2 Performance by Service
| Service | Standard | Premium | Notes |
|---|---|---|---|
| Blob Storage | GPv2 Standard | Premium Block Blobs | Premium = very low latency for small objects |
| Azure Files | Standard (HDD) | Premium (SSD) | Premium required for NFS 4.1 |
| Azure Disks | Standard HDD, Standard SSD | Premium SSD, Premium SSD v2, Ultra SSD | See Module 13 |
| Queue Storage | Standard only | — | No Premium tier |
| Table Storage | Standard only | — | No Premium tier |
10.3 Choosing the Right Account Type
| Account Type | Included Services | Performance |
|---|---|---|
| General Purpose v2 (GPv2) | Blob, Files, Queue, Table | Standard |
| Premium Block Blobs | Blob only | Premium |
| Premium File Shares | Files only | Premium |
| Premium Page Blobs | Blob (page blobs) only | Premium |
Module 11 – Storage Security
11.1 Authentication and Authorization Methods
graph TD
Client[Client / Application]
Client -->|Azure AD + RBAC| AAD[Azure Active Directory]
Client -->|Account Key| AK[Account Key\nFull Access]
Client -->|SAS Token| SAS[Shared Access Signature]
Client -->|Anonymous| PUB[Public Access\ncontainers/blobs]
AAD --> SA[Storage Account]
AK --> SA
SAS --> SA
PUB --> SA
11.2 RBAC Roles for Azure Storage
| Role | Scope | Access |
|---|---|---|
| Storage Blob Data Owner | Account / container | Read + write + delete + ACLs (ADLS Gen2) |
| Storage Blob Data Contributor | Account / container | Read + write + delete |
| Storage Blob Data Reader | Account / container | Read only |
| Storage Queue Data Contributor | Account / queue | Read/write/delete messages |
| Storage File Data SMB Share Contributor | Share | Read + write files via SMB |
11.3 Shared Access Signatures (SAS)
A SAS token is a signed URL that grants delegated, time-limited access without exposing the account key.
SAS types:
| Type | Signed by | Revocable | Recommended |
|---|---|---|---|
| Service SAS | Account key | Via Stored Access Policy | Limited use |
| Account SAS | Account key | Not directly | Avoid if possible |
| User Delegation SAS | Azure AD (user identity) | Yes (revoke AAD token) | Recommended |
SAS token structure:
https://myaccount.blob.core.windows.net/container/file.jpg
?sv=2023-08-03 ← API version
&st=2024-01-01T00:00:00Z ← start date
&se=2024-01-02T00:00:00Z ← expiry date
&sr=b ← resource: b=blob, c=container, s=share
&sp=r ← permissions: r=read, w=write, d=delete, l=list
&sig=<HMAC_signature> ← cryptographic signature
Stored Access Policy: define permissions and duration on the container and revoke a SAS by deleting the policy:
Container → Access policies → + Add policy
→ Identifier: readonly-policy
→ Permissions: Read, List
→ Start: 2024-01-01 Expiry: 2024-12-31
11.4 Firewall and Virtual Networks
Storage Account → Networking → Firewalls and virtual networks
→ Enabled from selected virtual networks and IP addresses
→ + Add existing virtual network → [select VNet/subnet]
→ Firewall: Add IP range: 203.0.113.0/24
→ Exceptions: Allow Azure services (for Azure Backup, Monitor, etc.)
Private Endpoints: connect the storage account via a private IP in your VNet — no public traffic.
Storage Account → Networking → Private endpoint connections → + Private endpoint
→ VNet: my-vnet Subnet: storage-subnet
→ DNS: Private DNS Zone integration (privatelink.blob.core.windows.net)
11.5 Encryption at Rest — Summary
| Option | Key Managed by | Rotation | Revocation | Use Case |
|---|---|---|---|---|
| Microsoft-managed keys (MMK) | Microsoft | Automatic | No | Default, standard compliance |
| Customer-managed keys (CMK) | Customer (Key Vault) | Manual / auto | Yes (delete key) | Strict compliance, GDPR |
| Customer-provided keys | Customer (per request) | N/A | Yes | Very specific cases (Blob REST API) |
Configure CMK:
Storage Account → Encryption → Customer-managed keys
→ Key store type: Key vault
→ Key vault: my-keyvault
→ Key: my-storage-key (version: auto)
→ Managed identity: System-assigned
11.6 Encryption in Transit
- HTTPS mandatory by default (
Secure transfer required = Enabled). - TLS 1.2 minimum recommended (
Minimum TLS version: TLS 1.2). - SMB 3.x encrypts data in transit by default.
Module 12 – Azure Files — Tiers and Quotas
Azure Files Tiers
| Tier | Hardware | Performance | Max IOPS | Protocols |
|---|---|---|---|---|
| Transaction optimized | Standard HDD | General use | ~1000 IOPS/TiB | SMB, REST |
| Hot | Standard HDD | Frequent access | ~1000 IOPS/TiB | SMB, REST |
| Cool | Standard HDD | Infrequent access | ~1000 IOPS/TiB | SMB, REST |
| Premium | NVMe SSD | Ultra-low latency | 100,000+ IOPS | SMB 3.x, NFS 4.1 |
Premium shares are in a
FileStorageaccount type (not GPv2). Billing is based on provisioned, not consumed, capacity.
Module 13 – Azure Disk Storage
13.1 Managed Disk Types
| Type | Hardware | Max IOPS | Max Throughput | Latency | Use Case |
|---|---|---|---|---|---|
| Ultra SSD | NVMe SSD | 160,000 | 2,000 MB/s | < 1 ms | SAP HANA, critical SQL Server, HPC |
| Premium SSD v2 | NVMe SSD | 80,000 | 1,200 MB/s | < 1 ms | Prod DBs, configurable high performance |
| Premium SSD | SSD | 20,000 | 900 MB/s | < 5 ms | Production VMs, databases |
| Standard SSD | Economical SSD | 6,000 | 750 MB/s | ~10 ms | Web servers, light dev/test |
| Standard HDD | HDD | 2,000 | 500 MB/s | ~20 ms | Backup, archiving, infrequent access |
13.2 Disk Roles
| Role | Description | Notes |
|---|---|---|
| OS Disk | Contains the operating system | 1 per VM, max size 4 TiB |
| Data Disk | Additional data disk | Up to 32 TiB, multiple per VM |
| Temporary Disk | Ephemeral local storage (non-persistent) | /dev/sdb Linux, D: Windows — lost if VM restarts |
The temporary disk is NOT a managed disk — it is physically attached to the Azure host. Never store important data there.
13.3 Diagram — Disks and VM
graph LR
VM[Azure Virtual Machine]
VM --> OSD[OS Disk\nPremium SSD\nPersistent]
VM --> DD1[Data Disk 1\nUltra SSD\nPersistent]
VM --> DD2[Data Disk 2\nStandard HDD\nPersistent]
VM --> TEMP[Temp Disk\nLocal SSD\nEphemeral]
13.4 Snapshots and Azure Disk Backup
Snapshots:
- Point-in-time copy of a managed disk.
- Stored as a page blob in a storage account (or managed snapshot).
- Can create a new managed disk from a snapshot.
Azure Disk Backup:
- Incremental backup based on snapshots.
- Integrated in Azure Backup Center.
- Configurable RPO (hourly or daily).
- No agent required — fully managed.
Azure Backup Center → + Backup
→ Datasource type: Azure Disks
→ Backup vault: my-backup-vault
→ Backup policy: every 4 hours, 7-day retention
→ Select disks → Configure backup
13.5 Advanced Features
| Feature | Description |
|---|---|
| Shared disks | Multiple VMs share a disk (Windows/Linux clusters) |
| Bursting | Temporary additional IOPS (Premium SSD, Standard SSD) |
| On-demand bursting | On-demand bursting with no time limit (Premium SSD 512 GiB+) |
| Encryption at host | Encrypts temp disk and cache via SSE |
| Disk access | Secure access via Private Link for export/import |
Module 14 – Data Migration
14.1 Overview of Tools
graph TD
SOURCE[Data Sources\nOn-premises / Other Cloud]
SOURCE -->|Small volumes < 1 TB\nScript automation| AZCOPY[AzCopy\nHigh-performance CLI tool]
SOURCE -->|Graphical interface\nInteractive management| ASE[Azure Storage Explorer\nDesktop application]
SOURCE -->|Large volumes > 1 TB\nNetwork too slow| DATABOX[Azure Data Box\nPhysical appliance]
SOURCE -->|ETL pipelines\nData integration| ADF[Azure Data Factory\nETL orchestration]
SOURCE -->|Offline migration\nLarge volumes| IMPORT[Azure Import/Export Service\nPhysical disks]
AZCOPY --> AZURE[Azure Storage]
ASE --> AZURE
DATABOX --> AZURE
ADF --> AZURE
IMPORT --> AZURE
14.2 AzCopy
Command-line tool optimized for data transfers to/from Azure Storage.
# Authentication (recommended: Azure AD)
azcopy login
# Copy a local file to Blob
azcopy copy 'C:\data\report.csv' 'https://myaccount.blob.core.windows.net/container/report.csv'
# Copy an entire folder (recursive)
azcopy copy 'C:\data\' 'https://myaccount.blob.core.windows.net/container/' --recursive
# Sync (copy only differences)
azcopy sync 'C:\data\' 'https://myaccount.blob.core.windows.net/container/' --recursive
# Copy between two storage accounts
azcopy copy 'https://source.blob.core.windows.net/cont?<SAS>' \
'https://dest.blob.core.windows.net/cont?<SAS>' --recursive
# List blobs
azcopy list 'https://myaccount.blob.core.windows.net/container'
14.3 Azure Storage Explorer
Desktop application (Windows/macOS/Linux) for visually managing storage resources:
- Browse blobs, files, queues, tables.
- Drag and drop for upload/download.
- Manage properties, metadata, ACLs.
- Generate SAS tokens visually.
- Connect via: Azure AD, account key, SAS, local emulator (Azurite).
14.4 Azure Data Box
Solution for migrating very large data volumes (terabytes to petabytes) when network is insufficient.
| Product | Usable Capacity | Use Case |
|---|---|---|
| Data Box Disk | Up to 35 TiB (5 × 8 TiB SSD) | Migrations < 40 TiB, easy to deploy |
| Data Box | 80 TiB | 40–80 TiB migrations, rugged appliance |
| Data Box Heavy | 770 TiB | Massive migrations, entire datacenter |
| Data Box Edge | Variable | Real-time processing + transfer (integrated compute) |
Data Box process:
1. Order the appliance in the Azure portal
2. Receive the physical appliance (AES-256 encrypted)
3. Copy data to the appliance
4. Return the appliance to Microsoft
5. Microsoft uploads the data to your storage account
6. Data available in Azure — appliance wiped (NIST 800-88)
14.5 Azure Data Factory
Data integration service (ETL/ELT) for complex pipelines:
- Connectors to 90+ sources (SQL, Oracle, Salesforce, AWS S3, Google GCS…).
- Copy activities, transformations (Mapping Data Flows), orchestration.
- Ideal for: continuous migrations, synchronization, data transformations.
14.6 Azure Import/Export Service
Manual alternative for shipping physical hard drives:
- You ship your own SATA HDD/SSD drives to a Microsoft datacenter.
- Import: on-premises data → Azure Blob or Files.
- Export: Azure Blob → physical disks returned to you.
- Useful when Data Box is not available in your region.
Module 15 – Azure Storage SDK (.NET)
15.1 NuGet Packages
<PackageReference Include="Azure.Storage.Blobs" Version="12.*" />
<PackageReference Include="Azure.Storage.Files.Shares" Version="12.*" />
<PackageReference Include="Azure.Storage.Queues" Version="12.*" />
<PackageReference Include="Azure.Identity" Version="1.*" />
15.2 BlobServiceClient — Connection
using Azure.Storage.Blobs;
using Azure.Identity;
// Recommended method: DefaultAzureCredential (Azure AD)
var serviceClient = new BlobServiceClient(
new Uri("https://myaccount.blob.core.windows.net"),
new DefaultAzureCredential());
// Alternative: connection string (dev/test only)
var serviceClient = new BlobServiceClient(
"DefaultEndpointsProtocol=https;AccountName=myaccount;AccountKey=<key>;...");
15.3 BlobContainerClient — Container Management
// Create or get a container
BlobContainerClient containerClient = serviceClient.GetBlobContainerClient("my-container");
await containerClient.CreateIfNotExistsAsync(PublicAccessType.None);
// List containers in the account
await foreach (BlobContainerItem container in serviceClient.GetBlobContainersAsync())
{
Console.WriteLine($"Container: {container.Name}");
}
// List blobs in a container
await foreach (BlobItem blob in containerClient.GetBlobsAsync())
{
Console.WriteLine($" Blob: {blob.Name} | Size: {blob.Properties.ContentLength} bytes");
}
15.4 BlobClient — Upload and Download
BlobClient blobClient = containerClient.GetBlobClient("images/photo.jpg");
// Upload from a local file
await blobClient.UploadAsync("C:\\photos\\photo.jpg", overwrite: true);
// Upload from a stream with options
var uploadOptions = new BlobUploadOptions
{
HttpHeaders = new BlobHttpHeaders { ContentType = "image/jpeg" },
Metadata = new Dictionary<string, string>
{
{ "author", "jane-doe" },
{ "category", "landscape" }
},
AccessTier = AccessTier.Hot
};
await using var stream = File.OpenRead("C:\\photos\\photo.jpg");
await blobClient.UploadAsync(stream, uploadOptions);
// Download to a local file
await blobClient.DownloadToAsync("C:\\downloads\\photo.jpg");
// Download to a stream
BlobDownloadResult downloadResult = await blobClient.DownloadContentAsync();
byte[] content = downloadResult.Content.ToArray();
15.5 Metadata and Properties
// Read properties
BlobProperties props = await blobClient.GetPropertiesAsync();
Console.WriteLine($"Content-Type : {props.ContentType}");
Console.WriteLine($"Size : {props.ContentLength}");
Console.WriteLine($"ETag : {props.ETag}");
Console.WriteLine($"Tier : {props.AccessTier}");
// Read metadata
foreach (var meta in props.Metadata)
Console.WriteLine($" {meta.Key} = {meta.Value}");
// Modify metadata
var metadata = new Dictionary<string, string>
{
{ "status", "processed" },
{ "processing_date", DateTime.UtcNow.ToString("yyyy-MM-dd") }
};
await blobClient.SetMetadataAsync(metadata);
// Modify Content-Type
await blobClient.SetHttpHeadersAsync(new BlobHttpHeaders
{
ContentType = "application/json"
});
15.6 Generating a SAS Token in Code
using Azure.Storage.Sas;
// SAS with User Delegation Key (recommended — Azure AD-based)
UserDelegationKey userDelegationKey = await serviceClient.GetUserDelegationKeyAsync(
DateTimeOffset.UtcNow,
DateTimeOffset.UtcNow.AddHours(1));
BlobSasBuilder sasBuilder = new BlobSasBuilder
{
BlobContainerName = "my-container",
BlobName = "images/photo.jpg",
Resource = "b", // b = blob, c = container
ExpiresOn = DateTimeOffset.UtcNow.AddHours(1)
};
sasBuilder.SetPermissions(BlobSasPermissions.Read);
BlobUriBuilder uriBuilder = new BlobUriBuilder(blobClient.Uri)
{
Sas = sasBuilder.ToSasQueryParameters(userDelegationKey, "myaccount")
};
Uri sasUrl = uriBuilder.ToUri();
Console.WriteLine($"SAS URL: {sasUrl}");
15.7 Changing a Blob’s Access Tier
// Move a blob to Archive
await blobClient.SetAccessTierAsync(AccessTier.Archive);
// Rehydrate from Archive to Hot (high priority)
await blobClient.SetAccessTierAsync(AccessTier.Hot, rehydratePriority: RehydratePriority.High);
Module 16 – Azure Storage CLI Commands
16.1 Storage Account
# Create a storage account
az storage account create \
--name myaccount \
--resource-group my-rg \
--location eastus \
--sku Standard_LRS \
--kind StorageV2 \
--access-tier Hot \
--min-tls-version TLS1_2 \
--allow-blob-public-access false
# List storage accounts
az storage account list --resource-group my-rg --output table
# Get access keys
az storage account keys list \
--account-name myaccount \
--resource-group my-rg
# Enable blob versioning
az storage account blob-service-properties update \
--account-name myaccount \
--resource-group my-rg \
--enable-versioning true
# Enable soft delete (14 days)
az storage account blob-service-properties update \
--account-name myaccount \
--resource-group my-rg \
--delete-retention-days 14 \
--enable-delete-retention true
# Change redundancy
az storage account update \
--name myaccount \
--resource-group my-rg \
--sku Standard_GRS
16.2 Blob Containers
# Set account variable (to avoid repeating it)
export AZURE_STORAGE_ACCOUNT=myaccount
export AZURE_STORAGE_KEY=$(az storage account keys list \
--account-name myaccount --resource-group my-rg \
--query '[0].value' --output tsv)
# Create a container
az storage container create \
--name my-container \
--account-name myaccount \
--public-access off
# List containers
az storage container list --account-name myaccount --output table
# Define a stored access policy
az storage container policy create \
--container-name my-container \
--name readonly-policy \
--account-name myaccount \
--permissions rl \
--expiry 2025-12-31
# Delete a container
az storage container delete \
--name my-container \
--account-name myaccount
16.3 Blobs
# Upload a file
az storage blob upload \
--container-name my-container \
--name images/photo.jpg \
--file C:\photos\photo.jpg \
--account-name myaccount \
--content-type image/jpeg
# Upload an entire folder
az storage blob upload-batch \
--destination my-container \
--source C:\photos\ \
--account-name myaccount \
--pattern "*.jpg"
# List blobs
az storage blob list \
--container-name my-container \
--account-name myaccount \
--output table
# Download a blob
az storage blob download \
--container-name my-container \
--name images/photo.jpg \
--file C:\downloads\photo.jpg \
--account-name myaccount
# Change access tier
az storage blob set-tier \
--container-name my-container \
--name images/photo.jpg \
--tier Archive \
--account-name myaccount
# Generate a SAS token for a blob
az storage blob generate-sas \
--container-name my-container \
--name images/photo.jpg \
--permissions r \
--expiry 2024-12-31T23:59:00Z \
--account-name myaccount \
--https-only \
--output tsv
# Delete a blob
az storage blob delete \
--container-name my-container \
--name images/photo.jpg \
--account-name myaccount
# Copy a blob between containers
az storage blob copy start \
--source-container source-cont \
--source-blob images/photo.jpg \
--destination-container dest-cont \
--destination-blob images/photo.jpg \
--account-name myaccount
16.4 Azure Files
# Create a file share
az storage share create \
--name my-share \
--account-name myaccount \
--quota 100
# List shares
az storage share list --account-name myaccount --output table
# Upload a file
az storage file upload \
--share-name my-share \
--source C:\config\app.config \
--path configs/app.config \
--account-name myaccount
# Create a directory
az storage directory create \
--share-name my-share \
--name configs \
--account-name myaccount
# List files in a directory
az storage file list \
--share-name my-share \
--path configs \
--account-name myaccount \
--output table
# Generate a SAS for the share
az storage share generate-sas \
--name my-share \
--permissions rl \
--expiry 2024-12-31T23:59:00Z \
--account-name myaccount \
--https-only \
--output tsv
# Delete a share
az storage share delete \
--name my-share \
--account-name myaccount
16.5 Queue Storage
# Create a queue
az storage queue create --name my-queue --account-name myaccount
# Send a message
az storage message put \
--queue-name my-queue \
--content "Task to process: id=42" \
--account-name myaccount
# Read the next message (without deleting it)
az storage message peek \
--queue-name my-queue \
--account-name myaccount
# Read and delete a message (dequeue)
az storage message get \
--queue-name my-queue \
--account-name myaccount
Module 17 – Review Questions
1. You store application logs that are never modified, only appended. Which blob type should you use?
Answer: Append Blob — optimized for sequential append operations, ideal for logs.
2. Your company must retain financial documents for 7 years with no modification or deletion possible, to satisfy a regulation. Which Azure Blob Storage feature do you use?
Answer: Immutable Storage (WORM) with a locked time-based retention policy.
3. Which redundancy option allows you to read data from the secondary region even without triggering a failover?
Answer: RA-GRS (Read-Access Geo-Redundant Storage) or RA-GZRS — the RA prefix means Read Access on the secondary region.
4. You have a 500 GB blob in Archive tier. An urgent process needs to access this blob within 30 minutes. What do you do?
Answer: Trigger a high-priority rehydration (
RehydratePriority.High) to Hot or Cool tier. Standard priority can take up to 15 hours; high priority is typically under 1 hour for blobs of reasonable size.
5. What is the difference between a user delegation SAS and an account SAS?
Answer: The user delegation SAS is signed with an Azure AD key (obtained via
GetUserDelegationKey) — it is tied to an AAD identity and can be revoked by revoking the AAD token. The account SAS is signed with the account key (HMAC-SHA256) — if the key leaks, all SAS tokens generated with that key are compromised.
6. You need to migrate 200 TiB of data from your on-premises datacenter to Azure. The available internet bandwidth is 100 Mbps. Which Microsoft service do you recommend and why?
Answer: Azure Data Box (80 TiB) or Azure Data Box Heavy (770 TiB). At 100 Mbps, transferring 200 TiB would take: 200 × 1024 × 8 / 100 / 3600 ≈ 457 hours (19 days). Data Box is much faster because physical transfer replaces the network.
7. Which performance option should you choose for a production SQL Server VM that requires latency below 1 ms and 80,000 IOPS?
Answer: Premium SSD v2 or Ultra SSD. Ultra SSD offers the lowest latency (< 1 ms) and up to 160,000 IOPS. Premium SSD v2 offers more flexible configuration at a slightly lower cost.
8. Your application needs to write messages to a queue so a worker can process them asynchronously. Which Azure storage service do you use?
Answer: Azure Queue Storage — designed for component decoupling via asynchronous messages (up to 64 KB per message, configurable TTL).
9. What is the difference between Azure Backup for Azure Files and Azure Files snapshots?
Answer: Azure Backup provides a complete managed solution (scheduling, retention, centralized vault, restore in multiple scenarios including Cross-Region Restore). Snapshots are manual or scripted point-in-time copies, stored directly in the share — ideal for fast recovery of small accidental changes but without central orchestration.
10. You enable Cloud Tiering in Azure File Sync. What happens to tiered files on the local server?
Answer: Infrequently accessed files are replaced by stubs (shortcuts) on the local server. They appear normally in File Explorer but occupy no local disk space. When a user opens a stubbed file, it is automatically retrieved from Azure Files transparently.
11. Which AzCopy command would you use to copy only changed files (incremental sync) from a local folder to a Blob container?
Answer:
azcopy sync 'C:\data\' 'https://myaccount.blob.core.windows.net/container/' --recursiveThe
synccommand compares source and destination files (by name and modification date) and transfers only differences, unlikecopywhich transfers everything.
12. Your storage account uses CMK (Customer-Managed Keys) via Azure Key Vault. An administrator revokes (disables) the key in Key Vault. What happens immediately?
Answer: Azure Storage can no longer decrypt the data encryption keys — all read and write operations on the storage account fail until the key is restored. This is a powerful control mechanism but must be used carefully (immediate impact on availability).
Extended Summary
| Concept | Key Points |
|---|---|
| Blob types | Block (general), Page (VHDs), Append (sequential logs) |
| Access tiers | Hot → Cool → Cold → Archive (storage cost ↓, access cost ↑) |
| Lifecycle policies | JSON automating blob movement/deletion |
| WORM / Immutability | Time-based retention or Legal hold — regulatory compliance |
| Versioning & Soft Delete | Protection against accidental deletion/overwrite |
| Redundancy | LRS < ZRS < GRS < GZRS; RA-GRS/RA-GZRS for active secondary read |
| Performance | Standard (HDD) vs Premium (NVMe SSD) — latency, IOPS, price |
| Disk types | Ultra SSD > Premium SSD v2 > Premium SSD > Standard SSD > Standard HDD |
| Authentication | Azure AD + RBAC > User Delegation SAS > Account SAS > Account Key |
| CMK | Key in Key Vault — full control, revoke = access blocked |
| Firewall / Private Endpoint | Restrict network access to the storage account |
| AzCopy | copy (all), sync (incremental), login (Azure AD) |
| Data Box | Physical migration — Disk (35 TiB), Box (80 TiB), Heavy (770 TiB) |
| SDK .NET | BlobServiceClient → BlobContainerClient → BlobClient |
| Azure File Sync | Local Windows cache + Cloud Tiering + multi-server sync |
| Temporary disk | Ephemeral, unmanaged — never store persistent data there |
Advanced Section – Use Cases, Questions, and Glossary
Advanced Azure Storage Use Cases
Backup and Disaster Recovery
Backup architecture with Blob Storage:
Application → Azure Backup (managed by Microsoft)
↓
Recovery Services Vault
├── VM backups (disk snapshots)
├── Azure SQL DB backups
└── Azure File Share backups
Direct storage in Blob (custom):
Application → Azure Blob Storage (RA-GZRS)
├── Hot tier (30 days)
├── Cool tier (90 days)
└── Archive tier (10 years, compliance)
Media and CDN
flowchart LR
UPLOAD["Multimedia content\n(videos, images)"] --> BLOB["Azure Blob Storage\nHot Tier"]
BLOB --> CDN["Azure CDN\n(global edge servers)"]
CDN --> USER1["User Paris\n< 5ms latency"]
CDN --> USER2["User New York\n< 5ms latency"]
CDN --> USER3["User Tokyo\n< 5ms latency"]
CDN -- "Cache MISS\n(new content)" --> BLOB
Big Data Pipeline with Data Lake Gen2
flowchart TD
IOT["IoT / Streaming Sources"] --> EH["Azure Event Hubs"]
DBASE["SQL Databases"] --> ADF["Azure Data Factory"]
FILES["CSV/Excel Files"] --> ADF
EH --> ADLS["Azure Data Lake Gen2\n(raw zone)"]
ADF --> ADLS
ADLS -- "Databricks Spark\n(transformation)" --> ADLS2["Data Lake Gen2\n(curated zone)"]
ADLS2 -- "Synapse SQL\n(aggregation)" --> ADLS3["Data Lake Gen2\n(serving zone)"]
ADLS3 --> PBI["Power BI\n(dashboards)"]
Static Website Hosting
Azure Blob Storage → Static Website:
Enable: Storage Account → Static website → Enabled
Files : $web container (automatically created)
URL : https://mystorageaccount.z13.web.core.windows.net/
Azure CDN integration:
→ HTTPS custom domain (myapp.com)
→ Latency < 5ms worldwide
→ Cache HTML/CSS/JS files
Use cases: SPA (React, Angular, Vue), static sites
Review Questions – Azure Storage
Question 1 You store application logs that are written sequentially and never modified. Which Blob type is most appropriate?
- A) Block Blob
- B) Page Blob
- C) Append Blob
- D) Archive Blob
Answer: C — The Append Blob is optimized for sequential data appending, like application logs. Blocks can only be appended, never modified.
Question 2 Your organization must retain backups for 10 years for regulatory reasons. The data will never be read except during an audit. Which Blob storage tier minimizes costs?
- A) Hot
- B) Cool
- C) Cold
- D) Archive
Answer: D — The Archive tier has the lowest storage cost. Data that is rarely (or never) accessed for long durations benefits most from this tier. Retrieval takes a few hours but is acceptable for regulatory archives.
Question 3 Your application needs to access a file in Azure Blob Storage from code, without storing storage keys in the configuration. Which approach is recommended?
- A) Connection string in source code
- B) SAS Token in an environment variable
- C) Managed Identity + RBAC role “Storage Blob Data Reader”
- D) Shared Key in Azure Key Vault
Answer: C — Managed Identity allows an Azure application to access Blob Storage without any stored credentials. This is the recommended “credentialless” approach for applications running on Azure.
Question 4 You must protect a Blob container against accidental or malicious deletions for 7 years (legal requirement). Which mechanism do you use?
- A) Soft Delete (7-day retention)
- B) Blob versioning
- C) WORM with Time-Based Retention Policy (7 years)
- D) Lifecycle Management (move to Archive)
Answer: C — The WORM (Write Once Read Many) policy with Time-Based Retention makes blobs immutable for a defined duration. Even administrators cannot delete data during this period. Ideal for regulatory compliance.
Question 5 Your company has 500 TB of on-premises data to migrate to Azure Blob Storage. The internet network has 50 Mbps of shared bandwidth. Which migration approach is most appropriate?
- A) AzCopy over the internet
- B) Azure Data Box (physical)
- C) Dedicated ExpressRoute
- D) Azure File Sync
Answer: B — With 500 TB and a 50 Mbps connection, the transfer over the internet would take years. Azure Data Box (or Data Box Heavy for 770 TiB) allows copying data to a physical appliance shipped by Microsoft, then returned to the datacenter for direct upload. This is the standard solution for massive migrations.
Question 6 Which Azure Storage redundancy offers the best protection against a complete Azure region failure, while allowing data reads from the secondary region without failover?
- A) LRS (Locally Redundant Storage)
- B) ZRS (Zone-Redundant Storage)
- C) GRS (Geo-Redundant Storage)
- D) RA-GZRS (Read-Access Geo-Zone-Redundant Storage)
Answer: D — RA-GZRS combines ZRS (3 zones in primary region) and GRS (replication to secondary region), with the ability to read from secondary region (Read-Access) without waiting for failover. This is the maximum level of protection and availability.
Glossary – Azure Storage
| Term | Definition |
|---|---|
| Storage Account | Top-level container for all Azure Storage services |
| Blob Storage | Object storage service for unstructured data (Binary Large Object) |
| Block Blob | Blob type for sequential block upload. General purpose |
| Page Blob | Blob type for random access (read/write). Used for VHDs |
| Append Blob | Blob type optimized for sequential append only (logs) |
| Container | Logical grouping of blobs within a Storage Account |
| Access Tier | Cost/performance level: Hot > Cool > Cold > Archive |
| Archive | Cheapest tier, offline data, hours to retrieve |
| Lifecycle Policy | JSON rule automating blob movement or deletion |
| LRS | Locally Redundant Storage — 3 copies in the same datacenter |
| ZRS | Zone-Redundant Storage — 3 copies in 3 different zones |
| GRS | Geo-Redundant Storage — LRS + async replication to secondary region |
| GZRS | Geo-Zone-Redundant Storage — ZRS + secondary region replication |
| RA-GRS | GRS with read access on the secondary region |
| RA-GZRS | GZRS with read access on the secondary region (maximum protection) |
| SAS Token | Shared Access Signature — temporary URL with limited permissions |
| Managed Identity | Azure AD identity for Azure resources, no credentials |
| CMK | Customer-Managed Key — encryption key in Azure Key Vault |
| WORM | Write Once Read Many — immutability for regulatory compliance |
| Soft Delete | Azure recycle bin: deleted blobs recoverable for N days |
| Blob Versioning | Automatically retain all versions of a blob |
| Static Website | Host a static site directly from Blob Storage |
| Azure Files | Managed SMB/NFS file shares in Azure |
| Azure File Sync | Bidirectional sync between Windows File Server and Azure Files |
| Cloud Tiering | File Sync feature: infrequently accessed files stored only in Azure |
| AzCopy | CLI tool for copying/syncing data to/from Azure Storage |
| Azure Data Box | Physical appliance for massive data migration (35 TiB to 1 PiB) |
| Azure Managed Disks | Virtual disks managed by Azure for VMs |
| Ultra SSD | Highest-performance disk — latency < 1ms, configurable IOPS |
| Premium SSD v2 | High performance with real-time configurable IOPS/throughput |
| Premium SSD | SSD for critical production workloads |
| Standard SSD | SSD for non-critical workloads |
| Standard HDD | Hard disk for dev/test and non-critical data |
| ADLS Gen2 | Azure Data Lake Storage Gen2 — Blob Storage + hierarchical namespace for analytics |
| Hierarchical Namespace | Real folder structure in ADLS Gen2 (vs. simulation in Blob) |
| Private Endpoint | Private VNet connection to Storage Account (no public internet) |
| Storage Firewall | Restricts network access to the Storage Account by IP/VNet |
Search Terms
azure · fundamentals · storage · core · infrastructure · microsoft · blob · disk · encryption · tiers · access · account · backup · data · cases · deep · diagram · dive · questions · service · types · containers · features · glossary