Amazon Elastic Block Store (EBS)

Amazon Elastic Block Store (EBS) is a high-performance, block-level storage service specifically designed for use with Amazon Elastic Compute Cloud (EC2) instances. It provides persistent, highly available, and scalable storage volumes that can be attached to EC2 instances, functioning like a traditional hard drive or solid-state drive.

Primary purpose and functionality:

Persistent Storage for EC2: EBS volumes serve as durable block storage for EC2 instances. This means data stored on an EBS volume persists independently of the life of the EC2 instance it’s attached to. If an EC2 instance is stopped or terminated, the data on its attached EBS volume remains.
Operating System, Application Data, and Databases: They are ideal for storing operating system files, application code, and databases, where consistent performance and data durability are critical.
Block-Level Storage: EBS operates at the block level, providing raw storage space that can be formatted with a file system (like ext4 or NTFS) and then mounted to a directory on your EC2 instance, just like a physical hard drive.
Attachment: An EBS volume can be attached to only one EC2 instance at a time. However, a single EC2 instance can have multiple EBS volumes attached to it.

Key features:

Scalability: Dynamically increase or decrease storage capacity, IOPS (Input/Output Operations Per Second), and throughput of your volumes without any downtime, accommodating changing workload requirements.
Durability: EBS volumes are designed for high durability. Data is automatically replicated within its Availability Zone (AZ) to protect against component failure. io2 Block Express volumes offer up to 99.999% durability. Other volume types typically provide 99.8% to 99.9% durability.
Persistence: Data on an EBS volume is retained even when the associated EC2 instance is stopped or terminated, ensuring data integrity and continuity.
EBS Snapshots: Create point-in-time backups of your EBS volumes. These snapshots are stored in S3 for high durability. Snapshots can be used to instantly restore new EBS volumes or migrate volumes to different Availability Zones, regions, or even other AWS accounts (since EBS volumes are AZ-bound).
High Performance: EBS offers various volume types (General Purpose SSD, Provisioned IOPS SSD, Throughput Optimized HDD, Cold HDD) tailored to specific workload needs, providing high performance in terms of IOPS and throughput.
Encryption: Secure your data with encryption. EBS volumes can be encrypted both at rest and data in transit between the EC2 instance and the EBS volume. This enhances data protection and helps with compliance requirements.
Pay-as-you-go Pricing: You pay only for the storage you provision, the IOPS you use, and data transfer costs, with different pricing options to optimize costs based on usage patterns.
EC2 Integration: EBS volumes are tightly integrated with EC2. They can be easily attached and detached from EC2 instances, providing flexible and manageable storage for virtual machines.

How EBS volumes work in practice:

Creation: You create an EBS volume in a specific size and type (e.g., 100 GB General Purpose SSD) within a particular Availability Zone.
Attachment: You attach this EBS volume to an EC2 instance that resides in the same Availability Zone.
Access and Formatting: Once attached, the EC2 instance recognizes the EBS volume as a raw block device. You then format it with a file system (e.g., mkfs.ext4) and mount it to a directory (e.g., /data) within the instance’s operating system.
Usage: The EC2 instance can now read from and write to the EBS volume as if it were a local hard drive.

EBS volumes are bound to a specific Availability Zone. This means:

An EBS volume created in AZ us-east-1a can only be attached to an EC2 instance in us-east-1a.
You cannot directly move an EBS volume to a different AZ or region by simply detaching and reattaching.
To move an EBS volume across AZs or regions, you must first create an EBS Snapshot of the volume. This snapshot can then be restored as a new EBS volume in any desired AZ or region.

EBS Provisioning

This section delves deeper into managing Amazon EBS volumes, focusing on snapshots for backup and migration, capacity provisioning for performance, and the “delete on termination” attribute.

EBS Snapshots (Backups and Recovery)

EBS snapshots are point-in-time backups of your EBS volumes, crucial for data protection and disaster recovery.

Incremental Backups: Snapshots are incremental. Only the blocks that have changed since the last snapshot are saved, which significantly reduces storage costs and improves backup efficiency.
Block-Level Operation: Snapshots capture all data blocks, metadata, and configuration settings associated with the EBS volume at the block level.
Data Consistency: Before creating a snapshot, AWS ensures data consistency by temporarily pausing I/O operations on the volume. This creates a consistent point-in-time image of the data. AWS manages this process automatically.
Storage in S3: Once created, EBS snapshots are stored in Amazon S3, leveraging S3’s high durability by replicating data across multiple Availability Zones within a given region.
Compression and Encryption: Snapshots are automatically compressed to minimize storage costs and are encrypted at rest. You can also integrate with AWS Key Management Service (KMS) for enhanced security and key management.
Lifecycle Management: AWS allows you to define lifecycle policies for your snapshots to automate their management. This includes setting rules for retention, deletion, and copying snapshots to other regions for disaster recovery or migration.
Data Recovery and Migration: You can create new EBS volumes from snapshots, restoring data to a previous state. To migrate an EBS volume to a different Availability Zone, region, or even another AWS account, you must first create a snapshot and then restore it in the desired new location.
Cost-Effectiveness: Snapshots are cost-effective for long-term data retention, especially when combined with lifecycle management to optimize costs by setting appropriate retention policies. Costs are based on the amount of data stored in S3.

Capacity Provisioning

Capacity provisioning involves configuring the size and performance characteristics (like IOPS and throughput) of your EBS volumes to meet specific workload requirements.

Volume Size: You specify the desired size of the volume in gigabytes, which determines its total storage capacity.
Volume Types: EBS offers various volume types, each optimized for different workloads:
- General Purpose SSD (GP2/GP3):
  - Provide a balance of price and performance, suitable for most workloads.
  - GP3 is the newer generation and generally better than GP2. It allows you to independently specify IOPS and throughput, meaning you can provision a smaller storage volume while maintaining high performance, potentially at a lower cost. Upgrading from GP2 to GP3 is seamless and recommended.
- Provisioned IOPS SSD (io1/io2/io2 Block Express):
  - Designed for I/O-intensive applications requiring consistent and predictable performance, such as large databases. You can provision a specific number of IOPS based on application needs. io2 Block Express offers the highest performance.
- Throughput Optimized HDD (st1):
  - Suitable for frequently accessed, throughput-intensive workloads like big data, log processing, and data warehouses, where throughput (MB/s) is more critical than IOPS.
- Cold HDD (sc1):
  - The lowest-cost option, suitable for infrequently accessed workloads where performance is not a primary concern, such as large cold data archives.
- Magnetic (Standard):
  - An older, lower-cost option for workloads with infrequent access and low performance requirements.
Provisioned IOPS: For Provisioned IOPS SSD types, you explicitly set the desired number of IOPS, ensuring consistent performance for latency-sensitive applications. This can be adjusted based on read/write requirements.
Volume Size and Performance (especially for GP2): For some volume types (like GP2), increasing the volume size can indirectly improve performance, as larger volumes often have a higher baseline performance and higher burst rates, allowing them to sustain higher I/O rates over longer periods.
Monitoring: Use Amazon CloudWatch to monitor EBS volume metrics such such as read/write operations, latency, and queue length. This helps identify bottlenecks and optimize resource usage.

Delete on Termination Attribute

The Delete on Termination attribute for an EBS volume determines whether the volume is automatically deleted when its associated EC2 instance is terminated.

Enabled: If enabled (which is the default for root volumes), the EBS volume will be automatically deleted when the EC2 instance it’s attached to is terminated. This helps save costs by ensuring you don’t pay for unused storage.
Disabled: If disabled (which is the default for additional, non-root EBS volumes), the EBS volume will persist even after its associated EC2 instance is terminated. This is useful for retaining data that you want to reuse with a new instance or back up separately.

This attribute can be configured when you launch an EC2 instance or modified for an existing EC2 instance through the AWS console or API.

EBS Volumes (Hands-on)

This hands-on demonstration covers creating an EC2 instance, attaching an EBS volume, understanding the delete on termination attribute, and observing volume persistence.

1. Create a New EC2 Instance (with an Additional EBS Volume):

Go to the EC2 Dashboard.
Click “Launch instance.”
Name: Give it a name (e.g., Basic-Sample-Instance).
Application and OS Images (AMI): Select “Ubuntu” (or any other preferred OS).
Instance type: Choose “t2.micro” (eligible for free tier).
Key pair (login): Select an existing key pair or create a new one. This is essential for connecting to your instance later.
Network settings: Leave as default.
Configure storage:
- You’ll see the root volume (e.g., 8 GB for Ubuntu).
- Click “Add new volume.”
- Volume Type: Choose “General Purpose SSD (gp2)” or “gp3.”
- Size (GiB): Enter a small size, e.g., 4.
- Leave other settings for this additional volume as default.
Click “Launch instance.”

2. Observe Instance Launch and Initial Volumes:

Go to “Instances” in the EC2 dashboard.
Your new instance will be in a “pending” state and then transition to “running.” This might take a minute or two.
While it’s launching, go to “Volumes” under the “Elastic Block Store” section in the left navigation pane.
You will see the root volume (e.g., 8GiB) and your additional 4GiB volume. Both will show as “in-use” and attached to your new EC2 instance, residing in the same Availability Zone as your instance.

3. Create and Attach an Independent EBS Volume:

While still on the “Volumes” page, click “Create volume.”
Volume Type: Choose “General Purpose SSD (gp2).”
Size (GiB): Enter a small size, e.g., 2.
Availability Zone: Crucially, select the exact same Availability Zone as your running EC2 instance (e.g., if your instance is us-east-1a, select us-east-1a for the volume). You can verify your instance’s AZ from its details page under the “Instances” section.
Leave other settings as default.
Click “Create volume.”
The new volume will appear in the “Volumes” list with a “status” of “available” (meaning it’s not yet attached to any instance):
Attach Volume:
- Select the newly created 2GiB volume (its status should be “available”).
- Click “Actions” → “Attach volume.”
- Instance: In the dropdown, select your Basic-Sample-Instance. (Note: Only instances in the same AZ will appear here).
- Device name: Leave the default or choose /dev/sdd.
- Click “Attach volume.”
The volume’s status will change to “in-use,” and it will show as attached to your instance.

4. Understand “Delete on Termination” and Test Instance Termination:

Go back to “Instances” in the EC2 dashboard.
Select your Basic-Sample-Instance.
Go to the “Storage” tab in the instance details.
You will see a list of attached volumes:
- The root volume (e.g., 8GiB) will have “Delete on termination” set to Yes (default).
- The additional 4GiB volume you created during instance launch will have “Delete on termination” set to No (default for non-root volumes added at launch).
- The 2GiB volume you created independently and attached will also have “Delete on termination” set to No (default for independently created volumes).
Terminate the Instance:
- With Basic-Sample-Instance selected, click “Instance state” → “Terminate instance.”
- AWS will display a message: “On an EBS-backed instance, the default action is for the root EBS volume to be deleted when the instance is terminated.”
- Confirm the termination.
Observe Volume Persistence:
- Go back to “Volumes” in the EC2 dashboard.
- Refresh the page periodically.
- Once the EC2 instance fully terminates, you will notice that the root volume (8GiB) disappears, as it was configured to be deleted on termination.
- However, your 4GiB and 2GiB volumes will remain, showing a status of “available” (not attached to any instance). This demonstrates that non-root volumes, or those explicitly set not to delete on termination, persist after instance termination.

5. Clean Up Resources:

To avoid incurring further charges, manually delete the remaining “available” EBS volumes.
Select the 4GiB and 2GiB volumes.
Click “Actions” → “Delete volume.”
Confirm the deletion.

This hands-on exercise clearly illustrates how EBS volumes function, their persistence, how they are tied to Availability Zones, and the behavior of the “delete on termination” attribute.

Amazon Elastic File System (EFS)

Amazon Elastic File System (EFS) is a serverless, fully elastic, shared file storage service designed for use with AWS cloud services and on-premises resources. Unlike EBS, which attaches to a single EC2 instance, EFS allows multiple EC2 instances (and other services) to access the same file system concurrently.

Core Purpose and Functionality:

Shared File System: EFS’s primary purpose is to provide a scalable and highly available shared file system that can be mounted by multiple EC2 instances simultaneously. This enables collaborative workflows, content management systems, web serving, and other scenarios where multiple compute resources need to access the same data.
Serverless and Elastic: EFS is fully managed by AWS, meaning you don’t need to provision or manage any underlying storage servers or infrastructure. It automatically scales storage capacity and performance up and down based on your actual usage, eliminating the need for manual provisioning and reducing costs.
High Availability and Durability: Data stored in EFS is replicated across multiple Availability Zones (AZs) within a given region, ensuring high availability and protection against single AZ failures.

Key Features:

Scalability and Elasticity:
- Automatically scales storage capacity from gigabytes to petabytes without manual intervention.
- Performance scales with the file system size and activity, accommodating varying workload requirements.
Shared Access:
- Allows multiple EC2 instances to access the same file system concurrently, facilitating collaborative work and shared data access for applications.
Broad Compatibility:
- Uses the Network File System (NFS) v4.1 protocol, ensuring compatibility with a wide range of Linux-based operating systems and applications.
- Full compliance with the POSIX (Portable Operating System Interface) standard, which enables easy portability of Unix-based applications.
Performance Modes:
- General Purpose Mode: Suitable for the vast majority of file system workloads (e.g., web serving, content management, development environments) that are latency-sensitive but don’t require the highest levels of aggregated throughput or IOPS. It automatically scales performance based on file system size.
- Max I/O Mode: Designed for workloads that require higher levels of aggregate throughput and IOPS (e.g., big data analytics, media processing, database backups). This mode scales to higher limits but at the cost of slightly higher latency for individual operations. Throughput needs to be manually provisioned for Max I/O mode.
Throughput Modes:
- Bursting Throughput Mode (Default): Designed for workloads with unpredictable or spiky access patterns. Your file system’s throughput scales with its size, and you earn burst credits that allow for higher throughput bursts for a period.
- Provisioned Throughput Mode: Ideal for workloads with predictable or constant throughput requirements. You specify a desired throughput level for your file system, ensuring consistent performance regardless of file system size.
Storage Classes:
- Standard: Data is replicated across multiple AZs for highest availability and durability.
- Infrequent Access (IA): Cost-optimized for data that is accessed less frequently. Data is automatically transitioned to this class after a period of inactivity. It’s still replicated across multiple AZs.
- One Zone Infrequent Access (One Zone IA): A lower-cost option for infrequently accessed data that does not require multi-AZ redundancy. Data is stored redundantly within a single Availability Zone. Suitable for secondary backups or data that can be easily recreated.
Pricing:
- Pay-as-you-go model with no upfront costs. You only pay for the storage consumed and data transfer.
Security:
- Data is encrypted. EFS supports encryption at rest (using AWS KMS for customer-managed keys with options for key rotation and deletion) and in transit.

AWS Backup

AWS Backup is a fully managed service that centralizes and automates your backup strategy across various AWS services. It provides a single pane of glass to configure, manage, and restore backups, streamlining data protection and ensuring compliance.

Key Capabilities:

Centralized Backup Management: Configure backup policies and apply them across multiple AWS services from a single console. Supported services include EC2 instances, Amazon RDS databases, Amazon EFS file systems, Amazon DynamoDB tables, AWS Storage Gateway volumes, Amazon EBS volumes, and more.
Automated Backup Process: Define backup plans to automate how often backups occur (e.g., daily, weekly, monthly) and how long they should be retained. This eliminates manual intervention and ensures consistent backup schedules.
Cross-Region and Cross-Account Copy: AWS Backup enables copying backups to different AWS regions and even to different AWS accounts. This is a critical feature for disaster recovery strategies, providing geographical redundancy and isolation.
Simplified Restoration: The service offers a straightforward process for restoring data. Depending on the service and backup type, you can restore individual files, directories, or entire systems.
Compliance and Retention: Set rules for the retention and lifecycle of your backups to meet specific compliance requirements.
IAM Integration: Integrates with AWS Identity and Access Management (IAM) to provide granular control over who can access and manage your backups.
Monitoring and Reporting: A centralized dashboard allows you to monitor the status of your backups and view detailed reports, helping to identify and address any issues promptly.

How AWS Backup Works:

Define a Backup Plan: Create a backup plan specifying the desired backup frequency (e.g., daily at 2 AM) and retention periods (e.g., retain for 30 days).
Assign Resources: Assign the AWS resources (e.g., specific EC2 instances, EFS file systems) that you want to include in the backup plan.
Automatic Backup and Storage: AWS Backup automatically initiates backups according to the plan. The backup data is then securely stored in an internal S3 bucket, specifically allocated for AWS Backup.
Monitor: Monitor the backup jobs and their status through the AWS Backup management console.

AWS Backup Vault Lock is a feature that enhances the security and compliance of your backups by applying immutable and tamper-proof safeguards to backup data stored within a backup vault.

In AWS Backup, a backup vault is a secure container where your recovery points (backups) are stored. You can create multiple vaults to organize backups based on criteria like department, application, or specific compliance needs.

Vault Lock helps meet stringent regulatory requirements for data immutability, ensuring that backups cannot be accidentally or maliciously deleted or altered for a specified period.

Immutability: Once a lock is applied to a backup vault, the policy you set becomes immutable for a defined duration. This means no one, including the AWS account root user, can alter or delete the recovery points (backups) stored in that vault before the lock period expires.
Compliance: This feature is vital for industries (e.g., finance, healthcare) that require data to be stored in an unaltered state for specific durations to comply with regulations (e.g., HIPAA, FINRA, SEC Rule 17a-4).

AWS Backup Vault Lock offers two distinct modes:

Compliance Mode:
- When enabled, the vault policy becomes completely immutable for the duration of the lock period.
- No one, not even the account root user, can change or delete the policy or any recovery points within the vault until the lock period expires.
- This mode ensures strict adherence to compliance requirements.
Governance Mode:
- This mode offers a balance between immutability and operational flexibility.
- It still protects recovery points from accidental or malicious deletion.
- However, specified IAM roles are allowed to manage and update the vault’s policies (e.g., adjust retention periods within the lock’s constraints, add new backup plans).
- It provides a controlled way to manage policies while still maintaining strong protection against data alteration or deletion.

AWS Backup (Hands-on)

This hands-on guide demonstrates how to configure backup plans in AWS Backup, assign resources, and understand the core features.

1. Navigate to AWS Backup Console:

Go to the AWS Management Console and search for “AWS Backup”.
On the dashboard, you’ll see options to create backup plans, view jobs, and manage vaults.

2. Create a Backup Plan:

In the left navigation pane, click “Backup plans”.
Click “Create backup plan”.
Choose a method:
- Select “Start with a template”. This provides pre-configured rules for common backup scenarios.
- From the “Backup plan template” dropdown, choose “Daily-Weekly-Monthly-5YearRetention” (or a similar template).
- Note: You could also choose “Build a new plan” for full customization or “Define a plan using JSON” for programmatic setup.
Customize Backup Rules (Optional but Recommended):
- The template populates several backup rules (e.g., “Daily-backups”, “Weekly-backups”, “Monthly-backups”).
- For “Daily-backups”:
  - Click “Edit” next to it.
  - Backup frequency: “Daily”
  - Backup window: Choose “Use default window” or customize to an off-peak time.
  - Lifecycle:
    - Transition to cold storage: Specify a duration (e.g., “30 days” to move to cheaper cold storage).
    - Retain backup: Set a total retention period (e.g., “35 days”).
  - Copy to a new destination (Optional):
    - You can enable “Copy to a new destination” to replicate backups to another AWS Region (e.g., “Canada (Central)”).
    - You can also specify an “Account ID” to copy to a different AWS account’s backup vault for disaster recovery.
  - Click “Save backup rule.”
- You can similarly inspect and adjust the “Weekly” and “Monthly” rules.
Backup plan name: Give your plan a descriptive name (e.g., My-Backup-Plan-Test).
Click “Create plan.”

3. Assign Resources to the Backup Plan:

After creating the plan, you’ll be redirected to its details page.
Click “Assign resources.”
Resource assignment name: Give it a name (e.g., My-Resource-Assignment).
IAM role: Choose “Default role.” (AWS Backup will create a service-linked role if it doesn’t exist, granting necessary permissions).
Assign resources:
- Include all resource types: Select this to back up all supported resource types in your account.
- OR Include specific resource types:
  - Select this and choose specific services like “S3.”
  - Then, you can choose to include “All S3 buckets” or “Specify S3 buckets” by name. For this hands-on, you might select a test S3 bucket you created earlier.
- Filter by tags (Optional): You can also use tags to include or exclude resources. For example, you could specify Key: Department, Condition: Equals, Value: Marketing to only back up resources tagged as belonging to the marketing department.
Click “Assign resources.”

4. Monitor Backup Jobs (Observation):

Backups will now be initiated based on the schedule defined in your backup plan.
In the left navigation pane, click “Jobs” → “Backup jobs.”
Here, you will see the status of your backup jobs (e.g., “Running,” “Completed,” “Failed”). It might take some time for the first backup jobs to appear and complete based on your defined backup window.

5. Clean Up Resources (Crucial Step):

Delete Resource Assignments:
- Go to your backup plan (My-Backup-Plan-Test).
- Go to the “Resource assignments” tab.
- Select your assignment (e.g., My-Resource-Assignment).
- Click “Delete.”
- Type the assignment name to confirm and click “Delete.”
Delete Backup Plan:
- Go to “Backup plans.”
- Select your backup plan (My-Backup-Plan-Test).
- Click “Delete.”
- Type the backup plan name to confirm and click “Delete.”
Check Backup Vaults:
- Go to “Backup vaults.”
- You might see recovery points in your default backup vault or any custom vaults you created.
- If there are recovery points, you might need to delete them manually before the vault can be deleted (though often the default vault persists). If you created a custom vault, ensure it’s empty before attempting deletion.

This hands-on exercise provides practical experience in configuring a centralized backup solution using AWS Backup, assigning resources, and understanding the automation aspects of data protection.

Quartz 4

Explorer

08. Other Storage Services