This tutorial makes file storage production-ready by transitioning uploaded images from local disk storage to AWS S3. The application now includes authentication, password resets, PostgreSQL with Alembic migrations, and file uploads where users upload profile pictures, process them with Pillow, and save them to a local media directory.

Local storage works fine for learning and development, but it’s not how production applications typically handle files.

The Problem with Local File Storage

Modern deployment platforms, especially container-based platforms, present a critical issue: the file system is not permanent. When applications restart or new versions deploy, containers can be wiped and rebuilt. Local files are not guaranteed to persist.

Another problem: files are tied to a single machine. When switching servers or running applications on multiple machines, code migration is straightforward, but uploaded media becomes problematic.

The standard solution: object storage. For AWS, that’s S3—probably the most popular solution. Many S3-compatible providers also exist that can run locally.

Object storage is a separate service built specifically for file storage. Files live independently from the application, providing durability, scalability, and the industry-standard approach.

This tutorial uses AWS S3 because it’s the most common solution encountered everywhere. Understanding S3 allows applying the same concepts to virtually any object storage provider.

Understanding Current Image Handling

Before changing code, examine how the application currently handles images to identify exactly what needs modification.

The project structure contains a media directory with a profile_pics folder where all uploaded images currently live.

In routers/users.py, the upload profile picture endpoint takes a file, validates size, and passes it to process_profile_image(). This function currently performs two tasks: processing the image using Pillow and saving it to disk.

In image_utils.py, the process_profile_image() function opens the image, processes it, and saves it directly to the media folder with image.save(). The user model’s image_path property returns that local location.

The goal: separate concerns. process_profile_image() should only process the image and return bytes. A new function will take those bytes and upload them to S3.

The critical point: image processing doesn’t change. Pillow still handles resizing, JPEG normalization, and other transformations. Only the storage layer changes.

Installing boto3

Install boto3, the official AWS SDK for Python:

uv add boto3

boto3 is maintained by Amazon and represents the standard library for interacting with AWS services. This is an essential skill—many applications interact with AWS services.

Setting Up AWS S3

An AWS account is required. AWS offers a free tier, and even beyond that, profile picture storage costs essentially nothing.

Creating an S3 Bucket

Open the AWS S3 console (search for “S3” in the AWS console). Click “Create bucket” to begin.

Bucket Configuration

Bucket naming requirements:

Globally unique
Lowercase only
No underscores

Choose a name unlikely to collide with existing buckets (e.g., fastapi-blog-uploads).

Bucket namespace options:

Global namespace (traditional): Bucket name must be unique across all of AWS
Account regional (newer): Name only needs to be unique within your account and region, but the system appends your account ID and region to the name (resulting in longer names). AWS tags this as the recommended approach going forward.

Either option works for this tutorial—the rest of the video remains consistent regardless of choice.

Select a region (e.g., US East 1). AWS often defaults to your most-used region.

Object Ownership

New buckets default to “Bucket owner enforced” with ACLs disabled—this is exactly what’s needed. Older tutorials sometimes used Access Control Lists (ACLs), but those are legacy. Modern setups manage access with bucket policies and IAM policies.

Block Public Access Settings

This is the critical configuration for this tutorial. By default, AWS blocks all public access. For serving content like images to users, public read access is required—a common use case. Profile pictures need public access because browsers must load them.

Uncheck “Block all public access.” Keep the first two options checked, but uncheck:

“Block public access to buckets and objects granted through new public bucket or access point policies”
“Block public and cross-account access to buckets and objects through any public bucket or access point policies”

Acknowledge the warning. This is intentional—a bucket policy will be written that only allows reads for a specific prefix, not the entire bucket.

Leave all other settings as defaults and click “Create bucket.”

Configuring Bucket Policy

The bucket now exists, but nothing can access it yet. Two things are needed:

A bucket policy allowing public image reads from a specific path
An IAM policy and IAM user allowing the application to upload and delete images

Creating the Bucket Policy

Click into the bucket, navigate to the “Permissions” tab, and scroll to “Bucket policy.” Click “Edit.” The policy is written in JSON:

{
    "Version": "2012-10-17",
    "Statement": [
        {
	        "Sid": "PublicReadProfilePics",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::fastapi-blog-uploads/profile_pics/*"
        }
    ]
}

The Version date of 2012-10-17 is not outdated—it’s the version of the policy language itself. AWS hasn’t needed to update the format since then, so every policy written today uses this same version. It’s a syntax version number that happens to be a date.

The policy has Effect: Allow with action s3:GetObject. The key part is the Resource field. The entire bucket is not made public—only objects inside the profile_pics prefix. Browsers can load profile pictures, but other bucket contents aren’t accidentally exposed.

Update the bucket name to match yours (fastapi-blog-uploads/profile_pics/*). Save the changes.

Common errors at this point:

Bucket name doesn’t exactly match what’s in the policy
AWS is still propagating bucket creation (wait 10-15 seconds and retry)

Creating an IAM Policy

Public reads are now allowed for profile pictures, but the application still needs permissions to upload and delete images.

Search for “IAM” in the AWS console and navigate to it. Close any pop-ups.

IAM (Identity and Access Management) handles AWS permissions. Navigate to “Policies” and click “Create policy.”

Switch to the “JSON” tab and paste:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::fastapi-blog-uploads/profile_pics/*"
        }
    ]
}

This follows the principle of least privilege. Only PutObject (upload) and DeleteObject (delete old pictures) are granted. GetObject isn’t needed—the application doesn’t download files; it generates URLs for browsers to fetch them directly.

The policy is scoped to the specific bucket and specific prefix. Update the bucket name to match yours. Click “Next.”

Name the policy (e.g., FastAPIBlogS3Policy) and click “Create policy.”

Creating an IAM User

A policy now exists, but an IAM user is needed for the application.

If deploying within the AWS ecosystem, IAM roles can be used instead of access keys, and boto3 picks up credentials automatically. Access keys work great for local development or hosting outside AWS (VPS, other clouds).

Navigate to “Users” in IAM and click “Create user.” Enter a username (e.g., fastapi-blog-s3). Click “Next.”

In permissions, select “Attach policies directly” and search for the policy just created (fastapi-blog-s3-policy). Select it and click “Next.” Everything should look correct—click “Create user.”

Creating Access Keys

The user exists but needs access keys. Click on the user, navigate to the “Security credentials” tab, scroll to “Access keys,” and click “Create access key.”

AWS asks what the key is for. Select “Application running outside AWS.” Click “Next.”

Add an optional description (e.g., “FastAPI Blog”) and click “Create access key.”

Critical: AWS shows the secret access key only once. Copy both values immediately:

Access key ID
Secret access key

Store them somewhere safe temporarily (they’ll be added to .env shortly). Once this page closes, the secret access key cannot be retrieved again.

Configuring Application Settings

Adding Credentials to .env

Open .env and add the S3 configuration:

S3_BUCKET_NAME=fastapi-blog-uploads
S3_REGION=us-east-1
S3_ACCESS_KEY_ID=your_access_key_id_here
S3_SECRET_ACCESS_KEY=your_secret_access_key_here

Critical reminder: .env must be in .gitignore. Never commit credentials to version control. This is a common mistake even large companies make—it’s a severe security issue.

Updating config.py

Open config.py and add S3 settings to the Settings class:

class Settings(BaseSettings):
    model_config = SettingsConfigDict(
        env_file=".env",
        env_file_encoding="utf-8"
    )
    
    database_url: str
    secret_key: SecretStr
    # ... other settings ...
    
    # S3 Configuration
    s3_bucket_name: str
    s3_region: str = "us-east-1"
    s3_access_key_id: SecretStr | None = None
    s3_secret_access_key: SecretStr | None = None
    s3_endpoint_url: str | None = None

Why are credentials optional (| None)? For local development or VPS hosting, these are set in .env and boto3 uses them directly. If deploying within the AWS ecosystem (EC2, ECS), IAM roles can be used instead. boto3’s default credential chain picks up temporary credentials automatically—access keys aren’t needed at all.

Making these optional means the same code works in both situations.

The s3_endpoint_url field: For real AWS S3, this is None. S3-compatible services that run locally can be used by pointing boto3 at a local endpoint with this setting.

Refactoring Image Utilities

Open image_utils.py. Currently, process_profile_image() performs two jobs: processing with Pillow and saving to disk. With S3, these concerns must be separated. The processing function should only process the image and return processed bytes plus the generated filename. The storage layer takes those bytes and uploads them to S3.

Updating Imports

Remove the pathlib import (no longer working with the file system):

# REMOVE: from pathlib import Path

Add new imports:

import boto3
from starlette.concurrency import run_in_threadpool
 
from config import settings

boto3 handles S3 operations. run_in_threadpool offloads blocking boto3 calls. Settings provide S3 configuration.

Remove the PROFILE_PICS_DIR constant (no longer saving to disk).

Creating the S3 Client Helper

Add a helper function to get the S3 client:

def _get_s3_client():
    """Get configured S3 client. Leading underscore indicates private helper."""
    return boto3.client(
        "s3",
        region_name=settings.s3_region,
        aws_access_key_id=settings.s3_access_key_id.get_secret_value() if settings.s3_access_key_id else None,
        aws_secret_access_key=settings.s3_secret_access_key.get_secret_value() if settings.s3_secret_access_key else None,
        endpoint_url=settings.s3_endpoint_url,
    )

This handles passing credentials if they exist or letting boto3 find them automatically. The leading underscore signals this is a private helper—meant for internal module use, not direct import from other files.

Modifying process_profile_image()

Currently, the function returns a filename string and saves to disk. Change it to return image bytes and filename:

def process_profile_image(content: bytes) -> tuple[bytes, str]:
    """
    Process uploaded profile image: resize, convert format, save with unique name.
    Returns the saved filename (not the full path).
    """
    with Image.open(BytesIO(content)) as original:
        ...
 
        output = BytesIO()
 
        img.save(output, "JPEG", quality=85, optimize=True)
        output.seek(0)
 
    return output.read(), filename

Key changes: Instead of image.save() going to a file path, it saves to an in-memory BytesIO object. seek(0) rewinds to the beginning so it can be read. The function returns both processed bytes and the filename.

The function is now completely decoupled from the file system.

Removing Old Delete Function

Delete the delete_profile_image() function (deletes from local disk). It will be replaced with an S3 version:

# DELETE THIS ENTIRE FUNCTION
# def delete_profile_image(filename: str | None) -> None:
#     ...

Adding S3 Upload and Delete Functions

Add synchronous S3 operations:

def _upload_to_s3(file_bytes: bytes, key: str) -> None:
    """Upload bytes to S3 (synchronous, blocking operation)."""
    client = _get_s3_client()
    client.upload_fileobj(
        BytesIO(file_bytes),
        settings.s3_bucket_name,
        key,
        ExtraArgs={"ContentType": "image/jpeg"}
    )
 
 
def _delete_from_s3(key: str) -> None:
    """Delete object from S3 (synchronous, blocking operation)."""
    client = _get_s3_client()
    client.delete_object(
        Bucket=settings.s3_bucket_name,
        Key=key
    )

Both use the S3 client helper. Note ContentType: "image/jpeg" in upload. Without this, browsers sometimes download the image instead of displaying it.

These boto3 calls are blocking (not asynchronous). As learned earlier with Pillow, blocking work directly in async endpoints blocks the event loop, preventing the server from handling other requests.

The pattern: if something blocks, offload it to the thread pool using run_in_threadpool. That’s the usual pattern for using any blocking library in an async FastAPI app.

Adding Async Wrappers

Create async wrappers that use run_in_threadpool:

async def upload_profile_image(file_bytes: bytes, filename: str) -> None:
    """Upload profile image to S3 (async, non-blocking)."""
    key = f"profile_pics/{filename}"
    await run_in_threadpool(_upload_to_s3, file_bytes, key)
 
 
async def delete_profile_image(filename: str) -> None:
    """Delete profile image from S3 (async, non-blocking)."""
    if filename is None:  
	    return
    key = f"profile_pics/{filename}"
    await run_in_threadpool(_delete_from_s3, key)

These create the full S3 key by appending the profile_pics/ prefix. Anywhere in the application, await upload_profile_image() now uploads to S3 in the thread pool without blocking the server.

Note delete_profile_image() is now async. Previously it was synchronous (just deleting a local file). Now it makes an S3 API call requiring await.

The key format matches the S3 bucket policy: profile_pics/*.

Updating the User Model

The image_path property currently returns a local path like /media/profile_pics/filename.jpg. It must return a full S3 URL.

Open models.py and add the settings import

from config import settings

Update the image_path property in the User model:

@property
def image_path(self) -> str:
    if self.image_file:
        return f"https://{settings.s3_bucket_name}.s3.{settings.s3_region}.amazonaws.com/profile_pics/{self.image_file}"
    return "/static/profile_pics/default.jpg"

This is the S3 URL format: https://bucket-name.s3.region.amazonaws.com/prefix/filename.

The default image remains on local storage. It’s part of the application (not user-uploaded content), so keeping it on the local filesystem is acceptable.

Updating Routes

Three endpoints need updates: upload profile picture, delete profile picture, and delete user (should clean up profile pictures in S3).

Updating Imports in users.py

Open routers/users.py and add the boto error handling import:

from botocore.exceptions import ClientError

Update the image_utils import to include the new upload function:

from image_utils import delete_profile_image, process_profile_image, upload_profile_image

Updating delete_user Endpoint

Locate delete_user and add await to the delete call:

# Clean up profile picture (only after successful commit)
if old_filename:
    await delete_profile_image(old_filename)  # Now async

Updating upload_profile_picture Endpoint

The beginning remains the same (check authorization, read bytes, validate size). The logic changes at image processing:

...
try:  
    processed_bytes, new_filename = await run_in_threadpool(process_profile_image, content)  
except UnidentifiedImageError as err:  
    raise HTTPException(  
        status_code=status.HTTP_400_BAD_REQUEST,  
        detail="Invalid image file. Please upload a valid image (JPEG, PNG, GIF, WebP).",  
    ) from err  
  
# Upload to S3 (also runs in threadpool via async wrapper)  
try:  
    await upload_profile_image(processed_bytes, new_filename)  
except ClientError as e:  
    raise HTTPException(  
        status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,  
        detail=f"Failed to upload image. {str(e)}",  
    )  
  
old_filename = current_user.image_file  
  
current_user.image_file = new_filename  
await db.commit()  
await db.refresh(current_user)  
  
if old_filename:  
    await delete_profile_image(old_filename)  
  
return current_user

Key changes:

process_profile_image() now returns a tuple: (processed_bytes, filename)
Upload to S3 happens after processing
Database updates follow upload
Old image deletion happens last (after successful commit)

Order of operations matters. If database update fails, the worst case is an orphan file in S3 (cleanable later). The alternative—deleting the old image first—could leave the user with no profile picture at all.

Updating delete_user_picture Endpoint

Add await to the delete call:

# Delete file (only after successful commit)
await delete_profile_image(old_filename)  # Now async

Removing Media Directory Mount

Open main.py. Previously, the media directory was mounted so FastAPI could serve local files. Local media is no longer used.

Remove the media mount:

app.mount("/static", StaticFiles(directory="static"), name="static")
# REMOVE THIS:
# app.mount("/media", StaticFiles(directory="media"), name="media")

Keep the static directory mount (still needed for CSS, JavaScript, and the default profile picture).

Testing S3 Connection

Before testing the full application, verify S3 credentials and permissions work correctly. A small script helps confirm upload and delete capabilities.

The tutorial includes check_s3.py:

"""
Quick script to verify AWS S3 credentials and permissions.
 
Run with: uv run check_s3.py
 
This checks that your .env credentials can upload to and delete from your S3 bucket
without needing to go through the full application flow.
"""
 
from io import BytesIO
 
from botocore.exceptions import BotoCoreError, ClientError
 
from config import settings
from image_utils import _get_s3_client
 
 
def check_s3_connection():
    s3 = _get_s3_client()
 
    print(f"Bucket: {settings.s3_bucket_name}")
    print(f"Region: {settings.s3_region}")
    print()
 
    test_key = "profile_pics/test.txt"
 
    # Test upload
    try:
        s3.upload_fileobj(
            BytesIO(b"test"),
            settings.s3_bucket_name,
            test_key,
            ExtraArgs={"ContentType": "text/plain"},
        )
        print("Upload: SUCCESS")
    except (BotoCoreError, ClientError) as e:
        print(f"Upload: FAILED - {e}")
        return
 
    # Test delete
    try:
        s3.delete_object(Bucket=settings.s3_bucket_name, Key=test_key)
        print("Delete: SUCCESS")
    except (BotoCoreError, ClientError) as e:
        print(f"Delete: FAILED - {e}")
        return
 
    print()
    print("All tests passed! Your S3 configuration is working.")
 
 
if __name__ == "__main__":
    check_s3_connection()

Run it:

uv run check_s3.py

Expected output:

Bucket: fastapi-blog-uploads
Region: us-east-1
✓ Upload successful
✓ Delete successful
✓ All tests passed - configuration is working

Troubleshooting common issues:

Upload fails: IAM policy is wrong, bucket name is wrong, or credentials are wrong
Upload works but delete fails: DeleteObject permission is missing from IAM policy
Script works but images later show 403 Forbidden in browser: Bucket policy isn’t set correctly, or the two “block public access” bucket policy settings weren’t unchecked when creating the bucket

This script helps narrow down problems—if it doesn’t work, the application won’t work either.

Testing the Full Application

Start the development server:

uv run fastapi dev main.py

Navigate to the application. Existing profile pictures are broken—expected because those files sit on local disk while the application now points to S3. Once a new image uploads, it goes to S3 and loads correctly.

Navigate to the account page and log in (e.g., test@example.com / testpassword1!). Upload a new profile picture. The upload succeeds and the image displays.

Verify it’s coming from S3: Right-click the image and select “Open image in new tab.” The URL shows: https://fastapi-blog-uploads.s3.us-east-1.amazonaws.com/profile_pics/...

Check the AWS console. Navigate to S3, click the bucket (fastapi-blog-uploads), and open the profile_pics folder. One object appears—the uploaded image.

Upload a different profile picture to test that the old one is deleted. Return to the account page, upload another image, and refresh the S3 console. Only one object remains—the new profile picture. The old one was deleted.

Summary

This tutorial made file storage production-ready by transitioning from local disk to AWS S3:

Set up an S3 bucket with proper permissions (bucket policy for public reads, IAM policy for application uploads/deletes)
Added boto3 to the project and configured S3 settings
Refactored image processing to separate it from storage layer
Wrote upload and delete functions using run_in_threadpool to prevent blocking the async application
Updated routes and models to work with S3
Removed the old local media mount (everything now serves from S3)

The stack now resembles a real production stack: PostgreSQL for the database, S3 for file storage, async endpoints for database work, and thread pool offloading for blocking operations.

The next tutorial covers testing—writing tests for the FastAPI application and examining patterns for testing routes, authentication, database interactions, and more.

Quartz 4

Explorer

16. AWS S3 and Boto3 - Moving File Uploads to the Cloud