Backend systems encompass numerous components, but this guide focuses on the fundamental topics used in approximately 90% of codebases. Understanding these core concepts provides the foundation needed for most backend development work.

HTTP Protocol: The Foundation of Client-Server Communication

HTTP serves as the primary medium through which browsers communicate with servers, enabling both data transmission and retrieval. While various protocols exist for client-server communication, HTTP remains the most widely adopted and forms the basis of modern web architecture.

The Two Core Principles of HTTP

1. Statelessness

Statelessness means HTTP has no memory of past interactions. Each HTTP request must carry all necessary information for the server to process it, including headers, URLs, and methods. Once the server responds, it completely forgets about that request. When a client makes another request, the server treats it as an entirely new, unrelated event.

This concept of self-contained requests is crucial: since the server doesn’t remember past requests, each request must include all necessary data—such as authentication tokens or session information—to handle that specific interaction. For example, when accessing a user profile, the client must provide credentials (cookies or tokens) on every single request so the server can identify which user is requesting the data.

Benefits of the Stateless Model:

Simplicity: Stateless design simplifies server architecture because the server doesn’t need to store session information, which would otherwise require additional resources and complexity.
Scalability: The stateless protocol makes it easy to distribute requests across multiple servers, as no single server needs to track session state.
Reliability: If a server crashes, it doesn’t affect client interactions since there’s no session or request memory that needs to be restored.

Despite HTTP’s stateless nature, developers often implement state management techniques like cookies, sessions, or tokens to maintain continuity in interactions where needed, such as user logins or shopping carts.

2. Client-Server Model

In a typical HTTP request flow, there are always two parties: a client and a server.

The Client (typically a web browser or application) initiates communication by sending requests to the server. The client is responsible for providing all information the server needs, such as the resource URL and headers.

The Server hosts resources like websites, APIs, or other content and waits for incoming requests from clients. When the server receives a request, it processes it and sends back the appropriate response—whether that’s a web page, data, an error message, a JSON file, a text file, or any other content type.

Remember: HTTP protocol dictates that communication is always initiated by the client to receive some kind of response from the server.

HTTP vs HTTPS

Throughout this discussion, HTTP and HTTPS can be considered interchangeable. HTTPS is essentially a more secure version of HTTP, but the underlying principles remain the same. HTTPS adds security features like encryption, security certificates, and TLS (Transport Layer Security), which border more into network engineering domain.

Connection Establishment

Before sending requests or receiving responses, the client and server must establish a connection mechanism. HTTP uses TCP (Transmission Control Protocol) as its underlying transport protocol. While HTTP doesn’t strictly require the transport protocol to be connection-based, it does require reliability—the protocol must not lose messages and should present errors when issues occur.

Among the two most common transport protocols on the internet (TCP and UDP), TCP is considered more reliable, which is why HTTP relies on it. TCP is connection-based, meaning it establishes a dedicated connection before data transfer begins.

The OSI Model and Backend Engineering

The OSI model is often referenced when discussing data transmission over networks. As backend engineers, we primarily work with Layer 7 (the Application Layer). Concepts like TCP handshakes, connection establishment, and TLS encryption often fall into the lower network layers. While understanding these concepts is beneficial, diving too deep into them can become a rabbit hole requiring extensive network engineering knowledge. Therefore, this guide focuses on the application layer with brief overviews of network layer concepts as needed.

HTTP Versions: Evolution Over Time

HTTP has evolved through several versions, each redefining how clients and servers exchange data:

HTTP 1.0: Each request opened a new connection, leading to inefficiencies. A connection had to be established and closed for every request-response cycle, which significantly slowed performance.

HTTP 1.1: Introduced persistent connections, allowing multiple requests and responses over the same TCP connection. This significantly improved performance by eliminating the need to establish a new connection for each request. It also added chunked transfer encoding and better caching mechanisms.

HTTP 2.0: Introduced multiplexing, allowing multiple requests and responses over a single connection. It uses binary framing instead of text-based transmission and supports header compression with HPACK. It also enables server push, allowing servers to send resources before the client explicitly requests them.

HTTP 3.0: Built on the QUIC protocol, a transport layer protocol over UDP instead of TCP. This design choice improved performance with faster connection establishment, reduced latency, and better handling of packet loss. It continues to support multiplexing without head-of-line blocking, which remains an issue in HTTP 2.0.

Key Takeaway: The fundamental concept to remember is that clients and servers establish a network connection through which messages are sent and received.

HTTP Messages: Structure and Components

HTTP communication involves two types of messages: request messages (sent by the client) and response messages (sent by the server).

Request Message Structure

A typical HTTP request message contains:

Request Method (GET, POST, PUT, etc.)
Resource URL (the endpoint being requested from the server)
HTTP Version (e.g., HTTP/1.1, currently the most widely used)
Host (the domain of the frontend or API)
Headers (key-value pairs containing metadata)
Blank Line (signifying the end of headers and start of body)
Request Body (data the client wants to send to the server)

Response Message Structure

A typical HTTP response message contains:

HTTP Version
Status Code (e.g., 200)
Status Text (e.g., “OK”)
Response Headers (key-value pairs containing metadata)
Blank Line (signifying the end of headers and start of body)
Response Body (the data being returned to the client)

HTTP Headers: The Metadata Layer

HTTP headers are key-value pairs that transmit metadata about the request or response. This is a critical concept because headers form a major part of both requests and responses.

Why Do We Need Headers? To understand the purpose of headers, consider a real-life analogy: when sending packages via courier, do we keep the recipient’s address and phone number inside the package or write it on top? We write it on top because the courier needs this information to successfully transmit the package through different modes of transportation. If all this information were kept inside the package, they would have to open it repeatedly to check the address.

Keeping metadata on top of the parcel provides a quick way to check information about the package without opening it. HTTP headers serve a similar purpose but with broader applications.

Categories of HTTP Headers

1. Request Headers

Request headers are sent by the client to the server to provide information about the request itself:

User-Agent: Identifies the type of client (browser, Postman, mobile app, server, etc.)
Authorization: Sends credentials like Bearer tokens to identify the user
Accept: Specifies what content types are expected (JSON, text, HTML, etc.)

Request headers help servers understand the client’s environment, preferences, and capabilities.

2. General Headers

General headers are used in both requests and responses and contain metadata about the message itself:

Date: The timestamp of the message
Cache-Control: Caching mechanisms like no-cache or max-age
Connection: Connection information such as keep-alive or close

3. Representation Headers

Representation headers deal with the representation of the resource being transmitted, whether in the request body or response:

Content-Type: Describes the media type (JSON, HTML, etc.)
Content-Length: Specifies the size of the resource in bytes
Content-Encoding: Indicates any encoding applied (gzip, deflate, etc.)
ETag: A unique identifier, mostly used for caching

These headers ensure clients and servers know how to interpret and process the message body.

4. Security Headers

Security headers enhance the security of requests and responses by controlling behaviors related to content loading, cookies, and encryption:

HTTP Strict Transport Security (HSTS): Ensures the client only communicates with the server over HTTPS, preventing protocol downgrade attacks
Content-Security-Policy (CSP): Restricts the sources from which content like JavaScript, CSS, and images can be loaded, helping prevent cross-site scripting (XSS) attacks
X-Frame-Options: Prevents the web page from being embedded in an iframe, mitigating clickjacking attacks
X-Content-Type-Options: Ensures the browser doesn’t try to guess the MIME type of content, preventing MIME type sniffing attacks
Set-Cookie with HttpOnly or Secure flags: Secures cookies by making them inaccessible to JavaScript and ensuring they’re sent only over HTTPS

Security headers protect both client and server from various attacks by controlling browser behavior and enforcing security policies.

Key Properties of HTTP Headers

Extensibility

HTTP is highly extensible because headers can be easily added or customized without altering the underlying protocol. Headers can be defined and used for various purposes, making HTTP adaptable to new technologies and use cases:

Security Enhancements: Headers like Strict-Transport-Security enforce secure connections
Custom Headers: Developers can create custom headers (e.g., X-Custom-Header) for specific application needs
Content Negotiation: Headers like Accept, Accept-Language, and Accept-Encoding allow servers to serve different versions of content based on client preferences

Remote Control Concept

HTTP headers act as a form of remote control for the server. They allow the client to send instructions or preferences to the server, influencing how the server responds or processes requests:

Content Type Negotiation: Clients can request specific formats using the Accept header, and the server responds with the appropriate format (HTML if requested, JSON if requested)
Caching and Expiration Control: The server uses headers like Cache-Control or Expires to control how long a resource should be cached by the client
Authentication: The client authenticates itself through the Authorization header, influencing access control decisions

These capabilities provide powerful mechanisms on top of basic message exchange, proving useful in numerous scenarios.

HTTP Methods: Defining Intent

HTTP methods exist to represent different kinds of actions that a client can request from a server. Instead of every request doing the same thing, methods define the intent of the interaction, giving clear semantic meaning to each type of action.

Common HTTP Methods:

GET: Used to fetch data from the server. It should not modify anything on the server.
POST: Used to create new data on the server. POST requests have a body, which makes sense because you need a way to send user data to the server.
PATCH: Used to update existing data. For example, when a user updates their name on a profile page. PATCH also has a request body to send the updated data to the server.
PUT: Also used to update data, but with a key difference from PATCH. Whatever data comes in the PUT request body should completely replace the previous instance. PATCH can be thought of as an append action or selective replacement, while PUT is a complete replacement. Many developers use PUT when they should be using PATCH, going against the semantic intent. The rule of thumb: always use PATCH unless you have a specific use case for PUT.
DELETE: Used to delete a resource from the server.

Idempotent vs Non-Idempotent Methods

A prevalent concept in HTTP methods is the distinction between idempotent and non-idempotent operations.

Idempotent methods can be called multiple times with the same result:

GET: Fetching data multiple times doesn’t modify anything on the server, so the result remains the same. GET is obviously idempotent.
PUT: Since PUT completely replaces a resource, it doesn’t matter how many times you replace the old data with the new data—the result will always be the same.
DELETE: You can only delete a resource from the server once. After that, it’s already deleted. You cannot perform the action multiple times and expect different results. The resource can only be deleted once, so subsequent DELETE requests produce the same result (resource not found).

Non-idempotent methods produce different results when called multiple times:

POST: When a user submits a request to create a new resource (like a note in an app), the first submission creates one note. If they submit the same request a second time, a second note is created. There are two different results for the same kind of request, which is why POST is considered non-idempotent.

The OPTIONS Method and CORS

The OPTIONS method has a specific use case related to CORS (Cross-Origin Resource Sharing) flow. You probably won’t use this method directly as a developer, but you’ll see it occasionally in your browser’s network tab in preflight requests.

The OPTIONS method is used to fetch the capabilities of the server for a cross-origin request. This is necessary because browsers enforce the Same-Origin Policy, which restricts web pages from making requests to domains different from the one serving the web page.

Your JS code
    |
    |  (1) Preflight request  -->   OPTIONS /check?
    |
 Browser (security guard)
    |                            Server
    |  <-- Rules (CORS headers) --
    |
    |  (2) Real request  --->   POST /data

CORS is a security mechanism enforced by browsers to control how web applications interact with resources hosted on different domains (cross-origin). Without CORS, browsers block requests made from a web application running on one origin (like example.com) to a different origin (like api.example.com) for security reasons. CORS allows servers to specify who can access their resources and how.

Understanding Same-Origin Policy

By default, browsers follow the Same-Origin Policy, which means a web page can only make requests to the same domain that served the web page. For a request to be considered same-origin, three components must match:

Protocol (http vs https)
Domain (example.com vs api.example.com)
Port (5173 vs 3000)

If any of these differ, the request is considered cross-origin.

There are two primary types of CORS flows:

Simple Request Flow
Pre-flighted Request Flow

A simple request doesn’t require a pre-flight check. Here’s how it works:

Client sends request: The browser automatically adds the Origin header to indicate the request’s origin. The request uses a simple method (typically GET, POST, or HEAD).
Server processes request: The server checks the Origin header against its CORS policy. If the origin is allowed, the server includes the Access-Control-Allow-Origin header in the response.
Server responds: The server responds with the resource and includes the necessary CORS headers, particularly Access-Control-Allow-Origin.
Browser validates: When the browser receives the response, it checks whether the server’s response includes the Access-Control-Allow-Origin header. If the header value matches the client’s origin (or contains * meaning all origins are allowed), the browser allows the response to pass through to the JavaScript client. If this header is missing or doesn’t match, the browser blocks the response and throws a CORS error in the console.

Example Response Headers:

Access-Control-Allow-Origin: https://example.com

Access-Control-Allow-Origin: *

A pre-flight request is required when certain conditions are met. The browser makes an OPTIONS request before the actual request to verify the server’s capabilities and permissions. A request qualifies as pre-flighted when:

The request is cross-origin (different protocol, domain, or port), AND
ONE of the following conditions is true:
- The method is not GET, POST, or HEAD (e.g., PUT, DELETE, PATCH)
- The request includes non-simple headers (e.g., Authorization or custom headers)
- The request has a Content-Type other than application/x-www-form-urlencoded, multipart/form-data, or text/plain (e.g., application/json)

Since most modern applications use JSON data, most cross-origin requests are pre-flighted.

Pre-flight Request Structure: The browser sends an OPTIONS request that looks like this:

Method: OPTIONS
URL: /api/resource
HTTP Version: 1.1
Host: api.example.com
Origin: https://example.com
Access-Control-Request-Method: PUT
Access-Control-Request-Headers: authorization

This request asks the server:

Do you support the PUT method for this route?
Do you allow the Authorization header?

Server Response to Pre-flight: If the server properly handles CORS, it responds with:

Status: 204 No Content
Access-Control-Allow-Origin: https://example.com
Access-Control-Allow-Methods: GET, POST, PUT, DELETE
Access-Control-Allow-Headers: authorization, content-type
Access-Control-Max-Age: 86400

Let’s break down these headers:

Access-Control-Allow-Origin: Specifies which origin is allowed (or * for all origins)
Access-Control-Allow-Methods: Lists the HTTP methods the server supports for this resource
Access-Control-Allow-Headers: Lists the non-simple headers the server accepts
Access-Control-Max-Age: Tells the browser not to make more pre-flight requests for the specified duration (in seconds). This saves bandwidth by caching the CORS configuration.

Final Request: After the pre-flight request succeeds (returns status 204), the browser validates all the conditions. If everything checks out, the browser sends the original request (in this case, the PUT request). The server then processes this request and responds normally with the requested operation’s result.

HTTP Response Status Codes

Response status codes communicate the result of a request in a standardized way. You can quickly determine whether a request succeeded, failed, or requires further action just by looking at the status code, without examining the response body or structure.

Status codes provide immediate feedback about the request outcome. They help clients handle errors by providing specific codes to identify problems. For example, a 401 (Unauthorized) status allows the client to automatically log the user out and prompt for re-authentication. A 400 (Bad Request) error due to invalid form input allows the client to ask the user to correct their submission.

Standardization is crucial. HTTP response codes are standardized across all web services, enabling consistency in how servers communicate with clients regardless of platform or language. Whether you’re building a server in Python, Go, Rust, JavaScript, or Ruby, you must follow these standards: 200 for success, 201 for resource creation, and so on.

Before HTTP status codes existed, clients had to guess the outcome based on response content, leading to inconsistencies and inefficiencies. Status codes provide a universal language that all clients and servers understand, streamlining interactions and error handling.

Response codes are three-digit numbers starting with 1, 2, 3, 4, or 5. The first digit categorizes the response:

1xx: Informational responses
2xx: Success responses
3xx: Redirection responses
4xx: Client errors
5xx: Server errors

1xx: Informational Responses

100 Continue: Sent by the server to indicate it has received the headers and the client can proceed to send the request body. Commonly used in large uploads—the client sends headers first, and if the server approves, it sends 100 Continue so the client can send the rest of the body.

101 Switching Protocols: Indicates the server is switching protocols as requested by the client, such as upgrading from HTTP to WebSocket.

These status codes aren’t frequently encountered in day-to-day development.

2xx: Success Responses

200 OK: The most common success code. Indicates the request was successful and the server is returning the requested resource or performing the requested action. Used for successful GET requests where a resource is retrieved.

201 Created: Indicates the request has been fulfilled and resulted in the creation of a new resource. Used for POST requests that create new resources (like form submissions).

204 No Content: Indicates the request was successful, but there’s no content to return. Seen in CORS pre-flight OPTIONS requests where the server provides information via headers only. Sometimes used for DELETE requests—the server confirms deletion but has no content to return.

3xx: Redirection Responses

301 Moved Permanently: The requested resource has been permanently moved to a new URL, and future requests should use this new URL. For example, if you initially had a /user route and later moved it to /person, you’d add a 301 response for the old route to redirect to the new one, maintaining backwards compatibility for old applications.

302 Found (Temporary Redirect): The requested resource is temporarily located at a different URL, but the client should continue using the original URL for future requests. Used for temporary situations like campaigns where you want to redirect traffic for a short period but eventually revert to the original route.

304 Not Modified: Indicates the resource hasn’t been modified since the last request. Used in conjunction with conditional GET requests for efficient caching. When using ETags, this tells the client to use the cached response instead of downloading a new one.

4xx: Client Errors

As a backend engineer, you’ll frequently work with these errors since they’re triggered by client behavior.

400 Bad Request: Triggered when the client sends invalid or illogical data. For example, expecting a number but receiving an array or string, or expecting an email but receiving a phone number. It tells the client something is wrong with the request format and they need to fix it before resubmitting.

401 Unauthorized: Triggered when a request requires authentication but the client either failed to provide valid credentials or isn’t authenticated at all. For example, if you’re expecting a JWT token and either the token has expired or the client hasn’t sent one, you respond with 401.

403 Forbidden: The server understood the request but refuses to authorize it. This can happen even if the client is authenticated. For example, when User A tries to delete a resource belonging to User B, the server responds with 403, indicating the user doesn’t have the necessary permissions.

404 Not Found: The most famous status code. Fired when the client requests a resource that’s unavailable, either because the URL is incorrect or the resource has been deleted.

405 Method Not Allowed: Fired when an invalid HTTP method is used, such as trying to PUT to a resource that only accepts GET or POST. This often happens due to typos—doing a PATCH instead of PUT, or POST instead of PUT on the frontend.

409 Conflict: Has numerous use cases. For example, if your app allows users to create folders with unique names, and they try to create a folder with a name that already exists, you respond with 409. The client understands a conflict exists and can prompt the user to choose a different name.

429 Too Many Requests: Used when rate limiting the client. Rate limiting means restricting the number of requests a client can make within a specific time period. If your server is configured to allow at most 60 requests per second per client, and the client exceeds that limit, you respond with 429.

5xx: Server Errors

500 Internal Server Error: Used for unexpected conditions on the server—when a process breaks or exceptions are raised that aren’t handled. Instead of returning an empty response or hanging the request, you respond with 500 so the client knows something went wrong server-side.

501 Not Implemented: Used when the server doesn’t support the requested HTTP method or functionality but plans to add it in the future. This communicates to the client that the feature isn’t currently available but may be implemented later.

502 Bad Gateway: Usually seen in proxies like Nginx. Occurs when a server acting as a proxy (in a load-balanced system or reverse proxy) receives an invalid response from the upstream server. This isn’t something you return intentionally—it’s handled by proxies and load balancers.

503 Service Unavailable: Used when the service is temporarily unable to handle requests, such as during high traffic periods or maintenance. This tells the client the service is currently unavailable and they should try again later.

504 Gateway Timeout: Similar to 502 but specifically means the upstream server failed to respond within the timeout period. For example, if you’re using Nginx and it couldn’t get a response from your original server running behind it, Nginx responds with 504, indicating it didn’t receive a response within the timeout window.

HTTP Caching

HTTP caching is a technique to store copies of responses for reuse, reducing the need for repeated requests to the server. This improves load times, reduces bandwidth consumption, and decreases server load. The client doesn’t need to download large amounts of data, and the server doesn’t need to send data repeatedly if nothing has changed—the client can simply reuse the old data.

How Caching Works

The caching mechanism involves several key headers:

Cache-Control: Tells the client how long to maintain the cache for a resource. For example, max-age=10 means the cache should be maintained for a maximum of 10 seconds.
ETag: A hash computed from the response. For production systems, the server typically hashes the response content and sends that hash as the ETag. This unique identifier helps determine if the resource has changed.
Last-Modified: Indicates the last time the request resource was modified. Clients can use this timestamp to decide whether to use the cached resource or request a new one.

The Caching Request Cycle

Initial Request:

Client makes a GET request to /api/resource
Server responds with:
- Status: 200 OK
- Cache-Control: max-age=10
- ETag: 3141 (hash of the response)
- Last-Modified: [timestamp]
- Response body with the actual data

Subsequent Request (Within Cache Period):

Client makes another GET request to the same resource
Client includes:
- If-None-Match: 3141 (the ETag from the cached response)
- If-Modified-Since: [timestamp] (from Last-Modified)
Server checks these conditions:
- Does the current ETag match the provided one?
- Has the resource been modified after the provided timestamp?
If nothing has changed, server responds with:
- Status: 304 Not Modified
- No response body (saving bandwidth)
Browser uses the cached version

After Resource Update:

Client updates the resource (POST/PUT/PATCH request)
Server responds with:
- Status: 200 OK
- New ETag: 2943
- New Last-Modified timestamp
Next GET request from client includes the old ETag: 3141
Server detects the mismatch (current ETag is 2943)
Server responds with:
- Status: 200 OK
- New complete response body
- Updated ETag and Last-Modified headers
Client updates its cache with the new data

Caching Considerations

In production settings, caching becomes more complicated because the server must manually implement and manage all ETags. If you accidentally forget to update an ETag, the client will continue using the cached version with outdated data, which creates serious problems.

Modern caching solutions like React Query provide complete client-side caching control. The client has full power over when to use cached resources and when to refetch, at what intervals, and with what capabilities. This approach is generally superior to traditional HTTP-based caching for most applications.

However, HTTP-based caching remains valuable for simple use cases where you don’t need complex caching logic.

Content Negotiation

Content negotiation is a mechanism by which client and server agree on the best format to exchange data. The client indicates its preferred format (JSON, XML, HTML, etc.), and the server attempts to respond with a compatible format or a suitable fallback.

Types of Content Negotiation

1. Media Type Negotiation The client specifies the desired format through the Accept header:

Accept: application/json (requesting JSON)
Accept: application/xml (requesting XML)
Accept: text/html (requesting HTML) 2. Language Negotiation The client requests content in a specific language using the Accept-Language header:
Accept-Language: en (English)
Accept-Language: es (Spanish) 3. Encoding Negotiation The client specifies which encoding it supports using the Accept-Encoding header:
Accept-Encoding: gzip, deflate, br, zstd

The server responds with the requested compression format.

Content Negotiation in Practice

Request Example:

GET /api/resource
Accept: application/json
Accept-Language: en
Accept-Encoding: gzip

Server Response:

Status: 200 OK
Content-Type: application/json
Content-Language: en
Content-Encoding: gzip
[Response body in JSON format, in English, compressed with gzip]

If the client changes preferences:

Accept: application/xml
Accept-Language: es

The server responds accordingly:

Content-Type: application/xml
Content-Language: es
[Response body in XML format, in Spanish]

Benefits of Content Negotiation

The client can communicate its preferences regarding data format and language. Based on these preferences, the server can respond in the appropriate format, making the client’s life easier by adhering to its specified requirements.

HTTP Compression

HTTP compression is a subset of content negotiation focused specifically on reducing response size through encoding.

Consider a large file with 11,000 entries:

With gzip compression enabled:
- File size: 3.8 MB
- Header: Content-Encoding: gzip

Without compression:
- File size: 26 MB

That’s a massive difference! Imagine every client having to download 26 MB instead of 3.8 MB. This consumes enormous bandwidth. Compression allows the server to send data in a compressed format, which the browser automatically decompresses on the client side, resulting in the same usable data but with dramatically reduced transfer size.

How Compression works:

Client sends request with: Accept-Encoding: gzip, deflate, br, zstd
Server compresses the response using one of the supported encodings (typically gzip)
Server responds with: Content-Encoding: gzip
Browser automatically decompresses the data
Client receives the full, uncompressed data

This process happens transparently to the application layer—you work with uncompressed data on both ends, but the transmission is optimized.

Persistent Connections and Keep-Alive

In the early days of HTTP (specifically HTTP 1.0), each request-response cycle required a separate connection to the server. This created significant inefficiencies since establishing and closing TCP connections is resource-intensive and slow.

To address this problem, HTTP 1.1 introduced persistent connections. With persistent connections, a single TCP connection can be reused for multiple requests and responses, avoiding the overhead of opening and closing a connection for every interaction.

The Keep-Alive Mechanism

Keep-Alive enables persistent connections. It allows the client and server to reuse the same connection for multiple request-response cycles until one of them decides to close it.

Key Points:

Default Behavior: In HTTP 1.1, connections are persistent by default. You don’t need to do anything explicitly—they remain open for further requests unless explicitly closed.
Multiple Requests: Multiple sequential requests and responses can be sent over a single connection, reducing latency and saving resources since fewer connections need to be established.
Keep-Alive Header: While persistent connections are default in HTTP 1.1, the Connection: keep-alive header can still be used to explicitly ask the server to keep the connection open. This header can include options like:
- timeout: How long the connection should remain open
- max: How many requests can be sent before the connection closes
Closing Connections: When Connection: close is specified, the connection closes after the response is sent. This was the default behavior in HTTP 1.0 and can still be explicitly enforced in HTTP 1.1.

You usually won’t work directly with these headers—the default values work fine for most use cases. This is good-to-know information for understanding how HTTP manages connections under the hood.

Handling Large Requests and Responses

Modern applications frequently need to transfer large files (videos, images, audio, documents) between clients and servers. HTTP provides mechanisms for handling these large transfers efficiently.

Multipart Requests: Sending Large Files to the Server

Multipart requests are used for sending large files from client to server. The key difference from typical JSON request bodies is that in multipart requests, the file’s binary data is transferred in parts—hence the name “multipart.”

Multipart Request structure:

POST /upload
Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Length: [size]

------WebKitFormBoundary7MA4YWxkTrZu0gW
[Binary data of the file]
------WebKitFormBoundary7MA4YWxkTrZu0gW--

Important Components:

Content-Type: multipart/form-data indicates this is a multipart request
Boundary: A unique delimiter that separates different parts of the binary data. Since the file data is transferred in parts, we need a code that separates these parts. The boundary appears at the start of the binary data and again at the end when all binary data transfer is complete.
Content-Length: The total size of the request

The server reads the file using the boundary markers to identify where each part begins and ends, then reconstructs the complete file.

Streaming Large Responses: Server to Client

For sending large files from server to client, HTTP supports chunked transfer encoding or streaming.

Streaming Response Structure:

GET /large-file
Accept: text/event-stream

Response:
Content-Type: text/event-stream
Connection: keep-alive
Transfer-Encoding: chunked

How It Works:

Content-Type: text/event-stream indicates the server will stream data through events rather than sending a complete text file
Connection: keep-alive keeps the connection open until all data is sent
Chunked Transfer: The server sends the file in chunks continuously
Client Processing: The client receives chunks in different events and keeps appending the data, constructing the complete file progressively

This approach is efficient because:

The client doesn’t have to wait for the entire file to download before processing
Memory usage is optimized—the client can process data as it arrives
The user sees progress as data streams in, improving perceived performance

SSL, TLS, and HTTPS

While we don’t explicitly work with these protocols day-to-day, understanding them provides important context for secure communication.

SSL (Secure Sockets Layer)

SSL was the original protocol for securing communications between clients (like web browsers) and servers. It encrypted data so sensitive information (passwords, credit card numbers) couldn’t be intercepted by attackers.

Current Status: SSL is now outdated due to security vulnerabilities and has been completely replaced by TLS.

TLS (Transport Layer Security)

TLS is the modern, more secure version of SSL. It encrypts data in transit, ensuring that any data sent between client and server is protected from interception and tampering.

How TLS works:

Uses certificates to authenticate the server and establish an encrypted connection
Prevents eavesdropping and data breaches
Continuously updated with newer versions offering better security
Current recommended version: TLS 1.3

HTTPS (HTTP Secure)

HTTPS is simply HTTP with added security features provided by TLS (initially SSL, now TLS). The underlying mechanism uses TLS.

How HTTPS works:

When you visit a website using HTTPS, TLS encrypts the communication between your browser and the server
This protects sensitive data like login credentials from being intercepted by attackers
The padlock icon in your browser indicates a secure HTTPS connection
All data transmitted over this connection is encrypted

What You Need to Know

This level of understanding is sufficient for application-level development. The key takeaway: HTTPS uses TLS to provide encrypted, secure communication between clients and servers, protecting sensitive data from interception and tampering.

Quartz 4

Explorer

05. Understanding HTTP for backend engineers, where it all starts