ShuWa

1. Basic Concepts of HTTP#

1. What is HTTP?#

HTTP stands for HyperText Transfer Protocol.
The name "HyperText Transfer Protocol" can be broken down into three parts:

• HyperText: Text that goes beyond ordinary text; it is a mixture of text, images, videos, etc., and most importantly, it contains hyperlinks that allow navigation from one hypertext to another.
• Transfer: Bidirectional data transmission between two points.
• Protocol: Establishes a standard for communication between computers (more than two participants).

2. What are the common HTTP status codes?#

• 1xx status codes are informational and represent an intermediate state in protocol processing, and are rarely used in practice.
• 2xx status codes indicate that the server successfully processed the client's request, which is the status we most want to see.
  "200 OK" is the most common success status code, indicating everything is normal. If it is not a HEAD request, the response headers returned by the server will include body data.
  "204 No Content" is also a common success status code, similar to 200 OK, but the response headers do not contain body data.
  "206 Partial Content" is used for HTTP chunked downloads or resuming downloads, indicating that the body data returned in the response is not the entire resource, but part of it, and it is also a successful status from the server's perspective.
• 3xx status codes indicate that the resource requested by the client has changed, and the client needs to resend the request with a new URL to obtain the resource, which is known as redirection.
  "301 Moved Permanently" indicates a permanent redirect, meaning the requested resource no longer exists and a new URL must be used for access.
  "302 Found" indicates a temporary redirect, meaning the requested resource still exists but must be accessed using another URL temporarily.
  Both 301 and 302 will use the Location field in the response headers to specify the URL to redirect to, and the browser will automatically redirect to the new URL.
  "304 Not Modified" does not imply a redirect; it indicates that the resource has not been modified, redirecting to an existing cached file, also known as cache redirection, which tells the client to continue using the cached resource for cache control.
• 4xx status codes indicate that there is an error in the message sent by the client, and the server cannot process it, which is the meaning of error codes.
  "400 Bad Request" indicates that there is an error in the message sent by the client, but it is a general error.
  "403 Forbidden" indicates that the server forbids access to the resource, and it is not an error in the client's request.
  "404 Not Found" indicates that the requested resource does not exist on the server or cannot be found, so it cannot be provided to the client.
• 5xx status codes indicate that the client's request message is correct, but an internal error occurred on the server during processing, which belongs to server-side error codes.
  "500 Internal Server Error" is a general error code, similar to the 400 type, indicating that something went wrong on the server, but we do not know what.
  "501 Not Implemented" indicates that the functionality requested by the client is not supported, similar to "coming soon, please stay tuned."
  "502 Bad Gateway" is usually an error code returned when the server acts as a gateway or proxy, indicating that the server itself is functioning normally, but an error occurred when accessing the backend server.
  "503 Service Unavailable" indicates that the server is currently busy and cannot respond to the client temporarily, similar to "the network service is busy, please try again later."

3. What are the common HTTP fields?#

• Host field: Used by the client to specify the server's domain name when sending a request.
• Content-Length field: When the server returns data, it will include the Content-Length field, indicating the length of the data in this response. The HTTP protocol uses carriage return and line feed as boundaries for HTTP headers, and the Content-Length field as the boundary for the HTTP body; both methods are to solve the "sticky packet" problem.
• Connection field: The Connection field is most commonly used by the client to request the server to use the "HTTP persistent connection" mechanism for reusing other requests. The default connection in HTTP/1.1 is a persistent connection, but to maintain compatibility with older versions of HTTP, the value of the Connection header field must be specified as Keep-Alive.
• Content-Type field: The Content-Type field is used by the server in its response to inform the client of the format of the data being sent.
• Content-Encoding field: The Content-Encoding field indicates the compression method of the data, specifying what compression format is used for the data returned by the server, such as gzip.

4. GET vs POST#

• The GET method is safe and idempotent because it is a "read-only" operation; regardless of how many times the operation is performed, the data on the server remains safe, and the result is the same each time. Therefore, GET request data can be cached, which can be done on the browser itself (completely avoiding the browser sending requests) or on proxies (like nginx), and GET requests can be saved as bookmarks in the browser.
• POST, being an operation that "adds or submits data," modifies resources on the server, so it is not safe, and submitting data multiple times will create multiple resources, making it not idempotent. Therefore, browsers generally do not cache POST requests and cannot save POST requests as bookmarks.

5. HTTP Caching#

1. Strong Caching#

Strong caching is implemented using the following two HTTP response header fields, which indicate the validity period of the resource in the client's cache:

• Cache-Control, which is a relative time;
• Expires, which is an absolute time;

Cache-Control takes precedence over Expires.
The process is as follows:

• When the browser first requests a server resource, the server will return this resource along with the Cache-Control in the Response header, setting the expiration time.
• When the browser requests the same resource again, it will first compare the request time with the expiration time set in Cache-Control to determine if the resource has expired; if not, it will use the cached version; otherwise, it will request the server again.
• When the server receives the request again, it will update the Cache-Control in the Response header.

2. Negotiated Caching#

When the response code is 304, it tells the browser that it can use the locally cached resource; this method of informing the client whether it can use the cache is known as negotiated caching.

The first method: The If-Modified-Since field in the request header and the Last-Modified field in the response header work together. The meanings of these two fields are:

• The Last-Modified in the response header: Indicates the last modification time of this response resource.
• The If-Modified-Since in the request header: When the resource expires and the response header has a Last-Modified declaration, the next time a request is made, the Last-Modified time is included. When the server receives the request and finds If-Modified-Since, it compares it with the last modification time of the requested resource (Last-Modified). If the last modification time is newer (greater), it means the resource has been modified, and it returns the latest resource, HTTP 200 OK; if the last modification time is older (less), it means there are no new modifications to the resource, and it responds with HTTP 304 to use the cache.

The second method: The If-None-Match field in the request header and the ETag field in the response header work together. The meanings of these two fields are:

• The ETag in the response header: Uniquely identifies the response resource.
• The If-None-Match in the request header: When the resource expires, if the browser finds an ETag in the response header, it will include the If-None-Match value as the ETag value when making a request to the server again. The server compares the resource; if it has not changed, it returns 304; if it has changed, it returns 200.

If the ETag and Last-Modified field information is sent to the server, the ETag takes precedence.

Why does ETag take precedence? This is because ETag can solve several difficult problems that Last-Modified cannot:

1. The last modification time of a file may change without modifying the file's content, which can lead the client to believe that the file has been changed and thus request it again.
2. Some files may be modified within seconds, and If-Modified-Since can only check to the granularity of seconds; using ETag can ensure that the client can refresh multiple times within 1 second.
3. Some servers cannot accurately obtain the last modification time of a file.

2. Differences Between HTTP and HTTPS#

1. What are the differences between HTTP and HTTPS?#

• HTTP is the HyperText Transfer Protocol, where information is transmitted in plaintext, posing security risks. HTTPS addresses the security flaws of HTTP by adding the SSL/TLS security protocol between the TCP and HTTP layers, allowing messages to be transmitted securely.
• Establishing an HTTP connection is relatively simple; after the TCP three-way handshake, HTTP message transmission can occur. In contrast, HTTPS requires an additional SSL/TLS handshake process after the TCP three-way handshake before entering encrypted message transmission.
• The default ports for the two protocols are different; the default port for HTTP is 80, while for HTTPS it is 443.
• The HTTPS protocol requires a digital certificate to be applied for from a CA (Certificate Authority) to ensure the server's identity is trustworthy.

2. What problems does HTTPS solve for HTTP?#

Because HTTP transmits in plaintext, it has the following three security risks:

• Eavesdropping risk, where communication content can be intercepted on the communication link, making user information vulnerable.
• Tampering risk, such as forcibly injecting spam ads, causing visual pollution, and making it hard for users to see.
• Impersonation risk, such as impersonating a Taobao website, leading to potential financial loss for users.

How does HTTPS address these three risks?

• It achieves confidentiality of information through a hybrid encryption method, addressing the eavesdropping risk.
• It uses a hashing algorithm to ensure integrity, generating a unique "fingerprint" for the data, which is used to verify data integrity, addressing the tampering risk.
• It places the server's public key in the digital certificate, addressing the impersonation risk.

Hybrid Encryption#

HTTPS employs a hybrid encryption method that combines symmetric and asymmetric encryption:

• Asymmetric encryption is used to exchange the "session key" before communication begins, after which asymmetric encryption is no longer used.
• During communication, symmetric encryption is used to encrypt plaintext data with the "session key."

The reason for using a "hybrid encryption" approach:

• Symmetric encryption uses only one key, which is fast but must be kept secret, making secure key exchange difficult.
• Asymmetric encryption uses two keys: a public key and a private key, where the public key can be distributed freely while the private key must remain confidential, solving the key exchange problem but being slower.

Hashing Algorithm + Digital Signature#

To ensure that the transmitted content is not tampered with, we need to calculate a "fingerprint" for the content and then transmit it along with the content.
Upon receiving the content, the recipient calculates a "fingerprint" for the content and compares it with the "fingerprint" sent by the sender. If the fingerprints match, it indicates that the content has not been tampered with; otherwise, it can be determined that the content has been altered.
In computing, a hashing algorithm (hash function) is used to calculate the hash value of the content, which is the content's "fingerprint." This hash value is unique and cannot be used to derive the original content.
While hashing algorithms can ensure that content has not been tampered with, they cannot guarantee that the "content + hash value" has not been replaced by a man-in-the-middle, as there is a lack of proof that the message received by the client originates from the server.
To avoid this situation, asymmetric encryption algorithms are used, which involve two keys:

• One is the public key, which can be shared with everyone;
• The other is the private key, which must be managed by the owner and kept secret.
• Public key encryption and private key decryption ensure the security of content transmission, as content encrypted with the public key cannot be decrypted by others; only the holder of the private key can decrypt it to obtain the actual content.
• Private key encryption and public key decryption ensure that messages cannot be impersonated, as the private key must remain confidential. If the public key can successfully decrypt content encrypted with the private key, it proves that the message was sent by the holder of the private key.

Thus, the primary use of asymmetric encryption is to confirm the identity of messages through the method of "private key encryption and public key decryption." The digital signature algorithm we commonly refer to uses this method, but the content encrypted with the private key is not the content itself but the hash value of the content.

Digital Certificate#

What if the identity verification step is missing? What if the public key is forged?

Through an authoritative CA (Certificate Authority), the server's public key is placed in a digital certificate (issued by the CA). As long as the certificate is trustworthy, the public key is also trustworthy.

3. How is a connection established in HTTPS? What interactions occur during this process?#

The "handshake phase" of TLS involves four communications, using different key exchange algorithms, and the TLS handshake process may vary. Currently, two commonly used key exchange algorithms are RSA and ECDHE.
The detailed process of establishing a TLS connection:

1. ClientHello
   First, the client initiates an encrypted communication request to the server, known as the ClientHello request.
   At this step, the client primarily sends the following information to the server:
   (1) The TLS protocol version supported by the client, such as TLS 1.2.
   (2) A random number generated by the client (Client Random), which will be used as one of the conditions for generating the "session key."
   (3) A list of cipher suites supported by the client, such as the RSA encryption algorithm.
2. ServerHello
   After receiving the client's request, the server responds with a SeverHello message. The server's response includes the following:
   (1) Confirmation of the TLS protocol version; if the browser does not support it, encrypted communication will be terminated.
   (2) A random number generated by the server (Server Random), which will also be used as one of the conditions for generating the "session key."
   (3) A list of confirmed cipher suites, such as the RSA encryption algorithm.
   (4) The server's digital certificate.
3. Client Response
   After receiving the server's response, the client first verifies the authenticity of the server's digital certificate using the CA public key in the browser or operating system.
   If the certificate is valid, the client retrieves the server's public key from the digital certificate and uses it to encrypt a message to send to the server, including:
   (1) A random number (pre-master key). This random number will be encrypted with the server's public key.
   (2) A notification to change the encryption algorithm, indicating that subsequent information will be encrypted using the "session key."
   (3) A notification that the client handshake is complete, indicating that the client's handshake phase has ended. This also includes a summary of all previously sent data for the server to verify.
   The first item, the random number, is the third random number in the entire handshake phase, which will be sent to the server, so both the client and server will have the same random number.
   With these three random numbers (Client Random, Server Random, pre-master key), the server and client will use the agreed encryption algorithm to generate the "session key" for this communication.
4. Final Server Response
   After receiving the client's third random number (pre-master key), the server calculates the "session key" for this communication using the agreed encryption algorithm.
   Then, the server sends the final information to the client:
   (1) A notification to change the encryption algorithm, indicating that subsequent information will be encrypted using the "session key."
   (2) A notification that the server handshake is complete, indicating that the server's handshake phase has ended. This also includes a summary of all previously sent data for the client to verify.

At this point, the entire TLS handshake phase is complete. Next, the client and server enter encrypted communication, which is conducted using the ordinary HTTP protocol, but with the content encrypted using the "session key."

RSA Algorithm#

The connection process described above is the process of establishing a connection using the RSA algorithm, which ensures secure communication between the client and server. However, it still has flaws:
The main problem with using the RSA key exchange algorithm is that it does not support forward secrecy.
When the client transmits the random number (one of the conditions for generating the symmetric encryption key) to the server, it is encrypted with the public key. The server, upon receiving it, decrypts it with the private key. Therefore, if the server's private key is compromised, all previously intercepted TLS communication ciphertext can be decrypted.
To solve this problem, the ECDHE key exchange algorithm was introduced, which is now used by most websites.

ECDHE Algorithm#

The ECDHE key exchange algorithm is an evolution of the DH algorithm, so we will start with the DH algorithm.

DH Algorithm#

The DH algorithm is an asymmetric encryption algorithm, which can be used for key exchange. The core mathematical concept of this algorithm is discrete logarithms.
Discrete logarithms are based on logarithmic operations with "modular arithmetic," meaning taking remainders, which corresponds to the programming language operator "%" and can also be represented as mod. The concept of discrete logarithms is illustrated below:

In the above diagram, the base a and modulus p are public parameters of the discrete logarithm, meaning they are public, while b is the true number, and i is the logarithm. Knowing the logarithm allows calculation of the true number using the formula above. However, knowing the true number makes it very difficult to deduce the logarithm.
Especially when the modulus p is a large prime number, even if the base a and true number b are known, it is nearly impossible to calculate the discrete logarithm with current computing capabilities, which is the mathematical basis of the DH algorithm.

In the above diagram: a and b are the respective private keys, A and B are the respective public keys, G and P are public information, and K is the symmetric encryption key used between Xiaohong and Xiaoming, which can serve as the session key.

DHE Algorithm#

E stands for ephemeral (temporary), meaning that the private keys of both parties are randomly generated and temporary for each key exchange communication.
Thus, even if a skilled hacker cracks the private key of a particular communication process, the private keys of other communication processes remain secure, as each communication process's private key is independent, ensuring "forward secrecy."

ECDHE Algorithm#

Due to the poor computational performance of the DHE algorithm, which requires extensive multiplication, the ECDHE algorithm has emerged, which is widely used for key exchange.

The ECDHE algorithm utilizes the properties of elliptic curves to compute public keys and the final session key with less computational effort.

The process of Xiaohong and Xiaoming using the ECDHE key exchange algorithm is as follows:

• Both parties agree in advance on which elliptic curve to use and the base point G on the curve; these two parameters are public.
• Each party randomly generates a random number as their private key d and multiplies it with the base point G to obtain the public key Q (Q = dG). At this point, Xiaohong's public and private keys are Q1 and d1, while Xiaoming's public and private keys are Q2 and d2.
• Both parties exchange their public keys, and finally, Xiaohong calculates the point (x1, y1) = d1Q2, while Xiaoming calculates the point (x2, y2) = d2Q1. Since multiplication on the elliptic curve satisfies the commutative and associative laws, d1Q2 = d1d2G = d2d1G = d2Q1, the x-coordinates for both parties are the same, thus forming a shared key, which is the session key.

In this process, both parties' private keys are randomly and temporarily generated and are not public. Even with public information (elliptic curve, public key, base point G), it is still very difficult to calculate the discrete logarithm on the elliptic curve (private key).

Handshake Process#

First TLS Handshake
The client first sends a "Client Hello" message, which includes the TLS version number used by the client, the list of supported cipher suites, and the generated random number (Client Random).
Second TLS Handshake
The server receives the client's "hello" and responds with a "Server Hello" message, which includes the TLS version number confirmed by the server, a random number (Server Random), and a suitable cipher suite selected from the client's list.
Next, to prove its identity, the server sends a "Certificate" message, which includes the certificate.
This step differs significantly from the RSA handshake process because the server has chosen the ECDHE key exchange algorithm, so after sending the certificate, it sends a "Server Key Exchange" message.
In this process, the server does three things:

• It selects an elliptic curve named x25519, which also determines the base point G on the elliptic curve, both of which will be made public to the client.
• It generates a random number as the server's private key and keeps it locally.
• It calculates the server's elliptic curve public key based on the base point G and the private key, which will be made public to the client.

To ensure that this elliptic curve public key is not tampered with by a third party, the server will use the RSA signature algorithm to sign the server's elliptic curve public key.
Subsequently, a "Server Hello Done" message is sent, indicating to the client, "This is the information I provide; the greeting is complete."
Third TLS Handshake
After the client receives the server's certificate, it must verify whether the certificate is valid. If the certificate is valid, the server's identity is confirmed to be trustworthy. The verification process involves checking the certificate chain step by step to confirm its authenticity and using the public key from the certificate to verify the signature, thus confirming the server's identity. Once confirmed, the process can continue.
The client generates a random number as the private key for the client's elliptic curve, then generates the client's elliptic curve public key based on the information provided by the server earlier, and sends it to the server using a "Client Key Exchange" message.
The final session key is generated using the "client random number + server random number + x (the shared key calculated using the ECDHE algorithm)."
After calculating the session key, the client sends a "Change Cipher Spec" message to inform the server that subsequent communications will use symmetric encryption.
Next, the client sends an "Encrypted Handshake Message," which summarizes the previously sent data and encrypts it with the symmetric key for the server to verify whether the generated symmetric key can be used normally.
Fourth TLS Handshake
Finally, the server performs a similar operation, sending "Change Cipher Spec" and "Encrypted Handshake Message" messages. If both parties verify that encryption and decryption are functioning correctly, the handshake is officially complete. They can then normally send and receive encrypted HTTP requests and responses.

4. How does HTTPS ensure the integrity of application data?#

TLS is implemented in two layers: the handshake protocol and the record protocol:

• The TLS handshake protocol is the four handshake processes we discussed earlier, responsible for negotiating encryption algorithms and generating symmetric keys, which are then used to protect application data (i.e., HTTP data).
• The TLS record protocol is responsible for protecting application data and verifying its integrity and origin, so HTTP data encryption is performed using the record protocol.

The TLS record protocol primarily handles the compression, encryption, and authentication of messages (HTTP data), as illustrated in the following diagram:

The specific process is as follows:

• First, the message is divided into multiple shorter segments, and each segment is compressed separately.
• Next, the compressed segments are appended with a message authentication code (MAC value, generated using a hashing algorithm) to ensure integrity and authenticate the data. By adding the MAC value, tampering can be detected. Additionally, to prevent replay attacks, the encoding of the segment is included when calculating the MAC.
• Then, the compressed segments along with the MAC value are encrypted using symmetric encryption.
• Finally, the encrypted data is combined with a header consisting of the data type, version number, and compressed length to form the final message data.

After the record protocol is completed, the final message data is transmitted to the transport control protocol (TCP) layer for transmission.

5. Is HTTPS secure against man-in-the-middle attacks?#

Is HTTPS always secure?
When the client initiates an HTTPS request through the browser, it may be forwarded to a "man-in-the-middle server" by a "fake base station." Thus, the client completes the TLS handshake with the "man-in-the-middle server," which then completes the TLS handshake with the actual server.

Conclusion: During the TLS handshake process, the man-in-the-middle server sends a forged certificate to the browser, which can be recognized as invalid by the browser (client), prompting a warning about the certificate issue.
If the user clicks to accept the man-in-the-middle server's certificate, the man-in-the-middle can decrypt the data in the HTTPS request initiated by the browser and also decrypt the HTTPS response data sent from the server to the browser. Essentially, the man-in-the-middle can "eavesdrop" on the data exchanged between the browser and the server.
Therefore, the HTTPS protocol itself has no vulnerabilities to date; even if a man-in-the-middle attack is successful, it fundamentally exploits a vulnerability in the client (the user clicking to continue or being maliciously imported with a forged root certificate), not a lack of security in HTTPS.

3. Evolution of HTTP/1.1, HTTP/2, and HTTP/3#

1. What performance improvements does HTTP/1.1 have over HTTP/1.0?#

• Connection Method: HTTP/1.0 uses short connections, while HTTP/1.1 supports persistent connections.
• Status Response Codes: HTTP/1.1 introduces a large number of new status codes, with 24 new error response status codes alone. For example, 100 (Continue) — a preliminary request before requesting a large resource, 206 (Partial Content) — an identifier for range requests, 409 (Conflict) — indicating a conflict with the current resource's specifications, and 410 (Gone) — indicating that the resource has been permanently moved with no known forwarding address.
• Caching Mechanism: In HTTP/1.0, caching decisions primarily relied on the If-Modified-Since and Expires headers. HTTP/1.1 introduces more caching control strategies, such as Entity tag, If-Unmodified-Since, If-Match, If-None-Match, and more options for controlling caching strategies.
• Bandwidth: In HTTP/1.0, there were instances of bandwidth wastage, such as when the client only needed part of an object, but the server sent the entire object without supporting partial downloads. HTTP/1.1 introduces the range header, allowing requests for only a portion of a resource, returning a 206 (Partial Content) status code, which facilitates developers' choices to fully utilize bandwidth and connections.
• Host Header Handling: HTTP/1.1 introduces the Host header field, allowing multiple domain names to be hosted on the same IP address, thus supporting virtual hosting. HTTP/1.0 lacks the Host header field, making virtual hosting impossible.

2. What performance improvements does HTTP/2.0 have over HTTP/1.1?#

• IO Multiplexing: HTTP/2.0 allows multiple requests and responses to be transmitted simultaneously over the same connection (considered an upgrade of HTTP/1.1's persistent connection). In contrast, HTTP/1.1 uses a serial approach, requiring independent connections for each request and response. This makes HTTP/2.0 more efficient in handling multiple requests, reducing network latency and improving performance.
• Binary Frames: HTTP/2.0 uses binary frames for data transmission, while HTTP/1.1 uses text-based messages. Binary frames are more compact and efficient, reducing the amount of data transmitted and bandwidth consumption.
• Header Compression: HTTP/1.1 supports body compression but does not support header compression. HTTP/2.0 supports header compression, reducing network overhead.
• Server Push: HTTP/2.0 supports server push, allowing the server to push related resources to the client when a resource is requested, reducing the number of requests and latency for the client. In contrast, HTTP/1.1 requires the client to send requests to retrieve related resources.

3. What optimizations does HTTP/3 implement?#

• Transport Protocol: HTTP/2.0 is implemented based on the TCP protocol, while HTTP/3.0 introduces the QUIC (Quick UDP Internet Connections) protocol for reliable transmission, providing security comparable to TLS/SSL, with lower connection and transmission latency. You can think of QUIC as an upgraded version of UDP, adding many features such as encryption and retransmission. HTTP/3.0 was previously named HTTP-over-QUIC, indicating that the most significant change in HTTP/3 is the use of QUIC.
• Connection Establishment: HTTP/2.0 requires the classic TCP three-way handshake process (typically 3 RTTs). Due to the characteristics of the QUIC protocol, HTTP/3.0 can avoid the delay of the TCP three-way handshake, allowing data to be sent during the first connection (0 RTT, zero round-trip time).
• Head-of-Line Blocking: In HTTP/2.0, multiple requests share a single TCP connection, and if a packet is lost, it blocks all HTTP requests. Due to the characteristics of the QUIC protocol, HTTP/3.0 mitigates head-of-line blocking to some extent, allowing multiple independent data streams to be established within a single connection. If one data stream experiences packet loss, it does not affect the other data streams (essentially a combination of multiplexing and polling).
• Error Recovery: HTTP/3.0 has a better error recovery mechanism, allowing for faster recovery and retransmission in the event of packet loss, delays, or other network issues. In contrast, HTTP/2.0 relies on TCP's error recovery mechanism.
• Security: Both HTTP/2.0 and HTTP/3.0 have high security requirements and support encrypted communication, but they differ in implementation. HTTP/2.0 uses the TLS protocol for encryption, while HTTP/3.0 is based on the QUIC protocol, which includes built-in encryption and authentication mechanisms, providing stronger security.