Retrieving objects using AWS SDK faster than fetching via URL over HTTP?

General
1

While implementing S3 object streaming using the AWS SDK with Bun and Hono, I noticed that the performance was noticeably better than fetching assets directly via the S3 URLs.

Comparison of fetch times:

S3 Raw Object URL:
curl -H "Content-Type: application/octet-stream" -w "Total time: %{time_total}s\n" -o /dev/null -s https://s3-url-some-region.com/foo.tgz

Total time: 0.365313s
AWS SDK GetCommand:
curl -H "Content-Type: application/octet-stream" -w "Total time: %{time_total}s\n" -o /dev/null -s https://app.local/foo.tgz

Total time: 0.135964s
AWS SDK Code:
const command = new GetObjectCommand({
  Bucket: 'bucket',
  Key: 'foo.tgz',
});

const obj = await client.send(command);
const readableStream = obj.Body?.transformToWebStream();

return stream(c, async (_stream) => {
  c.header('Content-Type', object.ContentType);

  await _stream.pipe(readableStream);
});

As shown, the time difference is quite noticeable. The AWS SDK method (GetObjectCommand) is significantly faster than directly fetching the object via the raw S3 URL.

Does the AWS S3 SDK utilize internal caching, or does it rely on persistent HTTP connections to minimize TCP handshake overhead?

2

The noticeable difference in performance between fetching an asset directly from S3 via a raw URL and using the AWS SDK (GetObjectCommand) is indeed interesting. To answer your question, the AWS SDK does not inherently rely on internal caching of objects or persistent HTTP connections in the way you might think. However, there are several factors that could explain why the SDK-based approach is faster:

1. Persistent HTTP Connections (HTTP Keep-Alive)

The AWS SDK, like many HTTP clients, does leverage persistent connections (also known as connection reuse or HTTP keep-alive). This means that once a connection is established between your client (e.g., using the SDK) and the S3 service, it stays open and can be reused for subsequent requests without needing to perform a full TCP handshake every time. This reduces the latency involved in establishing new TCP connections, which is a key factor in reducing total response time.

When you use a direct curl request to an S3 URL, a new HTTP request is typically initiated each time, which means a new TCP connection has to be established, including DNS resolution and the initial handshake. This can add noticeable latency to each individual request.

In contrast, the AWS SDK can keep the connection open across multiple requests (if using a long-lived connection pool), reducing the overhead associated with frequent handshakes.

2. Transfer-Encoding / Optimizations via SDK

The AWS SDK might apply certain optimizations when interacting with S3 that are not immediately obvious. For example, when streaming an object, the SDK can use more efficient transfer encodings or chunked transfer encoding mechanisms to send or receive data in a way that minimizes overhead and maximizes throughput.

By using the GetObjectCommand, you’re initiating a streamed download of the object directly to your application, potentially benefiting from internal optimizations that handle data transfers more efficiently than a simple HTTP request via curl. The transformToWebStream() method also ensures that the data is streamed as a continuous stream, which might avoid buffering delays that might occur in a raw HTTP request.

3. Regional Endpoint Optimization and AWS Infrastructure

When using the AWS SDK, the SDK may automatically route requests through the most optimal endpoint based on your region and configuration. This might mean that your request is optimized for lower latency or less congested routes. The SDK can also take advantage of S3’s internal optimizations, which might result in better throughput for fetching data.

In contrast, when you access an S3 URL directly using curl, the request is being made to a public URL, which may not be as optimized as the SDK’s internal mechanisms for routing the request and handling the object download.

4. Server-Side Caching / Transfer Efficiency

While the AWS SDK does not explicitly cache the object on the client side, S3 and AWS services might implement caching at the server side, such as Amazon CloudFront caching, if the request is routed through a CDN. Additionally, S3 might cache recently accessed objects or optimize the transfer based on various parameters like request frequency or object size.

5. Network Configuration

The difference in performance might also stem from the network configuration or underlying infrastructure used by the AWS SDK. The SDK might be running on AWS infrastructure, or it might be using network optimizations that aren’t available in the raw curl request. For instance, if the SDK runs within an AWS region close to the S3 bucket, the network path will be faster compared to a curl request that might go through public internet paths with higher latency.

6. HTTP Headers and Request Optimization

When using the AWS SDK, you might also benefit from certain request headers and optimizations that aren’t included in a typical curl request. For example, the SDK might automatically set headers such as x-amz-processed-time or x-amz-server-side-encryption (if applicable) that could improve performance by negotiating the most efficient transfer parameters with S3.

Summary of Key Factors:

  • Persistent connections (keep-alive): The SDK can reuse HTTP connections, reducing the TCP handshake overhead.
  • Optimized network routing: The SDK may route requests via more optimized paths or endpoints within AWS infrastructure.
  • Efficient streaming: Using streams and transfer encodings might make data transfer more efficient than a raw HTTP download.
  • Internal SDK optimizations: The SDK may have additional optimizations, such as more efficient handling of large object transfers or leveraging S3/CDN caching mechanisms.

Conclusion:

While the AWS SDK does not use caching in the traditional sense (i.e., it doesn’t store objects locally or reuse previously fetched objects from an internal cache), it does leverage persistent connections and several other optimizations, such as efficient streaming and better network routing, which can significantly improve performance over direct access via S3 URLs. This is likely why you’re seeing a noticeable improvement in fetch times when using the SDK, especially when streaming large objects like the foo.tgz file.

If you’re looking to further optimize your setup, you can explore AWS S3 Transfer Acceleration, which is a service that uses Amazon CloudFront’s edge locations to speed up uploads and downloads to/from S3.