Efficient File Handling: Using AWS Presigned URLs to Manage Large Uploads
Introduction
When dealing with uploading files to S3, the typical method involves sending the files to a server, which then uploads them to S3. However, when the files are large, this approach can be time-consuming as it essentially involves uploading the files twice. Additionally, it requires increasing the memory and adjusting request timeouts to manage these large requests effectively. Fortunately, AWS offers a solution that allows files to be uploaded directly to S3 from the front-end application using presigned URLs.
What Are Presigned URLs
Presigned URLs are specially generated links that provide temporary access for uploading files directly to an S3 bucket without requiring user authentication. These URLs are created by the AWS API, allowing a front-end application to upload a file directly by sending a PUT request to the presigned URL. The application must then monitor the response to determine the status of the upload, as the back-end remains unaware of whether the upload has been successfully completed or not. This method allows for secure, direct file uploads while maintaining separation between client and server operations. Presigned URLs typically look like this:
http://aws_url/bucket_name/uploads/file_name.extension?X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=fakeAccessKey%2F20240626%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240626T102233Z&X-Amz-SignedHeaders=host&X-Amz-Expires=300&X-Amz-Signature=943c0f9ffeb1f558f2cfc3a364ced2a4f1c1490e7b10488dc550c765d96ffa12
Let’s walk through the following Vue.js and Laravel implementation of presigned URL uploads, as demonstrated in the following GitHub repo:
Presigned URL
Step 1: Generating a Presigned URL
Once the user selects a file to upload, the front-end requests a presigned URL from the back-end. To accommodate this, the back-end should include an endpoint that receives the file name and type, and then generates the specified URL:
This method generates a URL that specifies the directory in S3 where the file will be stored, as well as the file’s storage name. It also creates a record in the database for the file with its S3 path and a created status. The files table and its corresponding Eloquent model are defined as follows:
Step 2: Uploading to S3
Once the front-end receives the presigned URL, it uploads the file using a PUT request and monitors the response. Regardless of whether the file was uploaded successfully or not, the front-end provides a status update to the back-end:
Step 3: Displaying the File (optional)
Once the back-end receives confirmation that the file has been successfully uploaded, it generates and sends a temporary URL for the file back to the front-end, along with other file-related data, for display:
References