Uploading Files Directly to AWS S3 in Laravel 5.3
In a recent project we use AWS S3 to store files. Originally I was uploading the files like normal then using Laravel’s built in File Storage to move the files to S3. However this seemed like a waste to upload files to my server that were going to get moved to another server. So I figured I would try uploading files directly to S3. This post describes mainly doing this for images, however, it can be manipulated pretty easily to handle any type of file.
Research
First step was I did some research and found some documentation in AWS’s documentation that explained how to upload files from and html form. http://docs.aws.amazon.com/AmazonS3/latest/dev/HTTPPOSTExamples.html
Setup
Next I setup a few environment variables in my .env file for s3. The S3_KEY_ID and the S3_SECRET_KEY are generated in AWS with IAM. Make sure you set your bucket name and region correctly also. Here is what those look like in my .env file:
S3_KEY_ID=**************************
S3_SECRET_KEY=***********************************
S3_BUCKET=bucketname
S3_REGION=us-east-1
Server Side Code
I was creating a blog post that allowed for multiple images per blog post so this is what my controller actions looked like:
public function create(Request $request)
{
list($policy, $signature) = $this->getPolicyAndSignature($request);
return view('blogs.form', [
'blog' => new Blog(),
'route' => 'blogs',
'policy' => $policy,
'signature' => $signature,
]);
}
protected function getPolicyAndSignature(Request $request)
{
return Media::getAwsPolicyAndSignature(
getenv('S3_BUCKET'),
'blogs/',
'public-read',
'image/'
);
}
You will see a reference to a Media model in the getPolicyAndSignature() function. The model looks something like this:
class Media extends Model
{
protected $table = 'medias';
protected $fillable = ['user_id','info'];
protected $dates = ['created_at','updated_at','deleted_at'];
protected $casts = ['info' => 'array'];
public function user()
{
return $this->belongsTo(User::class);
}
public static function getAwsPolicyAndSignature($bucket,$startsWith,$acl,$contentType)
{
$policy = base64_encode(json_encode([
"expiration" => "2100-01-01T00:00:00Z",
"conditions" => [
["bucket"=> $bucket],
["starts-with", '$key', $startsWith],
["acl" => $acl],
["starts-with", '$Content-Type', $contentType],
["success_action_status" => '201'],
]
]));
$signature = base64_encode(hash_hmac('sha1',$policy,getenv('S3_SECRET_KEY'),true));
return [$policy, $signature];
}
public static function makeS3Image($requestData,$user_id)
{
$data = $requestData['awsData'];
$awsData = new \SimpleXMLElement($data);
$awsData = (array)$awsData;
$jsData = $requestData['fileData'];
$media = Media::create([
'info' => [
'driver' => 'AwsS3Image',
'aws'=> $awsData,
'js' => $jsData,
],
'user_id' => $user_id,
]);
return $media;
}
}
Here is a migration similar to what I used for the medias table:
Schema::create('medias', function (Blueprint $table) {
$table->increments('id');
$table->integer('user_id')->unsigned();
$table->text('info')->nullable();
$table->timestamps();
$table->softDeletes();
$table->foreign('user_id')->references('id')->on('users');
});
I also created a pivot table to connect the Media to the Blog posts:
Schema::create('blog_medias', function (Blueprint $table) {
$table->integer('blog_id')->unsigned();
$table->integer('media_id')->unsigned();
$table->timestamps();
$table->foreign('blog_id')->references('id')->on('blogs');
$table->foreign('media_id')->references('id')->on('medias');
$table->unique(['blog_id','media_id']);
});
Frontend HTML
Now on the frontend here is what my blade view looked like:
<form action="/blogs" method="POST">
<input type="hidden" id="hidden_medias" name="medias" value="[]" />
<div class="form-group">
<label for="title" class="col-md-4 control-label">Title</label>
<div class="col-md-6">
<input class="form-control" maxlength="255" name="title" type="text" id="title">
</div>
</div>
<div class="form-group">
<label for="title" class="col-md-4 control-label">Content</label>
<div class="col-md-6">
<textarea class="form-control" name="title" id="body"></textarea>
</div>
</div>
<div id="image_container" class="form-group">
<label class="col-md-4 control-label">New Image Upload</label>
<div class="col-md-6">
<input type="file" style="width: 60%; display: inline-block;" class="form-control" id="image_uploader" name="file" accept="image/*" />
<div id="upload_now" class="btn btn-info btn-sm">
<i class="fa fa-upload"></i> Upload Now
</div>
</div>
</div>
<div class="form-group">
<label class="col-md-4 control-label">Images Uploaded</label>
<div id="preview_images" class="col-md-6"></div>
</div>
<div class="form-group">
<div class="col-md-6 col-md-offset-4">
<button type="submit" class="btn btn-primary">
<i class="fa fa-btn fa-save"></i> Save
</button>
</div>
</div>
</form>
Ok so we have inputs for a title, some body content, an image uploader and an image previewer.
Frontend Javascript
Now for the javascript, I obviously am using bootstrap and jQuery, but you could rewrite this to not require it. Also I used npm to install an md5 package ( https://www.npmjs.com/package/md5 ). Here is what the javascript looks like:
var uploading = false;
var doneUploading = false;
var previewImages = function() {
var generatedHtml = "";
var currentImages = JSON.parse($('#hidden_medias').val());
for(var x=0;x<currentImages.length;x++) {
var currentImage = currentImages[x];
var xmlDoc = jQuery.parseXML(currentImage.awsData);
var image = xmlDoc.getElementsByTagName('Location')[0].innerHTML;
generatedHtml += "<img class='upload_image_preview' src='" + image + "'>";
}
$('#preview_images').html(generatedHtml);
};
var uploadImage = function(callback) {
var file = $('#image_uploader')[0].files[0];
if(file !== undefined && uploading === false && doneUploading === false) {
uploading = true;
var data = new FormData();
var filename = md5(file.name + Math.floor(Date.now() / 1000));
var filenamePieces = file.name.split('.');
var extension = filenamePieces[filenamePieces.length - 1];
data.append('acl',"public-read");
data.append('policy',"{!! $policy !!}");
data.append('signature',"{!! $signature !!}");
data.append('Content-type',"image/");
data.append('success_action_status',"201");
data.append('AWSAccessKeyId',"{!! getenv('S3_KEY_ID') !!}");
data.append('key',"/blogs/" + filename + '.' + extension);
data.append('file', file);
var fileData = {
type: file.type,
name: file.name,
size: file.size
};
$.ajax({
url: 'https://[[bucket_name]].s3.amazonaws.com/',
type: 'POST',
data: data,
processData: false,
contentType: false,
success: function (awsData) {
var xmlData = new XMLSerializer().serializeToString(awsData);
var currentImages = JSON.parse($('#hidden_medias').val());
currentImages.push({
awsData: xmlData,
fileData: fileData
});
$('#hidden_medias').val(JSON.stringify(currentImages));
callback();
},
error: function (errorData) {
console.log(errorData);
}
});
}
};
$(document).ready(function(){
var form = $('#image_container').parents('form')[0];
var $form = $(form);
$('#upload_now').click(function() {
var self = $(this);
self.attr('disabled','disabled');
self.html('<i class="fa fa-btn fa-spinner fa-spin"></i> Uploading');
uploadImage(function() {
self.removeAttr('disabled');
self.html('<i class="fa fa-upload"></i> Upload Now');
$('#image_uploader').val('');
doneUploading = true;
uploading = false;
previewImages();
});
});
$form.submit(function(e){
var submit = $form.find('button[type="submit"]')[0];
var $submit = $(submit);
$submit.attr('disabled','disabled');
$submit.html('<i class="fa fa-btn fa-spinner fa-spin"></i> Saving');
uploadImage(function(){
doneUploading = true;
uploading = false;
previewImages();
$form.submit();
});
if(uploading === true) {
e.preventDefault();
}
});
});
Ok now for some explanation of all that javascript. First, we need to keep track of when stuff is uploading and when uploads are done. Also we need to catch the form submission as we need to upload files to S3 before we can let the form submit to our server.
We can’t send a post right away to our server when an image is done uploading as the blog post hasn’t been created so we can’t add an entry to a pivot table. So instead the hidden input field hidden_medias stores a JSON array of the upload file objects that gets submitted when the form is submitted.
previewImages() is a function that converts that JSON array of uploaded file objects into actual html <img> tags so we can see the previews of the images already uploaded. If you were uploading files that weren’t images obviously you would want to do something else to show what files have been uploaded.
uploadImage() is our main function that actually uploads the file to AWS, then stores an object in our temporary JSON array. Before an image gets uploaded I generate my own filename for the file. I don’t want two people to upload a file named “profile.jpg” and the first image gets overwritten. Also I don’t want people to be able to randomly guess file names and see images they might not have permission to view on S3. So I take the filename, add a timestamp to it then md5() that to generate a filename, then append the original extension onto the filename. I also create a object var fileData to store some of the file’s original info that will be stored in our database on our Media model.
I then generate a FormData javascript object and add all the required fields that need to be sent to AWS. These fields need to match what we put in our policy when we generated it. Also the order is important to AWS, I had issues with this. You will see I have {!! getenv('S3_KEY_ID') !!}, {!! $policy !!} and {!! $signature !!} these are variables that need to be passed from the controller to the javascript in some way.
Finally, we are ready to send our ajax request to S3. Make sure you set your bucket name correctly in the url. If the upload is successful we will receive some XML data back from S3. This XML data gets combined with fileData and added to out temporary JSON array.
The last part of the javascript is simply the jQuery listening for clicks on the Upload Now button or the submit button. If the Upload Now button is clicked it uploads the file currently selected right away, updates the previews and then clears the file input so another file can be uploaded. If the form is submitted, it checks to see if there is a file selected but not uploaded, if so uploads it first, then finally submits the form to our server.
Server Side Code – Handle Post
The frontend is done, now how does the controller save this info? Here is what the store() function in the BlogController looks like (fyi isInvalid() comes from a package I use for validation):
public function store(Request $request)
{
$blog = Blog::create($request->only(['title', 'body']));
$blog->user_id = $request->user()->id;
if ($blog->isInvalid()) {
return back()->withErrors($blog->getErrors())->withInput();
}
$blog->save();
$blog->attachMedia(json_decode($request->input('medias'), true), $request->user()->id);
return redirect()->route('blogs.index')->with('success', 'Blog Created!');
}
This is taking the data coming in from the response creating an instance of Blog and saving it. If it is valid and saves then we attach the media to the blog. Here is what the attachMedia() function and the relationship for media looks like in my Blog model:
public function medias()
{
return $this->belongsToMany(Media::class,'blog_medias')->withTimestamps();
}
public function attachMedia($medias,$user_id)
{
foreach($medias as $media) {
$media = Media::makeS3Image($media,$user_id);
$this->medias()->attach($media->id);
}
}
The Media::makeS3Image() is back up in our Media model. This generates a model instance and saves it in the db, then we just need to attach it to our blog instance.
Accessing Media / Displaying Images
Final step, accessing the files. If you are uploading documents or something other than images obviously you could do something similar to this. We want to actually display the images in HTML output. My first thought was doing something like this:
<img src="{!! $media->info['aws']['Location'] !!}" />
This works fine, however, we wanted some protection, we only wanted authenticated users to be able to see the images. So instead here is what I decided to do:
<img src="/medias/{{ $media->id}}" />
This allows us to run this route through our auth middleware, and any other middleware/permission checking we want. If the user has permission then we send a valid response. The media controller action that actually handles this:
public function view(Media $media)
{
return (new Response('',301,['Location' => $media->info['aws']['Location']]));
}
So what this does is when the request comes in, it gives a 301 redirect response, and sets a header location to point at the s3 file. The browser handles this without a problem. The only way the user could possibly ever find the actual s3 file url would be to open up developer tools, look at the network tab, and look at the response headers.
Conclusion
As you can tell directly uploading files to s3 isn’t necessarily an easy task. In this case it was quite complex because I wanted the ability to upload multiple images, upload images before the actual blog post was saved in the db, and add some ability to somewhat limit visibility of the images to authenticated and non-authenticated users.