How To Sync Local Files And Folders To AWS S3 With The AWS CLI

So you have your AWS S3 bucket ready, and you have some local files and folders you want to sync up to the cloud. One method to accomplish this task would be to drag and drop the files and folders into the bucket manually, via the AWS Management Console. Since we prefer to use command line utilities (and eventually, automation) to accomplish our tasks, we should get comfortable with AWS CLI.

In this snip, Josh will walk you through using AWS CLI to upload individual files and folders to your S3 buckets, you how to delete any files that exist in the destination but not in the source, and how to set basic permissions during the upload.

Prerequisites include: AWS CLI Installed AWS Account AWS S3 Bucket Created

When you want to start automating the synchronization of local files up to an Amazon S3 bucket. You need to start getting familiar with the AWS Command Line Interface. So to start. I'm going to list. All of my available buckets as you can see. I've got 2. Already one of them was automatically created by Elastic Beanstalk. But the other one techsnips dash sync is the one I'll be using for this demonstration also quickly check the contents of that bucket. This is more just to show that I don't currently have anything in there before we get on to the rest of this demonstration. So let's go ahead and copy a single file up there. Note that when I'm doing a single file. I need to use the copy or CP command rather than sync and my destination needs to have a trailing slash on the end. Otherwise, the CLI tool might think that I wanted a file at the destination without a extension. This isn't too much of an issue when I'm saving to the root of my bucket. But if you were saving into a sub folder, then it would become more of an issue so. With my very important memo uploaded let's move on to syncing. So I've got a directory of backups here and I'm going to synchronize them with a sub folder on my bucket called backups. Note that rather than CP of this time specified sync. Alright so that commands finished now that was probably spit up in the video, but that took only about 20 to 30 seconds. And I wanted to illustrate just how fast that was pushing through data up to AWS so that directory had 1120 files totaling about 138 megabytes. So the synchronization is actually really quick, but let's start giving on to something a little bit more complex. So I've gotta scripts directory where I store a number of PS one files, but I've accidentally stored a couple of trec files. From these techsnips recordings and I don't really want those synchronized to AWS so ignoring the fact that this is a long command on multiple lines. Notice that I'm using the S 3 sync command again. My sources that scripts directory and my target is my techsnip sync bucket. And I've specified the subdirectory of scripts. I've also included exclude and include options, so by default everything in the directory is included so there's no point having an include flag without also having an exclude flag So what I've done here is of excluded everything specified by a wild card and I've included just PS one files. This means that nothing from this folder will be synchronized unless it has a PS one. Extension I'm also using the grants option here and what I'm doing is allowing all users IE anyone on the Internet read access to the files that I'm uploading here, so if you go ahead and run. This my PS. One files are uploaded and we can check that by listing the contents of my scripts directory. Open my S3 bucket and note that the only files contained within that PS one files so a synchronization means any new files are going to be synchronized up there. But what about if we delete a file so I'll delete one of those PS one scripts from my scripts directory. And then I'll invoke that expression again, but note that nothing's changed this because by default. The command line tools aren't going to delete anything in my bucket unless I explicitly tell it that's what I want to do so. I'm re running the same command as before, except on the last line. I'm using the delete option. So when I run this command. Now I see that one files being deleted being the PS one file that I deleted from my local directory. So let's go ahead and copy some other scripts into my scripts directory. And we'll re run that long command again. And you can see that 4 new files have been uploaded to their directory or my S3 bucket. So, just to wrap things up. Let's have a look at the listing for all of my techsnip sync bucket again. We can see my very important memo sitting there in the root and we can see my 2 subfolders backups and scripts and if we have a look in scripts again. We can see all of my PS. One files, including the 4 that were added after the fact so that's an introduction to synchronizing local files with an Amazon S3 bucket. Thanks for watching.