Sunday, November 29, 2009

How to sync local folders with Amazon S3 using CloudBerry S3 Explorer PRO and PowerShell

Note: this post applies to CloudBerry Explorer 1.6.3 PRO and later.

We continue enhancing PowerShell command line interface for Amazon S3 in CloudBerry Explorer and recently we added a command that complement the new Sync Folder feature. Check out our previous post if you want to learn more about that feature.

In this post we will explain how you can configure automated synchronization using the new PowerShell command. Here is an example

First you have to configure proxy settings

Set-CloudOption -ProxyAddress {PROXY} -ProxyPort 8080

Then you have to connect to your Amazon S3 account with access key and security key

$s3 = Get-CloudS3Connection -Key XXXX -Secret XXXX

Select the path on Amazon S3

$source = $s3 | Select-CloudFolder -Path boooks/sync

Connect to local file system and select local folder

$local = Get-CloudFileSystemConnection

$temp = $local | Select-CloudFolder C:\temp\sync

Synchronize contents from $source to $temp, in other words from Amazon S3 to local folder

$source | Copy-CloudSyncFolders $temp

Or synchronize content in both ways.

$source | Copy-CloudSyncFolders $temp –Bidirectional

You can save the whole script into the files and schedule it using Windows Task Schedule to run synchronization periodically.

If you came to this post by chance you should know that CloudBerry S3 Explorer PRO is a Windows program that helps managing Amazon S3 storage and CloudFront . You can download it at http://cloudberrylab.com/ It is currently in beta and free for all users. You can download it at http://cloudberrylab.com/

Like our products? Please help us spread the word about them. Learn here how to do it.

5 comments:

Anonymous said...

If you want to also include sub folders use this command:

$source | Copy-CloudSyncFolders $temp -IncludeSubFolders

I got stuck on it for a few hours, maybe this can help someone.

Anonymous said...

Is there a way to exclude specific file types in the sync?

andy said...

yes, you can exclude the files of a certain type. you should look for ExcludeFiles command that allows to exclude certain files from sync using the standard wildcards (for example: *.exe; *.dll; d*t.doc; *.t?t). More info is in the PowerShell snap-in documentation

Anonymous said...

The synchronization works fine when saving backup files from EC2 to S3. However, when synchronizing from S3 to a local folder, these backup files (with different date names) start to accumulate in the local folder. If I don't manually delete them, I will run out of disk space. I have the DeleteOnTarget flag set. Could there something I am missing with this?

Prodmetrics IT Services said...

very informative.