How to sync local folders with Amazon S3 using CloudBerry S3 Explorer PRO and PowerShell

Note: this post applies to CloudBerry Explorer 1.6.3 PRO and later.

We continue enhancing PowerShell command line interface for Amazon S3 in CloudBerry Explorer and recently we added a command that complement the new Sync Folder feature. Check out our previous post if you want to learn more about that feature.

In this post we will explain how you can configure automated synchronization using the new PowerShell command. Here is an example

First you have to configure proxy settings

Set-CloudOption -ProxyAddress {PROXY} -ProxyPort 8080

Then you have to connect to your Amazon S3 account with access key and security key

$s3 = Get-CloudS3Connection -Key XXXX -Secret XXXX

Select the path on Amazon S3

$source = $s3 | Select-CloudFolder -Path boooks/sync

Connect to local file system and select local folder

$local = Get-CloudFileSystemConnection

$temp = $local | Select-CloudFolder C:\temp\sync

Synchronize contents from $source to $temp, in other words from Amazon S3 to local folder

$source | Copy-CloudSyncFolders $temp

Or synchronize content in both ways.

$source | Copy-CloudSyncFolders $temp –Bidirectional

You can save the whole script into the files and schedule it using Windows Task Schedule to run synchronization periodically.

If you came to this post by chance you should know that CloudBerry S3 Explorer PRO is a Windows program that helps managing Amazon S3 storage and CloudFront . You can download it at http://cloudberrylab.com/ It is currently in beta and free for all users. You can download it at http://cloudberrylab.com/

Like our products? Please help us spread the word about them. Learn here how to do it.

How to install CloudBerry Backup for Windows Home Server

In this post we will explain how to install or re-install CloudBerry Backup add-on for Windows Home Server.

Originally it is from here http://usingwindowshomeserver.com/2009/04/19/how-to-install-or-remove-an-add-in-for-windows-home-server/

1. Open WHS server shared folder: \\YOUR WHS\Shared Folders\Software\Add-Ins
2. Copy the add-in (it is likely to be called WHS_CloudBerryOnlineBackupSetup_v1.3.1.371o.msi) to the Add-Ins folder.
3. Launch the Windows Home Server Console.
4. Click the Settings button on the Windows Home Server Console to open the Settings screen.
5. Click Add-ins in the left pane.
6. Click the Available tab. You should now see CloudBerry Online Backup add-in listed.

Please note you should uninstall previous versions of CloudBerry Online Backup if you ever had it before. After uninstalling the previous version you need to restart your WHS Console.

___

CloudBerry S3 Backup for Windows Home Server is WHS add-on that automates data backup and restore to online storage powered by Amazon S3 cloud storage. It comes with onetime fee of $29.99 (US) and you can download fully functional 15 day evaluation version at http://www.cloudberrylab.com/default.aspx?page=backup-whs

Like our products? Please help us spread the word about them. Learn here how to do it.

Search across your S3 buckets with CloudBerry Explorer PRO

Note: this post applies to CloudBerry Explorer 1.7PRO and later.

In CloudBerry Lab we are always thinking how to bring more value to our users. Since we use Amazon S3 heavily ourselves it is easy sometimes to identify things that will make CloudBerry Explorer a better tool and the ability to Search for a specific data is one of them.

How Search works

Nothing can be simpler: just click Ctrl+F on your keyboard or the lens icon next to the path bar.

Then type a text into the search text box and click enter. You will get the results that match your pattern

You can also use wildcard characters such as (*)

Or slightly more complex

Note: we don’t search across the contents of your files. We can only search across key names (file names). Please also note that Search is only available in our PRO version

___

If you came to this post by chance you should know that CloudBerry S3 Explorer is a Windows freeware product that helps managing Amazon S3 storage and CloudFront . You can download it at http://cloudberrylab.com/

If you came to this post by chance you should know that CloudBerry S3 Explorer PRO is a Windows program that helps managing Amazon S3 storage and CloudFront . You can download it at http://cloudberrylab.com/ It is currently in beta and free for all users. You can download it at http://cloudberrylab.com/

Like our products? Please help us spread the word about them. Learn here how to do it.

A few things to consider when doing software project outsourcing.

We recently completed an outsourcing project for a customer that went relatively well. I just want to share some thoughts on outsourcing and when it works well. I don't think it is new for most of the people, but it good to remind yourself every once in a while. Anyways, here you go.

1. don't do outsourcing if you under a pressure to make the date. Most likely you will not make the date

2. understand that outsourcing team is not part of yours. they have different work culture.

3. understand that their deliverables might not be exactly what you expect and plan enough time to put it in order

4. employ outsourcing when you need something to be done but you have relatively enough time

5. employ outsourcing if you want to put your key (internal) resource on the key area of your product while slowly moving towards strategic goal with the help of outside team.

6. make outsourced task as self contained as possible. the more integration requirement and inter-team communications you have the better chance for failure.

I have been in this area for ~7 year and we use outsourcing in CloudBerry Lab too. We employ some freelancers and their initial work is ... complete crap. But we realize at the beginning what we will get and we're slowly improving. At the end we are all happy.

Hope some of our readers will find it useful

How to generate CloudFront Private URLs using Canned Policy

Note: this post applies to CloudBerry Explorer 1.6.5 and later.

This is a second article related to CloudFront Private Content feature that allows you restrict the access to your content. Apparently CloudFront private content configuration is not straight forward and involves many steps as described in our previous post. You have to configure your distribution to support private content, generate keys, create policies and finally sign URLs. CloudFront users wanted something simple similar to Amazon S3 Query String authentication.

Coming Canned Policies

CloudFront team have been quick to react on user feedback and introduced so called Canned Policies. Canned Policies unlike Custom Policies are generated automatically, depending on the resource you want to generate the signed URL for and the expiration time. In this case the expiration time is passed as a query string parameter in the URL:

http://mycloudront.com/folder/file1.jpg?Expire=1258247342

Creating Canned Policies with CloudBerry Explorer

First, you have to create the canned policy. Got to Tools | Policies in the program menu to open Add New Policy dialog. Choose Canned Policy as shown on the screen. See how IP Range and Resource Mask fields become disabled.

Specify Private Key file and Key pair ID. Click ok to create the canned policy.

Note: Amazon CloudFront checks the signature with a public key that is stored in Amazon (it can be uploaded if you use your own private key, or created by Amazon if you use Amazon’s key generator). For Amazon to know with which key it should check the signature, the Key Pair Id is passed in the URL as a parameter.

Generating URLs using a Canned Policy

There is nothing new to Generate Web URL dialog. It is just that you have to choose the canned policy in the list.

Note: The policy is placed in WebURL as a query parameter (url-safe Base64-encoded). Only accounts set up as a Trusted Signer for a distribution can sign the Policy. Otherwise the signed URL will not be valid.

What’s next

We are going to make it even easier to generate protected URLs using Canned Policies in the future release. You won’t have to create a canned policy separately and you will be able to generate URL right on the Web URL screen. Stay tuned!

___

If you came to this post by chance you should know that CloudBerry S3 Explorer is a Windows freeware product that helps managing Amazon S3 storage and CloudFront . You can download it at http://cloudberrylab.com/

If you came to this post by chance you should know that CloudBerry S3 Explorer PRO is a Windows program that helps managing Amazon S3 storage and CloudFront . You can download it at http://cloudberrylab.com/ It is currently in beta and free for all users. You can download it at http://cloudberrylab.com/

Like our products? Please help us spread the word about them. Learn here how to do it.

What’s new in CloudBerry S3 Explorer Sync Folders

Note: this post applies to CloudBerry Explorer 1.6.5 and later.

A few weeks ago we released a preview of Sync Folders feature that allows automatically synchronize local folders with Amazon S3 buckets. Over a few weeks we got we gathered some feedback from our user community and we are happy to announce some improvements.

Sync folders based on names only

This option works exactly like its name implies. In other words it will only copy files if it finds the files with new names. This option works faster as we don’t have to identify date modified for each files.

Sync folder apply ACL automatically

In the previous release you had to manually set ACLs to the files copied to the bucket using Sync Folder features. In the latest release the bucket ACL can be applied to new files automatically. To make it work you have to set Permission inheritance option in Tools | Options | Copy/Move tab:

A bit of usability…

And last but not least some of our users complained that they were confused once they created a sync folder. What should I do next? Does it run automatically or I should do something to run it? We added a message explaining what you should do next to run sync.

___

If you came to this post by chance you should know that CloudBerry S3 Explorer is a Windows freeware product that helps managing Amazon S3 storage and CloudFront . You can download it at http://cloudberrylab.com/

If you came to this post by chance you should know that CloudBerry S3 Explorer PRO is a Windows program that helps managing Amazon S3 storage and CloudFront . You can download it at http://cloudberrylab.com/ It is currently in beta and free for all users. You can download it at http://cloudberrylab.com/

Like our products? Please help us spread the word about them. Learn here how to do it.

Managing CloudFront Private Content with CloudBerry S3 Explorer

Note: this post applies to CloudBerry Explorer 1.7 and later.

CloudBerry S3 Explorer is by far the most popular Amazon S3 and CloudFront manager on Windows platform. At the same time Amazon rapidly enhance their services to meet growing customer expectations. To maintain our leadership position we are trying to say on top of Amazon developments and support all recent enhancements in CloudBerry Explorer.

Today CloudFront team introduced a new exciting capability that will help to protect your content on the distributions. You might want to use CloudFront to deliver a digital asset you’ve sold online, or use it to distribute objects only to your company’s employees. In these circumstances, you need detailed control over who can download your “private” content. As a result today Amazon added an ability to handle private content in Amazon CloudFront.

How CloudBerry Explorer supports CloudFront Private Content?

First you have to configure a distribution to serve private content. You can configure and existing distribution or a new distribution, but a distribution should be created before you configure private content. Check out here to learn how to create a CloudFront distribution.

To configure a distribution for private content click on the distribution properties on the toolbar on in the content menu and go to the Private Content tab.

Click the checkbox to enable Private Content. Now there are a few things that might confuse you at first.

Origin Access Identity: this is a "virtual" user that is used for managing permissions. Since you are able to restrict access you need to set permissions for objects. But for whom? Amazon provides some "virtual" users for this purpose. The Origin Identity that you see CloudBerry Explorer generates for you automatically as you should have at list one. You can also generate them using CloudFront API.

Identity S3 canonical user ID. This is the user ID that you can use in ACL editor to grant permissions. It is associated with the Origin Access Identity.

Trusted Signers:
You can only obtain the access to objects with signed URLs.
Trusted signers are Amazon S3 accounts numbers of users who can generate singed URLs. If one of them will generate signed URL with his private key, it will be valid. If someone else tries to generate signed URL - it will be invalid.

To find your account number go to Your Account | Account Activity on AWS website. Look for Account Number as shown on the screen:

Add myself to trusted signers: by default you are not added to Trusted Signers. So if you want to generate signed URLs you have to check it.

Granting permissions to origin access identity

Notice that ACL dialog has been extended to include Origin Access Identity account automatically for CloudFront distributions configured for Private Content, You have to grant Read access to CloudFront Origin Access Identity account to make the files available for Private Distribution.

Update: There is now a more advanced way to grant Origin Access Identity access to the files in the bucket. You can configure the Bucket Policy to always grant Read access to Origin Access Identity. Read our recent blog post for details.

Security Policies

Before we go any further we have to learn what Security Policy is. Click Tools | Policies in the program menu to open policies window and click New Policy. It helps you to create the policy which will basically define an IP range and address range to limit access to your private content.

Specify a path to your Private Key file in the private key field. This key will be used to sign the Policy.

Key Pair ID is something you will get from Amazon when you generate a Key Pair using Amazon web interface our upload your own key.

The policy on the screen below will grant access to every IP address 0.0.0.0/0 and to every file in your bucket http://*

Note: that the policies are stored locally on your computer. So if you run CloudBerry Explorer on another computer or somebody else manage your S3 account the policies are not shared.

What is Private Key and how I can get it?

The private key - is the file in PEM format. The key pair (public (for Amazon to check your signature) + private (to be stored securely on your computer)) can be generated by Amazon on your account page.

If you already have a key pair, you can upload PUBLIC key to Amazon S3 (using your account web interface), and use your PRIVATE key to sign the URLs.

Using a Key Pair Generated by AWS

If you don't already have key pair, you can have AWS generate a pair and automatically associate the public key with your AWS account.

To have AWS create a key pair for you

1. Go to the Amazon Web Services web site at http://aws.amazon.com .

2. Point to Your Account and click Security Credentials.

3. Log in to your AWS account.

The Security Credentials page is displayed.

4. In the Access Credentials section of the page, click the Key Pairs tab.

5. Click Create a New Key Pair.

Your new public and private key are generated, along with an ID for the key pair. Amazon keeps the public key and gives you the private key.

6. From the dialog box, download your private key file to a local directory, and record the corresponding key ID.

Here's how it looks like

Important: If you're an Amazon EC2 user, you probably already have at least one other key pair, which you use to connect to your instances through SSH or Windows Remote Desktop. Your Amazon EC2 key pairs work only with Amazon EC2, and your Amazon CloudFront keys work only with CloudFront. You can't use the private key from one service with the other service.

To upload your own public key

1. Go to the Amazon Web Services web site at http://aws.amazon.com.

2. Point to Your Account and click Security Credentials.

3. Log in to your AWS account. The Security Credentials page is displayed.

4. In the Access Credentials section of the page, click the Key Pairs tab.

5. Click Upload Your Own Public Key.

6. Follow the instructions presented to upload your public key.

Note: CloudBerry Explorer also supports PKCS#8 non-encrypted keys that starts with

-----BEGIN PRIVATE KEY-----

How to generate Private Content URL?

Generate Web Url dialog has been extended. The new option only shows up when creating a URL for objects residing on CloudFront Distributions configured for Private Content.

Conclusion

We hope CloudFront Private Content support in CloudBerry Explorer will make it easier to explore new CloudFront functionality and make CloudBerry Explorer your tool of choice for managing S3 and CloudFront.

If you came to this post by chance you should know that CloudBerry S3 Explorer is a Windows freeware product that helps managing Amazon S3 storage and CloudFront . You can download it at http://cloudberrylab.com/

If you came to this post by chance you should know that CloudBerry S3 Explorer PRO is a Windows program that helps managing Amazon S3 storage and CloudFront . You can download it at http://cloudberrylab.com/ It is currently in beta and free for all users. You can download it at http://cloudberrylab.com/

Like our products? Please help us spread the word about them. Learn here how to do it.

How to setup CloudBerry S3 Backup notification

Note: this post applies to CloudBerry Backup and CloudBerry Backup for WHS 1.3.1 and later.

CloudBerry S3 Backup is a powerful Windows backup and restore that automate backup and restore processes to Amazon S3 cloud storage.

As always we are adding features that our customers are requesting. One of the things our customers have been asking us ever since the first beta release is an option to get an email when backup completes successfully or failed. Today we are proud to announce it.

How to setup email notification?

Nothing could be easier. We added another step to the end of the Backup Plan wizard where you can configure a notification. You can choose between getting emails after every run or only when backup plan fails. If it fails you can also optionally send a diagnostic log to CloudBerry Lab support so that we can look into your problem immediately.

Confirming email address

When you enter email address you will have to confirm it.

We will send you a confirmation email and you will have to click the link to confirm it. If you don’t confirm your email address or enter a fake email address you will never receive the notification as we don’t want to send unsolicited emails.

Notification

When everything is set you will be getting an email according to your settings in case of successful or failed backup. The email will looks like this:

Hope you will like our new feature and as always we are welcome your feedback.

If you came to this post by chance you should know that CloudBerry Backup is a Windows program that leverages Amazon S3 storage.You can download the trial version at http://cloudberrydrive.com/

Your account is not signed-up for Amazon S3

We continue our series of articles devoted to common Amazon S3 issues. In our previous article we discussed the difference between the requested time and the current time is too large error.

In this article we will discuss how to handle Your account is not signed up for Amazon S3 issue. This error occurs when you are surprisingly enough… not signed up for Amazon S3. Apparently, there is a two steps process to get an Amazon S3 account. First you have to create an Amazon account

Second, you have to sign up to each service separately. Go to http://aws.amazon.com/s3 and click large Sign Up for Amazon S3 button.

We described the sign-up process in details at http://blog.cloudberrylab.com/2009/02/how-to-sign-up-for-amazon-s3-service.html

How to verify if my account is signed up for a service?

The best way to verify if you are in fact signed-up for the service is to go to Your Account. This is the top right link on the AWS website. Choose Account Activity in pop-up menu.

Then you should see the list of the services you are signed up for.

If you don’t see Amazon Simple Storage Service (S3) in the list you are not signed up for it.

+++++

If you came to this post by chance you should know that CloudBerry S3 Explorer is a Windows freeware product that helps managing Amazon S3 storage and CloudFront . You can download it at http://cloudberrylab.com/ (yes, we do this kind of post to promote our product! ;-) )

Like our products? Please help us spread the word about them. Learn here how to do it.

How to send diagnostic info with CloudBerry Explorer

Note: this post applies to CloudBerry Explorer 1.6.3 and later.

In CloudBerry Lab we are striving to make our customer service better. In this competitive market with the abundance of free offerings this is the only way to stay afloat. One of the ways to keep customers happy is to be very responsive when it comes to support request resolution. Should you delay the request and the customer will turn to a competitor.

Sometimes it takes a lot of time to understand what the issue is and what caused it. This is very diagnostic log helps dramatically. In the new version of CloudBerry Explorer we made it even easier to send us the diagnostic information.

All you have to do is to go to Tools| Diagnostics and you will get this window

Now click Send button and you will get the following screen. Here you can explain what the issue to help us a little more. Finally you have an option to post the message to our discussion forum at http://www.cloudberrylab.com/forum/default.aspx so that other CloudBerry Lab users benefit from the resolution too.

Now click Send button and the log files along with your message will be automatically transferred to our web service.

----------

If you came to this post by chance you should know that CloudBerry S3 Explorer is a Windows freeware product that helps managing Amazon S3 storage and CloudFront . You can download it at http://cloudberrylab.com/

If you came to this post by chance you should know that CloudBerry S3 Explorer PRO is a Windows program that helps managing Amazon S3 storage and CloudFront . You can download it at http://cloudberrylab.com/ It is currently in beta and free for all users. You can download it at http://cloudberrylab.com/

Like our products? Please help us spread the word about them. Learn here how to do it.

CloudBerry Backup - Purging options explained

Note: this post applies to CloudBerry Backup 1.3 and later.

CloudBerry S3 Backup is a powerful Windows backup and restore that automate backup and restore processes to Amazon S3 cloud storage. In this post we will try to explain you how to configure purging options and why you need them.

CloudBerry Backup creates a version of the file on backup storage each time the file is modified. Using purging options you basically configure how many versions to keep. This helps you keep the number of versions to the necessary minimum and cut down on storage requirements.

Versioning

If you want to see the versions of a particular file simply go to the Backup Storage tab. Make sure that you click All Versions button on the toolbar. This will show the panel at the bottom and you will be able to restore a specific version of the file from there.

Purging options

To configure default Purging Options go the main menu Tools | Options and click on the Purging tab. The options are

1. Purge versions older than. It will delete all versions from the backup storage older than specified time interval that you configure

2. Keep number of version. This is absolute number of the most recent file version to keep.

CloudBerry Backup will always keep one version (the last version) of the file regardless of how purging options are configured. Furthermore, even if you delete a file on your computer it will still stay on your backup storage.
We will add an option that will delete file on online storage when you delete it locally in the future release.

Note: you can overwrite purging options when configuring Backup Plans

History purging options

The other two options control the History which is used mostly for troubleshooting and doesn’t add-up to storage requirements. You can look at the history records at the backup tab as shown on the screen below. In most of the cases you should leave default values. Again you can delete the records older then a give time interval or you can select how many records to keep.

Note, the history is stored on your local computer, not on the online storage and it doesn’t consume much disk space. Anyways, don’t worry too much about the History.

If you came to this post by chance you should know that CloudBerry Backup is a Windows program that leverages Amazon S3 storage. you can learn more about it at http://cloudberrydrive.com/