CloudBerry S3 Explorer and GoodData on demand CloudFront log analysis

Note: this post applies to CloudBerry Explorer 1.5.2 and later.

CloudBerry Explorer is by far one of the most popular tool for managing Amazon S3 and CloudFront accounts. One of the aspect that the tool is designed to manage is CloudFront access logs. Check out our previous post on how to enable CloudFront logging. CloudBerry Explorer also provides some basic reports to get started with CloudFront logging, but if you have a heavily loaded production environment that leverages CloudFront as CDN and you want to slice and dice the data 3^rd party services such as S3stat, SiSense or Good Data can help. In this post we will introduce Good Data and how CloudBerry Explorer can help GoodData users.

What is GoodData?

Good Data provides an on demand service to analyze Amazon CloudFront performance and traffic logs. Pre-built reports provide a comprehensive view into your content delivery performance across technology, geography, and time. Ad hoc analysis enables you to discover the underlying causes of performance issues. Best of all, there’s no software to install, no licenses to buy, and no IT support required. The service is completely web based. It is currently in a limited beta and if you are interested in taking part you are invited to sign-up

Why I need CloudBerry Explorer?

One of the problems that the services like GoodData experiencing is that only the owner of the bucket gets access to the log file. Consequently GoodData doesn’t have access to newly created log files and can’t generate reports.

To address this limitation we have implemented a capability to propagate ACL settings on a scheduled base. In other words at a regular intervals CloudBerry Explorer will automatically check for new files and modify their ACL to match the ACL setup for the parent bucket.

This will help to provide GoodData service account with the access to all files so that the service can keep generating reports.

How it works

Just select a bucket setup as a CloudFront distribution and click Schedule ACL on the task bar as shown on the picture.

It will open a dialog that will help you to schedule CloudBerry Explorer service process using Windows Task scheduler. The process will basically check for the new files every hour by default and apply the ACL list of the parent bucket to all files in it.

Once the process is setup you can enjoy advanced reporting generated using GoodData . This feature is not limited to GoodData service and can be used for other services such as steamincloud.com that provides for a FREE FLV converter – check here for more info.

If you came to this post by chance you should know that CloudBerry S3 Explorer is a Windows freeware product that you can download at http://cloudberrylab.com/

Copy files from FTP to Amazon S3 with CloudBerry Explorer PRO

A few weeks ago CloudBerry Lab started the first beta of CloudBerry Explorer PRO for Amazon S3. In the PRO version of CloudBerry Explorer we are going to offer some advanced functionality on top of basic functionality offered in our freeware version.

Among pro features are Compression, Encryption, Chucking and FTP support. In our previous post we will talked about the first three and now we will explain how FTP support works.

FTP support

FTP support has been by far one of the most sought after feature that Amazon S3 users asked on Amazon forum and when sending feedback to our support.

Imagine the situation where you want to host your media files on Amazon S3. Currently they reside on you hosting provider’s server. Almost all hosting providers offer you FTP access to your data. Now all you have to do is to copy your files from your hosting provider to Amazon S3. Before you had to use one of the FTP clients to copy your files locally and then use CloudBerry Explorer or another S3 client to upload files to S3.

With FTP support in CloudBerry Explorer this task is made even easier. You don’t need a separate FTP client anymore. Just configure a connection to FTP server and copy files to your bucket. From the user perspective FTP accounts look exactly like S3 buckets.

NOTE: CloudBerry Explorer still downloads files to your local computer and then uploads them to Amazon S3. It is just that it makes this process simpler.

Once you set up your FTP account you can start copying the files between FTP servers and S3 buckets. On the screen below you can see FTP connection to the right and S3 connection to the left.

Conclusion

With FTP support in addition to compression, encryption and chunking CloudBerry Explorer is becoming even more compelling product for professional Amazon S3 users. Please visit our website and try out the product http://www.cloudberrylab.com/default.aspx?page=s3-explorer-pro

Note: Beta 1 will expire on July, 1 2009. Please don’t use beta version for production purposes. Prices for PRO version will be announced later when we release the product to the market.

How to view Amazon S3 server log with CloudBerry S3 Explorer

Note: this post applies to CloudBerry Explorer 1.5.1 and later.

In our previous post we talked about how to enable Amazon S3 server log. In this post we will show you some of the cool CloudBerry Explorer features that allow you to view Amazon S3 server log in a human readable form.

How to generate a report

To generate a report simply right click the bucket and choose View Server Access Log from the context menu. This option will be available if you have enabled the log for the bucket previously. You can click to the Logging Setting to enable the server log.

What reports are available

Reports are available in a tabular form and a chart form. Reports are not available instantly. When log files are loaded and a report is generated you will experience a delay that depends on the volume of your log files and see a progress.

There are two charts available:

1. the total number of requests per day

2. the total traffic per day (shown on the screen)

Conclusion

CloudBerry Explorer provides you only with basic means with several built-in reports for analyzing your S3 server log. If you want to have more advanced and more flexible log analysis I suggest that you look for other tools such as SiSense Prism Viewer with Amazon S3 Dashboard . Check out our full review of this product here.

You can also check out an online service called S3Stat that generate reports from your log files using Webalizer and delivers them to your S3 bucket. These guys have also recently added CloudFront log support. Check out their latest blog post on the subject Code on the Road: Cloudfront Analytics from S3stat

If you came to this post by chance you should know that CloudBerry S3 Explorer is a Windows freeware product that you can download at http://cloudberrylab.com/

How to control your bandwidth with CloudBerry S3 Explorer

Note: this post applies to CloudBerry Explorer 1.5.1 and later.

As always we are adding features that our customers are requesting. This time we decided to add a bandwidth control. In other words now you can control how much of your available network bandwidth you want to use for Amazon S3 related stuff. This might be helpful when you are actively working on the net, surfing the website, sending and receiving email messages, etc while uploading / downloading files to Amazon S3.

Bandwidth control allows you to limit network utilization by CloudBerry Explorer and make it available to other program, so that you can perform other tasks as efficient as possible.

To control bandwidth simply go to Tools| Options, Connection tab. The default value is Unlimited, but you can change it to whatever speed you want.

If you came across this post by chance you should know that CloudBerry S3 Explorer is a Windows freeware product that you can download at http://cloudberrylab.com/

CloudBerry Explorer PRO for Amazon S3

About two weeks ago CloudBerry Lab started the first beta of CloudBerry Explorer PRO for Amazon S3. In the PRO version of CloudBerry Explorer we are going to offer some advanced functionality on top of basic functionality offered in our freeware version. 

Among pro features are Compression, Encryption, Chucking and FTP support. In this post we will talk about the first three and we will explain how FTP support works in our follow-up post.

Compression and encryption

Compression when used properly can significantly lower your storage and transfer bills.  We use industry standard ZIP compression format. To enable compression just go to Tools| Options| Advanced and click appropriate check box.

For encryption we offer a number of industry standard algorithm and it can further protect your data stored on your S3 account from unauthorized access.

Chunking

Chunking is a configuration option that breaks down your large files into the smaller pieces to speed up transfer and make it more reliable. This is helpful when you upload large files such as virtual machine images that can weigh up to 20 GB or even more.  With chunking you will break it down into several smaller files and upload each separately. In this case if for some reason transfer fails you will only have to re-upload several failed pieces and not the entire file.

By default the chunk is ~10MB and you can configure this option as you see fit.  You can also use chunk transparency option. When the option is “on” the file broken down into chunks will look like a single file on Amazon S3 storage.

 

Note: if you use chunking you need CloudBerry Explorer to download them back and combine them again to a single file.  Chunked file is not combined back on S3 storage and resides there as a number of small files.

Note 2: currently there is a bug in the beta version that doesn’t allow you to use chunking when MD5 checksum is on.

I turned Use chunks on, intentionally set Chunk size to 100KB and copied 1 MB Picture 005.jpg file to S3.  Then I turned Chunk transparency off. This is how the file looks like on S3 bucket.

If I turn Chunk transparency on you will see only one file even though physically there are 13 files stored.

Other benefits

Among other benefits of CloudBerry Explorer PRO are extended support and no expiration date.

Conclusion

With compression, encryption and chunking CloudBerry Explorer is becoming even more compelling product for professional Amazon S3 users.  Please visit our website and try out the product http://www.cloudberrylab.com/default.aspx?page=s3-explorer-pro

Note: Beta 1 will expire on July, 1 2009. Please don’t use beta version for production purposes. Prices for PRO version will be announced later when we release the product to the market. 

How to sign-up for Amazon S3 and CloudBerry Backup

This post will focus on CloudBerry Online Backup in hope that this product will become your tool of choice for automating backup and restore processes.  I am going to show you how easy it is s to sign-up for Amazon S3 account and register it with CloudBerry Backup.

Just remember, signing up for Amazon S3 is a two-step process. First you should create an Amazon account. Second you have to sign-up for S3 service.  If you ever bought a book or any other goods on Amazon you should have that account already and you can safely skip the first step and move to the second.

How to create Amazon account

Go to http://aws.amazon.com/ website and click the link at the very top right corner of the screen.

This will take you to the next screen, where you can create a new account or use your existing Amazon account.

This will take you to the next step, where you confirm your email and specify your password

This will take you to the next step where you have to fill in your address information and accept a license agreement.

The last step will take you to the confirmation step where you can specify Amazon Web Service you want to sign up to – in our case it is Amazon Simple Storage Service of Amazon S3.

Note: you have to sign-up for each Amazon S3 service separately.

How to sign-up for Amazon S3 service

This is a second part of the sign-up process. After you created an Amazon account you have to sign-up for the service itself.  The link will take you to the Amazon S3 service page and you have to click “Sing Up For Amazon S3” button.

Next page will walk you thru Amazon S3 low pricing structure and you will have to specify your credit card details at the very bottom of the page.

Don’t worry! My personal Amazon S3 monthly bills rarely exceed a couple of bucks!

Pass a few more steps to complete sign-up and you will get to a confirmation screen

By now you are successfully signed-up for Amazon S3 service.

How to obtain CloudBerry Backup

Note: CloudBerry Backup is a Windows only tool. If you are working on other platforms you will have to search for other tools.

Go to http://cloudberrydrive.com/   and follow the link to download page.

You will have to fill a registration form and we will send you an email with the download link.  Please don’t enter fake emails as you will never get a download link and waste your time. Click the link in the email to download the installation package. Install the product and run it.

Once we finish beta testing phase and release the product we will remove the registration form. For now we want to be able to keep touch with the beta testers.

Now you have to tell CloudBerry Backup your S3 account. Run CloudBerry Backup and click Backup Wizard button on the toolbar as shown on the screen. 

Create Backup Plan Wizard will come up. Just skip the welcome step and move to the second step where you have to specify your Amazon S3 account. Getting Access and Secret keys (also called Access Identifiers) is a little tricky and we will explain it in the next section.

How to get Access Identifiers

Access and Secret keys are also known as Access Identifiers and you can think of them as of your username and password.  To get them go to the http://aws.amazon.com/ website and click Access Identifiers link as shown below.

Scroll the page down a little bit to get to “Access Key ID and Secret Access Key” section. For obvious reasons I don’t show my keys here.

Click show button to display your Secret key.

Copy keys and past them in to the CloudBerry Backup Create Backup Plan Wizard screen as shown on the picture below.  Once you enter your keys the Bucket name list will get enabled. If you have followed my instructions and just created an account you don’t have any buckets just yet and the list should be empty.

Note: if you are not familiar with Amazon S3 terms you can think of the bucket as of a top level folder.

If you had S3 account before the Bucket name list should show every bucket you created already.

I don’t have a bucket, so what?

For this particular case we have an option to create a bucket for you. The bucket name should be globally unique across all Amazon S3 accounts (kind of domain names).  Check out Amazon S3 documentation if you want to learn more about bucket naming rules.  Currently we default the name of the bucket to your computer name. To ensure the name uniqueness in the future versions  we will generate a default name based on a globally unique identifier. 

You have a choice of creating a bucket in the US or Europe data center. The decision should be made primarily based on your physical location as you have to choose the closest data center to speed up data transfer.  You may also want to check out Amazon S3 storage and transfer fees as EU buckets are slightly more expensive.  

Note that you have an option to use secure SSL connection to Amazon S3 to protect your data during transfer. 

If you don't have S3 account we provide you a link to Amazon Web Services website where you can create one. 

Click OK to create a bucket. The last thing you have to specify is a Backup path. It could be anything you want  like mybackuppath that I entered on the screen below.  

Leave Save these S3 settings as default checked if you don’t want to look this complicated screen anymore.

Note: you can always change your account settings later in the Tools|Options General tab.

Now just go through the rest of the Wizard steps and create your backup plan. You can also check out our previous blog post if you want to get an idea of the main functionality provided by CloudBerry Backup. We also suggest that you watch the video on our website.

Conclusion

CloudBerry Backup is a powerful program that automates your backup and restores processes. We advocate and idea that you should set and forget about it. However, setup process could be a little tricky even though we try to simplify it as much as possible.

 Now you are ready to enjoy Amazon S3 with CloudBerry Backup, you can create Backup Plans, copy files to S3 and schedule the process.  We wish you never need to use our restore functionality.  

If you came here by chance and don’t have CloudBerry Backup yet you can download it at http://cloudberrydrive.com/

How to set custom http headers for Amazon S3 objects with CloudBerry Explorer

In CloudBerry Lab we are trying to make what works best for our customers and custom http headers is no exception. This is something that was brought up many times.   For instance renowned blogger Amit Agarwal brought it up in his post on CloudBerry Explorer.

Why should I care about custom http headers?

If you are using Amazon S3 to serve your files over http sometimes you need an additional flexibility that custom http headers could provide.  For instance if you properly set Cache-Control http header, it will help you to reduce your Amazon S3 bills. For more information check out another Amit Agarwal’ post.

There are two ways you can configure http headers using CloudBerry Explorer. You can set up http-headers for the files already located in S3 buckets or configure them right before copying files to S3.

How to set custom http header for files already stored on Amazon S3

Just select the file you want to set headers for and right click it. You will see an option called Set HTTP Headers in the context menu. The same option is available on the toolbar.

You will get the windows with the list of http headers like the one below. The list shows the headers that are already assigned to the file. Most of them are assigned by Amazon S3 by default.

If you want to add a custom http header just click add button and choose the header from the list. You can also type a name of the header manually if it is not there.   For our example with will set Cache-Control header.

Then choose the value for the header from the list.  Note that the list is filled with predefined values, but you can always type in your own value.

 Click ok and the new header will appear in the list.

Click ok once again and ok on the confirmation dialog. The header will be applied to the file on S3.  Next time someone requests this file from the web browser, the new http header will take effect. 

Can I set headers for multiple files at once?

Yes! Just multi-select the files that you want to set headers for and follow the instructions above.

How to set custom http headers for files that I want to copy to S3

It is very easy to set custom http headers to the files when they are copied to Amazon S3.  When you copy files a confirmation dialog now has Set http headers link. If you click on it the same window will come up and you will be able to manage http headers. Just follow the steps above.

Cache Control and Amazon CloudFront

When you use Amazon CloudFront service Cache-Control header is becoming even more important.  CloudFront edge servers will check S3 to see if the object has changed after the expiration time you set on your object.  If you have set no expiration time on your object, the default is 24 hours.  When that check is made, if the object is S3 has changed, the new object will go into the edge location.

Looking ahead

Besides the caching we are also looking into GZIP support that will help you to serve compressed files from S3. Not only will it help you to lower you bills but also to serve your contents faster!

Conclusion

With the ability to set custom http headers CloudBerry Explorer is becoming a compelling tool for Amazon S3 users, in particular to web site developers who want to have granular control on how the files are handled by web browser. What’s more caching is what can significantly lower your Amazon S3 monthly bills.