Sometimes people need to copy a large number of files between two Amazon S3 accounts. This process can be time consuming in particular when you copy between a bucket located in the US and the bucket located in EU. This is where CloudBerry Explorer PowerShell interface comes handy.
The advantage of using CloudBerry Explorer for copying files between buckets
The good thing about CloudBerry Explorer is that it copies the files in Amazon S3 w/o the need to download the files to a local computer. This dramatically reduces the time needed to run such copy and user expenses. For copying between the buckets in the same geographical location it is absolutely free. If you want to copy between buckets in different locations (between US and EU) you still have to pay transfer fees. PowerShell command line interface further simplifies things as you can automate the whole process and literally forget about it.
Getting Started with PowerShell Console
Note: if you don’t have PowerShell installed on your computer please refer “How to get Microsoft PowerShell” and “How to register PowerShell snap-in” sections below.
Note: to run the following example your PowerShell Execution Policy should be set to at least RemoteSigned. If you don’t know how to do it see at the bottom of the post.
Run PowerShell console from start menu.
To add a CloudBerry explorer snap-in to PowerShell console run the following command
Add-PSSnapin CloudBerryLab.Explorer.PSSnapIn
Create an object for a source S3 account connection
$s3 = Get-CloudS3Connection -Key [accesskey] -Secret [secretkey] -ProxyAddress [proxy] -ProxyPort [port]
Where [accesskey] and [secretkey] your Amazon S3 account access and secret keys respectively
[proxy], [port] - a name or an IP address and a port number of your proxy server respectively (if needed)
Create an object for a target S3 account connection
$s3target = Get-CloudS3Connection -Key [accesskey] -Secret [secretkey] -ProxyAddress [proxy]ProxyPort [port]
Create an object for a target bucket and a folder
$destination = $s3target | Select-CloudFolder -Path [target bucket path]
Where [target bucket path] is the name of the bucket followed by the path to your files on a target Amazon S3 account
Note that you can specify just a bucket without folders underneath it if you want to copy files to the bucket root.
Select a source bucket and a folder and run copy process
$s3 | Select-CloudFolder -Path [source bucket path] | Copy-CloudItem -Destination $destination -Filter *
Where [source bucket path] is the name of the bucket followed by the path to your files on a source Amazon S3 account
While copy process is running you can monitor the progress.
Just replace your [accesskey] , [secretkey] , [target bucket path] and [source bucket path] with those applicable to your environment.
How to move files instead of copying them
If you want to move files you can use Move-CloudItem command. It has exactly the same syntax as Copy-CloudItem command. Here is an example:
$s3 | Select-CloudFolder -Path [source bucket path] | Move-CloudItem -Destination $destination -Filter *
How to create a PowerShell batch file
Create a text file and paste a script above to it. Name it with '.ps1' extension, like “CopyS3files.ps1”
Now you can run scripts from the console.
c:\myscripts\CopyS3files.ps1
or if it is in current directory just
.\ CopyS3files.ps1
Note: to run PowerShell batch files the default Execution Policy might need to be changed. See below on how to set Execution Policy.
How to schedule a PowerShell batch file
You can also schedule PowerShell script file with Windows Task Scheduler
Just paste the following line to “Run” field as depicted below.
%SystemRoot%\system32\WindowsPowerShell\v1.0\powershell.exe c:\myscripts\ CopyS3files.ps1
If -noexit is specified PowerShell console will not be closed after script execution.
More information on how to run scripts can be found on the Microsoft website at: http://www.microsoft.com/technet/scriptcenter/topics/winpsh/manual/run.mspx
How to get Microsoft PowerShell
Microsoft PowerShell can be downloaded for free from Microsoft website at http://www.microsoft.com/technet/scriptcenter/topics/msh/download.mspx
How to register CloudBerry Explorer PowerShell Snap-In for Amazon S3
If the PowerShell was installed prior to installation of CloudBerry Explorer the Snap-in will be installed automatically as a part of the CloudBerry Explorer installation process. Otherwise, run the following command in the CloudBerry Explorer installation folder
c:\Program Files\CloudBerryLab\CloudBerry Explorer for Amazon S3
C:\Windows\Microsoft.NET\Framework\v2.0.50727\installutil.exe CloudBerryLab.Explorer.PSSnapIn.dll
Note: You can do this from command line or PowerShell.
You can verify that the CloudBerry Explorer Snap-in is registered. Run the following command using PowerShell console:
Get-PSSnapin –Registered
PowerShell displays registered Snap-Ins. Check that CloudBerryLab.Explorer.PSSnapIn is in the list.
How to set Execution Policy
To allow running scripts in batch files default execution policy must be changed, it must be set only once. By default execution policy is set to Restricted (Running scripts or configuration files is prohibited).
Note: You can still execute individual commands. The default policy should be changed only if you want to execute PowerShell batch files.
Possible options:
Unrestricted – all scripts are permitted without signing.
· RemoteSigned – local scripts can be executed without signing. Remote (downloaded using Outlook, Internet explorer, etc) must be signed with certificate. On execution message will appear weather you trust publisher.
· AllSigned – only signed scripts can be executed. The message about publisher will appear.
To run the script in our example set Execution Policy to RemoteSigned as shown on the screenshot.
Setting Execution Policy on Vista
If you are running Vista you will need to run the console as Administrator or you won’t be able to change Execution Policy. This is because Execution Policy is controlled by a Windows Registry and access to registry on Vista is restricted by UAC.
You will likely get "Set-ExecutionPolicy : Access to the registry key ..." error message when running Set-ExecutionPolicy command.
To work around the problem start the PowerShell console as an Administrator and run the command. You can do this by right-clicking the PowerShell icon in the Start Menu and selecting "Run as Administrator".
You can donwload CloudBerry Explorer for free at http://cloudberrylab.com/
You can learn more about CloudBerry Explorer PowerShell interface at http://cloudberrylab.com/default.aspx?page=amazon-s3-powershell
Posts
