Backup / migrating files into Amazon S3 bucket using RSYNC


The idea is simple, we want to rsync our data into Amazon S3. FYI, I use Fedora 17 at this example.
We can use this powerfull tools called “s3fs” to mount S3 into our local. Download the latest version in http://s3fs.googlecode.com/files/s3fs-1.61.tar.gz.

1. Install dependecies
To be able using this tools, you need to have :

1
sudo yum install fuse fuse-devel libcurl libxml2-devel libcurl-devel libstdc++-devel openssl-devel curl-devel mailcap

On Debian / Ubuntu :

1
apt-get install build-essential libfuse-dev fuse-utils libcurl4-openssl-dev libxml2-dev mime-support

2. Compile and install
Now you can extract by “tar -xvvf s3fs-1.61.tar.gz” and start compile :

1
2
3
./configure
make
sudo make install


Now you should see “s3fs” in your command console.

3. Setup Credentials
We need to configure AWS ACCESS and SECRET Key before using this tools. There are several way to setup this credentials :

A. Temporary by export variable

1
2
export AWSACCESSKEYID=X**********7
export AWSSECRETACCESSKEY=X********************7

B. Permanent by create configuration file

1
2
vim ~/.passwd-s3fs
accessKeyId:secretAccessKey

And don’t forget to “chmod 600 ~/.passwd-s3fs”

3. Mounting your bucket into /mnt
Now we can test to mount our S3 bucket into /mnt by :

1
2
sudo chmod 777 /mnt
s3fs <your-bucket> /mnt/

Here is the results on “df -hT” :

1
2
3
4
/dev/sda6      ext4       167G   12G  154G   7% /
tmpfs          tmpfs      3.9G     0  3.9G   0% /sys/fs/cgroup
tmpfs          tmpfs      3.9G     0  3.9G   0% /media
s3fs           fuse.s3fs  256T     0  256T   0% /mnt

Now, whenever you put files into /mnt, it will send into your amazon buckets.

4. Tips
List all files that will synced but without copying it. See -n option in rsync (mean test).

1
rsync -n -v -r –size-only /path/to/sourcefolder/ /path/to/destfolder/

Mirror all files from local to remote with deleting file in mirror that doesn’t exists in local.

1
rsync -av –delete /backup/directory /mount/point>

5. Problem
If you’re facing problem like :

1
fuse: bad mount point – Transport endpoint is not connected

The solutions are :

1
2
3
kill the sshfs process(es)
umount -l /mnt/foo (as root)
mount the sshfs again

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.