The idea is simple, we want to rsync our data into Amazon S3. FYI, I use Fedora 17 at this example.
We can use this powerfull tools called “s3fs” to mount S3 into our local. Download the latest version in http://s3fs.googlecode.com/files/s3fs-1.61.tar.gz.
1. Install dependecies
To be able using this tools, you need to have :
1 | sudo yum install fuse fuse-devel libcurl libxml2-devel libcurl-devel libstdc++-devel openssl-devel curl-devel mailcap |
On Debian / Ubuntu :
1 | apt-get install build-essential libfuse-dev fuse-utils libcurl4-openssl-dev libxml2-dev mime-support |
2. Compile and install
Now you can extract by “tar -xvvf s3fs-1.61.tar.gz” and start compile :
1 2 3 | ./configure make sudo make install |
Now you should see “s3fs” in your command console.
3. Setup Credentials
We need to configure AWS ACCESS and SECRET Key before using this tools. There are several way to setup this credentials :
A. Temporary by export variable
1 2 | export AWSACCESSKEYID=X**********7 export AWSSECRETACCESSKEY=X********************7 |
B. Permanent by create configuration file
1 2 | vim ~/.passwd-s3fs accessKeyId:secretAccessKey |
And don’t forget to “chmod 600 ~/.passwd-s3fs”
3. Mounting your bucket into /mnt
Now we can test to mount our S3 bucket into /mnt by :
1 2 | sudo chmod 777 /mnt s3fs <your-bucket> /mnt/ |
Here is the results on “df -hT” :
1 2 3 4 | /dev/sda6 ext4 167G 12G 154G 7% / tmpfs tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup tmpfs tmpfs 3.9G 0 3.9G 0% /media s3fs fuse.s3fs 256T 0 256T 0% /mnt |
Now, whenever you put files into /mnt, it will send into your amazon buckets.
4. Tips
List all files that will synced but without copying it. See -n option in rsync (mean test).
1 | rsync -n -v -r –size-only /path/to/sourcefolder/ /path/to/destfolder/ |
Mirror all files from local to remote with deleting file in mirror that doesn’t exists in local.
1 | rsync -av –delete /backup/directory /mount/point> |
5. Problem
If you’re facing problem like :
1 | fuse: bad mount point – Transport endpoint is not connected |
The solutions are :
1 2 3 | kill the sshfs process(es) umount -l /mnt/foo (as root) mount the sshfs again |