Open Source Musings The thoughts, ideas, and opinions of an open source guy

Taking a quick look at rsync

A terminal window If I had cobble together a list my favourite utilities, rsync would be near the top. It’s simple, secure, and flexible.

But what’s rsync, you’re asking? rsync is a command line utility for transfering and synchronizing files and directories across a network. rsync has a number of uses and, like most command line tools, a number of options.

Chances are, you’ll use rsync for a handful of tasks. I do, and that means I only use a few of its options.

Here’s a quick look at how I use rsync, and the options I use.

Backing up a website

While my website is relatively static, I do make the occasional change to it. When I do that, I’m often not at my main computer. I use ssh to log into the site, then make the change. Keeping track of those changes can become a chore.

I could use FileZilla to transfer the entire site to my hard drive (I like to have a local copy), but that’s wasteful. With rsync, I can grab only the files that have changed.

Simply running the rsync command isn’t enough. To back only up the changed files on the website, I prefer to log in using SSH. Luckily, my hosting provider offers shell access; not all do. I also need to specify the remote directory to look in and the directory on my computer into which I want to copy the files.

Here’s the command string that I use:

rsync -e ssh -avz –recursive –exclude=weblog [email protected]:/remote_directory/ /home/scott/website/

Let’s break that down:

  • [email protected] is the login name for the web host
  • The -e ssh option tells rsync to run using SSH to secure the connection
  • The -avz tells rsync to grab only the changed files (a), give me information from rsync as the files are transfered (v), and to compress the files as they’re transferred (z)
  • The –recursive option goes into the subdirectories on the server and grabs changed files from them
  • The -exclude option tells rsync to ignore certain files and directories (in the example above, the folder for one of my blogs)

You can exclude multiple files and folders by specifying an –exclude for each of them.

Using it locally

As you may or may not know, I’ve moved a couple of my blogs and a website to Jekyll — you can learn more about that in this post. The files for those sites are on my hard drive. But, as we all know, not nice things can happen to hard drives.

So, I like to backup my Jekyll sites to my local ownCloud folder. That syncs the files with ownCloud on the web.

To do that, I run the command:

/home/scott/Jekyll Sites/” && rsync –exclude _site/ –exclude .sass-cache -rtvu . “/home/scott/ownCloud/Jekyll Sites Backup”

Resuming a failed transfer

Part of my work involves working at client sites. Which means I use their hardware and their networks. Sometimes, I need to move large files across the network — usually from a server in North America or Europe to my computer in New Zealand. To do that, I use the command:

rsync [email protected]:/path/to/file /path/to/local_folder

For a variety of reasons, the transfers occasionally fail part way through. Instead of restarting a lengthy transfer from the beginning, why not pick up where it left off. To do that with rsync, use the command:

rsync -av –partial [email protected]:/path/to/file /path/to/local_folder

You’ll remember the -av option from earlier. The –partial option tells rsync to keep partially transferred files and start

Using an alias with rsync

All those options can be a bit of a chore to remember or type, especially if you don’t use rsync as regularly as you should. But you don’t have to remember or type those commands. Instead, use a command line alias.

Here are the aliases I use to back up my Jekyll files:

alias jbackup=’cd “/home/scott/Jekyll Sites/” && rsync –exclude _site/ –exclude .sass-cache -rtvu . “/home/scott/ownCloud/Jekyll Sites Backup” && cd’

Imagine typing those commands every time you need to use them!

When you need to use rsync, I just open a terminal window and run the command associated with the alias. It’s a little bit of work up-front, but after that it’s just a matter of letting the alias take over.

As I mentioned at the beginning of this post, rsync is a very powerful tool with a number of options. I’ve only really scratched the surface of what rsync can do. For my needs, what I outlined above is enough.

Thoughts? Let's start a conversation on Twitter.

Did you enjoy this post or find it useful? Then please consider supporting this blog with a micropayment via Liberapay. Thanks!