magpiebrain

Sam Newman's site, a Consultant at ThoughtWorks

If you’ve been using computers for a while, chances are you’ve lost valuable data more than once. Most computers nowadays come with CD writers (and many with DVD writers) which is great for the occasional manual backup – there is even the nice OSX-specific iSync tool which remind you when to perform the backup, and can manage the files being backed up too. However what would be better would be an automatic solution.

The rsync tool ships with OSX, Linux, and even Windows. Rather than copying and overwriting files on a remote location rsyc performs incremental transfer of files – only those files not already present will be copied – this drastically reduces the time taken to perform backups.

For my purposes, I wanted files to be backed up off site. I decided to sign up for a Strongspace account. Strongspace provide external, secure storage – you can access your files either via the web interface, SFTP, or (handy for our purposes) via rsync. While the rest of the article (and following parts) assume you’ll also be backing up to a Strongpsace account, it is a small matter to instead copy your files to an external disk drive or even another computer on your network. Note that Strongpsace supports rsync by running an rsync server – so these instructions should work equally well for backing up to any other rsync server.

Assuming you have your Strongspace account setup already, lets get started. To start with, I simply want to backup the contents of my Documents folder:


	

rsync -azv /Users/johnsmith/Documents johnsmith@johnsmith.strongspace.com:/home/johnsmith/backups/mac

When run, this will copy the contents of Documents to the remote directory /home/johnsmith/backups/mac. Unfortunately, since the release of Tiger my Documents folder has become stuffed with lots of Dashboard widgets I never use. To ignore these, I specify the exclude option:


	

rsync <del>azv </del>-exclude "Widgets" /Users/johnsmith/Documents johnsmith@johnsmith.strongspace.com:/home/johnsmith/backups/mac

The exclude option can take a regular expression, so you can specify multiple directories if you want to. If you want to backup files from multiple locations, then consider having a single backup directory, then create symlinks to the directories you want backed up. You’ll need to add the L option so that rsync copies the referenced directory rather than simply copying the symlink itself:


	

rsync -azvL /Users/johnsmith/backup johnsmith@johnsmith.strongspace.com:/home/johnsmith/backups/mac

Notice that when you run the command, you get prompted every time for your strongspace password. In part two we’ll start using an SSH key to avoid this, and we’ll be well on the way to providing an automatic backup solution.

Advertisements

9 Responses to “Trouble Free Backups, Part One – rsync and Strongspace”

  1. Sam Newman

    Ah – I wasn’t aware of that, thanks Jon (although I have been using the Fink build of rsync rather than the built in one). I’ll get round to updating the tutorial in a day or so.

    Reply
  2. Sam Newman

    Google’d a bit, and found that RsyncX won’t work perfectly if the destination server isn’t also OSX (apparently suome of the extra metadata can get lost). There is a “patch(rsync+hfsmode patch)”:http://www.quesera.com/reynhout/misc/rsync+hfsmode/ for the OSX rsyc which enables the copying of HFS+ resource forks and finder metadata.

    Some more digging turns up at least one reference to rsync supporting resource forks in Tiger – I’ll try and test this tonight – so from the sounds of it you should only need rsyncx/the HFS+ patch if running Panther or earlier.

    Reply
  3. Sam Newman

    More googling implies rsync has a bug with resource forks in Tiger:

    bq. on Tiger, rsync has a critical bug when handling files with resource forks. The bug makes rsync generate a new copy of files and directories with resource forks in each increment.

    (from “this topic(AppleTalk forum)”:http://forums.appletalk.com.au/index.php?s=ba6dc11860726cb374cc170f466d6464&showtopic=11327&st=0&#entry93258)

    *sigh* – for me this is probably OK though, I really only backup code and my documents to strongpsace. The bulk of my stuff gets backed up to an external drive, so I guess I can use RsyncX (or even Backup 3) for that.

    Reply
  4. Jon Tirsen

    Yeah, it’s a bit of pain in the backside those resource forks. Please do blog if you come up with a good solution for using rsync to a non-OSX machine that preserves resource forks.

    (Sometimes I just wish Apple would’ve dropped resource forks when they switched to Unix.)

    Reply
  5. Sam Newman

    They _are_ deprecated according to Apple – they’ve been trying to get people to move away from them for a while, but I guess they had to keep them for backwards compatibility.

    For the moment it looks like I’ll have to use a local full backup to either an external drive or another mac, and use my strongspace stuff for things like my documents, code and the like.

    Reply
  6. Oliver

    What I’ve done to back up several machines on a staff network is:
    1. Create a central server which uses rsync to back up each Mac
    2. That central server nfs-mounts a Unix box (but you could do it with sftp virtual drives if nfs is not available or desirable)
    3. The central server then saves the data to a disk image using hdiutil on the unix server. Voila: resource forks preserved, and all you have to do is download a handy disk image when you want to restore đŸ™‚

    Reply
  7. Daryl

    So, I still have not had a consistent answer on what happens if you *do* use rsync to try and backup to Strongspace (which I have by dint of my Joyent textdrive account).

    Basically, an rsync backup of my Mac would kick a** (particularly as my flat was burgled and drive stolen few months back).

    So, let’s say I rsync my entire Users folder to Strongspace and then need to restore the entire thing back to a new machine because of a complete hard drive failure on the PB.

    Has anyone actually tried to see what would happen ? with or without the -E flag ? (in particular, mail archives, documents, photos, sites and sources directories and perhaps a compressed automated mysql dump)

    Basically, I just want to know that I’d be able to reconstruct critical data in the event of catastrophic loss.

    let me know,
    Daryl.

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Basic HTML is allowed. Your email address will not be published.

Subscribe to this comment feed via RSS

%d bloggers like this: