magpiebrain

Sam Newman's site, a Consultant at ThoughtWorks

Posts from the ‘howto’ category

At a recent ThoughtWorks techy meetup, we recorded a couple of videos for internal use. One of them involved someone speaking to slides. The problem was that we had no projector, and our flip camera wasn’t really up to doing a great job of filming the screen. So the problem I was faced with was the best way to synchronize the audio track from the video to the slides.

What follows is a far from efficient workflow that allowed me to sync the audio track with the slides. The only constraint was my skills, and the fact that I only had freeware software and whatever came with the mac (iLife 08).

0. Convert Flip AVI -> mp4

I’m not sure if I needed to do this, but I figured the OSX tools would play nicer with m4v than with the avi’s from the Flip. I didn’t have the Flip software installed. I used Handbrake to convert the video.

1. Import into GarageBand to rip out the audio track

I probably could of done this via Handbrake, but it was fairly easy to do in GarageBand. I imported the video, selected the video track, and removed it.

2. Convert the Keynote slides to Images

Given that I was in GarageBand, I figured I’d create a podcast, using the Keynote slides as artwork for the podcast. Exporting the slides as JPEG is a simple job – I chose to have Keynote create one image for each stage in a transition. I then imported the JPEGs into Aperture so I could access them via GarageBand’s media browser.

3. Add the artwork to the podcast

You’ll need to make sure the Podcast track and the media browser is visible. Then, it’s a simple matter of dragging in the relevant slide in sync with the audio track. The UI is pretty intuitive.

4. Export the podcast

Now it gets trickier. At this point, GarageBand thinks we have an audio, rather than video, podcast. I tried exporting into iTunes, but just got an audio track – I’m sure the artwork would show up on my iPhone, but I want a movie for uploading to our internal video hosting service (we have Google apps, so get Google Video).

What I ended up doing was exporting to iWeb, exporting out the resulting Podcast site, then grabbing the resulting quicktime movie. This quicktime movie was almost what I wanted – a video, with the audio track from the flip, with synchronized slides. But I wanted to do a couple more things – namely I wanted to split the presentation into two parts (5-10min videos are easier for people to digest than a single 20min presentation), and add some additional titles.

5. Split and add titles

Luckily, iMovie can handle this. I’m fairly sure it was able to directly import the quicktime movie. Adding the titles is a little fiddly, far worse is the simple task of splitting the movie itself. I hope to try iMove 09 soon to see if this is improved.

It was at this point I realised I had an issue – namely that the artwork you embed in podcasts via GarageBand is limited to a square, 300×300 pixel dimensions. This led to some clipping of the slides. If I have to use this workflow again, I’ll make sure to export square slides from keynote in the first place.

6. Export and upload

Which is finally where things got easy.

Notes

  • The GarageBand project was massive. The two resulting videos from iMovie were less than 15MB – the GarageBand project which just contained the slides and the audio track was over 450MB.
  • I’d love to find a better option for syncing the artwork – being limited to 300×300 pixels is a pain, especially when I’m uploading to a video hosting service capable of much higher quality widescreen playback.
  • I like the power of iMovie, but I dislike the UI intensely. I’d love to be able to increase/decrease clip selection using the keyboard for more control.
  • For some reason Google Video too ages to process the two small videos – significantly larger videos uploaded afterwards were available within a few minutes. It is possible that the small, squashed size of the movie gave the transcoding a problem, or it may just of been a transient glitch, but I ended up having to wait overnight for them to finally be available.
  • I’d love to know of a better workflow using free/cheap software, so please get in touch if you have any ideas!
February 27, 2010

I knocked this up to help testing on something I’ve been working on in my spare time. It would be a trivial exercise to extend this to build pages for specific URLs – this example returns the same markup for any example. The old codehaus site for Jetty contain lots of examples of how to configure an embedded server.

import org.eclipse.jetty.server.handler.AbstractHandler
import org.eclipse.jetty.server.Handler
import org.eclipse.jetty.server.Server
import org.eclipse.jetty.server.Request
import javax.servlet.http.HttpServletRequest
import javax.servlet.http.HttpServletResponse
import scala.xml.Elem

class HttpServer {

  val handler = new MutableHandler()

  def run(port: Int) = {
    val server = new Server(port)
    server.setHandler(handler)
    server.start()
  }

  def updateHtml(html: Elem) = {
    handler.html = html
  }
}

protected class MutableHandler extends AbstractHandler {
  var html = <h1>Hello</h1>

  override def handle(target: String, request: HttpServletRequest, response: HttpServletResponse) = {
    response.setContentType("text/html");
    response.setStatus(HttpServletResponse.SC_OK);
    response.getWriter().println(html.toString());
    (request.asInstanceOf[Request]).setHandled(true);
  }
}
May 16, 2009

Now we’ve worked out what files you want backed up, and we’ve sorted automatic authentication, the only thing left to do is schedule our backups. Unfortunately we can’t make use of the very handy iSync or Backup tools to do this (Apple don’t seem keen on opening up these tools up, as they no-doubt help sell expensive .Mac accounts).

h3. cron

cron is available on all *NIX systems, and is a well established solution to running all kinds of house keeping tasks. On OSX 10.4 however cron is being phased out in favour of launchd which we’ll look at next.

launchd

As of OSX 10.4 a new daemon called launchd has been added. Eventually this will replace tools such as cron, rc and xinetd – however it’s low on the userfriendly scale right now. Unlike cron’s simple crontab file, launchd requires XML files which can be a pain to edit without something like the shareware program launchd editor. Hopefully later OSX releases might bundle a decent UI for managing it.

Anacron + launchd – for those who like to switch their computers off

Both cron and the newer launchd suffer from the same problem when it comes to running routine housekeeping tasks – if the computer isn’t on when a task is scheduled, the task won’t get run, and even when you switch the computer on again, neither tool can work out that it should try and catch up by running the task on startup.

So using either cron or launchd means that if we use either task to manage our backups, the computer has to be on when the backup is scheduled to run.

Anacron attempts to solve these deficiencies. It stores a record of when each job was last run, so on startup it can work out if it needs to catch up. This is especially handy for laptops which are probably only on for a few hours a day.

Depending on your operating system, you can either get anacron setup to work with cron, or launchd. Assuming you take the anacron + launcd approach, grab Ronald Florence’s highly useful Anacron for OSX 10.4 installer. Once installed, you should have a file /etc/anacrontab which will look like this:


	
#period delay   job-identifier  command
1       5       cron.daily      periodic daily
7       10      cron.weekly     periodic weekly
30      15      cron.monthly    periodic monthly

The first field is the frequency of the job – 1 means daily, 7 means weekly and 30 means monthly. The second value is the delay in minutes between anacron realising a job needs to be run, and it actually getting scheduled. The third field is a unique identifier for the job, and finally the command itself.

To simplify things we should create a simple script which contains our rsync command(s) – for now you can place it into your bin directory, and call it something sensible like backup. Next, assuming you want to run the backup job daily, add the following line to /etc/anacrontab:


	
1       10      backup.user      /Users/johnsmith/bin/backup

To make sure anacron is working (and is picking up your job) simply start Console.app, and select Open System Log from the File menu – you should see its output. If you need to tweak your backup job, you can remove the record of the previous run /var/spool/anacron/ – it should contain one file per record in anacrontab. Then, you can force anacron to rerun the job by executing anacron -n.

And there you have it! Trouble free, secure backups that will get run even if you switch your computer on for only a few hours each day.

October 31, 2005

In our last part, we used rsync to connect to a remote server to perform incremental backups. The problem is that we really want this to be automatic. Scheduling when a backup occurs is actually fairly simple. What is more work is performing automatic authentication so our backup can occur without user intervention.

Authentication using an SSH key

We’ll be using an SSH key to authenticate ourselves with Strongspace (or any other rsync server for that matter). Thanks have to go to Jens-Christian Fischer’s post on his own backup solution, which helped get me started. To start with, open a terminal and enter the following command:


	
$ ssh-keygen -t dsa -b 1024

Leave the key location unchanged, but enter a passphrase. You can choose not to enter a passphrase (and this will simplify things) however this is pretty insecure. If anyone gets hold of your key they’ll be able to access your Strongspace files without the need for a password.

Now we’ve generated our key, we can use it to authenticate ourselves when connecting to the remote server. The server we’re connecting to needs a copy of our public key:



	
$ cp ~/.ssh/id_dsa.pub authorized_keys

Log into Strongspace via the web interface, create a directory called .ssh and copy the authorized_keys file into this directory. To test this, run the rsync command we created in part one:


	
$ rsync -azvL 
/Users/johnsmith/backup 
johnsmith@johnsmith.strongspace.com:/home/johnsmith/backups/mac

This time, rather than the Strongpsace server asking you for your password, you’ll get prompted for your key’s passphrase.

SSHKeychain

So at this point you must be thinking “Well, what was the point of that – you’ve replaced the need to enter the password for the remote server, with the need to enter a passphrase for your key!” – and you’d be right. What we need is some kind of software that can be used to automatically handle unlocking our key.

SSHKeychain is an OSX specific SSH key management tool. When using tools like rsync or ssh, SSHKeychain can automatically lookup your passphrase. Download and install the tool, then start it up. Open the Preferences pane, select the Environment tab and enable Manage global environment variables – this will allow other applications to use the keys managed by SSHKeychain. Check the keys tab and ensure your key (~/.ssh/id_dsa) is visible. Finally select Agent/Key Status from the Agent menu, enter your passphrase for your key, and enable the option to add the passphrase to your keychain – this means that when you log on, SSHKeychain will automatically have access to your SSH key’s passphrase. So long as you’re logged on you’ll have no need to type it in again.

For SSHKeychain to start managing your key, you’ll need to log off and back on, but before you do it’s a good idea to add SSHKeychain to your startup items so you don’t have to remember to start it up every time.

Once you’ve logged off and back on, SSHKeychain will be silently managing your keys. The first time you use an application like ssh or rsync, SSHKeychain will look in your OSX Keychain, locate the passphrase for your key and automatically authenticate you. Note that you will need to be logged in for this to work, as SSHKeychain needs to be running, and your OSX Keychain needs to be unlocked.

To test this, after logging back in run our rsync command again – this time you shouldn’t be prompted for any password.

In part three we’ll complete our backup solution by creating an automated backup script.

October 31, 2005

If you’ve been using computers for a while, chances are you’ve lost valuable data more than once. Most computers nowadays come with CD writers (and many with DVD writers) which is great for the occasional manual backup – there is even the nice OSX-specific iSync tool which remind you when to perform the backup, and can manage the files being backed up too. However what would be better would be an automatic solution.

The rsync tool ships with OSX, Linux, and even Windows. Rather than copying and overwriting files on a remote location rsyc performs incremental transfer of files – only those files not already present will be copied – this drastically reduces the time taken to perform backups.

For my purposes, I wanted files to be backed up off site. I decided to sign up for a Strongspace account. Strongspace provide external, secure storage – you can access your files either via the web interface, SFTP, or (handy for our purposes) via rsync. While the rest of the article (and following parts) assume you’ll also be backing up to a Strongpsace account, it is a small matter to instead copy your files to an external disk drive or even another computer on your network. Note that Strongpsace supports rsync by running an rsync server – so these instructions should work equally well for backing up to any other rsync server.

Assuming you have your Strongspace account setup already, lets get started. To start with, I simply want to backup the contents of my Documents folder:


	
rsync -azv /Users/johnsmith/Documents 
johnsmith@johnsmith.strongspace.com:/home/johnsmith/backups/mac

When run, this will copy the contents of Documents to the remote directory /home/johnsmith/backups/mac. Unfortunately, since the release of Tiger my Documents folder has become stuffed with lots of Dashboard widgets I never use. To ignore these, I specify the exclude option:


	
rsync <del>azv </del>-exclude "Widgets" 
/Users/johnsmith/Documents 
johnsmith@johnsmith.strongspace.com:/home/johnsmith/backups/mac

The exclude option can take a regular expression, so you can specify multiple directories if you want to. If you want to backup files from multiple locations, then consider having a single backup directory, then create symlinks to the directories you want backed up. You’ll need to add the L option so that rsync copies the referenced directory rather than simply copying the symlink itself:


	
rsync -azvL 
/Users/johnsmith/backup 
johnsmith@johnsmith.strongspace.com:/home/johnsmith/backups/mac

Notice that when you run the command, you get prompted every time for your strongspace password. In part two we’ll start using an SSH key to avoid this, and we’ll be well on the way to providing an automatic backup solution.

October 29, 2005