magpiebrain

In the spirit of making my mistakes in public – something which I have a long history of on this blog – I thought I’d post up a solution I came up with for a relatively simple problem. I’m not unhappy with the solution – it works – but I can’t help thinking I’m missing something and there is a more elegant solution out there.

The Problem

For our input we have a set of time series records. Each record contains a list of name value pairs. For each record, the keys are not fixed – they may vary. Let’s imagine that we’re recording viewing figures for the major UK soap operas – some soaps are on every day, some are only on a few days a week.

In Clojure, we’re got our data in the following form:

(("Monday" {:eastenders 6.5, :thearchers 2.3, :corrinationstreet 5.6})
 ("Tuesday" {:eastenders 6.8, :thearchers 1.4})
 ...)

We want to convert this into a single table of data, with the keys from the source data representing the columns, with each row representing a different timestamp, so that we can visualise the data with something like gnuplot, Incanter or just plain old excel. So we want to get to something like this (yes, I know The Archers is on at the weekend too, but this is just an example):

The challenge here (such as it is) is that we want a sparse table, and that our code cannot know beforehand the total universe of soap names (what if a new soap launched?).

The Header Row

The first part of this problem as I saw it was to determine which soaps our records represented to create a header row. The solution I came up with was to stick the keys for all records into a set:

user=> (def soaps 
'(("Monday" {"eastenders" 6.5, "thearchers" 2.3, "corrinationstreet" 5.6})
("Tuesday" {"eastenders" 6.8, "thearchers" 1.4})))

#'user/records
user=> (defn as-columns [records]
(apply (partial conj #{}) (mapcat keys (map second records)))))
#'user/columns

user=> (as-columns soaps)
#{"corrinationstreet" "eastenders" "thearchers"}

Assuming we want a CSV file to store our content, a function to create the header row becomes:

user=> (str "Date," (apply str (interpose "," (as-columns soaps))))
"Date,corrinationstreet,eastenders,thearchers"

The Data

Now we have a list of all possible keys (in this example, the names of the soaps), we can use this to extract data from the records for each row. Getting data from a map is straightforward – even handling the not-there case is simple enough:

user=> (def some-map 
{"eastenders" 6.5, "thearchers" 2.3, "corrinationstreet" 5.6})
#'user/some-map

user=> (get some-map "eastenders" "-")
6.5

user=> (get some-map "hollyoaks" "-")
-

Given a list of known columns, we can use a list comprehension to extract the data in a consistent order:

user=> (for [col #{"eastenders" "thearchers" "corrinationstreet"}] 
(get {"eastenders" 6.8, "thearchers" 1.4} col "-"))
(["-"] [6.8] [1.4])

Pulling It All Together

Taking those various strands, we end up with the following solution:

(defn as-columns 
  [records]
  (apply (partial conj #{}) (mapcat keys (map second records))))

(defn header-row 
  [records]
  (str "Date," (apply str (interpose "," (as-columns records)))))

(defn values-for-record
  [columns values]
  (for [col columns] (get values col "-")))

(defn as-row
  [record columns]
  (let [day (first record)
        values (second record)]
   (str day ","
    (apply str (interpose "," (values-for-record columns values))))))

(defn as-data
  [records]
  (apply str 
    (interpose "n" 
      (for [record records] 
          (as-row record (as-columns records))))))

(defn as-table
  [records]
    (apply str (header-row records) "n" (as-data records)))

Things I like with the solution: it works.

Things I don’t like with the solution:

• Duplicated call to as-columns, in both as-data and header-row
• Still don’t think I’ve got the indentation right
• Still worried I’m creating functions which are too large, or that are not readable
• This solution was arrived at by hacking on the code in a REPL – not TDD. My Clojure skills are still lacking, so I have to embark on the occasional hack-a-thon to learn some things – this being one such exercise. I plan to re-implement this using TDD with what I’ve learnt to see what I end up with. It will be interesting to see if TDDing this allays my fears about function size.
• Lots of (apply str (interpose… duplication going around – I should factor that out
• Not sure if the list comprehension here is needed – have I missed something obvious?

Updated to reflect some feedback and one example of using commons-exec as an alternative to the plain old Runtime.exec

Second Update to reflect use of shell-out – thanks Scott!

Basic

Making use of clojure.contrib.duck-streams:

(ns utils
 (:use clojure.contrib.duck-streams))

(defn execute [command]
  (let [process (.exec (Runtime/getRuntime) command)]
    (if (= 0 (.waitFor  process))
        (read-lines (.getInputStream process))
        (read-lines (.getErrorStream process)))))

...
user=> (execute "ls")
("MyProject.iml" "lib" "out" "src" "test")

It could be improved obviously – for example catching some of the potential IOExceptions that can result to rethrow additional information, such as the command being executed, or the ability to take a seq of program arguments.

Error & Argument Handling

This version adds some basic (and ugly) exception handling, and also handles spacing out arguments passed in (so passing "ls" "-la" gets processed into "ls -la"):

(defn execute
  "Executes a command-line program, returning stdout if a zero return code, else the
  error out. Takes a list of strings which represent the command & arguments"
  [& args]
  (try
    (let [process (.exec (Runtime/getRuntime) (reduce str (interleave args (iterate str " "))))]
      (if (= 0 (.waitFor  process))
          (read-lines (.getInputStream process))
          (read-lines (.getErrorStream process))))
    (catch IOException ioe
      (throw (new RuntimeException (str "Cannot run" args) ioe)))))

Using commons-exec

I had some problems with hanging processes, so knocked up a version using Apache’s commons-exec. This version has the added advantage of killing long-running processes, and I folded in Steve’s suggestion for a better way of splicing in the spaces in the command line args (see his comment). commons-exec is part of the special sauce inside Ant, so is a rock solid way of launching command-line processes (well, as rock solid as Java gets).

The use of the ByteArrayOutputStream is probably inefficient, and again, decent error handling is left as an exercise to the reader.

(defn alternative-execute
  "Executes a command-line program, returning stdout if a zero return code, else the
  error out. Takes a list of strings which represent the command & arguments"
  [& args]
  (let [output-stream (new ByteArrayOutputStream)
        error-stream (new ByteArrayOutputStream)
        stream-handler (new PumpStreamHandler output-stream error-stream)
        executor (doto
                  (new DefaultExecutor)
                  (.setExitValue 0)
                  (.setStreamHandler stream-handler)
                  (.setWatchdog (new ExecuteWatchdog 20000)))]
     (if (= 0 (.execute executor (CommandLine/parse (apply str (interpose " " args)))))
       (.toString output-stream)
       (.toString error-stream))))

Using clojure.contrib.shell-out

Many thanks to Scott for this. clojure.contrib supplies the very neat shell-out:

user=> (use 'clojure.contrib.shell-out)
nil
user=> (sh "ls" "-la")

I haven’t probed further to see if this deals with my hanging process problem, but it certainly doesn’t seem to have any support for killing timeout processes. If you’re worried about runaway tasks, the commons-exec version above might be the right choice for you.

I’ve been playing around with partially applied functions in Clojure, and have hit an interesting snag when dealing with Java interop. First, lets examine what partial does in Clojure, by cribbing off an example from Stuart Halloway’s Programming Clojure:

user=> (defn add [one two] (+ one two))
#'user/add
user=> (add 1 2)
3
user=> (def increment-by-two (partial add 2))
#'user/increment-by-two
user=> (increment-by-two 5)
7

What partial is doing is partially applying the function – in our case we have applied one of the two arguments our add implementation requires, and got back another function we can call later to pass in the second argument. This example is obviously rather trivial, but partially applied functions can be very handy in a number of situations.

Anyway, this wasn’t supposed to be a discussion of partial in general, but one problem I’ve hit with it when trying to partially apply a call to a Java static method. So, let’s implement our trivial add method in plain old Java:

public class Functions {
    public static int add(int first, int second) {
        return first + second;
    }
}

Then try using partial as before:

user=> (import com.xxx.yyy.Functions)
com.xxx.yyy.Functions
user=> (Functions/add 1 2)
3
user=> (def increment-by-two (partial Functions/add 1))
java.lang.Exception: Unable to find static field: add in class com.xxx.yyy.Functions (NO_SOURCE_FILE:3)
user=> 

So it seems like the partial call can’t handle static calls in this situation. But what if I wrap the call in another function?

user=> (defn java-add [arg1 arg2] (Functions/add arg1 arg2))
#'user/java-add
user=> (def increment-by-two (partial java-add 2))
#'user/increment-by-two
user=> (increment-by-two 10)
12

Which works. There is probably a reason why, but I can’t quite work it out right now.

Posted in the “I hope no-one else has to go through this” category in the hope that Google surfaces this for some other poor soul.

Picture a rather trivial split function:

(defn split [str delimiter]
  ((seq (.split str delimiter))))

Which helpfully spits out:

java.lang.ClassCastException: clojure.lang.ArraySeq cannot be cast to clojure.lang.IFn

The issue here is the additional set of parentheses – a hang-over from a previous edit. Removing this fixed the trouble. These parentheses were causing Clojure to expect a function call…