Selenium is a very good in-browser testing tool. It has bindings for many different languages (including Java, Python, Perl, C! and PHP). With it you can create a suite of tests which can run tests in multiple browsers (including IE, Opera, Firefox and Safari).
The ability to run tests inside a browser is a huge boon to those of us who have to worry about cross-browser compatible websites. By having an automated test suite (and having it run regularly, perhaps using Continuous Integration) you can automatically run a set of tests repeatedly on a number of platforms, on a number of browsers, whenever code changes. Whilst this doesn’t do away with the need for normal exploratory testing, nor is it always possible (or sensible) to automate all tests, this can dramatically reduce the QA time required prior to a go live.
Selenium is a very good tool, designed to be much simpler to use than similar tools (such as Mercury’s products). It is a very good tool that you probably don’t need all that much.
Selenium Slowdown
Selenium’s strength – that it tests web applications in the browser – is also it’s weakness. Testing in browsers is slow. Not only do you have the overhead of starting and marshaling an external process (the browser) but the fact that the tests have be rendered on screen means that a sizable Selenium test suite can take an awfully long time to run.
There are techniques which can be used to handle long running tests suites (more later perhaps) but I suspect for most of you, you don’t need to worry about them
Testing the DOM
Think about what it is you want to test in your web application. You need to simulate some user activity (clicking a link, entering text) and test that some result is displayed to the user. Selenium is as good as most things out there at doing that – but as we’ve already said, it’s slow. What is the alternative?
Figure 1: An overview of a browser driver
Well what is it we are really testing here? Let’s start with the user input. For the most part (I’m excluding AJAX interaction here – more later), when a user interacts with a web page, they end up creating a HTTP request to the server. Your server acts on that request, and returns some HTML, which the browser converts into a Document Object Model, and which in turn gets rendered to the user.
So when we want to check what is displayed to the user, what is it we are actually doing? Our testing tools don’t look at the screen rendering – all they need to do is carry out assertions on the DOM itself.
So to test most web applications, we need to create a HTTP request, and perform assertions on the DOM. And Selenium certainly isn’t the fastest way of doing that.
Faking the browser
The reason that Selenium is slow, is the browser. We are using Selenium to drive the browser, which in turn submits a request for us. The browser then handles the response, creates (or manipulates) the DOM, and renders the response. Why not simply remove the browser altogether?
Tools like HTTPUnit (for Java) or Twill (for Python) let you do just that. With them, you can create a request, submit it directly to the server, handle the response and interrogate the DOM. HTTPUnit and Twill are effectively emulating the browser’s ability to create a DOM from a server response.
Figure 2: An overview of a browser-emulation testing
Test suites using browser emulation tools like this will be an order of magnitude faster than similar Selenium test suites.
No place for Selenium?
There is certainly a place for in-browser testing. In our overview of browser testing above, we implied that the DOM for any given page is created entirely as a result of a response from the server, but the world isn’t that simple.
Using Javascript, web developers have for a long time been able to manipulate the DOM by executing on the client side with no interaction with the server. Selenium (and similar tools) are still very useful for testing these kinds of situations – however for most of us there will be much less need for these (slower) tests.
Conclusion
The tools available for browser testing have come on leaps and bounds in recent years. There is a place for browser drivers (like Selenium or Sahi) and for suites based on browser emulation techniques (such as HTTPUnit or Twill). Knowing which to use and when can result in significant time savings when running your test suites.
magpiebrain 
10 Responses to “Selenium rocks – and you don’t need it”
Selenium comes in very handy when your application makes heavy use of Javascript. For example, HtmlUnit currently chokes on trying to parse the standard Dojo include files, so any application which uses Dojo isn’t testable by HtmlUnit. Selenium, since it uses the actual browser, doesn’t have this problem. So for my team at least, it’s Selenium or nothing…
Which was the point of the article – anything (not just Javascript) which manipulates the DOM after the page has been served will require in-browser testing – either manual or automated. But understand when you make that choice (even Javascript heavy apps will serve some static data).
One approach to making Javascript-heavy apps testable using static, outside-browsr verification is to decompose the content of each page into a series of components, each one available on it’s own URL.
http://htmlunit.sf.net is another powerful java browser emulator
There is another approach.
It is a bit of a bruteforce:
It is quite possible to test Ajax app with Selenium. I successfully test a complex Ajax app built with Dojo for living. Testing ALL the app functionality and performing stress test using the same techniques.
Imagine a Dojo grid that is running in a separate thread in the browser and updating itself every so often, showing the user bids placed by other users in close to real time. Thus the user in question DOES NOT make any interaction with the page at all. No posts, no mouse clicks nothing. Yet the data is changing on the page and the change has to be tested.
I had to write a harness for Dojo widgets using HTMLParser, UI Declaration and some repeaters. But once written it is quite possible to parse the HTML that Dojo throws in to the browser. The DOM will change under us so testing DOM in Ajax app is rather pointless. We are never guaranteed to see an expected value in a static unit of time in an asynchronus app. Therefore making a single call to find an object and verify its value will not work.
More over, the value in question may be in the DOM but is hidden from the user and therefore not “seen” on in the browser window. Dojo has a lot of stuff like that.
Ajax is fluid so the tests must be fluid too.
The solution is to repeatedly capture the HTML from the browser,
… selenium.getHtmlSource(); …
parse it and interrogate the snapshot with use of a repeater class callback that looks something like this:
class Repeater{
public final static int TIMEOUT = 10000;
private static final int PAUSE = 250;
public static interface RepeaterCallback {
T whatToRepeat();
}
public static T repeat(long timeout, RepeaterCallback repeater) {
return repeat(timeout, PAUSE, repeater);
}
public static T repeat(long timeout, long pause, RepeaterCallback repeater) {
long exitTime = System.currentTimeMillis() + timeout;
while (true) {
T result = repeater.whatToRepeat();
if (result != null) return result;
if (System.currentTimeMillis() + pause >= exitTime) return null;
if (pause > PAUSE)
sleepNoThrow(pause);
else
sleepNoThrow(PAUSE);
}
}
}
The pattern is: Build, Operate Check.
Fire up the repeater with a given time out. And inside the callback call your test code. The test code will grab the HTML from the page in to an immutable variable and interrogate it. If unsuccessful (the value not found) it will repeat the process again and again until it either finds what you are looking for or times out.
Wrap the test case in to a method and:
@Test
public testSomething(){
MyTestParameters param …..
setupMyTest(param);
changeCondition(param);
assertEquals(“ExpectedValue”, testSomethingInAjax(param));
}
private String testSomethingInAjax(final MyTestParameters param) {
return Repeater.repeat(15000, 0, new Repeater.RepeaterCallback() {
public String whatToRepeat() {
return findExpectedValue(param);
}
});
}
private setupMyTest(MyTestParameters p){
// build test conditions
}
private changeCondition(MyTestParameters p){
// mock something, perform some actions
}
private String findExpectedValue(MyTestParameters p){
// Test the results in with use of the repeater
// return pass or fail
}
With a little code you can test anything in GWT, Dojo, Yahoo tool kit etc etc.
Wrap your UI components in to java classes with use of parser and test the expected value with a repeater, because you can NEVER be sure that the value will be there when you expected 🙂
Just my $0.02
I really think you DO need selenium to test your website properly in various browsers. JS that might work for FF does always not work for IE or Safari. True, selenium does not really catch js exceptions, but some exceptions make the entire test fail, so we do get some feedback on this.
Emulating a browser is not the same thing as running it in a browser
I have to agree, it is terribly slow, but we just run it on our dev server every night, and get an email every morning with newly introduced bugs.
It really works perfectly for us.
I have used selenium for more than 2 years and I am totally impressed how easy it makes to integration test your application across multiple browsers. I have also used httpunit for 4+ years and the biggest problem that I have faced with that is the javascript incompatility of httpunit and firefox/IE.
Overall I would say that Selenium is the way to go for testing web applications and
Maven Selenium for java web applications, but when it comes to load testing I’ve had better luck with JMeter.
We have found that the answer to all these questions and issues is to use Selenium and TestMaker by PushtoTest. We get the best of both worlds – browser compatibility tests and high speed regression test runs, performance tests and production scripted performance monitoring. All with the same selenium script – we now convert our selenium scripts using the TestMaker “transformer” to python scripts and now have programatic control on our scirpts with data driven.
@Alexander:
I’m trying to use Selenium to test Dojo apps, but i’m having some problems with some components like dijit.form.FilteringSelect do you have some documentantion that can help me?
Thanks,
João
Your approach worked great when I used it 5 years ago on classic page-click-page websites, when we all regarded JavaScript as something to be avoided. SInce then, JavaScript has been rehabilitated (largely due to the success of GMail), and now client-side programming is a key part of any decent web application. As a result, you can’t avoid having the browser in the testing loop any more.
@Joao – I just started using Selenium to test a Dojo app I’m working on and ran into the trouble with FilteringSelect as well. Specifically, I couldn’t record selecting a field from the select dropdown. I manually edited the script to do the following:
1. MouseDown on the down caret for the FilteringSelect.
2. MouseOver the dropdown value you want selected (the mouseover triggers a ‘highlighted selection’ in the FilteringSelect code.
3. MouseUp on the dropdown value you want to select.
mouseDown
//div[@id=’widget_dijit_form_FilteringSelect_3′]/div/input
mouseOver
css=#dijit_form_FilteringSelect_3_popup10
mouseUp
css=#dijit_form_FilteringSelect_3_popup10
Hope that helps,
Rob