Tuesday, September 14, 2004

Testing web-based applications with FIT

I have had a need to automate testing of our web-based applications because manual testing could never be completed in a reasonable timeframe.

For the last two weeks, I've been writing a fixture for FIT specifically to test web-based applications. The idea is to use very sophisticated HtmlUnit library for actual HTTP interactions and dealing with HTML (and Javascript) and to use FIT for specifing, running and reporting on test cases.

update: project is now hosted on public subversion repository. See http://opensvn.csie.org/webfixture.

For those who have missed it, here is a short description of the FIT framework. FIT is a web-based tool for specifying and executing acceptance tests for all kinds of application, with a particular accent on simplicity of the specs (they could be understood and may be even written by a non-programmer, i.e. customer). The central role in the FIT framework is devoted to a fixture. A fixture implements a vocabulary, which is used by user to describe things he wants FIT to perform. For example, to press a button or to follow a link. Or to evaluate 2+2 and to check that result is 4. Thus, user writes these checks using vocabulary and programmer implements this vocabulary with a fixture in a meaningful way. If you're intrigued, Michael Feathers has written a more elaborate intro in his Pitching a Fit post.

Originally, I tried to write a custom ActionFixture for each web page I had to handle but soon discovered that it was a bad idea. Firstly, it was too much work -- to write a Java class for each page I need to test. Secondly, and more important, each fixture would use different vocabulary for each page which means that test writer had to be familiar with an underlying Java class. Therefore I decided it would be better to write a single class (a fixture) that could be used to exercise any web page. Luckily web languages' expressiveness is rather limited so the resulting vocabulary to describe test cases should be manageable.

Shortly after I started to work on my WebFixture, I discovered a (semi?)-official HtmlFixture, created for the same purpose and which even uses the same HtmlUnit library! Nevertheless, I decided to continue to develop my own fixture, for these reasons:
  1. To get experience with Java platform.

    One of my goals with this mini-project has been to try out a Java platform. I've been on most C++, Python, Tcl camps these years and my Java programs never went beyond the "Hello, world!" exercise. This project seems like a perfect opportunity to learn a new platform.

  2. To control fixture's development.

    I want a control over this fixture to add things I want in a way I want. While the HtmlFixture is mature, its source code is undocumented and poorly written (to me). This probably a funny thing to say from a novice Java developer, but still, the source code seems like a mess to me and don't want to dive into it.

  3. To design it differently.

    If I decided to base my work on HtmlFixture there would be still no guarantee that my changes would ever go into original codebase. Why? Because I my approach is different and therefore there is a good chance that changes I propose won't "fit" author's concepts. Don't get me wrong, HtmlFixture is a fine work, I just want to implement the same concept a bit differently.

So how my fixture does look like?

The important design goal to me is to make fixture's vocabulary as simple and as high-level as possible. It should be similar in terms to those used by a web browser metaphor. It's better to sacrifice some underlying versatility of HtmlUnit to gain in simplicity and consistency for a user. It's better to overload verbs (where appropriate) than to invent new ones.

For example, I use verb click to follow links and submit buttons and use verb enter to enter data to input boxes, text areas and select boxes. This way, the same verb can performs technically different things as long as it means one thing to the user.

There are some other features I've added so far: site configuration, Java delegates, erroneous page detection.

The site command allows you to set some site options (or preferences) which include site root, timing information, error page patterns and more. These preferences are kept in a static storage so that you could set them in SetUp page and use on test pages.

The error page patterns used to detect erroneous web pages - normal HTML pages which are returned with HTTP response code 200 but which nevertheless are a sign of some internal error in the web application. For example, some application servers (like WebKit) show crash reports in this way and fixture could be configured to detect these automatically.

Java delegates are extension point to be able to hook custom checks that are beyond normal web processing, like verifying the database state. A delegate is passed a current HtmlElement and should return a boolean result - whether a test has passed or not.

And finally, some closing words on my first impressions on Java platform.

The IDE for a statically-typed language could do amazing things and Eclipse actually does them! Moreover, it is very capable in other aspects making handling of Java verbosity almost imperceptible. With advanced browsing support and context assist I managed to grasp an essence of both FIT and HtmlUnit library very quickly. And QuickFix assist helps greatly to correct my naive syntax errors. Overall, I must say, the development process goes much more smoothly and is much more fun that I have expected, thanks to IDE.

15 Comments:

Blogger Ģirts Kalniņš said...

is it possible to see sources of tests?

8:35 PM  
Blogger Max Ischenko said...

I can mail them to you.

12:04 PM  
Blogger Ģirts Kalniņš said...

Would be nice, that's anything at smejmoon.lv :)

Testing generated HTML seems like realy hard problem for me.

1:47 AM  
Anonymous alabinsky said...

Max,

I'm going to introduce FitNesse for acceptance testing of our web apps.
I've tried HtmlFixture with a simple login form.
We are using pure HtmlUnit for functional testing of our web UI, and when I saw a link to HtmlFixture, I expected some more abstract wrapper with keywords imitating user actions in browser.

I see the 2 problems with HtmlUnit:

1) The resulting test tables are for programmers, they are not customer friendly (Element, Type, Focus Element, Set Value etc - not a business language)

2) It seems it does not work correctly for a form with submit buttons. HtmlFixture does not allow clicking a particular input type="submit", it allows submitting a form itself. As a result, it looks like the name/value of the submit button does not get passed to the server, and the form does not get processed properly.

So I started looking for existing solutions on similar problems and found your post in the yahoo group.
I like how your test table looks - this is about how I initially imagined the acceptance tests for web UI should look like. Would be nice to look at sources. Would you share it?
Have you seen the another alternative - JWebFixture (http://groups.yahoo.com/group/fitnesse/message/2103)? It's possibly close to your approach.


Best regards,
Anna

1:09 AM  
Blogger Max Ischenko said...

Hello Anna,

I share your attitude towards HtmlUnit, in webfixture I tried to make an API as high-level and customer-friendly as possible, even if it may be a bit less precise.

And yes, jWebUnit's API is similar (as far as you can get in Java-vs-FIT) so you may explore it instead. The problem with webfixture is that I don't use it now and hence do not evolve it. Of course, this may change if I get to work on web testing again...OTOH, if there is enough interest I could put it on some public subversion server and allow everyone to contribute.

What do you think?

If you provide an e-mail I can send sources directly to you.

1:52 PM  
Anonymous alabinsky said...

Max,

I'm very interested to look at your webfixture.

I've tried both the JWebFit and HtmlFixtue and none of them is not suitable four our needs as is. Unfortunately, JWebFit has strange assumptions hardcoded - e.g. that a field label always comes before the form input or that selecting a checkbox sets it's value to 'on', etc.

Ideally I would like to use/reuse/write a compact user friendly keywords language based on HtmlUnit (as in my personal opinion this is the best web testing JUnit extension at the moment).

I think it would be great if you decide to publish the fixture – at least people would have more options to select from. Also I like the idea about using svn server.

I've sent you a private email - looking forward to receive your fixture code in a reply.

Thanks,

Anna

9:38 PM  
Blogger Max Ischenko said...

Anna, sent it.

Btw, another option you may want to consider is Selenium. It become a hot topic recently

7:49 AM  
Anonymous pschrier@exoftware.com said...

HI Max,

I am playing with your tool, but one thing puzzles me. I open the Google site and want to have the 'Advertisment Prgrograms'-link clicked by WebFixture...how should I do that??? When I use
| click | Advertisment programs | Fitnesse claims that it is not a clickable item...what to do?

Peter

8:42 PM  
Anonymous Anonymous said...

Testing HTML code is a very difficult task. I read through the fixture code at fitnesse.org to figure out how to do it.

-------------------------
Need a phat deal?
http://deals.phatsite.com
-------------------------

3:29 AM  
Anonymous Anonymous said...

Hi Max!

I have tried your webfixture and like the high level and effective commands. I tried to write a test case with a input button that starts a javascript. I tried the code below, but without luck. It seems that the code does not succeed to submit the page. Do you have any idea?

:-)
Bård


!| org.maxischenko.fit.fixtures.WebPageActionFixture |
| go | https://wwtest/delta/portal/forsikring/ois/biltariffering.jsp?C=FORMHANDLER&p=pageBiltariffering&flow=bil&pp=gn |
| check | title | Gjensidige - Privat/Våre forsikringer/Bil, båt og MC|
|enter|firstname|Bård|
|enter|lastname|Nymoen|
|enter|pnbr|1111111111|
| select | input | value | Next | ! Next |
| click |
| select | input | value | Avbryt | ! Avbryt |

11:52 AM  
Anonymous Anonymous said...

Hey Max,

I just got a hold of your fixture and it looks promising but is there any documentation for it and how to use it??? Thanks

11:27 PM  
Blogger Max Ischenko said...

For documentation, check out WebFixturePublicAPI.java source file. I'm afraid that's the best docs currently available.

6:18 AM  
Blogger Max Ischenko said...

Hi Bård,

I'm afraid I can't tell from the glance why it doesn't work for you. Try to use 'check' after 'select input' to assert that you've actually selected proper control. You can also try to use 'submit' keyword.

6:22 AM  
Anonymous Anonymous said...

I'm not to good and submitting patches, but the following code makes errors better to track down. in the method
private void handleInvocationException(Parse cells, Throwable target add the following } else {
exception(cells, new FitFailureException(target.getMessage()));
}

6:16 PM  
Blogger H. Nakajima said...

Keep it up. I enjoy your nice blog. check out my data entry home base business
site. It pretty much covers data entry home base business
related stuff.

2:02 PM  

Post a Comment

<< Home