Demo Showcase Suite – A History – Part 6

Honestly, I’m trying to do these once a day, but I took 3 days off for a vacation, and it’s taken me a couple of days to get caught back up.  Here’s the next installment on my series of goals that I had when we stated on Demo Showcase.

4. I didn’t want to get to a week from the delivery date, need to make a change, and have to regression test everything by hand.

As I write these entries, what I thought were distinct topics, all seem to be related.  This is talking about unit testing, right?  No.  Wrong. Different topic altogether.  Application regression testing has always been a thorn in my side.  I’ve never had the chance to put a good test automation tool to work for me to help with UI regression testing.  Along with unit tests, I wanted to have a tool that would let the QA’s quickly and efficiently automate their tests so that the tests they did in March would be instantaneously rerunable in July.  Sure, unit testing will get you part way there, but nothing beats a human looking at the screen and seeing if the right result is a reasonable result.

If there was a swing and a miss on a goal on this project, this was the big one.  In fact, I didn’t even get the bat of my shoulder.  In the post mortem discussions I’ve been having with the team, this one popped up to the top of the list for the QAs as something that directly affected their personal satisfaction with the project and with their jobs.  With the rapid pace that we were kicking out builds each day, the QA’s rarely got a chance to work on a stable version of the application for more than a few hours, and some tests would take an hour to go through by hand if they didn’t run into bugs.  They had a ‘happy path’ test they repeated each time, and there were many versions of the application where  they never got to complete the full test.  If we were to look at the number of hours spent testing on this project, I would think that we spent far more hours testing than the value we got from it.  That is, since so much time was spent testing the same thing over and over, little time was available to test edge cases until the build really stabilized.

So coming out of this project, we still have this as an open item to find an automated regression test suite that works for what we do.  I don’t think you can really evaluate the return on this investment prior to becoming proficient in the use of the tool, so it may take a few projects before we really see the benefits.  But it is definitely something we will be carefully considering before the end of the year, and likely before the next release cycle.