Schedule and Events


Thursday, October 30, 2008

When should a test run unattended? - III

First off, I've revised the title of of the series. I'm all for automating work that can be described and /precisely/ evaluated.

For example, let's say you have a PowerOf function. To test it, you could write a harness that takes input from the keyboard and prints the results, or you could write something like this:

is(PowerOf(2,1),1, "Two to the first is two");
is(PowerOf(2,2),4, "Two to the second is four");
is(PowerOf(3,3),27, "Three to the third is twenty-seven");
is(PowerOf(2,-1,undef,"PowerOf doesn't handle negative exponents yet");
is(PowerOf(2,2.5,undef,"PowerOf doesn't handle fractional exponents yet");

And so on.

When you add fractional or negative exponents, you can add new tests and re-run all the old tests, in order.

That is to say, this test can now run unattended and it will be very similar to what you would manually. Not completely - because if the powerOf function takes 30 seconds to calculate the answer, which is unacceptable, it will still eventually "Green Bar" - but hopefully, when you run it by hand, you notice this problem. (And if you are concerned about speed, you could wrap the tests in timer-based tests.)

Enter The GUI

As soon as we start talking about interactive screens, the number of things the human eye evaluates goes up. Wayyy up. Which brings us back to the keyword or screen capture problem - either the software will only look for problems I specify, or it will look for everything.

Let's talk about a real bug in the field

The environment: Software as a service web-based application that supports IE6, IE7, Firefox 2, Firefox 3, and Safari. To find examples, I searched bugzilla for "IE6 transparent", where we've had a few recently. (I do not mean to pick on IE; I could have searched for Safari or FF and got a similar list.) That does bring up an interesting problem: Most of the bugs below looked just fine in other browsers.

Here are some snippets from actual bug reports.

1) To reproduce, just go to IE6 and resize your browser window to take up about half your screen. Then log into dashboard, and see "(redacted element name)" appear too low and extra whitespace in some widget frames.

2) Page includes shows missing image in place of "Edit" button in IE6 and IE7

3) In IE6 only, upload light box shows up partly hidden when browser is not maximized.

4) In IE6 and IE7, comment's editor has long vertical and horizontal scroll bar.

5) In IE6 at editor UI, there is a thick blue spaces between the buttons and rest of the editor tools

6) To reproduce, in IE6, create some (redacted), then check out the left-most tab of (redacted 2). The icons for type of even are not the same background color as the widget itself. (see attachment)

All of these bugs were caught by actual testers prior to ship. I do not think it is reasonable to expect these tests to be automated unless you were doing record/playback testing. Now, if you were doing record/playback testing, you'd have to run the tests manually first, in every browser combination, and they'd fail, so you'd have to run them again and again until the entire sub-section of the application passed. Then you'd have a very brittle test that worked under one browser and one operating system.

That leaves writing the test after the fact, and, again, you'll get no help from keyword-driven frameworks like Selenium - "Whitepace is less than a half and inch between elements X and Y" simply isn't built into the tool, and the effort to add it would be prohibitive. If you wanted to write automated tests after the bugs were found, you'd have to use a traditional record/playback tool and now have two sets of tests.

That brings up a third option - slideshow tests that are watched by a human being, or that record periodic screen captures that a human can compare, side-by-side, with yesterday's run. We do this every iteration at Socialtext to good effect, but those tests aren't run /unattended/. Thus I change the name of this series.

I should also add, that problems like "too much whitespace" or "a button is missing but there is a big red X you can push" are fundamentally different from a crash or timeout. So if you have a big application to test, it might be a perfectly reasonable strategy to make hundreds of thousands of keyword-driven tests that make sure the basic happy-path of the application returns correct results (of the results you can think of when you write the tests.)

To Recap

We have discussed unit and developer-facing test automation along with three different GUI-test driving strategies. We found that the GUI-driving, unattended strategies are really only good for regression - making sure what worked yesterday still works today. I've covered some pros and cons for each, and found a half-dozen real bugs from the field that we wouldn't reasonably expect these tests to cover.

This brings up a question: What percentage of bugs are in this category, and how bad are they, and how often do we have regressions, anyway?

More to come.

Monday, October 27, 2008

Programming Parables





Here it is.


Thursday, October 23, 2008

Sometimes, words aren't enough






Go check it out.


Wednesday, October 22, 2008

On Being An "Old Dude"











The rest ... aren't.


I think you get the point.

Monday, October 20, 2008

Asperger's Syndrome




















--Matt Heusser

Cloud Computing is the new XML

In April of 2000 I took a development course and got a free copy of XML Magazine. I didn't get what the actual value of the technology was. A few months later, I realized that there was nothing to get - if XML had value, the guys hyping the magazine didn't know it yet.

And if you don't get the same feeling listening to the gurus talk about cloud computing, I suspect you haven't been listening close enough.

But first, the Good News
Just like the XML, Cloud Computing does have some places where it can add value. I argue that, in the next few yeas, we will see some generally useful applications and niches for cloud computing. Eventually, over time, some organizations will be be able to to 'give up' their data centers and turn on web servers like a we turn on electricity or tap water - but anyone familiar with virtual web hosting allready has a deal like that. Eventually, it might be possible to place our servers and databases in the grid and turn up the number of servers when demand spikes. Still, Tim Chou wrote "The End Of Software" four years ago, and the sad fact is that that capability is still years out. What do I believe about cloud computing today?

The Bad News
- Cloud Computing will have limited applications compared to the over-hype it has now,
- It will take years to realize those applications,
- Exactly what you should be doing with cloud computing will probably be very different from what todays 'visionaries' are telling you
- The 'visionaries' who are currently hyping cloud computing will probably shake out of the market before the true value and applicability of cloud computing is realized

It would be nice to be proven wrong. For the time being, I believe the smart money is against the hype machine.

Cloud Computing is the new XML.

Remember: You heard it here first.

Friday, October 17, 2008

Blank Sheet of Paper Syndrome

(No, I haven't forgotten about Test Automation. I'm just trying to leverage my time in the best possible way. Most of this post came out of a recent discussion on the SW-IMPROVE discussion list ...)

One thing that I found helpful when gathering requirements is something I call "blank sheet of paper syndrome." That is to say, if you give a (brand new) analyst a template, they will often react with something like this:

"oh, easy, I can do this. Project Name, Project Manager, Executive Sponsor, Initiated Date, Today's Date, Desired Date ... I can fill this out." (... 30 minutes pass ...) "Here's your requirements doc."

On the other hand, if you give them a blank sheet of paper, you are more likely to have a reaction like this:

"But ... well ... this is blank! I have no idea what to write! I don't really know what the customer desires! I had better go find out!"

In my experience, this second reaction is much more likely to result in figuring out the /essential/ requirements of a software system, instead of gathering requirements that "all look like each other."

The argument /for/ requirements templates is that without the template you would forget something. My personal conclusion is that, for the vast majority of projects I have worked on, I will gladly run the risk of forgetting something on the template, in trade for the benefit of, hopefully, capturing something more essential on a personally-written document.

Now, I would be remiss if I did not add that the Planning Game in Extreme Programming - where you have no person in the middle and negotiate the requirements, is one valid implementation of blank sheet of paper concept. You cards could even have a slot for title, points, priority, and description, and I wouldn't be offended. :-)

In other words, Jerry Weinberg's Rule of three, in that you should strive for at least three options for every problem, still applies. "No Template/Blank Sheet" and "All Template/No Customization Sheet" is a false dichotomy - you can always do more or less, allow some customization, have a half-dozen questions instead of a three-page template, and so on.

All that I am saying here is that in the push for stable/predictable/repeatable, some organizations try to make all projects look the same. Sometimes, for the project team, that can do more harm than good.

Thursday, October 16, 2008

So what's a Privateer Scholar, anyway?

A few months back, I changed my LinkedIn title from dev/tester (or whatever it was) to "Privateer Scholar."

The title is derived from James Bach, who took it from Buckminster Fuller's Operating Manual for Spaceship Earth.

Here's the basic idea in Bach's words:

* A buccaneer-scholar is anyone whose love of learning is not muzzled, yoked or shackled by any institution or authority; whose mind is driven to wander and find its own voice and place in the world.

* This way of being has sometimes been called autodidact, individualist, anarchist, non-conformist, contrarian, bohemian, skeptic, hacker, hippie, slacker, seeker, philosoph, or free thinker. None of those fit for me.

Now, Bach refers to himself as a buccaneer because he is an independent contractor/consultant - he's really on his own.

I, on the other hand, like to try to work within organizations. I work within a specific authority (Socialtext, or Calvin College, or Maybe BZMedia) against it's enemies (competition - and at Calvin, the competition is called "Ignorance"). In piratical terms, for the time being - I seek a letter of marquee.

For more detail, see Bach's presentation on the subject.

Monday, October 13, 2008

When should a test be automated - II

Before we can dive in, let's take a step back.






click_ok, field_last_name
type_ok, field_last_name, heusser
click_ok, field_first_name
type_ok, field_first_name, matthew
click_ok, submit_button
wait_for_element_present_ok, //body, hello matthew heusser, 30000





The question is what, when, and how much.

Stay tuned.

Friday, October 10, 2008

When should a test be automated - I







More to come.

Wednesday, October 08, 2008

Complete test automation is the Holy Grail?








Let me put that in testing terms:




Monday, October 06, 2008

New Annoucements up




Something's gotta give.




Warm Regards,

--matt heusser