

Friday, January 17, 2014

A test passes or fails through what?

  1. 1.
    move or cause to move in a specified direction.
    "he passed through towns and villages"
    synonyms:goproceedmoveprogress, make one's way, traveldriveflyMore
  2. 2.
    go past or across; leave behind or on one side in proceeding.
    "on the way to the station she passed a cinema"
  1. 1.
    an act or instance of moving past or through something.
    "repeated passes with the swipe card"
  2. 2.
    a success in an examination, test, or course.
    "an A-level pass in Music"

If your meaning of Pass is Noun(2) then your notion of Pass in Software Testing maybe weak. If you were to think about Pass in Software Testing as Verb(2) then you are giving yourselves an awesome power to your testing.

Frequently asked questions

Q: How many tests passed?
Q: How many tests failed?
Q: Did that test pass?
Q: Did that test fail?

Not so frequently asked questions

Q: What does a test need to pass through to be called as pass or fail?
A: A test needs an expected result or an oracle to pass through to determine pass or fail.

Q: How does Pradeep think when he exploratory tests?
A: He constructs a test (idea - a heuristic) and runs them across several oracles allowing himself to find more problems per test (heuristic) run.

Q: How does this help Pradeep achieve better test coverage?
A: If he were to run the test with just one oracle (or expected result, you dummy) he would achieve limited coverage and limit himself from finding the bugs he can per test run.

Q: Can you give an example?
A: Sure.

Let us take a test that needs to be run to determine if the GUI writes to the database when a Submit button is pressed. The most obvious oracles is - the values provided in the GUI by the user should be written to database. This can be considered a Requirements or Functional Oracle. The time it takes to write can be another oracle. My test may pass through Requirements and Functional but may not pass through Time Oracle. If it fails the Time oracle then I need to investigate if this is about my internet connection speeds or about the code or about the server. That investigation can help me to make recommendations of what speeds of internet are required to make the write faster. Now, if there is a time out implementation to the write command then I could try delaying it and seeing what happens to the data on the GUI considering that the user has spent 20 minutes typing into several forms before hitting the submit button (which we always do. Always. Let me repeat - ALWAYS). If that fails, then we have tested the test against Recovery / Error / Data Retention / User oracles.

All this takes hardly any noticeable time. I pass through a test through so many oracles to make my Pass a Power Pass. Scripted tests are Power-less tests because there are one (or maybe two) oracles at max people have to pass it through to determine Pass/Fail.

My tests passes or fails but not the way you think it passes but the way I see it pass through my powerful construct of oracles.

No comments: