Hosted by:
Unit testing expertise at your fingertips!
Home | Discuss | Lists

Assertion Method

Please see page 362 of xUnit Test Patterns for the latest information.
How do we make tests self-checking?

Call a utility method to evaluate whether an expected outcome has been achieved.

Sketch Assertion Method embedded from Assertion Method.gif

A key part of writing Fully Automated Tests (see Goals of Test Automation on page X) is to make them Self-Checking Tests (see Goals of Test Automation) to avoid having to inspect the outcome of each test for correctness each time it is run. This involves finding a way to express the expected outcome in a way that can be verified automatically by the test itself.

Assertion Methods give us a way to express the expected outcome in a way that is both executable by the computer and useful to the human reader who can then use Tests as Documentation (see Goals of Test Automation).

How It Works

We encode the expected outcome of the test as a series of assertions that state what should be true for the test to pass. The assertions are realized as calls to Assertion Methods that encapsulate the mechanism that causes the test to fail. The Assertion Methods may be provided by the Test Automation Framework (page X) or by the test automater as Custom Assertions (page X).

Why We Do This

Encoding the expected outcome using Conditional Test Logic (page X) is very verbose and makes tests hard to read and understand. It is also much more likely to lead to Test Code Duplication (page X) and Buggy Tests (page X). Assertion Methods help us avoid all these issues by moving that complexity into reusable Test Utility Methods (page X) that can be verified to work correctly using Test Utility Tests (see Test Utility Method).

Implementation Notes

All the members of the xUnit family provide Assertion Methods but it is an area where there is a fair degree of variability. The key implementation considerations are:

Calling Built-in Assertion Methods

The way the Assertion Methods are called from within the Test Method (page X) varies from language to language and framework to framework. The language features determine what is possible and preferable while the framework builders chose which of several options to use. The names they choose for the Assertion Methods are influenced by how they chose to access them. The most common variations for accessing the Assertion Methods are:

Assertion Messages

Assertion Methods typically take an optional Assertion Message as a text parameter that is included in the output when the assertion fails. This allows the test automater to explain to the test maintainer exactly which Assertion Method failed and to better explain what should have occurred. The error detected by the test will be much easier to debug if the Assertion Method provides more information about why it failed. Choosing the right Assertion Method goes a long way to achieving this because many of the built in Assertion Methods provide useful diagnostic information about the values of the arguments. This is especially true for Equality Assertions.

One of the biggest difference between members of the xUnit family is where the optional Assertion Message is in the argument list. Most members tack it on to the end as an optional argument but at least one, JUnit makes it the first argument when it is present.

Choosing the Right Assertion

We have two goals for the calls to Assertion Methods in our Test Methods:

To achieve these goals we must strive to use the most appropriate Assertion Method. While the syntax and naming conventions vary from one member of the xUnit family to the next, most provide a basic set of assertions that fall into the following categories:

We need to verify different kinds of outcomes in a Fully Automated Test so most members of the xUnit family provide several different forms of Assertion Method:

Variation: Equality Assertion

Equality Assertions are the most common examples of Assertion Methods. They are used to compare the actual outcome with an expected outcome that is expressed in the form of a constant Literal Value (page X) or an Expected Object (see State Verification on page X) By convention, the expected value is specified first and the actual value follows it. The diagnostic message output by xUnit typically depends on this order. The equality of the two objects is usually determined by invoking the equals method on the expected object. If the SUT's definition of equals makes this impractical, we can either do Equality Assertions on individual fields of the object or we can use a Test-Specific Subclass (page X) that implements our test-specific equality as the Expected Object.

Variation: Fuzzy Equality Assertion

When we cannot guarantee an exact match due to variations in precision or expected variations in value, it may be appropriate to use a Fuzzy Equality Assertion. Typically, these look just like an Equality Assertion with the addition of an extra "tolerance" or "comparison map" parameter that specifies how close the actual argument must be to the expected one. The most common example of a Fuzzy Equality Assertion is the comparison of floating point numbers where the limitations of arithmetic precision needs to be accounted for by providing a tolerance (the maximum acceptable distance between the two values.)

I have used the same approach when comparing XML documents where direct string comparisons may result in failure due to certain fields having unpredictable content. In this case, the "fuzz" specification is a "comparison schema" that specifies which fields need to match or which ones should be ignored. This particular kind of equality assertion is very similar to asserting that a string conforms to a regular expression or other form of pattern matching.

Variation: Stated Outcome Assertion

Stated Outcome Assertions are a way of saying exactly what the outcome should be without passing an expected value as an argument. The outcome must be common enough to warrant a special Assertion Method. The most common examples of this are:

Stated Outcome Assertions are often used as Guard Assertions (page X) to avoid Conditional Test Logic.

Variation: Expected Exception Assertion

In languages that support block closures, we can use a variation of Stated Outcome Assertion that takes an additional parameter specifying the kind of exception we expect. We can use this Expected Exception Assertion to say "run this block and verify that the following exception is thrown." This is more compact than using a try/catch construct. Typical examples are:

Variation: Single Outcome Assertion

A Single Outcome Assertion always behaves the same. The most commonly used Single Outcome Assertion is fail which causes a test to be treated as a failure. It is most commonly used in two circumstances:

One circumstance in which we really should not be using Single Outcome Assertions is in Conditional Test Logic. There is almost never a good reason to include conditional logic in a Test Method as there is usually a more declarative way to do it using other styles of Assertion Methods. For example, use of Guard Assertions results in tests that are more easily understood and less likely to yield incorrect results.

Motivating Example

The following illustrates the kind of code that would be required for each item we wanted to verify if we did not have Assertion Methods. All we really want to do is

      if (x.equals(y)) {
         throw new AssertionFailedError( "expected: <" + x.toString() +
               "> but found: <" + y.toString() + ">");
      } else  { // OK, continue // ...}
Example UnsafeChecking embedded from java/com/xunitpatterns/misc/LifeWithoutAssertions.java

but this will cause a NullPointerException if x is null and it would be hard to distinguish this from an error in the SUT. So we have to put some guard clauses around this so that we always throw an AssertionFailedException:

      if (x == null) { // cannot do null.equals(null)
         if (y == null ) {  // they are both null so equal
         } else {
            throw new AssertionFailedError(
               "expected null but found: <" + y.toString() +">");
      } else if (!x.equals(y)) { // comparable but not equal !
      } // equal
Example BetterChecking embedded from java/com/xunitpatterns/misc/LifeWithoutAssertions.java

Yikes! That got pretty messy. And we'll have to do this for every attribute we want to verify? This is not good. There must be a better way.

Refactoring Notes

Luckily for us, the inventors of xUnit realized this and have already done the requisite Extract Method[Fowler] refactoring to create a library of Assertion Methods that we can call instead. We simply replace the mess of in-line if statements and thrown exceptions with a call to the appropriate Assertion Method. The example below is the code for the JUnit assertEquals method. Note that although the intent is the same as the code we wrote, it has been written in terms of guard clauses that identify when things are equal.

    * Asserts that two objects are equal. If they are not
    * an AssertionFailedError is thrown with the given message.*/
      failNotEquals(message, expected, actual);
Example EqualityAssertionImplementation embedded from java/junit/framework/Assertions.java

The method failNotEquals is a Test Utility Method that fails the test and provides a diagnostic assertion message.

Example: Equality Assertion

Here is the same assertion logic recoded to take advantage of JUnit's Equality Assertion:

      assertEquals( x, y );
Inline code sample

Here is the same assertion coded in C#. Note the classname qualifer and the resulting difference in the method naming:

      Assert.AreEqual( x, y );
Inline code sample

Example: Fuzzy Equality Assertion

To compare two floating point numbers (which are rarely ever really equal, we specify the acceptable different using a Fuzzy Equality Assertion:

      assertEquals( 3.1415, diameter/2/radius, 0.001);
      assertEquals( expectedXml, actualXml, elementsToCompare );
Example FuzzyEqualityAssertion embedded from java/com/xunitpatterns/misc/SampleAssertionUsage.java

Example: Stated Outcome Assertion

To insist that a particular outcome has occurred, we use a Stated Outcome Assertion such as:

      assertNotNull( a );
      assertTrue( b > c );
      assertNonZero( b );
Example StatedOutcomeAssertion embedded from java/com/xunitpatterns/misc/SampleAssertionUsage.java

Example: Expected Exception Assertion

This is an example of how we verify the correct exception was raised when we have blocks. In Smalltalk's SUnit, it looks like this:

         should: [Flight new mileage: -1122]
         raise: RuntimeError new 'Should have raised error'
Example ExpectedExceptionAssertionSUnit embedded from Smalltalk/Test Templates.st

The should: indicates the block of code to run (surrounded by square brackets) while the raise: specifies the expected exception object. In Ruby, it looks like this:

      assert_raises( RuntimeError, "Should have raised error")
                  {flight.setMileage(-1122) }
Example ExpectedExceptionAssertionRubyUnitCompact embedded from Ruby/TestTemplates.rb

Ruby also lets us use this more "control structure" style syntax by delimiting the block using do/end instead of curly braces:

   assert_raises( RuntimeError, "Should have raised error") do
Example ExpectedExceptionAssertionRubyUnit embedded from Ruby/TestTemplates.rb

Example: Single Outcome Assertion

To fail the test, use the Single Outcome Assertion:

      fail( "Expected an exception" );
Example SingleOutcomeAssertion embedded from java/com/xunitpatterns/misc/SampleAssertionUsage.java

Page generated at Wed Feb 09 16:39:45 +1100 2011

Copyright © 2003-2008 Gerard Meszaros all rights reserved

All Categories
Introductory Narratives
Web Site Instructions
Code Refactorings
Database Patterns
DfT Patterns
External Patterns
Fixture Setup Patterns
Fixture Teardown Patterns
Front Matter
Result Verification Patterns
Test Double Patterns
Test Organization
Test Refactorings
Test Smells
Test Strategy
Value Patterns
XUnit Basics
xUnit Members
All "XUnit Basics"
Test Method
--Four-Phase Test
Assertion Method
--Equality Assertion
--Fuzzy Equality Assertion
--Stated Outcome Assertion
--Expected Exception Assertion
--Single Outcome Assertion
--Assertion Message
Testcase Class
Test Runner
Testcase Object
Test Suite Object
--Test Discovery
--Test Enumeration
--Test Selection