Why use more that XSLT for testing XSLT?

Why should you prefer an XSLT unit testing framework that uses more that just XSLT? Two reasons: xsl:message and multiple output documents.

Most XSLT unit testing frameworks are written purely in XSLT; that is, the framework uses an XSLT processor and XSLT stylesheet to run the tests (or a massaged form of the tests) on the stylesheet under test. The test result output is typically XML that is processed by another stylesheet to produce HTML for viewing.

Using XSLT to test XSLT may be partly explained by XSLT being an XSLT practitioner’s favourite tool and partly by Eric van der Vlist’s XSLT-based XSLTunit being the first known framework. Jeni Tennison’s possibly unnamed framework is also purely XSLT (though XSLT 2.0 instead of the XSLT 1.0 in XSLTunit), and it inspired the Tennison Tests framework, which automates running the multiple tests but hasn’t tinkered with the XSLT nature of the tests, and also XTC.

By comparison, the frameworks that rely on compiling code to run tests are, it seems, much less well known. They include Juxy, which compiles Java code for JUnit tests, and XMLUnit, which has versions in Java for use with JUnit and C# for use with NUnit.

IMO, the extra-XSLT frameworks have the advantage that you can (or should be able to) make assertions about aspects of the stylesheet to which a pure XSLT framework is either blind or oblivious.

Inasmuch as you typically use xsl:message to output messages when there’s an error in the source document or there’s something that the stylesheet can’t handle, you should want to make assertions about whether or not a message has been emitted. This is particularly true when you are writing “dirty” tests that you expect will trigger error handling (as opposed to “clean” tests that you expect to work), and even more true when you expect to execute a xsl:message that has ‘terminate="yes"‘.

Pure XSLT frameworks don’t provide a way to tell when a message is emitted, and any <xsl:message terminate="yes"> will terminate the testing framework along with the stylesheet under test, leaving you with nothing. A good extra-XSLT framework will let you make assertions about xsl:message and will not evaporate when the stylesheet terminates itself.

When a stylesheet may create multiple result documents, you probably want to make assertions about the existence and content of those result documents. That could be left to a post-process, but if the file names and the content relate to the content of the source document, it is simpler and more self-contained when you could make your assertions in the same unit tests as you use to make assertions about the rest of the operation of the stylesheet. A pure XSLT framework can’t provide any indication that a result document was or was not created, though it may be possible to access a secondary result document as part of the unit tests (provided the XSLT processor finishes writing the secondary result document while the transformation-that-is-the-running-of-the-unit-tests is still in progress). An extra-XSLT framework typically lets you make assertions after the stylesheet-under-test has finished.

So while there are some aspects of XSLT processing, such as what happens on a xsl:message, that you as a stylesheet writer don’t need to concern yourself about, you as a stylesheet tester do want to know about them, so you need more that just an XSLT processor in the framework for your unit tests.

2 thoughts on “Why use more that XSLT for testing XSLT?

  1. Tony,

    Another good reason for going outside an XSLT-based framework is to test how stylesheet parameters should change the output.

    I have a dream of having an platform-neutral XSLT test document format which could be implemented in any language. One method would be a transformation into an XProc pipeline, which you then run. XProc has the advantage of being able to set parameters, catch errors (messages), and passes on multiple result documents as one of the outputs of the XSLT step. I have no doubt that something similar could be achieved today through a transformation into an ant script.

    By the way, I’ve been doing some work on a new Behavior-Driven-Development-based framework, which I have named(!) XSpec. More to come soon…


  2. Jeni,

    Thanks for the hint about parameters. You can do that with Juxy and with UTF-X (http://utf-x.sourceforge.net/). Juxy also lets you set global variables (I think at least one of the XSLT frameworks does too). It’s not covered in the XMLUnit documentation, but inasmuch as XMLUnit just runs a JAXP transformation (and presumably does something similar for C#), setting parameters should be possible with XMLUnit, too.

    I think we all have dreams of making the hub format for testing XSLT (for a very small value of “all”). I have a XML format for Juxy tests that I’ve considered could be usefully transformed into tests for other frameworks.

    I guess any framework that can run a stylesheet multiple times and recover from terminating stylesheets would do the job. One issue for an XProc pipeline would be how to generate reports such that you can see the results for tests that terminated alongside results for tests that didn’t.

    The Behavior-Driven-Development stuff that I’ve seen talks in the same breath about using mock objects when testing. XSLT isn’t exactly object oriented, and stylesheets seldom make any pretense of presenting an API, so in addition to being generally interested in seeing XSpec when it’s done, I’d be interested in seeing what it would say about testing XSLT.

Comments are closed.