mod_perl logo perl icon
previous page: Preparing mod_perl modules for CPANpage up: General Documentationnext page: Issuing Correct HTTP Headers

Running and Developing Tests with the Apache::Test Framework

mod_perl Pocket Reference

mod_perl Pocket Reference

By Andrew Ford
Writing Apache Modules with Perl and C

Writing Apache Modules with Perl and C

By Lincoln Stein, Doug MacEachern
Embedding Perl in HTML with Mason

Embedding Perl in HTML with Mason

By Dave Rolsky, Ken Williams
mod_perl2 User's Guide

mod_perl2 User's Guide

By Stas Bekman, Jim Brandt
Practical mod_perl

Practical mod_perl

By Stas Bekman, Eric Cholet
The mod_perl Developer's Cookbook

The mod_perl Developer's Cookbook

By Geoffrey Young, Paul Lindner, Randy Kobes

Table of Contents


The title is self-explanatory :)


This chapter discusses the Apache-Test framework, and in particular explains how to:

  1. run existing tests
  2. setup a testing environment for a new project
  3. develop new tests

For other Apache::Test resources, see the References section at the end of this document.


Basics of Perl Module Testing


If you have ever written or looked at the tests that come with most Perl modules, you'll recognize that Apache::Test uses the same concepts. The script t/TEST executes all the files ending with .t that it finds in the t/ directory. When executed, a typical test prints the following:

  1..3     # going to run 3 tests
  ok 1     # the first  test has passed
  ok 2     # the second test has passed
  not ok 3 # the third  test has failed

Every ok or not ok is followed by a number that identifies which sub-test has passed or failed.

t/TEST uses the Test::Harness module, which intercepts the STDOUT stream, parses it and at the end of the tests, prints the results of the tests: how many tests and sub-tests were run and how many passed, failed, or were skipped.

Some tests may be skipped by printing:



By default, print statements in the test script are filtered out by Test::Harness. If you want the test to print what it does (for example, to debug a test) use the -verbose option. So for example if your test does this:

  print "# testing : feature foo\n";
  print "# expected: $expected\n";
  print "# received: $received\n";
  ok $expected eq $received;

in the normal mode, you won't see any of these prints. But if you run the test with t/TEST -verbose, you will see something like this:

  # testing : feature foo
  # expected: 2
  # received: 2
  ok 2


A simpler approach is to use the Test::More module in your test scripts. This module offers many useful test functions, including diag, a function that automatically escapes and passes strings to print to bypass Test::Harness:

  use Test::More;
  diag "testing : feature foo\n";
  diag "expected: $expected\n";
  diag "received: $received\n";
  ok $expected eq $received;

In fact, for an example such as this, you can just use Test::More's is function, which will output the necessary diagnostics in the event of a test failure:

  is $received, $expected;



The Writing Tests section documents several helper functions that make simplify the writing of tests.

For more details about the Test::Harness module please refer to its manpage. Also see the Test and Test::More manpages for documentation of Perl's test suite.



In order to use Apache::Test it has to be installed first.

Install Apache::Test using the familiar procedure:

  % cd Apache-Test
  % perl Makefile.PL
  % make && make test && make install

If you install mod_perl 2.0, Apache::Test will be installed with it.


Running Tests


You can look at the modperl-2.0's or httpd-test's (perl-framework) testing environments, both of which use Apache::Test for their test suites.


Testing Options


  % t/TEST -help



Basic Testing

Running tests is just like for any CPAN Perl module; first we generate the Makefile file and build everything with make:

  % perl Makefile.PL [options]
  % make


  % make test

But this approach adds quite an overhead, since it has to check that everything is up to date (the usual make source change control). Therefore, you have to run it only once after make; for re-running the tests, it's faster to run them directly via:

  % t/TEST

When make test or t/TEST is run, all tests found in the t directory (files ending with .t are recognized as tests) will be run.


Individual Testing

To run a single test, simply specify it at the command line. For example, to run the test file t/protocol/echo.t, execute:

  % t/TEST protocol/echo

Notice that the t/ prefix and the .t extension for the test filenames are optional when you specify them explicitly. Therefore the following are all valid commands:

  % t/TEST   protocol/echo.t
  % t/TEST t/protocol/echo
  % t/TEST t/protocol/echo.t

The server will be stopped if it was already running and a new one will be started before running the t/protocol/echo.t test. At the end of the test the server will be shut down.


  % t/TEST -verbose protocol/echo


  % ./t/TEST modules protocol/echo

will run all the tests in t/modules/ directory, followed by t/protocol/echo.t test.


Repetitive Testing

By default, when you run tests without the -run-tests option, the server will be started before the testing and stopped at the end. If during a debugging process you need to re-run tests without the need to restart the server, you can start it once:

  % t/TEST -start-httpd

and then run the test(s) with -run-tests option many times:

  % t/TEST -run-tests

without waiting for the server to restart.

When you are done with tests, stop the server with:

  % t/TEST -stop-httpd

When the server is running, you can modify .t files and rerun the tests without restarting it. But if you modify response handlers, you must restart the server for changes to take an effect. However, if the changes are only to perl code, it's possible to arrange for Apache::Test to handle the code reload without restarting the server.

The -start-httpd option always stops the server first if any is running.

Normally, when t/TEST is run without specifying the tests to run, the tests will be sorted alphabetically. If tests are explicitly passed as arguments to t/TEST they will be run in the specified order.


Parallel Testing

Sometimes you need to run more than one Apache-Test framework instance at the same time. In this case you have to use different ports for each instance. You can specify explicitly which port to use using the -port configuration option. For example, to run the server on port 34343, do this:

  % t/TEST -start-httpd -port=34343

You can also affect the port by setting the APACHE_TEST_PORT evironment variable to the desired value before starting the server.

Specifying the port explicitly may not be the most convenient option if you happen to run many instances of the Apache-Test framework. The -port=select option helps such situations. This option will automatically select the next available port. For example if you run:

  % t/TEST -start-httpd -port=select


There is one problem that remains to be resolved, though. It's possible that two or more servers running -port=select will still decide to use the same port, because when the server is configured it only tests whether the port is available but doesn't call bind() immediately. This race condition needs to be resolved. Currently the workaround is to start the instances of the Apache-Test framework with a slight delay between them. Depending on the speed of your machine, 4-5 seconds can be a good choice, as this is the approximate the time it takes to configure and start the server on a quite slow machine.


Verbose Mode


  % t/TEST -verbose

In verbose mode, the test may print useful information, like what values it expects and what values it receives, given that the test is written to report these. In silent mode (without -verbose), these printouts are filtered out by Test::Harness. When running in verbose, mode usually it's a good idea to run only problematic tests in order to minimize the size of the generated output.

When debugging tests, it often helps to keep the error_log file open in another console, and see the debug output in the real time via tail(1):

  % tail -f t/logs/error_log

Of course this file gets created only when the server starts, so you cannot run tail(1) on it before the server starts. Every time t/TEST -clean is run, t/logs/error_log gets deleted; therefore, you'll have to run the tail(1) command again once the server starts.


Colored Trace Mode

If your terminal supports colored text you may want to set the environment variable APACHE_TEST_COLOR to 1 to enable any colored tracing when running in the non-batch mode. Colored tracing mode can make it easier to discriminate errors and warnings from other notifications.


Controlling the Apache::Test's Signal to Noise Ratio

In addition to controlling the verbosity of the test scripts, you can control the amount of information printed by the Apache::Test framework itself. Similar to Apache's log levels, Apache::Test uses these levels for controlling its signal to noise ratio:

  emerg alert crit error warning notice info debug

where emerg is the for the most important messages and debug is for the least important ones.

Currently, the default level is info; therefore, any messages which fall into the info category and above (notice, warning, etc) will be output. This tracing level is unrelated to Apache's LogLevel mechanism, which Apache-Test sets to debug in t/conf/httpd.conf and which you can override t/conf/

Let's assume you have the following code snippet:

  use Apache::TestTrace;
  warning "careful, perl on the premises";
  debug "that's just silly";

If you want to get only warning messages and above, use:

  % t/TEST -trace=warning ...

now only the warning message

  careful, perl on the premises

will be printed. If you want to see debug messages, you can change the default level using -trace option:

  % t/TEST -trace=debug ...

now the last example will print both messages.

By default the messages are printed to STDERR, but can be redirected to a file. Refer to the Apache::TestTrace manpage for more information.

Finally, you can use the emerg(), alert(), crit(), error(), warning(), notice(), info() and debug() methods in your client and server side code. These methods are useful when, for example, you have some debug tracing that you don't want to be printed during the normal make test or .Build test. However, if some users have a problem you can ask them to run the test suite with the trace level set to 'debug' and, voila, they can send you the extra debug output. Moreover, all of these functions use Data::Dumper to dump arguments that are references to perl structures. So for example your code may look like:

  use Apache::TestTrace;
  my $data = { foo => bar };
  debug "my data", $data;

and only when run with -trace=debug it'll output:

  my data
  $VAR1 = {
            'foo' => 'bar'

Normally it will print nothing.


Stress Testing


The Problem



The Solution


The Apache-Test framework provides a few options useful for stress testing.


Resolving Sequence Problems





Apache::TestSmoke Solution


You should create a small script to drive Apache::TestSmoke, usually t/SMOKE.PL. If you don't have it already, create it:

  use strict;
  use warnings FATAL => 'all';
  use FindBin;
  use Apache::TestSmoke ();

Usually Makefile.PL converts it into t/SMOKE while adjusting the perl path, but you can create t/SMOKE in first place as well.

t/SMOKE performs the following operations:

  1. Runs the tests randomly until the first failure is detected. Or non-randomly if the option -order is set to repeat or rotate.


  3. It reports all the successful reductions as it goes to STDOUT and report file of the format: smoke-report-<date>.txt.


  4. Goto 1 and run again using a new random seed, which potentially should detect different failures.


  1. Binary search: first try the upper half then the lower.


You can get the usage information by executing:

  % t/SMOKE -help


  % t/SMOKE

If you want to work on certain tests you can specify them in the same way you do with t/TEST:

  % t/SMOKE foo/bar foo/tar



-order=rotate is used just to override the default -order=random, since in this case we want to preserve the order. We also specify -times=1 for the same reason (override the default which is 50).



Finally, any other options passed will be forwarded to t/TEST as is.


RunTime Configuration Overriding

After the server is configured during make test or with t/TEST -config, it's possible to explicitly override certain configuration parameters. The override-able parameters are listed when executing:

  % t/TEST -help

Probably the most useful parameters are:

For a complete list of override-able configuration parameters see the output of t/TEST -help.


Request Generation and Response Options


The debugging options -debug and -breakpoint are covered in the Debugging Tests section.

For a complete list of available switches see the output of t/TEST -help.


Batch Mode

When running in the batch mode and redirecting STDOUT, this state is automagically detected and the no color mode is turned on, under which the program generates a minimal output to make the log files useful. If this doesn't work and you still get all the mess printed during the interactive run, set the APACHE_TEST_NO_COLOR=1 environment variable.


Setting Up Testing Environment

We will assume that you have setup your testing environment even before you have started coding the project, which is a very smart thing to do. Of course it'll take you more time upfront, but it'll will save you a lot of time during the project developing and debugging stages. The extreme programming methodology says that tests should be written before starting the code development.


Know Your Target Environment



Basic Testing Environment

So the first thing is to create a package and all the helper files, so later we can distribute it on CPAN. We are going to develop an Apache::Amazing module as an example.

  % h2xs -AXn Apache::Amazing
  Writing Apache/Amazing/
  Writing Apache/Amazing/Makefile.PL
  Writing Apache/Amazing/README
  Writing Apache/Amazing/
  Writing Apache/Amazing/Changes
  Writing Apache/Amazing/MANIFEST



We want our module to live in the Apache-Amazing directory, so we do:

  % mv Apache/Amazing Apache-Amazing
  % rmdir Apache

From now on the Apache-Amazing directory is our working directory.

  % cd Apache-Amazing

We don't need the, as we are going to create a whole testing environment:

  % rm

We want our package to reside under the lib directory, so later we will be able to do live testing, without rerunning make every time we change the code:

  % mkdir lib
  % mkdir lib/Apache
  % mv lib/Apache

Now we adjust lib/Apache/ to look like this:

  package Apache::Amazing;
  use strict;
  use warnings;
  use Apache2::RequestRec ();
  use Apache2::RequestIO ();
  sub handler {
      my $r = shift;
      return Apache::OK;
  ... pod documentation goes here...

The only thing our modules does is set the text/plain header and respond with "Amazing!".

Next, you have a choice to make. Perl modules typically use one of two build systems: ExtUtils::MakeMaker or Module::Build.

ExtUtils::MakeMaker is the traditional Perl module build system, and comes preinstalled with Perl. It generates a tradiational Makefile to handle the build process. The code to generate the Makefile resides in Makefile.PL.

Module::Build is a new build system, available from CPAN, and scheduled to be added to the core Perl distribution in version 5.10, with the goal of eventually replacing ExtUtils::MakeMaker. Module::Build uses pure Perl code to manage the build process, making it much easier to override its behavior to perform special build tasks. It is also more portable, since it relies on Perl itself, rather than the make utility.

So the decision you need to make is which system to use. Most modules on CPAN use ExtUtils::MakeMaker, and for most simple modules it is more than adequate. But more and more modules are moving to Module::Build so as to take advantage of its new features. Module::Build is the future of Perl build systems, but ExtUtils::MakeMaker is likely to be around for some time to come.

Fortunately, Apache::Test makes it easy to use either build system.

Next we create the test suite, which will reside in the t directory:

  % mkdir t

First we create t/TEST.PL which will be automatically converted into t/TEST during perl Makefile.PL stage:

  use strict;
  use warnings FATAL => 'all';
  use lib qw(lib);
  use Apache::TestRunPerl ();

This script assumes that Apache::Test is already installed on your system and that Perl can find it. If not, you should tell Perl where to find it. For example you could add:

  use lib qw(Apache-Test/lib);

to t/TEST.PL, if Apache::Test is located in a parallel directory.

As you can see we didn't write the real path to the Perl executable, but #!perl. When t/TEST is created the correct path will be placed there automatically.

Note: If you use Apache::TestMB in a Build.PL script, the creation of the t/TEST.PL script is optional. You only need to create it if you need it to do something special that the above example does not.

Next we need to prepare extra Apache configuration bits, which will reside in t/conf:

  % mkdir t/conf

We create the t/conf/ file, which will be automatically converted into t/conf/extra.conf before the server starts. If the file has any placeholders like @documentroot@, these will be replaced with the real values specific for the Apache server used for the tests. In our case, we put the following configuration bits into this file:

  # where Apache::Amazing can be found
  PerlSwitches -I@ServerRoot@/../lib
  # preload the module
  PerlModule Apache::Amazing
  <Location /test/amazing>
      SetHandler modperl
      PerlResponseHandler Apache::Amazing

As you can see, we just add a simple <Location> container and tell Apache that the namespace /test/amazing should be handled by the Apache::Amazing module running as a mod_perl handler. Notice that:

      SetHandler modperl


      SetHandler perl-script

which also works for mod_perl 2.0.

Now we can create a simple test:

  use strict;
  use warnings FATAL => 'all';
  use Apache::Amazing;
  use Apache::Test;
  use Apache::TestUtil;
  use Apache::TestRequest 'GET_BODY';
  plan tests => 2;
  ok 1; # simple load test
  my $url = '/test/amazing';
  my $data = GET_BODY $url;
  ok t_cmp(
           "basic test",

Now create the README file.

  % touch README

Don't forget to put in the relevant information about your module, or arrange for ExtUtils::MakeMaker::WriteMakefile() to do this for you with:

      dist  => {
          PREOP => 'pod2text lib/Apache/ > $(DISTVNAME)/README',

Or for Module::Build to generate the README with:

  my $build = $build_pkg->new(
      create_readme => 1,

In these cases, README will be created from the documenation POD sections in lib/Apache/, but the file must exist for make dist or ./Build.PL dist to succeed.

And finally, we adjust or create the MANIFEST file, so we can prepare a complete distribution. Therefore we list all the files that should enter the distribution including the MANIFEST file itself:

  Makefile.PL # and/or Build.PL

You can automate the creation or updating of the MANIFEST file using make manifest with Makefile.PL or ./Build manifest with Build.PL.

That's it. Now we can build the package. But we need to know the location of the apxs utility from the installed httpd server. We pass its path as an option to Makefile.PL or Build.PL. To build, test, and install the module with Makefile.PL, do this:

  % make
  % make test
  All tests successful.

To install the package run:

  % make install

Now we are ready to distribute the package on CPAN:

  % make dist

This build command will create the package which can be immediately uploaded to CPAN. In this example, the generated source package with all the required files will be called: Apache-Amazing-0.01.tar.gz.

The same process can be accomplished with Buiild.PL like so:

  % ./Build
  % ./Build test
  All tests successful.
  % ./Build install
  % ./Build dist

The only thing that we haven't done and hope that you will do is to write the POD sections for the Apache::Amazing module, explaining how amazingly it works and how amazingly it can be deployed by other users.


Extending Configuration Setup

Sometimes you need to add extra httpd.conf configuration and perl startup-specific code to your project that uses Apache::Test. This can be accomplished by creating the desired files with an extension .in in the t/conf/ directory and running:

  panic% t/TEST -config

which for each file with the extension .in will create a new file, without this extension, convert any template placeholders into real values and link it from the main httpd.conf. The latter happens only if the file have the following extensions:

Files whose name matches the following pattern:


will be included very last in httpd.conf. This is especially useful if you want to include Apache directives that would need a running Perl interpreter (see When Does perl Start To Run) without conflicting with Apache::Test's use of PerlSwitches.

Make sure that you don't try to create, it is not going to work, since httpd.conf is already generated by Apache-Test.

As mentioned before the converted files are created, any special tokens in them are getting replaced with the appropriate values. For example the token @ServerRoot@ will be replaced with the value defined by the ServerRoot directive, so you can write a file that does the following:
  PerlSwitches -I@ServerRoot@/../lib

and assuming that the ServerRoot is ~/modperl-2.0/t/, when my-extra.conf will be created, it'll look like:


The valid tokens are defined in %Apache::TestConfig::Usage and also can be seen in the output of t/TEST -help's configuration options section. The tokens are case insensitive.

For a complete list see the Apache::TestConfig manpage.


Special Configuration Files

Some of the files in the t/conf directory have a special meaning, since the Apache-Test framework uses them for the minimal configuration setup. But they can be overriden:


Inheriting from System-wide httpd.conf

Apache::Test tries to find a global httpd.conf file and inherit its configuration when autogenerating t/conf/httpd.conf. For example it picks LoadModule directives.

It's possible to explicitly specify which file to inherit from using the -httpd_conf option. For example during the build:

  % perl Makefile.PL -httpd_conf /path/to/httpd.conf

or with Build.PL:

  % perl Build.PL -httpd_conf /path/to/httpd.conf

or during the configuration:


Certain projects need to have a control of what gets inherited. For example if your global httpd.conf includes a directive:

  LoadModule apreq_module "/home/joe/apache2/modules/"

And you want to run the test suite for Apache::Request 2.0, inheriting the above directive will load the pre-installed and not the newly built one, which is wrong. In such cases it's possible to tell the test suite which modules shouldn't be inherited. In our example Apache-Request has the following code in t/TEST.PL:

  use base 'Apache::TestRun';
  sub pre_configure {
      my $self = shift;
      # Don't load an installed mod_apreq

it subclasses Apache::TestRun and overrides the pre_configure method, which excludes the module mod_apreq.c from the list of inherited modules (notice that the extension is .c).


Apache::Test Framework's Architecture



In some cases the response part of the test runs the test inside itself, so all it requires from the request part of the test, is to generate the request and print out a complete response without doing anything else. In such cases Apache::Test can auto-generate the client part of the test for you.


Developing Response-only Part of a Test

If you write only a response part of the test, Apache::Test will automatically generate the corresponding test part that will generated the response. In this case your test should print 'ok 1', 'not ok 2' responses as usual tests do. The autogenerated request part will receive the response and print them out automatically completing the Test::Harness expectations.


  my $client_file = catfile 't',

Notice that the leading /^test/ part is removed. Here are some examples of that translation:

  t/response/MyApache/       => t/myapache/write.t
  t/response/TestApache/     => t/apache/write.t
  t/response/TestApache/Mar/ => t/apache/mar/write.t

If we look at the autogenerated test t/apache/write.t, we can see that it starts with the warning that it has been autogenerated, so you won't attempt to change it. Then you can see the trace of the calls that generated this test, in case you want to figure out how the test was generated. And finally the test loads the Apache::TestRequest module, imports the GET shortcut and prints the response's body if it was successful. Otherwise it dies to flag the problem with the server side. The latter is done because there is nothing on the client side, that tells the testing framework that things went wrong. Without it the test will be skipped, and that's not what we want.

  use Apache::TestRequest 'GET_BODY_ASSERT';
  print GET_BODY_ASSERT "/TestApache__write";



So t/response/TestApache/ becomes: /TestApache__write.

Now a simple response test may look like this:

  package TestApache::write;
  use strict;
  use warnings FATAL => 'all';
  use constant BUFSIZ => 512; #small for testing
  use Apache2::RequestIO;
  use Apache2::RequestRec; 
  sub handler {
      my $r = shift;

[F] Apache2::Const is mod_perl 2.0's package, if you test under 1.0, use the Apache::Constants module instead [/F].

The configuration part for this test will be autogenerated by the Apache-Test framework and added to the autogenerated file t/conf/httpd.conf when make test or ./Build test or t/TEST -configure is run. In our case the following configuration section will be added:

  <Location /TestApache__write>
     SetHandler modperl
     PerlResponseHandler TestApache::write

You should remember to run:

  % t/TEST -configure


Also notice that if you manually add configuration the <Location> path can't include ':' characters in the first segment, due to Apache security protection on WinFU platforms. So please make sure that you don't create entries like:

  <Location /Foo::bar/>

You can include ':' characters in the further segments, so this is OK:

  <Location /tests/Foo::bar/>



Developing Response and Request Parts of a Test




  use strict;
  use warnings FATAL => 'all';
  use Apache::Test;
  use Apache::TestUtil;
  use Apache::TestRequest 'GET_BODY';
  plan tests => 1; # plan one test.
  t_debug("connecting to $hostport");
  my $received = GET_BODY "/TestApache__cool";
  my $expected = "COOL";
  ok t_cmp(
           "testing TestApache::cool",

See the Apache::TestUtil manpage for more info on the t_cmp() function (e.g. it works with regexs as well).

And the corresponding response part:

  package TestApache::cool;
  use strict;
  use warnings FATAL => 'all';
  sub handler {
      my $r = shift;

Again, remember to run t/TEST -clean before running the new test so the configuration will be created for it.

As you can see the test generates a request to /TestApache__cool, and expects it to return "COOL". If we run the test:

  % ./t/TEST t/apache/cool

We see:

  All tests successful.


  # connecting to localhost:8529
  # testing : testing TestApache::cool
  # expected: COOL
  # received: COOL
  ok 1
  All tests successful.

So in case in our simple test we have received something different from COOL or nothing at all, we can immediately see what's the problem.

The name of the request part of the test is very important. If Apache::Test cannot find the corresponding test for the response part it'll automatically generate one and in this case it's probably not what you want. Therefore when you choose the filename for the test, make sure to pick the same Apache::Test will pick. So if the response part is named: t/response/TestApache/ the request part should be named t/apache/cool.t. See the regular expression that does that in the previous section.


Developing Test Response Handlers in C

If you need to exercise some C API and you don't have a Perl glue for it, you can still use Apache::Test for the testing. It allows you to write response handlers in C and makes it easy to integrate these with other Perl tests and use Perl for request part which will exercise the C module.


The httpd-test ASF project is a good example to look at. The C modules are located under: httpd-test/perl-framework/c-modules/. Look at c-modules/echo_post/echo_post.c for a nice simple example. mod_echo_post simply echos data that is POSTed to it.

The differences between vairous tests may be summarized as follows:



Request and Response Methods

If you have LWP (libwww-perl) installed its LWP::UserAgent serves as an user agent in tests, otherwise Apache::TestClient tries to emulate partial LWP functionality. So most of the LWP documentation applies here, but the Apache-Test framework provides shortcuts that hide many details, making the test writing a simple and swift task. Before using these shortcuts Apache::TestRequest should be loaded, and its import() method will fetch the shortcuts into the caller namespace:

  use Apache::TestRequest;

Request generation methods issue a request and return a response object (HTTP::Response if LWP is available). They are documented in the HTTP::Request::Common manpage. The following methods are available:

These are two special methods added by the Apache-Test framework:


  $content = $res->content;

to retrieve the content fo the respose and:


to retrieve specific headers.

Refer to the HTTP::Response manpage for a complete reference of these and other methods.


  $res = GET $url;
  $str = $res->content;

simply use:

  $str = GET_BODY $url;


Other Request Generation helpers

META: these methods need documentation

Request part:




Starting Multiple Servers

By default the Apache-Test framework sets up only a single server to test against.

In some cases you need to have more than one server. If this is the situation, you have to override the maxclients configuration directive, whose default is 1. Usually this is done in t/TEST.PL by subclassing the parent test run class and overriding the new_test_config() method. For example if the parent class is Apache::TestRunPerl, you can change your t/TEST.PL to be:

  use strict;
  use warnings FATAL => 'all';
  use Apache::TestRunPerl ();
  package MyTest;
  our @ISA = qw(Apache::TestRunPerl);
  # subclass new_test_config to add some config vars which will be
  # replaced in generated httpd.conf
  sub new_test_config {
      my $self = shift;
      return $self->SUPER::new_test_config;


Multiple User Agents

By default the Apache-Test framework uses a single user agent which talks to the server (this is the LWP user agent, if you have LWP installed). You almost never use this agent directly in the tests, but via various wrappers. However if you need a second user agent you can clone these. For example:



Hitting the Same Interpreter (Server Thread/Process Instance)

When a single instance of the server thread/process is running, all the tests go through the same server. However if the Apache::Test framework was configured to to run a few instances, two subsequent sub-tests may not hit the same server instance. In certain tests (e.g. testing the closure effect or the BEGIN blocks) it's important to make sure that a sequence of sub-tests are run against the same server instance. The Apache-Test framework supports this internally.

Here is an example from ModPerl::Registry closure tests. Using the counter closure problem under ModPerl::Registry:

  #!perl -w
  my $counter = 0;
  sub counter {
      #warn "$$";
      print ++$counter;


  my $url = "/same_interp/cgi-bin/";
  # should be no closure effect, always returns 1
  ok t_cmp(
      "the closure problem is there",
  sub req {
      my ($same_interp, $url) = @_;
                                                    \&GET, $url);
      return $res ? $res->content : undef;

In this test we generate two requests to cgi-bin/ and expect the returned value to increment for each new request, because of the closure problem generated by ModPerl::Registry. Since we don't know whether some other test has called this script already, we simply check whether the substraction of the two subsequent requests' outputs gives a value of 1.




Writing Tests

All the communications between tests and Test::Harness which executes them is done via STDOUT. I.e. whatever tests want to report they do by printing something to STDOUT. If a test wants to print some debug comment it should do it starting on a separate line, and each debug line should start with #. The t_debug() function from the Apache::TestUtil package should be used for that purpose.


Defining How Many Sub-Tests Are to Be Run




  print "1..5\n";

In Apache::Test this is done as follows:

  use Apache::Test;
  plan tests => 5;


Skipping a Whole Test

Sometimes when the test cannot be run, because certain prerequisites are missing. To tell Test::Harness that the whole test is to be skipped do:


The optional comment after # skipped will be used as a reason for test's skipping. Under Apache::Test the optional last argument to the plan() function can be used to define prerequisites and skip the test:

  use Apache::Test;
  plan tests => 5, $test_skipping_prerequisites;

This last argument can be:



Skipping Numerous Tests

Just like you can tell Apache::Test to run only specific tests, you can tell it to run all but a few tests.

If all files in a directory t/foo should be skipped, create:

  print "1..0\n";

Alternatively you can specify which tests should be skipped from a single file t/SKIP. This file includes a list of tests to be skipped. You can include comments starting with # and you can use the * wildcharacter for multiply files matching.


  # skip all files in protocol
  # skip basic cgi test
  # skip all filter/input_* files



The second pattern skips a single test modules/cgi.t. Note that you shouldn't specify the leading t/. And the .t extension is optional, so you can say:

  # skip basic cgi test

The last pattern tells Apache::Test to skip all the tests starting with filter/input.


Reporting a Success or a Failure of Sub-tests

After printing the number of planned sub-tests, and assuming that the test is not skipped, the test runs its sub-tests and each sub-test is expected to report its success or failure by printing ok or not ok respectively followed by its sequential number and a new line. For example:

  print "ok 1\n";
  print "not ok 2\n";
  print "ok 3\n";

In Apache::Test this is done using the ok() function which prints ok if its argument is a true value, otherwise it prints not ok. In addition it keeps track of how many times it was called, and every time it prints an incremental number, therefore you can move sub-tests around without needing to remember to adjust sub-test's sequential number, since now you don't need them at all. For example this test snippet:

  use Apache::Test;
  use Apache::TestUtil;
  plan tests => 3;
  ok "success";
  t_debug("expecting to fail next test");
  ok "";
  ok 0;

will print:

  ok 1
  # expecting to fail next test
  not ok 2
  not ok 3



Skipping Sub-tests

If the standard output line contains the substring # Skip (with variations in spacing and case) after ok or ok NUMBER, it is counted as a skipped test. Test::Harness reports the text after the pattern # Skip\S*\s+ as a reason for skipping. So you can count a sub-test as a skipped as follows:


or using the Apache::Test's skip() function which works similarly to ok():

  skip $should_skip, $test_me;

so if $should_skip is true, the test will be reported as skipped. The second argument is the one that's sent to ok(), so if $should_skip is true, a normal ok() sub-test is run. The following example represent four possible outcomes of using the skip() function:

  use Apache::Test;
  plan tests => 4;
  my $ok     = 1;
  my $not_ok = 0;
  my $should_skip = "foo is missing";
  skip $should_skip, $ok;
  skip $should_skip, $not_ok;
  $should_skip = '';
  skip $should_skip, $ok;
  skip $should_skip, $not_ok;

now we run the test:

  ok 1 # skip foo is missing
  ok 2 # skip foo is missing
  ok 3
  not ok 4
  # Failed test 4 in skip_subtest_1.t at line 13

As you can see since $should_skip had a true value, the first two sub-tests were explicitly skipped (using $should_skip as a reason), so the second argument to skip didn't matter. In the last two sub-tests $should_skip had a false value therefore the second argument was passed to the ok() function. Basically the following code:

  $should_skip = '';
  skip $should_skip, $ok;
  skip $should_skip, $not_ok;

is equivalent to:

  ok $ok;
  ok $not_ok;

However if you want to use t_cmp() or some other function call in the arguments to ok() that won't quite work since the function will be always called no matter whether the first argument will evaluate to a true or a false value. For example, if you had a function:


and now you want to run this sub-test if module HTTP::Date is available, changing it to:


will still run t_cmp() even if HTTP::Date is not available. Therefore it's probably better to code it in this way:

  if (eval {require HTTP::Date}) {
  else {
      skip "Skip HTTP::Date not found";


Running only Selected Sub-tests


  use Apache::Test;
  plan tests => 4;
  sok {1};
  sok {0};
  sok sub {'true'};
  sok sub {''};


  % ./t/TEST -verbose skip_subtest_2 1 3
  ok 1
  ok 2 # skip skipping this subtest
  ok 3
  ok 4 # skip skipping this subtest
  ok, 2/4 skipped:  skipping this subtest
  All tests successful, 2 subtests skipped.

Only the sub-tests 1 and 3 get executed.


  ok 1 # skip askipping this subtest
  not ok 2
  # Failed test 2
  ok 3
  not ok 4
  # Failed test 4



Todo Sub-tests

In a safe fashion to skipping specific sub-tests, it's possible to declare some sub-tests as todo. This distinction is useful when we know that some sub-test is failing but for some reason we want to flag it as a todo sub-test and not as a broken test. Test::Harness recognizes todo sub-tests if the standard output line contains the substring # TODO after not ok or not ok NUMBER and is counted as a todo sub-test. The text afterwards is the explanation of the thing that has to be done before this sub-test will succeed. For example:


In Apache::Test this can be done with passing a reference to a list of sub-tests numbers that should be marked as todo sub-test:


In this example sub-tests 3 and 6 will be marked as todo sub-tests.


Making it Easy to Debug



  use Apache::Test;
  use Apache::TestUtil;
  plan tests => 1;
  t_debug("testing feature foo");
  $expected = "a good value";
  $received = "a bad value";
  t_debug("expected: $expected");
  t_debug("received: $received");
  ok defined $received && $received eq $expected;

If in this example $received gets assigned a bad value string, the test will print the following:

  % t/TEST debug_comments
  debug_comments....FAILED test 1


  % t/TEST -verbose debug_comments
  # testing feature foo
  # expected: a good value
  # received: a bad value
  not ok 1




but no fear, Apache::TestUtil comes to help. The function t_cmp() does all the work for you:

  use Apache::Test;
  use Apache::TestUtil;
  ok t_cmp(
      "a good value",
      "a bad value",
      "testing feature foo");

t_cmp() will handle undef'ined values as well, so you can do:

  my $expected;

Finally you can use t_cmp() for regex comparisons. This feature is mostly useful when there may be more than one valid expected value, which can be described with regex. For example this can be useful to inspect the value of $@ when eval() is expected to fail:

  eval {foo();}
  if ($@) {

which is the same as:

  eval {foo();}
  if ($@) {
      t_debug("func eval");


Tie-ing STDOUT to a Response Handler Object


  sub handler {
      my $r = shift;
      return Apache2::Const::OK;

now the client should print the response to STDOUT for Test::Harness processing.

If the response handler is configured as:

  SetHandler perl-script

STDOUT is already tied to the request object $r. Therefore you can now rewrite the handler as:

  use Apache::Test;
  sub handler {
      my $r = shift;
      plan tests => 2;
      ok "true";
      ok "";
      return Apache2::Const::OK;



  use Apache::Test;
  sub handler {
      my $r = shift;
      plan $r, tests => 2;
      ok "true";
      ok "";
      return Apache2::Const::OK;

Yet another alternative to handling the test framework printing inside response handler is to use Apache::TestToString class.


  use Apache::Test;
  sub handler {
      my $r = shift;
      plan tests => 2;
      ok "true";
      ok "";
      my $output = Apache::TestToString->finish;
      return Apache2::Const::OK;

In this example Apache::TestToString intercepts and buffers all the output from and can be retrieved with its finish() method. Which then can be printed to the client in one shot. Internally it calls Apache::Test::test_pm_refresh() to make sure plan(), ok() and other functions() will work correctly more than one test is running under the same interpreter.


Helper Functions

Apache::TestUtil provides other helper functions, useful for writing tests, not mentioned in this tutorial:


See the Apache::TestUtil manpage for more information.


Auto Configuration


If your tests are comprised from the request and response parts, Apache::Test automatically adds the configuration section for each response handler it finds. For example for the response handler:

  package TestResponse::nice;
  ... some code

it will put into t/conf/httpd.conf:

  <Location /TestResponse__nice>
      SetHandler modperl
      PerlResponseHandler TestResponse::nice

If you want to add some extra configuration directives, use the __DATA__ section, as in this example:

  package TestResponse::nice;
  ... some code
  PerlSetVar Foo Bar

These directives will be wrapped into the <Location> section and placed into t/conf/httpd.conf:

  <Location /TestResponse__nice>
      SetHandler modperl
      PerlResponseHandler TestResponse::nice
      PerlSetVar Foo Bar

This autoconfiguration feature was added to:


Forcing Configuration Sections into the Top Level

If some directives are supposed to go to the base configuration, i.e. not to be automatically wrapped into <Location> block, you should use a special <Base>..</Base> block:

      PerlSetVar Config ServerConfig
  PerlSetVar Config LocalConfig

Now the autogenerated section will look like this:

  PerlSetVar Config ServerConfig
  <Location /TestResponse__nice>
     SetHandler modperl
     PerlResponseHandler TestResponse::nice
     PerlSetVar Config LocalConfig

As you can see the <Base>..</Base> block has gone. As you can imagine this block was added to support our virtue of laziness, since most tests don't need to add directives to the base configuration and we want to keep the configuration sections in tests to a minimum and let Perl do the rest of the job for us.


Bypassing Auto-Configuration

In more complicated cases, usually when virtual hosts containers are involved, the auto-configuration might stand in a way and you will simply want to bypass it. If that's the case, put the configuration inside the <NoAutoConfig>..</NoAutoConfig> container. For example:

      <VirtualHost TestPreConnection::note>
          PerlPreConnectionHandler TestPreConnection::note
          <Location /TestPreConnection__note>
              SetHandler modperl
              PerlResponseHandler TestPreConnection::note::response

Notice, that the internal sections will be still parsed, tokens @var@ will be substituted and VirtualHost sections will be rewritten with an automatically assigned port number and ServerName.


Virtual Hosts

Apache::Test automatically assigns an unused port for the virtual host configuration. Just make sure that you use the package name in the place where you usually specify a hostname:port value. For example for the following package:

  package MyApacheTest::Foo;
  <VirtualHost MyApacheTest::Foo>
      <Location /test_foo>

After running:

  % t/TEST -conf

Check the auto-generated t/conf/httpd.conf and you will find what port was assigned. Of course it can change when more tests which require a special virtual host are used.


  use Apache::TestRequest;
  my $module = "MyApacheTest::Foo;";


Running Pre-Configuration Code

Sometimes you need to setup things for the test. This usually includes creating directories and files, and populating the latter with some data, which will be used at request time. Instead of performing that operation in the client script every time a test is run, it's usually better to do it once when the server is configured. If you wish to run such a code, all you have to do is to add a special subroutine APACHE_TEST_CONFIGURE in the response package (assuming that that response package exists). When server is configured (t/TEST -conf) it scans all the response packages for that subroutine and if found runs it.

APACHE_TEST_CONFIGURE accepts two arguments: the package name of the file this subroutine is defined in and the Apache::TestConfig configuration object.


  package TestDirective::perlmodule;
  use strict;
  use warnings FATAL => 'all';
  use Apache::Test ();
  use Apache2::RequestRec ();
  use Apache2::RequestIO ();
  sub handler {
      my $r = shift;
      my ($class, $self) = @_;
      my $vars = $self->{vars};
      my $magic = __PACKAGE__;
      my $content = <<EOF;
  package ApacheTest::PerlModuleTest;
  \$ApacheTest::PerlModuleTest::MAGIC = '$magic';
      my $file = catfile $target_dir,
          'perlmodule-vh', 'ApacheTest', '';
      $self->writefile($file, $content, 1);



Controlling the Configuration Order

Sometimes it's important in which order the configuration section of each response package is inserted. Apache::Test controls the insertion order using a special token APACHE_TEST_CONFIG_ORDER. To decide on the configuration insertion order, Apache::Test scans all response packages and tries to match the following pattern:






Now if a new test is added and it has to be the first, add to this new test a token with a negative value whose absolute value is higher than -150, e.g.:










Threaded versus Non-threaded Perl Test's Compatibility


      ... configuration bits

Apache::Test will start the server with -DPERL_USEITHREADS if the Perl is ithreaded.

For example PerlOptions +Parent is valid only for the threaded perl, therefore you have to write:

      # a new interpreter pool
      PerlOptions +Parent


  if (have_perl 'ithreads'){
      # ithread specific code



Retrieving the Server Configuration Data


  use Apache::Test;


Module Magic Number

The following code retrieves the major and minor MMN numbers.

  my $info = $cfg->{httpd_info};
  my $major = $info->{MODULE_MAGIC_NUMBER_MAJOR};
  my $minor = $info->{MODULE_MAGIC_NUMBER_MINOR};

For example for MMN 20011218:0, this code prints:

  major=20011218, minor=0


Debugging Tests



Under C debugger


  % t/TEST -debug

in another shell:

  % t/TEST -run-tests

then the -debug shell will have a (gdb) prompt, type where for stacktrace:

  (gdb) where

You can change the default debugger by supplying the name of the debugger as an argument to -debug. E.g. to run the server under ddd:

  % ./t/TEST -debug=ddd

META: list supported debuggers

If you debug mod_perl internals you can set the breakpoints using the -breakpoint option, which can be repeated as many times as needed. When you set at least one breakpoint, the server will start running till it meets the ap_run_pre_config breakpoint. At this point we can set the breakpoint for the mod_perl code, something we cannot do earlier if mod_perl was built as DSO. For example:


will set the modperl_cmd_switches and modperl_cmd_options breakpoints and run the debugger.


  % ./t/TEST -debug -breakpoint=modperl_hook_init

In fact -breakpoint automatically turns on the debug mode, so you can run:

  % ./t/TEST -breakpoint=modperl_hook_init


Under Perl debugger


  % perl -debug

the flow control gets passed to the Perl debugger, which allows you to run the program in single steps and examine its states and variables after every executed statement. Of course you can set up breakpoints and watches to skip irrelevant code sections and watch after certain variables. The perldebug and the perldebtut manpages are covering the Perl debugger in fine details.

The Apache-Test framework extends the Perl debugger and plugs in LWP's debug features, so you can debug the requests. Let's take test apache/read from mod_perl 2.0 and present the features as we go:

META: to be completed

run .t test under the perl debugger

  % t/TEST -debug perl t/modules/access.t




  % t/TEST -debug lwp t/modules/access.t





To get Start the server under strace(1):

  % t/TEST -debug strace

The output goes to t/logs/strace.log.

Now in a second terminal run:

  % t/TEST -run-tests

Beware that t/logs/strace.log is going to be very big.



Using Apache::Test to Speed up Project Development


First of all, your Perl modules need to reside under the lib directory, the same way they reside in blib/lib. In the section Basic Testing Environment, we've already arranged for that. If resides in the top-level directory, it's not possible to perform 'require Apache::Amazing'. Only after running make or ./Build wil the file be moved to blib/lib/Apache/, which is when we can load it. But you don't want to run make or ./Build every time you change the file. It's both annoying and error-prone, since at times you'd make a change, try to verify it, and it will appear to be wrong for no obvious reason. What will really have happend is that you just forgot to run make or ./Build and the server was testing against the old unmodified version in blib/lib. Of course, if you always run make test or ./Build test, it'll always do the right thing, but it's not the most effecient approach to undertake when you want to test a specific test and you do it every few seconds.

The following scenario will make you a much happier Perl developer.

First, we need to instruct Apache::Test to modify @INC, which we could do in t/conf/ or t/conf/, but the problem is that you may not want to keep that change in the released package. There is a better way, if the environment variable APACHE_TEST_LIVE_DEV is set to a true value, Apache::Test will automatically add the lib/ directory if it exists. Executing:

 % APACHE_TEST_LIVE_DEV=1 t/TEST -configure

will add code to add /path/to/Apache-Amazing/lib to @INC in t/conf/ This technique is convenient since you don't need to modify your code to include that directory.

Second, we need to configure mod_perl to use Apache::Reload--to automatically reload the module when it's changed--by adding following configuration directives to t/conf/

  PerlModule Apache2::Reload
  PerlInitHandler Apache2::Reload
  PerlSetVar ReloadAll Off
  PerlSetVar ReloadModules "Apache::Amazing"

(For more information about Apache::Reload, depending on the mod_perl generation, refer to the mod_perl 1.0 documentation or the Apache2::Reload manpage for mod_perl 2.0.)

now we execute:

  % APACHE_TEST_LIVE_DEV=1 t/TEST -configure

which will generate t/conf/extra.conf and start the server:

  % t/TEST -start

from now on, we can modify Apache/ and repeatedly run:

  % t/TEST -run basic

without restarting the server.


Writing Tests Methodology

META: to be completed


When Tests Should Be Written


Other Webserver Regression Testing Frameworks


Got a question?

Post it to the Apache-Test dev list. The list is moderated, so unless you are subscribed to it it may take some time for your post to make it to the list.

For more information see:

For list archives and subscribing information, please see: Apache-Test dev list








