Test Driven .NET Development with FitNesse

When FitNesse is running as a Windows service or on a dedicated server, it is easy to integrate testing into automated build tools. FitSharp package also contains a test runner intended for running tests externally, using local DLLs and a remote FitNesse server.

To run a FitNesse test or suite with TestRunner, execute the following command:

Runner.exe -r fitnesse.fitserver.TestRunner,dotnet2\fit.dll Server Port TestPage

Server is the name or IP address of the machine where FitNesse is running; if you are running FitNesse on the same computer as the tests, it is localhost. Port is the port on which FitNesse is running; if you installed FitNesse as suggested in Setting up FitNesse this is 8888. TestPage is a test name or a test suite. For example, the following command executes the TicketReviewTests suite:

C:\services\FitNesse> dotnet2\Runner.exe -r fitnesse.fitserver.TestRunner,dotnet2\fit.dll localhost 8888 TicketReviewTests

Always execute TestRunner from the main FitNesse folder so that relative paths to DLLs in test scripts point to the same files as when running from FitNesse.

	Why does TestRunner not see my DLLs?
	If you start TestRunner on a remote machine, it may have problems finding DLLs from the paths specified in the FitNesse page. Double-check first that tests work OK from FitNesse. If they do, then try putting DLLs into the same folder as Runner.exe. Another option is to specify additional DLLs after the test page name, as the last argument of Runner.exe (include a semi-colon on the end).

To run FitNesse tests from a NAnt^[25] script, use the exec task to start Runner.exe. The server name and port probably won't change often, so I suggest creating a utility NAnt script that expects the test name to be passed as a parameter and has other arguments defined as local properties. Here is what the script looks like:

scripts/runfitnesse.build

2   <property name="fitnesse.dir" value="c:\services\fitnesse" />
3   <property name="fitnesse.server" value="localhost" />
4   <property name="fitnesse.port" value="8888" />
5   <target name="test">
6   	<exec program="${fitnesse.dir}\dotnet2\Runner.exe" 
7   	commandline="-r fitnesse.fitserver.TestRunner,dotnet2\fit.dll ${fitnesse.server} ${fitnesse.port} ${fitnesse.test}" 
8   	workingdir="${fitnesse.dir}"/>
9   </target>

It is fairly easy to integrate FitNesse with CruiseControl.NET, the most popular continuous integration server for .NET. However, there is a trick involved in formatting results. The .NET test runner does not support the XML format required by CruiseControl.NET out of the box, so we first have to run the tests and store results into a file. Then we have to format the results using the java test runner, which produces something CruiseControl.NET can read. Runner.exe has an additional option, -results outfile.txt, which causes it to store the output into outfile.txt. We need to execute the following two commands within CruiseControl.NET to integrate FitNesse tests:

C:\services\FitNesse> dotnet2\Runner.exe -r fitnesse.fitserver.TestRunner,dotnet2\fit.dll -results c:\temp\fitnesseres.txt localhost 8888 TicketReviewTests.WinningsRecordedCorrectly
C:\services\FitNesse> java -cp fitnesse.jar fitnesse.runner.FormattingOption c:\temp\fitnesseres.txt xml c:\temp\fitnesseres.xml localhost 8888 TicketReviewTests.WinningsRecordedCorrectly

Although we could use the executable^[26] task from the CruiseControl.NET tasks block directly, it is better to wrap these steps into a NAnt script instead. This allows us to control whether the build should break or not (acceptance tests should typically not break the build, but unit and component tests should). Using NAnt also allows us to print some debug messages and automatically record test execution timings.

We need to create two NAnt tasks that run tests and filter results:

scripts/runfitnesse.build

10  <target name="test">
11  	<echo message="running tests ${fitnesse.test}" />
12  	<delete file="${output.file}" />
13  	<delete file="${format.file}" />
14  	<exec program="${fitnesse.dir}\dotnet2\TestRunner.exe" 
15  	commandline="-results ${output.file} -r fitnesse.fitserver.TestRunner,dotnet2\fit.dll ${fitnesse.server} ${fitnesse.port} ${fitnesse.test}" 
16  	workingdir="${fitnesse.dir}"
17  	failonerror="true"/>
18  	<echo message="tests ${fitnesse.test} complete" />
19  </target>
20  <target name="format">
21  	<echo message="formatting ${fitnesse.test} results" />
22  	<delete file="${format.file}" />
23  	<exec program="java.exe" 
24  	workingdir="${fitnesse.dir}"
25  	commandline="-cp ${fitnesse.dir}\fitnesse.jar fitnesse.runner.FormattingOption ${output.file} xml ${format.file} ${fitnesse.server} ${fitnesse.port} ${fitnesse.test}" failonerror="false"/>
26  	<echo message="formatting ${fitnesse.test} results into ${format.file} complete" /gt;
27 </target>

Notice the failonerror="true" part in the first task. Set the value to false for acceptance tests, so that they do not break your build.

The first step should be executed from the tasks block and the second from the publishers block, because we want the results to be formatted even when a test fails (even more in this case, because we want to know what went wrong). The results can then be included into the build report in the merge^[27] block. Remember to include the xmllogger publisher at the end of the publishers block,^[28] to enable CruiseControl.NET to display regular build results.

FitNesse was designed to allow easy collaboration and testing, not really to be a 24/7 Internet server. At first it typically froze in our environment after a day or two of continuous testing, so we included a service restart before tests. Restarting the server also makes sure that FitNesse is using the latest versions of test scripts and not the ones cached in memory. When FitNesse is installed as a service (see Can FitNesse start automatically with Windows?), you can use net.exe to stop and re-start it. Don't use sc.exe because this command is asynchronous, and you definitely don't want the tests to start executing before the server is up.

Here is a complete CruiseControl.NET project configuration:

scripts/ccnet.config

1   <cruisecontrol>
2   <project name="Continuous-Test">
3       <workingDirectory>w:\ccnetbuild\source\test</workingDirectory>
4       <artifactDirectory>w:\ccnetbuild\artifact-cont\test</artifactDirectory>
5       <tasks>
6       <exec>
7               <executable>net.exe</executable>
8               <buildArgs>stop ccnetfitnesse</buildArgs>
9       </exec>
10      <exec>
11              <executable>net.exe</executable>
12              <buildArgs>start ccnetfitnesse</buildArgs>
13      </exec>
14      <nant>
15                <buildFile>w:\ccnetbuild\source\build\runfitnesse.build</buildFile>
16                <buildTimeoutSeconds>300000</buildTimeoutSeconds>
17                <buildArgs>-D:output.file=c:\temp\fitnesse-tx.log -D:format.file=c:\temp\fitnesse-tx.xml -D:fitnesse.test=TicketReviewTests</buildArgs>
18              <targetList><target>test</target></targetList>
19      </nant>
20      </tasks>
21       <publishers>
22              <nant>
23                <buildFile>w:\ccnetbuild\source\build\runfitnesse.build</buildFile>
24                <buildTimeoutSeconds>300000</buildTimeoutSeconds>
25                <buildArgs>-D:output.file=c:\temp\fitnesse-tx.log -D:format.file=c:\temp\fitnesse-tx.xml -D:fitnesse.test=TicketReviewTests</buildArgs>
26              <targetList><target>format</target></targetList>
27              </nant>
28        <merge>
29               <files>
30                    <file>c:\temp\fitnesse-tx.xml</file>
31               </files>
32           </merge>
33        <xmllogger />
34      </publishers>
35  </project>
36  </cruisecontrol>

If you build code and run tests in the same CruiseControl.NET project, remember to delete FitNesse results (both raw and and formatted) before the build. If you do not do this, when a build fails, the publishers block merges old FitNesse results with the failed build report. This gives a misleading report that the build failed but tests passed.

Automated build tools can signal errors and notify developers when the build fails. We can use this feature to keep track of the health of the repository code automatically. However, different types of tests call for different notifications.

Basic tests should ideally be executed on every source code change without anyone pushing the button, so it is a good idea to have these tests run as part of the basic build. This calls for basic tests to run quickly, as explained in Don't mix quick and slow tests. If basic tests are not passing, the complete build should fail. Ideally, this should stop people from committing unchecked code into the repository. In practice, however, unchecked code does somehow find its way into the source code repository from time to time. I strongly suggest setting the build server to run all quick tests after any code change in the repository, to get the fastest feedback.

Component and integration tests run more slowly so it is not practical to run them on every change. We typically execute them every couple of hours on a central system, so developers don't have to care about them. The build system should, however, fail and notify developers when one of these tests fail (and it is when, not if).

Tests that guide the development process typically do not pass for most of the time. When they all pass, the work is done. So, there's no reason to sound the fire alarm when an acceptance test fails. However, publishing acceptance test results periodically is a good idea, because it helps to answer the most frequent question we get from project managers: “how are we doing?”.