Wednesday, 11 January 2012 by haemoglobin
For a recent project I introduced SpecFlow to automate acceptance tests in true BDD style which I have been rather pleased with. SpecFlow’s HTML output report is very good for reporting the current behaviour and state of the system to stakeholders – an example SpecFlow report can be seen here.
Using TeamCity as our Continuous Integration platform (highly recommended by the way, I’m a bigfan of TeamCity and is free for smaller teams), what we wanted was to have the SpecFlow tests run by TeamCity after every check-in, and the SpecFlow output report displayed in an easily accessible tab of the build result.
In order to demonstrate how this can be done I’ve stripped it right back from the ground up using an example SpecFlow project which is available on GitHub.
These steps assume you have git installed, however the basic idea will work with any source control system.
- Download an install TeamCity, SpecFlow and NUnit.
- From a command prompt / git bash shell – navigate to where you want to clone/checkout the SpecFlow examples (e.g c:\git), and run:
git clone https://github.com/techtalk/SpecFlow-Examples.git
This will create the directory c:\git\SpecFlow-Examples containing the repository’s source. Being a DVCS however, this directory is a git repository itself that we can point TeamCity directly at for our example.
- In BowlingKata\BowlingKata-Nunit add a new batch file RunSpecFlow.bat with the following contents:
"C:\Program Files (x86)\NUnit 2.5.10\bin\net-2.0\nunit-console.exe"
=TestResult.txt /xml=TestResult.xml /framework=net-4.0 Bowling.Specflow\bin\Debug\Bowling.Specflow.dll
"C:\Program Files (x86)\TechTalk\SpecFlow\specflow.exe"
IF NOT EXIST TestResult.xml GOTO FAIL
IF NOT EXIST TestResult.html GOTO FAIL
echo ##teamcity[buildStatus status=
EXIT /B 1
The first line will run the SpecFlow tests which in essence are standard NUnit tests under the hood. The reason we need to run it like this from the command line instead of using the TeamCity inbuilt NUnit runner is that we need to have TestResult.txt and TestResult.xml available to specflow.exe for it to generate the report in the next line.
There are a couple of extra checks at the end to see if the output files exist otherwise fail the TeamCity build in case of catastrophic error running the commands. Step 10 will let TeamCity fail the build if any test fails.
Note that your installation locations to NUnit and SpecFlow may vary. In the real world, we actually included NUnit and SpecFlow as part of our repository so things would work on any build agent.
- Commit this new file to the repository with:
git add RunSpecFlow.bat
git commit –m “Added RunSpecFlow.bat”
- In TeamCity, add a new project “SpecFlow Integration Test”.
- Create a git VCS root called SpecFlow Examples with the Fetch URL of C:\git\SpecFlow-Examples or whatever you cloned the SpecFlow Examples repository to in step 2 above. All other settings can be left as the default.
- In the VCS settings section, tick “Clean all files before build”.
The reason for this is due to the last lines in the batch file checking for the existence of the output files. If we don’t clean the agent first, the old test output files will still be hanging around from the time before (if running on the same build agent) so we won’t get a failure if the files failed to be created for some reason. It’s slightly slower having this option on, but in practice I’ve had less strange issues with build agents if I always start with a clean checkout.
- Create a configuration in the project called “Run SpecFlow Tests”. In the artifacts path, add the following:
BowlingKata\BowlingKata-Nunit\TestResult.html=>. This will copy the SpecFlow output report into the artifacts for the build.
- Add an MSBuild step to compile the BowlingKata\BowlingKata-Nunit\Bowling.sln solution using the .NET 4.0 MSBuild version.
- Add a Command Line step using the working directory BowlingKata\BowlingKata-Nunit and our command executable RunSpecflow.bat
- Add a Build Feature of type XML report processing with BowlingKata\BowlingKata-Nunit\TestResult.xml in the monitoring rules section. This will allow TeamCity to display the number of tests run (with pass/fail counts) and will fail the build if any tests fail. Without this, the build will always be green.
- Navigate to TeamCity Administration (top right) –> Server Configuration –> Report TabsCreate a new Report Tab with:
This tells TeamCity that whenever it sees an artifact named TestResult.html at the root level of the output artifacts, add a new tab to the build output called SpecFlow Results, and display TestResult.html within it.
- Run the build!
Have a look at the lovely output below (I purposely made a test fail to show how it would look):
Of course, the next step is to just add a TeamCity trigger so all of this runs on every check-in to the repository.
I hope this shows how you can get SpecFlow integrated into TeamCity which I think is very effective, especially if your clients have access to your TeamCity server directly as a way of reporting progress or state of the system.