Generating SpecFlow reports for your TeamCity builds

http://jamesheppinstall.wordpress.com/2013/04/25/generating-specflow-reports-for-your-teamcity-builds/

You may or may not be aware that SpecFlow has the ability to produce some reports giving information about the features and steps after a test run has finished. The test execution report contains details about the SpecFlow test features, scenarios and steps. The step definition report shows the steps and their statuses, including unused steps and steps with no implementation.

The SpecFlow documentation details how the reports can be generated using SpecFlow.exe at the command line. Whilst it is useful to be able to manually generate these reports, it would be better if we could generate them as part of the TeamCity build process and make the reports available from within the build details of TeamCity. This makes the reports more accessible to all team members and especially useful when showing your product owner the scenarios covered by your acceptance test suite.

Lets run through getting things up and running.

1. Running the tests using nunit-console.exe

In order to generate the SpecFlow reports you must first run the tests using nunit-console.exe in order to generate the correct output files to feed into the report generation. The SpecFlow documentation tells us this is the command we need to run.

nunit-console.exe /labels /out=TestResult.txt /xml=TestResult.xml bin\Debug\BookShop.AcceptanceTests.dll

As we need to run nunit-console.exe it is not possible to use TeamCity’s NUnit runner. There are a few different runner types we could run this command such as Command line or Powershell, but the one that best fits our needs is probably MsBuild, but as always the same results can be achieved in lots of different ways.

In Visual Studio create a build file. I have given mine the imaginative name of nunit.build

<?xml version="1.0" encoding="utf-8" ?>

<Project ToolsVersion="3.5" DefaultTarget="Compile"
xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

<PropertyGroup>
    <NUnitHome>C:/Program Files (x86)/NUnit 2.6.2</NUnitHome>
    <NUnitConsole>&quot;$(NUnitHome)\bin\nunit-console.exe&quot;</NUnitConsole>
    <testResultsTxt>&quot;$(teamcity_build_checkoutDir)/TestResult.txt&quot;</testResultsTxt>
    <testResultsXml>&quot;$(teamcity_build_checkoutDir)/TestResult.xml&quot;</testResultsXml>
    <projectFile>&quot;$(teamcity_build_checkoutDir)\CrazyEvents.AcceptanceTests\CrazyEvents.AcceptanceTests.csproj&quot;</projectFile>
</PropertyGroup>
 
<Target Name="RunTests">
  <Exec Command="$(NUnitConsole) /labels /out=$(testResultsTxt) /xml=$(testResultsXml) $(projectFile)"/>
</Target>  
    
</Project>

I am not an MSBuild expert and it took me a while to get the paths correctly escaped.  If you are getting build errors check the paths first. TeamCity has some properties that you can reference in your build files to give you access the build related things. I am using teamcity_build_checkoutDir to specify that the output from the build is placed in the root of the checkout directory.

In TeamCity your build step should look something like this:

SpecflowTeamCityNUit1

With the build file in place and the step configured you may want to give things a run to make sure everything is working as expected. When you do you will notice that there is something missing from TeamCity.

SpecflowTeamCityNUnit2

If you had used the built in NUnit test runner you would expect to see a Test tab in TeamCity with the details of the tests run as part of the build. The tab is missing because TeamCity doesn’t know about the tests been run by the MsBuild task. The next step will show how to get the test tab and results back.

2. Getting the test tab to show in TeamCity

Luckily for us the good people at TeamCity have created an NUnit addin that does just that. To use the addin in needs to be present in the NUnit addin folder. The best way to ensure the addin is present in to add a Copy task to the MSBuild target. The full build file now looks like this:

<?xml version="1.0" encoding="utf-8" ?>

<Project ToolsVersion="3.5" DefaultTarget="Compile"
xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

    <ItemGroup>
        <NUnitAddinFiles Include="$(teamcity_dotnet_nunitaddin)-2.6.2.*" />
    </ItemGroup>

    <PropertyGroup>
        <NUnitHome>C:/Program Files (x86)/NUnit 2.6.2</NUnitHome>
        <NUnitConsole>&quot;$(NUnitHome)\bin\nunit-console.exe&quot;</NUnitConsole>
        <testResultsTxt>&quot;$(teamcity_build_checkoutDir)/TestResult.txt&quot;</testResultsTxt>
        <testResultsXml>&quot;$(teamcity_build_checkoutDir)/TestResult.xml&quot;</testResultsXml>
        <projectFile>&quot;$(teamcity_build_checkoutDir)\CrazyEvents.AcceptanceTests\CrazyEvents.AcceptanceTests.csproj&quot;</projectFile>
    </PropertyGroup>

    <Target Name="RunTests">
        <MakeDir Directories="$(NUnitHome)/bin/addins" />
        <Copy SourceFiles="@(NUnitAddinFiles)" DestinationFolder="$(NUnitHome)/bin/addins" />
        <Exec Command="$(NUnitConsole) /labels /out=$(testResultsTxt) /xml=$(testResultsXml) $(projectFile)"/>        
    </Target>

</Project>

An addins directory is created and the two addin files are copied from TeamCity to the directory. It is important to specify the correct version of the addin files in the NUnitAddinFiles section for the version on NUnit you are using. With the build file checked in the next build will have the missing Test tab, and the test summary will be displayed on the run overview.

SpecflowTeamCityNUnit3

3. Running the SpecFlow reports

To run the SpecFlow report I have used another build task to make two calls to SpecFlow.exe with the files generated as output by NUnit. The final build file looks like this:

<?xml version="1.0" encoding="utf-8" ?>

<Project ToolsVersion="3.5" DefaultTarget="Compile"
xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

    <ItemGroup>
        <NUnitAddinFiles Include="$(teamcity_dotnet_nunitaddin)-2.6.2.*" />
    </ItemGroup>

    <PropertyGroup>
        <NUnitHome>C:/Program Files (x86)/NUnit 2.6.2</NUnitHome>
        <NUnitConsole>&quot;$(NUnitHome)\bin\nunit-console.exe&quot;</NUnitConsole>
        <testResultsTxt>&quot;$(teamcity_build_checkoutDir)/TestResult.txt&quot;</testResultsTxt>
        <testResultsXml>&quot;$(teamcity_build_checkoutDir)/TestResult.xml&quot;</testResultsXml>
        <projectFile>&quot;$(teamcity_build_checkoutDir)\CrazyEvents.AcceptanceTests\CrazyEvents.AcceptanceTests.csproj&quot;</projectFile>
        <SpecflowExe>&quot;C:\Users\Admin\Documents\Visual Studio 2010\Projects\CrazyEventsNew\packages\SpecFlow.1.9.0\tools\specflow.exe&quot;</SpecflowExe>
    </PropertyGroup>

    <Target Name="RunTests">
        <MakeDir Directories="$(NUnitHome)/bin/addins" />
        <Copy SourceFiles="@(NUnitAddinFiles)" DestinationFolder="$(NUnitHome)/bin/addins" />
        <Exec Command="$(NUnitConsole) /labels /out=$(testResultsTxt) /xml=$(testResultsXml) $(projectFile)"/>        
    </Target>

    <Target Name="SpecflowReports">
        <Exec Command="$(SpecflowExe) nunitexecutionreport $(projectFile) /xmlTestResult:$(testResultsXml) /testOutput:$(testResultsTxt) /out:&quot;$(teamcity_build_checkoutDir)/SpecFlowExecutionReport.html&quot;"/>
        <Exec Command="$(SpecflowExe) stepdefinitionreport $(projectFile) /out:&quot;$(teamcity_build_checkoutDir)/SpecFlowStepDefinitionReport.html&quot;"/>
    </Target>

</Project>

Now we just need to get the html output of the reports to show in TeamCity.

4. Displaying the SpecFlow reports as tabs in the build output

Again, the people at TeamCity have made this really easy. First it is necessary to set the required files as build artifacts in the build configuration.

SpecflowTeamCityNUnit4

The final part of the jigsaw is to specify the report html file artifacts to be used as report tabs. Head over to Report tabs section of the Administration page and specify the new report tabs.

SpecflowTeamCityNUnit5

Now your tabs will be visible in TeamCity.

SpecflowTeamCityNUnit6

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s