Category: Cloud Computing

Much like the tool you use for unit testing, the tool you use for code coverage depends on the programming language and application framework you use.

When you target .NET Core applications to run on Linux, coverlet is a popular option. Coverlet is a cross-platform, code-coverage library for .NET Core. Before we add code coverage to the pipeline, let’s check in with the team.

How is code coverage done in .NET Core?

The way you collect code coverage depends on what programming language and frameworks you’re using, as well as what code coverage tools are available.

Mara and Andy do some investigation around code coverage for .NET Core applications. Here’s what they find:

• Visual Studio on Windows provides a way to perform code coverage.
• However, because the team is building on Linux, they can use coverlet , a cross-platform code coverage library for .NET Core.The unit test project requires the coverlet.msbuild NuGet package.
• Code coverage results are written to an XML file so that they can be processed by another tool. Azure Pipelines supports Cobertura and JaCoCo coverage result formats.Mara and Andy decide to try Cobertura.
• To convert Cobertura coverage results to a format that’s human-readable, they can use a tool called ReportGenerator .
• ReportGenerator provides a number of formats, including HTML. The HTML formats create detailed reports for each class in a .NET project.Specifically, there’s an HTML format called HtmlInline_AzurePipelines, which provides a visual appearance that matches Azure Pipelines.

How can I manage .NET Core tools?

A .NET Core tool such as ReportGenerator is a special NuGet package that contains a console application. You can manage a .NET Core tool as a global tool or as a local tool.

A global tool is installed in a centralized location and can be called from any directory. One version of a global tool is used for all directories on the machine.

A local tool is a more isolated copy of a .NET Core tool that’s scoped to a specific directory. Scope enables different directories to contain different versions of the same tool.

You use a manifest file to manage local tools for a given directory. This file is in JSON format and is typically named dotnet-tools.json. A manifest file enables you to describe the specific tool versions that you need to build or run your application.

When you include the manifest file in source control along with your application sources, developers and build systems can run the dotnet tool restore command to install all of the tools listed in the manifest file. When you need a newer version of a local tool, you simply update the version in the manifest file.

To keep things more isolated, in this module you work with local tools. You create a tool manifest that includes the ReportGenerator tool. You also modify your build pipeline to install the ReportGenerator tool to convert code coverage results to a human-readable format.

Run code coverage locally

Before Mara and Andy write any pipeline code, they decide to try things manually to verify the process. Follow along with their process:

1. In Visual Studio Code, open the integrated terminal.
2. Run the following dotnet new command to create a local tool manifest file.

 

dotnet new tool-manifest

The command creates a file named .config/dotnet-tools.json.

3. Run the following dotnet tool install command to install ReportGenerator:

dotnet tool install dotnet-reportgenerator-globaltool

This command installs the latest version of ReportGenerator and adds an entry to the tool manifest file.

Run the following dotnet add package command to add the coverlet.msbuild package to the Tailspin.SpaceGame.Web.Tests project:

dotnet add Tailspin.SpaceGame.Web.Tests package coverlet.msbuild

• 5. Run the following dotnet test command to run your unit tests and collect code coverage:

dotnet test --no-build \
  --configuration Release \
  /p:CollectCoverage=true \
  /p:CoverletOutputFormat=cobertura \
  /p:CoverletOutput=./TestResults/Coverage/

If the command fails, try running it like this:

 

• MSYS2_ARG_CONV_EXCL="*" dotnet test --no-build \
    --configuration Release \
    /p:CollectCoverage=true \
    /p:CoverletOutputFormat=cobertura \
    /p:CoverletOutput=./TestResults/Coverage/
  

  This command resembles the one you ran previously. The /p: flags tell coverlet which code coverage format to use and where to place the results.

6. Run the following dotnet tool run command to use ReportGenerator to convert the Cobertura file to HTML:

 

dotnet tool run reportgenerator \
  -reports:./Tailspin.SpaceGame.Web.Tests/TestResults/Coverage/coverage.cobertura.xml \
  -targetdir:./CodeCoverage \
  -reporttypes:HtmlInline_AzurePipelines

A number of HTML files appear in the CodeCoverage folder at the root of the project.

• In Visual Studio Code, expand the CodeCoverage folder, right-click index.htm, and then select Reveal in Explorer (Reveal in Finder on macOS or Open Containing Folder on Linux).
• In Windows Explorer (Finder on macOS), double-click index.htm to open it in a web browser.You see the coverage report summary.
• Scroll to the bottom of the page to see a coverage breakdown by class type.
• Select the link to TailSpin.SpaceGame.Web.LocalDocumentDBRepository`1 to view further details.Notice that the GetItemsAsync method is covered by unit tests, but the CountItemsAsync method has no coverage.This makes sense, because the FetchOnlyRequestedGameRegion test method calls the GetItemsAsync method but does not call the CountItemsAsync method. (To review the test code, see the DocumentDBRepository_GetItemsAsyncShould.cs file.)

Create a branch

Now that you can build a code coverage report locally, you’re ready to add tasks to your build best cheap web hosting pipeline, which performs the same tasks.

In this part, you create a branch named code-coverage, based on the unit-tests branch, to hold your work. In practice, you would ordinarily create this branch from the master branch.

1. In Visual Studio Code, open the integrated terminal.
2. In the terminal, run the following git checkout command to create a branch named code-coverage:

git checkout -b code-coverage

Commit your changes and push the branch to GitHub

Here you push your changes to GitHub and see the pipeline run. Recall that you’re currently in the code-coverage branch.

Although not required, here you add and commit each file separately so that each change is associated with a descriptive commit message.

1. In Visual Studio Code, go to the terminal.
2. Add and commit the Tailspin.SpaceGame.Web.Tests.csproj file, which now contains a reference to the coverlet.msbuild package:
   Bash

 

• git add Tailspin.SpaceGame.Web.Tests/Tailspin.SpaceGame.Web.Tests.csproj
  git commit -m "Add coverlet.msbuild package"
  

• Add and commit the tool manifest file, dotnet-tools.json:

• git add .config/dotnet-tools.json
  git commit -m "Add code coverage"
  

• Add and commit azure-pipelines.yml, which contains your updated build configuration:

• git add azure-pipelines.yml
  git commit -m "Add code coverage"
  

• Push the code-coverage branch to GitHub.

 

5. git push origin code-coverage
   

Watch Azure Pipelines run the tests

Here you see the tests run in the pipeline and then visualize the results from Azure Test Plans.

1. In Azure Pipelines, trace the build through each of the steps.
2. When the build finishes, navigate back to the summary page and select the Code Coverage tab.You view the same results that you did when you ran the tests locally.

As an optional step, you can explore the results from Azure Pipelines.

Add the dashboard widget

In the previous part, you added the Test Results Trend widget to your dashboard, which lets others quickly review test result trends over time.

Here you add a second widget that summarizes code coverage.

1. In a new browser tab, go to marketplace.visualstudio.com .
2. On the Azure DevOps tab, search for code coverage.
3. Select Code Coverage Widgets (published by Shane Davis).
4. Select Get it free.
5. In the drop-down list, select your Azure DevOps organization.
6. Select Install.
7. Go back to Azure DevOps.
8. Go to Overview > Dashboards.
9. Select Edit.
10. Search for Code Coverage, and then select Code Coverage.
11. Drag Code Coverage to the canvas.
12. Select the gear icon to configure the widget.
13. Keep all the default settings, except for:
    • Width: Enter 2.
    • Build definition: Select your pipeline.
    • Coverage measurement: Enter Lines.
14. Select Save.
15. Select Done Editing.

The widget shows the percentage of code your unit tests cover.

You now have code coverage set up in your pipeline. Although your existing code coverage is low, you have a baseline that you can improve over time.

Later, you can configure coverlet to check to see whether your tests provide a minimum threshold of coverage. Your threshold might be 30 percent, 50 percent, or 80 percent coverage, depending on your requirements. The build will fail if less than this amount is covered by your tests.

Remove code coverage files

Recall that when you ran Reportgenerator earlier, a number of HTML files appeared in the CodeCoverage folder at the root of the project.

These HTML files are not intended to be included in source control, and you no longer need them. Although the project’s .gitignore file is already set up to ignore anything in the CodeCoverage directory, it’s a good idea to delete these files so that they’re not added to your Git repository in future modules.

Andy is going to work with Mara to add unit tests to the automated build that Mara created with Microsoft Azure Cloud Pipelines. Regression bugs are creeping into their code and breaking the leaderboard’s filtering functionality. Specifically, the wrong game mode keeps appearing.

The following image illustrates Amita’s problem. When Amita selects “Milky Way” to show only scores from that game map, she gets results from other game maps, such as Andromeda.

Both Andy and Mara want to catch the error before it reaches Amita, the tester. Unit tests are a great way to automatically test for regression bugs.

Andy also thinks that adding the unit tests now will give them a head start as they improve the Space Game web app. The application uses a document database to store high scores and player profiles. Right now, it uses local test data. Later, they plan to connect the app to a live database.

A number of unit test frameworks are available for C# applications. Mara chooses NUnit because it’s popular with the community and she’s used it before.

Here’s the unit test you’re working with, along with Mara and Andy.

 

[TestCase("Milky Way")]
[TestCase("Andromeda")]
[TestCase("Pinwheel")]
[TestCase("NGC 1300")]
[TestCase("Messier 82")]
public void FetchOnlyRequestedGameRegion(string gameRegion)
{
const int PAGE = 0; // take the first page of results
const int MAX_RESULTS = 10; // sample up to 10 results

// Form the query predicate.
// This expression selects all scores for the provided game region.
Expression<Func<Score, bool>> queryPredicate = score => (score.GameRegion == gameRegion);

// Fetch the scores.
Task<IEnumerable> scoresTask = _scoreRepository.GetItemsAsync(
queryPredicate, // the predicate defined above
score => 1, // we don't care about the order
PAGE,
MAX_RESULTS
);
IEnumerable scores = scoresTask.Result;

// Verify that each score's game region matches the provided game region.
Assert.That(scores, Is.All.Matches(score => score.GameRegion == gameRegion));
}

 

You can filter the leaderboard by any combination of game type and game map.

This test queries the leaderboard for high scores and verifies that each result matches the provided game map.

In an NUnit test method, TestCase, provides inline data to use to test that method. Here, NUnit calls the FetchOnlyRequestedGameRegion unit test method like this:

C#
FetchOnlyRequestedGameRegion("Andromeda");
FetchOnlyRequestedGameRegion("Pinwheel");
FetchOnlyRequestedGameRegion("NGC 1300");
FetchOnlyRequestedGameRegion("Messier 82");
FetchOnlyRequestedGameRegion("Milky Way");
FetchOnlyRequestedGameRegion("Andromeda");
FetchOnlyRequestedGameRegion("Pinwheel");
FetchOnlyRequestedGameRegion("NGC 1300");
FetchOnlyRequestedGameRegion("Messier 82");

Notice the call to the Assert.That method at the end of the test. An assertion is a condition or statement that you declare to be true. If the condition turns out to be false, that could indicate a bug in your code. NUnit runs each test method using the inline data you specify and records the result as a passing or failing test.

Many unit test frameworks provide verification methods that resemble natural language. These methods help make tests easy to read and help you map the tests to the application’s requirements.

Consider the assertion made in this example:

C#

Assert.That(scores, Is.All.Matches<Score>(score => score.GameRegion == gameRegion));

 

You might read this line as:

Assert that the game region of each returned score matches the provided game region.

Here’s the process to follow:

1. Fetch a branch from the GitHub repository that contains the unit tests.
2. Run the tests locally to verify that they pass.
3. Add tasks to your pipeline configuration to run the tests and collect the results.
4. Push the branch to your GitHub repository.
5. Watch your Azure Pipelines project automatically build the application and run the tests.

Fetch the branch from GitHub

Here you fetch the unit-tests branch from GitHub and check out, or switch to, that branch.

This branch contains the Space Game project that you worked with in the previous modules and an Azure Pipelines configuration to start with.

1. In Visual Studio Code, open the integrated terminal.
2. Run the following git commands to fetch a branch named unit-tests from the Microsoft repository, and then switch to that branch.

   Bash
   
   git fetch upstream unit-tests
   git checkout -b unit-tests upstream/unit-tests

   The format of this command enables you to get starter code from the Microsoft GitHub repository, known as upstream. Shortly, you’ll push this branch to your GitHub repository, known as origin.

3. As an optional step, in Visual Studio Code, open the azure-pipelines.yml file and familiarize yourself with the initial configuration. The configuration resembles the basic one you created in the Create a build pipeline with Azure Pipelines module. It builds only the application’s Release configuration.

 

Run the tests locally

It’s a good idea to run all tests locally before you submit any tests to the pipeline. Here you do that.

1. 1. In Visual Studio Code, open the integrated terminal.
   2. Run dotnet build to build each project in the solution.

      dotnet build --configuration Release

   3. Run the following dotnet test command to run the unit tests:

      dotnet test --configuration Release --no-build

The –no-build flag specifies not to build the project before running it. You don’t need to build the project, because you built it in the previous step.

You see that all five tests pass.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.

Test Run Successful.
Total tests: 5
     Passed: 5
 Total time: 0.9320 Seconds

In this example, the tests took about one second to run.

Notice that there were five total tests. Although we define just one test method, FetchOnlyRequestedGameRegion, that test is run five times, once for each game map as specified in the TestCase inline data.

4. Run the tests a second time. This time, provide the –logger option to write the results to a log file.

dotnet test Tailspin.SpaceGame.Web.Tests --configuration Release --no-build --logger trx

You see from the output that a TRX file is created in the TestResults directory.

A TRX file is an XML document that contains the results of a test run. It’s a popular format for NUnit tests because Visual Studio and other tools can help you visualize the results.

Later, you’ll see how Azure Pipelines can help you visualize and track your tests’ results as they run through the pipeline.

5. As an optional step, in Visual Studio Code, open the DocumentDBRepository_GetItemsAsyncShould.cs file from the Tailspin.SpaceGame.Web.Tests folder and examine the test code. Even if you’re not interested in building .NET Core apps specifically, you might find the test code useful because it resembles code you might see in other unit test frameworks.

 

Push the branch to GitHub

Here you push your changes to GitHub and see the pipeline run. Recall that you’re currently on the unit-tests branch.

1. In the integrated terminal, add azure-pipelines.yml to the index, commit the changes, and push the branch up to GitHub.
   Bash

1. git add azure-pipelines.yml
   git commit -m "Run and publish unit tests"
   git push origin unit-tests
   

Watch Azure Pipelines run the tests

Here you see the tests run in the pipeline and then visualize the results from Microsoft Azure Test Plans. Azure Test Plans provides all the tools you need to successfully test your applications. You can create and run manual test plans, generate automated tests, and collect feedback from stakeholders.

1. In Azure Pipelines, trace the build through each of the steps.You see that the Run unit tests – Release task runs the unit tests just as you did manually from the command line.
2. Navigate back to the pipeline summary.
3. Move to the Tests tab.You see a summary of the test run. All five tests have passed.
4. In Azure DevOps, select Test Plans, and then select Runs.You see the most recent test runs, including the one you just ran.
5. Double-click the most recent test run.You see a summary of the results.In this example, all five tests have passed. If any tests failed, you could navigate to the build task to get additional details.

   You can also download the TRX file so you can examine it through Visual Studio or another visualization tool.

Although Mara and Andy have added only one test, it’s a good start and it fixes the immediate problem. Now, the team has a place to add more tests and run them as they improve their process.

Merge your branch into master

Mara and Andy are happy with their results, so they decide to merge the unit-tests branch to master. In practice, you would do the same. But for brevity, we’ll skip that process for now.