Assignment 11: Test Coverage Report

Due

Monday, November 8^th at 8am.

Context

Tests are a very important part of a long-lived software project, especially in dynamic programming languages. They enable ongoing confidence for developers, since if the changes they make break an existing feature, they’ll know about that before deploying anything.

However, it is a common problem on software projects that writing tests never gets prioritized, especially for developers (presumably like most of you) who haven’t learned the discipline of it.

A common way to measure the health of the test suite is to measure its coverage: what percentage of the lines of non-test code is executed when running the test suite? 100% coverage means that every line of code is exercised by at least one test in the suite. Few projects achieve 100% coverage, and it is usually wise to concentrate testing efforts on code that is more likely to be wrong. So coverage is not a perfect metric of test health, but it is at least objective, quantifiable, and reasonably useful.

Requirements

Depending on your project setup, you may need to analyze the test coverage of more than one body of code. For example, if you use separate languages (or separate repositories) for the frontend and the backend, you’ll need to analyze each separately.

For each part of your source code, two things are required: the test coverage report itself, and an interpretation. Each is detailed below.

Note that you will be graded on the conjunction of both elements. A low test coverage percentage can be redeemed by a thoughtful and reasonable interpretation; conversely, a high test coverage percentage can be damaged by a sloppy or uncareful interpretation. There’s no test coverage threshold that I expect all projects to exceed.

Test coverage report

Find a tool to produce a test coverage report for your language, configure it, and run it on your source code. Capture the results and post them to your web site. (I would prefer to view all results directly on your web site, but if that is a challenge, you may instead zip up the results directory and post the zip file to your web site.)

The report should include the following information:

What overall percentage of lines in the entire [part of the] code is covered by tests.
A coverage percentage per file.

Note that the test coverage should report the coverage for the automated test suite included in your repository, not any manual testing process you might perform yourself.

Interpretation

Write a brief (probably at most two page) statement that interprets the results and answers (at least) the following questions:

Which part of the code is this report for?
Do the results represent a healthy test suite, an unhealthy test suite, or somewhere in between? Why?
Are there parts of the code that don’t seem worth testing? If so, why not?
What are the top testing priorities in the current code? How important are they?
Are there any parts of the code that seem important to test, but you’re not sure how to test them?

Post the interpretation, along with the report itself, to your project web site.

Hints

The way to assess how important tests are for a particular unit (module, class, or function) is to consider the question: If a new developer were making changes to this unit, how likely would he or she be to break some important functionality? This question is not always easy to answer, and answers are not definite since it’s predicting a hypothetical future; nevertheless, it can be a useful prompt to consider.

Although the bulk of the work described here is in finding, choosing, and configuring a test coverage tool, you may want also to improve your quantitative coverage numbers and qualitative interpretation by writing more tests before the due date. In fact, that’s kind of the point of this assignment, so you’d be wise to budget some time for that.

You might want to use the ADR process described in Assignment 4 to structure the decision of which test coverage tool(s) to use.