What you should know about code coverage

Author: Wolfgang Meincke | Date: April 03, 2019

  • Code Coverage is an important metric in embedded software development and is considered in standards like ISO 26262
  • Measuring all coverage and robustness goals is a comprehensive task
  • This article gives an overview of relevant test goals in respect to the risk class, development level and project needs.

Each time we talk about testing, we also talk about code coverage. But why is it important to keep an eye on code coverage and what are possible code coverage goals?

What is code coverage?

Code coverage is the sum of all coverage goals that are covered by a set of test cases. Coverage goals are e.g. statements or decisions. The ISO 26262 names coverage goals like statement, decision/branch and modified condition/decision coverage for unit testing. Depending on the risk class (=ASIL level) a subset of these coverage goals are highly recommended. Nevertheless, these coverage goals are just a subset of the available coverage goals for C Code in general. There are more coverage goals like function, function call, switch case, relation operator coverage. In addition, it is useful to consider functional scenarios, derive equivalence classes, boundary value checks or individually specified critical situations. For all the coverage goals the tester wants to achieve a high (if possible) 100% coverage. Beside the code coverage goals named before, we should consider robustness goals like division-by-zero, critical down casts, range violations and unreachability as well.

It is not easy to reach high coverage results with handwritten test cases. However, based on a technology called Model Checking it is possible to generate tests automatically and to create a mathematical proof, that a certain coverage or robustness goal can never be reached. As a result, all available coverage goals are analyzed and get reported as either covered or unreachable.

Let’s have a look on some samples of typical coverage goals. Coverage goals can have a different amount of properties that can either be covered or unreachable. For all of the coverage goals high coverage should be achieved.

Coverage Goal Description ASIL relevance ISO 26262 2nd edition reference
Statement (or C0) coverage goal with one property executed once it is measured as covered

A: ++
B: ++
C: +
D: +

Part 6
Chapter 9.4.5
Table 11
Scope: Unit
  • driver_command = validated_up
  • driver_command = (X_S170 != 0) && (Aux_B  < 1)
Decision (or Branch coverage or C1) is a coverage goal with two properties that can become independently true or false A: +
B: ++
C: ++
D: ++
Part 6
Chapter 9.4.5
Table 11
Scope: Unit
  • S167_d1, e.g. in an if-Statement --> if(S167_d1){...}
  • if((a<5) && (b>0)) {...}
Condition coverage goal with two properties that can become independently true or false n.a.

Not explicitly named, it is a refinement of decision and MC/DC coverage

Scope: Unit
  • (a<5), e.g. first part of a decision --> if (a<5) && (b>0) {...}
  • (b>0), e.g. the second part of a decision --> if ( (a < 5) && (b > 0) ) { … }
  • (a != 0)
CDC, MC/DC Condition Decision Coverage (CDC) / Modified Condition Decision Coverage (MC/DC) A: +
B: +
C: ++
D: ++
Part 6
Chapter 9.4.5
Table 11
Scope: Unit
  • a > b && c == d
  • Every decision has taken all possible outcomes (true, false) at least once.
  • Every condition appearing in a decision has taken all possible outcomes (true, false) at least once.
  • Every condition is able to independently influence the result of the decision.
Function coverage goal with one property that shows that a certain function was called. A: +
B: +
C: ++
D: ++
Part 6
Chapter 10.4.6
Table 14
Scope: Architecture

void ICF_A5_main(void) { ... }

Function Call coverage goal with one or more properties depending on how often a certain function can be called in different places in the code. A: +
B: +
C: ++
D: ++
Part 6
Chapter 10.4.6
Table 14
Scope: Architecture
void func_a(void) {
     sub_func();
}

void func_b(void){
     sub_func();
}
Switch-Case coverage goal with a various amount of properties, depending on the number of cases. Technically it is a Decision. A: +
B: ++
C: ++
D: ++
Part 6
Chapter 9.4.5
Table 11
Scope: Unit
switch(expression)
{
case constant1:
   statements
     break;
case constant2:
   statements
     break;
   default:
     statements
}
Relational Operator coverage goal with several properties (true, false, +/- one LSB around the decision and the exact value).

n.a.

can be looked upon as an extended decision check

Scope: Unit

X_SSHZ_Memory_C >= 25

Goals: TRUE, FALSE, X_SSHZ_Memory_C=24, X_SSHZ_Memory_C=25, X_SSHZ_Memory_C=26

Equivalence class and boundary checks user defined coverage goal that slices the value range of a variable into several domains.

A: +
B: ++
C: ++
D: ++

Part 6
Chapter 10.4.4
Table 13
Scope: Software Integration

  • Signal A is an Int8. In this case, the value range can e.g. be sliced into 5 domains [-128 -60) [-60 –20) [-20 20) [20 60) [60 127].
  • To cover the boundaries, the ranges [-128] and [127] need to be taken into account in addition.
Individually specified This coverage goal allows a user specific definition of additional coverage goals

not applicable in general

depends on its definition

Scope: Unit, Software Integration, Architecture
  • Output A and output B can never be true at the same time.
  • Parameter A is true => Output C remains 0

 

Now let’s have a look at robustness goals. For these coverage goals an unreachable result is expected. Otherwise this might lead to unintended behavior.

Robustness Goal Description ASIL relevance ISO 26262 2nd edition reference
Down Cast robustness goal with two properties. For an unsigned datatype, only one critical down cast range is relevant. For signed data types there are two critical ranges. n.a.

Part 6
Chapter 8.4.4
f) Robustness
Scope: Unit

(UInt8) ((Ca1_control_ctr_a) + 1) //Ca1_control_ctr_a is an Int16
The critical ranges in this case are

  • [-2147483648, -1]
  • [256, 2147483647]
Division by Zero robustness goal with one property n.a. Part 6
Chapter 8.4.4
f) Robustness
Scope: Unit

(((UInt16) My_Nominator_Variable) / (((Uint16) (My_Denominator_Variable()) + 1)

Divisor becomes 0 by setting the output of this function My_Denominator_Varialbe() to 65535. To add up the “+ 1” produces an overflow and the term in the denominator becomes 0.
Range Violation is a user defined robustness goal

A: ++
B: ++
C: ++
D: ++

Part 6
Chapter 7.4.12
Table 4
Scope: Architecture                                                         

an output that has the data type sInt16 and in addition a min/max range from -4000 to 4000. The range violation plugin allows to prove, if these min/max ranges can be violated by any input combination over time.

Of course, not all projects focus on all of these coverage and robustness goals but a sufficient subset that fulfills the project safety goals. In general, there are three levels test goal are related to:

What test goals are taken into account for each level is based on the risk class of the component and is usually defined by the customer and the quality assurance.

The Author:

Wolfgang Meincke studied Computer Science at the University Ravensburg-Weingarten where he graduated in 2006. He then worked at EWE TEL GmbH where he was responsible for requirements engineering and project management for software development projects as well as agile software development processes. In 2014 he joined BTC as senior pilot engineer to support the customers to integrate the tools into their development processes as well as giving support and training on test methodologies regarding the ISO 26260 standard. One of his main areas of interest is the formalization of safety requirements and their verification based on formal testing and model-checking technologies for unit test, integration test and HIL real-time-testing.