Software Engineering - Coverage Summary

Software Engineering - Coverage Summary - Transcript

Introduction to Software Testing Chapters 1 5 Coverage Summary

Paul Ammann Jeff Offutt
http www cs gmu edu offutt softwaretest

From Chapter 1 Criteria Based on Four Structures
1

Graphs

Structures Ways to model software

2

Logical Expressions

not X or not Y and A and B A 0 1 1 B 600 700 800 C swe cs isa infs if x y z x y else z 2 x
Ammann Offutt 2

3

Input Domain Characterization

4

Syntactic Structures

Introduction to Software Testing Ch 1 5

Test Coverage Criteria
A tester s job is simple Define a model of the software then find ways to cover it
Test Requirements Specific things that must be satisfied or covered

during testing

Test Criterion A collection of rules and a process that define test

requirements

Testing researchers have defined dozens of criteria but they Testing are all really just a few criteria on four types of structures are
Introduction to Software Testing Ch 1 5 Ammann Offutt 3

Coverage Overview
Four Structures for Four Modeling Software Modeling Graphs
Applied to

Logic
Applied to Source

Input Space
FSMs DNF

Syntax
Applied to

Specs

Source Design

Specs Use cases

Source

Models Input
4

Integ
Ammann Offutt

Introduction to Software Testing Ch 1 5

Coverage Overview
Four Structures for Four Modeling Software Modeling Graphs
Applied to

Logic
Applied to Source

Input Space
FSMs DNF

Syntax
Applied to

Specs

Source Design

Specs Use cases

Source

Models Input
5

Integ
Ammann Offutt

Introduction to Software Testing Ch 1 5

Ch 2 Graph Coverage Criteria 1
Concepts for graphs
Visiting touring Tests and test paths Structural and data flow coverage Prime Paths Defs and uses

Simple graph coverage criteria

Node Coverage NC TR contains each reachable node in G in

Edge Coverage EC TR contains each reachable path of length up to 1 in G of
Introduction to Software Testing Ch 1 5 Ammann Offutt 6

Graph Coverage Criteria 2
Covering multiple edges

Edge Pair Coverage EPC TR reachable path of length up to 2 in G reachable

contains

each

Complete Path Coverage CPC TR contains all paths in G in

Specified Path Coverage SPC TR contains a set S of test paths where S is supplied as a parameter test

Introduction to Software Testing Ch 1 5

Ammann Offutt

7

Graph Coverage Criteria 3
Handling loops

Simple Round Trip Coverage SRTC TR contains at least one round trip path for each reachable node in G that begins and ends a round trip path that Complete Round Trip Coverage CRTC TR contains all round trip paths for each reachable node in G all Prime Path Coverage PPC TR contains each prime path in G path
Introduction to Software Testing Ch 1 5 Ammann Offutt 8

Graph Coverage Criteria 4
Data flow coverage

All defs coverage ADC For each set of du paths S du n v TR contains at least one path d in S du

All uses coverage AUC For each set of du paths to uses S du ni nj v TR contains at least one path d in du S

All du paths coverage ADUPC For each set S du du ni nj v TR contains every path d in S
Introduction to Software Testing Ch 1 5 Ammann Offutt 9

Graph Coverage Criteria Subsumption
Complete Path Coverage CPC Prime Path Coverage PPC Edge Pair Coverage EPC Edge Coverage EC Node Coverage NC
Ammann Offutt

All DU Paths Coverage ADUP All uses Coverage AUC All defs Coverage ADC

Complete Round Trip Coverage CRTC Simple Round Trip Coverage SRTC

Introduction to Software Testing Ch 1 5

10

Coverage Overview
Four Structures for Four Modeling Software Modeling Graphs
Applied to

Logic
Applied to Source

Input Space
FSMs DNF

Syntax
Applied to

Specs

Source Design

Specs Use cases

Source

Models Input
11

Integ
Ammann Offutt

Introduction to Software Testing Ch 1 5

Ch 3 Logic Coverage Criteria 1
Concepts in logical expressions
Active clauses Determination of the predicate

Simple logical expression criteria
Predicate Coverage PC For each p in P TR contains two TR requirements p evaluates to true and p evaluates to false

Clause Coverage CC For each c in C TR contains two TR requirements c evaluates to true and c evaluates to false

Introduction to Software Testing Ch 1 5

Ammann Offutt

12

Logic Coverage Criteria 2
Testing multiple clauses at once

Combinatorial Coverage CoC For each p in P TR has test TR requirements for the clauses in Cp to evaluate to each Cp possible combination of truth values possible Active Clause Coverage ACC For each p in P and each major clause ci in Cp choose minor clauses cj j i so that ci ci Cp choose cj so ci determines p TR has two requirements for each ci ci TR ci evaluates to true and ci evaluates to false ci

Introduction to Software Testing Ch 1 5

Ammann Offutt

13

Logic Coverage Criteria 3
General Active Clause Coverage GACC For each p in P and each major clause ci in Cp choose minor clauses cj j i ci Cp choose cj so that ci determines p TR has two requirements for each ci ci TR ci ci evaluates to true and ci evaluates to false The values ci ci chosen for the minor clauses cj do not need to be the same cj not when ci is true as when ci is false that is cj ci true cj ci ci ci false for all cj OR cj ci true cj ci false for all cj false cj cj ci Restricted Active Clause Coverage RACC For each p in P and each major clause ci in Cp choose minor clauses cj j i ci Cp choose cj so that ci determines p TR has two requirements for each ci ci TR ci ci evaluates to true and ci evaluates to false The values ci ci chosen for the minor clauses cj must be the same when ci is cj must ci true as when ci is false that is it is required that cj ci true ci cj ci false for all cj cj ci cj
Introduction to Software Testing Ch 1 5 Ammann Offutt 14

Logic Coverage Criteria 4
Practical version

Correlated Active Clause Coverage CACC For each p in P and each major clause ci in Cp choose minor clauses cj j i ci Cp choose cj so that ci determines p TR has two requirements for each ci ci TR ci ci evaluates to true and ci evaluates to false The values ci ci chosen for the minor clauses cj must cause p to be true for one cj value of the major clause ci and false for the other that is it ci is required that p ci true p ci false p ci

Introduction to Software Testing Ch 1 5

Ammann Offutt

15

Logic Coverage Criteria 5
Inactive clauses
Inactive Clause Coverage ICC For each p in P and each major clause ci ci in Cp choose minor clauses cj j i so that ci does not determine p TR Cp choose cj so ci does TR has four requirements for each ci 1 ci evaluates to true with p true 2 four ci 1 ci ci evaluates to false with p true 3 ci evaluates to true with p false and 4 ci ci ci evaluates to false with p false ci General Inactive Clause Coverage GICC For each p in P and each major clause ci in Cp choose minor clauses cj j i so that ci does not ci Cp choose cj so ci does determine p The values chosen for the minor clauses cj do not need to be The cj d o the same when ci is true as when ci is false that is cj ci true cj ci ci ci false for all cj OR cj ci true cj ci false for all cj false cj cj ci cj Restricted Inactive Clause Coverage RICC For each p in P and each major clause ci in Cp choose minor clauses cj j i so that ci does not ci Cp choose cj so ci does determine p The values chosen for the minor clauses cj must be the same The cj must when ci is true as when ci is false that is it is required that cj ci true ci ci cj ci false for all cj cj ci cj
Introduction to Software Testing Ch 1 5 Ammann Offutt 16

Logic Coverage Criteria Subsumption
Combinatorial Clause Coverage COC Restricted Active Clause Coverage RACC Correlated Active Clause Coverage CACC General Active Clause Coverage GACC Clause Coverage CC
Introduction to Software Testing Ch 1 5

Restricted Inactive Clause Coverage RICC General Inactive Clause Coverage GICC

Predicate Coverage PC
Ammann Offutt 17

Coverage Overview
Four Structures for Four Modeling Software Modeling Graphs
Applied to

Logic
Applied to Source

Input Space
FSMs DNF

Syntax
Applied to

Specs

Source Design

Specs Use cases

Source

Models Input
18

Integ
Ammann Offutt

Introduction to Software Testing Ch 1 5

Ch 4 Input Space Criteria 1
Concepts for input space criteria

Input domains No difference with level of software abstraction unit integration etc Partitioning domains blocks Constraints among partitions

Simple input space criterion

Each Choice EC One value from each block for each partition must be used in at least one test case partition

Introduction to Software Testing Ch 1 5

Ammann Offutt

19

Input Space Criteria 2
Combining values

Pair Wise PW A value from each block for each partition must be combined with a value from every block for each other partition partition

t Wise TW A value from each block for each group of t partitions must be combined partitions

Introduction to Software Testing Ch 1 5

Ammann Offutt

20

Input Space Criteria 2
Using domain knowledge of the program

Base Choice BC A base choice block is chosen for each partition and a base test is formed by using the base choice for each partition Subsequent tests are chosen by holding all but one base choice constant and using each non base choice in each other choice parameter parameter Multiple Base Choice MBC One or more base choice blocks are chosen for each partition and base tests are formed by using each base choice for each partition Subsequent tests are chosen by holding all but one base choice constant for each base test and using each non base choices in each other parameter using
Introduction to Software Testing Ch 1 5 Ammann Offutt 21

ISP Coverage Criteria Subsumption
All Combinations Coverage AC

T Wise Coverage TW

Multiple Base Choice Coverage MBC

Pair Wise Coverage PW

Base Choice
Coverage BC

Each Choice Coverage EC
Introduction to Software Testing Ch 1 5 Ammann Offutt 22

Coverage Overview
Four Structures for Four Modeling Software Modeling Graphs
Applied to

Logic
Applied to Source

Input Space
FSMs DNF

Syntax
Applied to

Specs

Source Design

Specs Use cases

Source

Models Input
23

Integ
Ammann Offutt

Introduction to Software Testing Ch 1 5

Ch 5 Syntax based Criteria 1
Concepts for syntax based testing
Grammars Regular expressions finite state machines Mutation testing

Simple syntax based criteria

Terminal Symbol Coverage TSC TR contains each terminal symbol t in the grammar G Production Coverage PC TR contains each production p in the grammar G the Derivation Coverage DC TR contains every possible string that can be derived from the grammar G that
Introduction to Software Testing Ch 1 5 Ammann Offutt 24

Mutation Testing

Grammars describe both valid and invalid strings Both types can be produced as mutants A mutant is a variation of a valid string Mutation is based on mutation operators and ground strings

Ground string A string in the grammar
The term ground is used as a reference to algebraic ground terms

Mutation Operator A rule that specifies syntactic variations of
strings generated from a grammar

Mutant The result of one application of a mutation operator
A mutant is a string
Introduction to Software Testing Ch 1 5 Ammann Offutt 25

Syntax based Criteria 2
Mutation criteria

Mutation Coverage MC For each m M TR contains TR exactly one requirement to kill m Mutation Operator Coverage MOC For each mutation operator TR contains exactly one requirement to create a mutated string m that is derived using the mutation operator Mutation Production Coverage MPC For each mutation operator TR contains several requirements to create one mutated string m that includes every production that can be mutated by that operator mutated
Introduction to Software Testing Ch 1 5 Ammann Offutt 26

Summary of Criteria
Chapter 2
Eleven criteria five recommended

Chapter 3
Eight criteria three recommended

Chapter 4
Six criteria two recommended

Chapter 5
Six criteria three recommended

Total Thirty one criteria thirteen recommended

Introduction to Software Testing Ch 1 5

Ammann Offutt

27