Hands-on CTLog

In this section we will present a full example on how to use ctlog to build CAs for the example in the Introduction to Combinatorial Testing section. To start, ctlog should be already installed in your system.

Defining the System Under Test

ctlog supports different representations for Systems Under Test (SUTs). See the Input SUT formats section for more details.

In the following we will represent the SUT of the Introduction to Combinatorial Testing section using the ACTS format:

[System]
Name: DefaultName

[Parameter]
OS (enum) : L,W,M,i,A
Pl (enum) : F,S,C,A
Re (enum) : K,F,H,W
Or (enum) : P,L

[Constraint]
C1: (OS != L || Or = L)
C2: (OS != L || Pl != A)
C3: (OS != W || Or = L)
C4: (OS != W || Pl != A)
C5: (OS != M || Or = L)
C6: (OS != M || Pl != A)
C7: (Pl != S || OS = M || OS = i)
C8: (OS != i || Re != K)
C9: (OS != A || Re != K)

You can check the ACTS format to find a full description of the ACTS format. For our example, we will define the parameters as enum. You can use the full expressiveness of the ACTS format to define non-CNF constraints (we handle them by using OptiLog’s Modelling module).

We will assume this SUT is stored under the sut.acts file. With this file we can now apply any of the algorithms implemented in ctlog to generate MCACs.

Generating an MCAC using CTLog

Once we have defined our SUT, we can generate an MCAC using any of the MCAC Algorithms implemented in CTLog. For this example, we will use the IPOG algorithm for strength 2:

ctlog mcac algorithms run IpogAlgorithm sut.acts 2

This command would produce the following output:

Test suite size: 22
c T: OS,Pl,Re,Or
c T: L,F,K,L
c T: W,F,F,L
c T: M,F,H,L
c T: i,F,W,P
c T: A,F,F,P
c T: M,S,K,L
c T: i,S,F,P
c T: L,C,F,L
c T: W,C,K,L
c T: M,C,W,L
c T: i,C,H,P
c T: A,C,H,L
c T: i,A,F,L
c T: A,A,W,P
c T: L,C,H,L
c T: L,C,W,L
c T: W,C,H,L
c T: W,C,W,L
c T: M,C,F,L
c T: M,S,H,L
c T: M,S,W,L
c T: A,A,H,L
Finished output of test cases. Size: 22
Validating MCAC...
VALIDATION OK
Num covered: 69
Num forbidden: 13

As we can see, the output MCAC is printed through standard output by default. However, there is no need to parse this output if we want to extract the MCAC, as we can directly write the MCAC to a file by using the --out-file parameter:

ctlog mcac algorithms run IpogAlgorithm sut.acts 2 --out-file mcac.csv

In this case the generated output would be:

Finished output of test cases. Size: 22
Validating MCAC...
VALIDATION OK
Num covered: 69
Num forbidden: 13

And the mcac.csv file:

OS,Pl,Re,Or
L,F,K,L
L,C,F,L
W,F,F,L
W,C,K,L
M,F,H,L
M,S,K,L
M,C,W,L
i,F,W,P
i,S,F,P
i,C,H,P
i,A,F,L
A,F,F,P
A,C,H,L
A,A,W,P
L,F,H,L
L,F,W,L
W,F,H,L
W,F,W,L
M,F,F,L
M,S,H,L
M,S,W,L
i,A,H,L

Using other MCAC algorithms

One of the main benefits of CTLog is the common interface that we provide for using the implemented MCAC algorithms. This way we will be able to try different algorithms over the same SUT without much hassle. For this example, we will use the RBOT-its to try to reduce the size of the returned MCAC. To this end, we will need to specify the --ub parameter, which we can set to 22 (according to test suited that returned IPOG before).

ctlog mcac algorithms run MaxSatMcacAlgorithm sut.acts 2 --ub 22

The output of this comand is the following:

Test suite size: 21
c T: OS,Pl,Re,Or
c T: L,C,K,L
c T: L,F,F,L
c T: L,C,H,L
c T: L,C,W,L
c T: W,F,K,L
c T: W,C,F,L
c T: W,F,H,L
c T: W,F,W,L
c T: M,S,K,L
c T: M,S,F,L
c T: M,C,H,L
c T: M,F,W,L
c T: i,C,F,P
c T: i,S,H,P
c T: i,S,W,P
c T: A,A,F,L
c T: A,A,H,P
c T: A,F,W,P
c T: i,A,W,L
c T: A,C,H,L
c T: i,F,H,L
Finished output of test cases. Size: 21
Validating MCAC...
VALIDATION OK
Num covered: 69
Num forbidden: 13

To obtain a list with all the supported MCAC algorithms you can run:

ctlog mcac algorithms list

This command output a list with the implemented algorithms, as well as whether they required an initial upper bound through the --ub command (such as MaxSatMcacAlgorithm) or not (such as IpogAlgorithm).

IpogAlgorithm. Requires UB: False
PRBOTitsAlgorithm. Requires UB: False
RemicaAlgorithm. Requires UB: False
CalotAlgorithm. Requires UB: True
MaxSatItsAlgorithm. Requires UB: False
MaxSatMcacAlgorithm. Requires UB: True
SatMcacAlgorithm. Requires UB: True
WeakenProductAlgorithm. Requires UB: False

To inspect the additional parameters that each algorithm supports you can run the help command. For example, for the IpogAlgorithm we have:

ctlog mcac algorithms help IpogAlgorithm

Parameters:
    --break_param_ties: type=<class 'bool'>, default=False
    --break_val_ties: type=<class 'bool'>, default=False
    --fill_tests: type=<class 'bool'>, default=True
    --seed: type=<class 'int'>, default=123456

In this case, we can run IpogAlgorithm specifying that we don’t want it to fill the empty values (i.e., values that don’t affect the coverage of the MCAC) in the generated test suite as follows:

ctlog mcac algorithms run IpogAlgorithm sut.acts 2 --args fill_tests=False

This command output the following MCAC, where * represent empty values:

Test suite size: 22
c T: OS,Pl,Re,Or
c T: L,F,K,L
c T: W,F,F,L
c T: M,F,H,L
c T: i,F,W,P
c T: A,F,F,P
c T: M,S,K,L
c T: i,S,F,P
c T: L,C,F,L
c T: W,C,K,L
c T: M,C,W,L
c T: i,C,H,P
c T: A,C,H,L
c T: i,A,F,L
c T: A,A,W,P
c T: L,*,H,L
c T: L,*,W,L
c T: W,*,H,L
c T: W,*,W,L
c T: M,*,F,L
c T: *,S,H,*
c T: *,S,W,*
c T: *,A,H,*
Finished output of test cases. Size: 22
Validating MCAC...
VALIDATION OK
Num covered: 69
Num forbidden: 13

We could replace any of the * by any of the parameter’s values, as these do not alter the coverage of the returned MCAC (as long as the replaced value is consistent with the SUT constraints within the test case).

For more information, you can check ctlog mcac algorithms run.