CLI Reference

In this section we provide a quick reference for the Command Line Interface (CLI) of CTLog.

SAT-based Combinatorial Testing tools and algorithms in Python

usage: ctlog [-h] [--version] {IPOG,PRBOT-its,maxsat-its,CALOT,maxsat-mcac,SUT-G} ...

Positional Arguments

command: Possible choices: IPOG, PRBOT-its, maxsat-its, CALOT, maxsat-mcac, SUT-G

Named Arguments

--version: show program’s version number and exit

Sub-commands

IPOG

In-Parameter Order General algorithm

ctlog IPOG [-h] [--verbose {CRITICAL,ERROR,WARNING,INFO,DEBUG}] [--in-format {acts,extacts}] [--strength STRENGTH] [--out-file OUT_FILE]
           [--out-format {csv,acts}] [--skip-validation] [--all-missing-tuples] [--seed SEED] [-bpt] [-bvt]
           in_file

Named Arguments

--verbose

Possible choices: CRITICAL, ERROR, WARNING, INFO, DEBUG

CTLOG verbosity level

Default: “INFO”

Input SUT

Input System Under Test (SUT) parameters

in_file

Input SUT in one of the supported formats: acts, extacts.

--in-format

Possible choices: acts, extacts

Format of the input SUT. If None, inferred from the extension of the input file.

--strength, -t

Desired strength for the MCAC

Default: 2

Output MCAC

Output Mixed Covering Array with Constraints (MCAC) parameters

--out-file

Output file where the MCAC will be written. If None -> stdout

--out-format

Possible choices: csv, acts

Output MCAC format

Default: “csv”

--skip-validation

Skips the validation of the generated MCAC (validation can consume huge amounts of time for strengths > 3)

Default: False

--all-missing-tuples

Print the number of missing tuples during the validation

Default: False

Algorithm

Parameters of the algorithm

--seed, -s: Default: 1
-bpt, --break-param-ties: Default: False
-bvt, --break-val-ties: Default: False

PRBOT-its

Pool-Refine Build-One-Test iterative test suite construction algorithm

ctlog PRBOT-its [-h] [--verbose {CRITICAL,ERROR,WARNING,INFO,DEBUG}] [--in-format {acts,extacts}] [--strength STRENGTH] [--out-file OUT_FILE]
                [--out-format {csv,acts}] [--skip-validation] [--all-missing-tuples] [--no-complete-tests] [--seed SEED] [--pv-heuristic {exact,approx,ipog}]
                [--time-limit TIME_LIMIT] [--pool-impl {py,nim}] [--max-pool-size MAX_POOL_SIZE | --max-mem-GB MAX_MEM_GB] [--maxsat-timeout MAXSAT_TIMEOUT]
                [--intermediate-maxsat-timeout INTERMEDIATE_MAXSAT_TIMEOUT] [--max-window-size MAX_WINDOW_SIZE] [--min-free-tests MIN_FREE_TESTS]
                [--parallel-num-pools PARALLEL_NUM_POOLS] [--max-parallel-jobs MAX_PARALLEL_JOBS] [--out-folder OUT_FOLDER]
                [--submitter-script SUBMITTER_SCRIPT] [--filter-tuples] [--progressive] [--start-tupl START_TUPL] [--end-tupl END_TUPL]
                [--ws-csv-folder WS_CSV_FOLDER]
                in_file

Named Arguments

--verbose

Possible choices: CRITICAL, ERROR, WARNING, INFO, DEBUG

CTLOG verbosity level

Default: “INFO”

--no-complete-tests

Do not complete the empty values for the tests in the final test suite

Default: True

Input SUT

Input System Under Test (SUT) parameters

in_file

Input SUT in one of the supported formats: acts, extacts.

--in-format

Possible choices: acts, extacts

Format of the input SUT. If None, inferred from the extension of the input file.

--strength, -t

Desired strength for the MCAC

Default: 2

Output MCAC

Output Mixed Covering Array with Constraints (MCAC) parameters

--out-file

Output file where the MCAC will be written. If None -> stdout

--out-format

Possible choices: csv, acts

Output MCAC format

Default: “csv”

--skip-validation

Skips the validation of the generated MCAC (validation can consume huge amounts of time for strengths > 3)

Default: False

--all-missing-tuples

Print the number of missing tuples during the validation

Default: False

BOT-its

Arguments for the BOT-its algorithm. By default this will be the algorithm that will be executed.

--seed

RNG seed

Default: 1

--pv-heuristic

Possible choices: exact, approx, ipog

Heuristic for choosing the best param-value

Default: “exact”

--time-limit

Time limit in seconds after which the solution will be printed

--pool-impl

Possible choices: py, nim

Implementation of the pool

Default: “py”

P(R)BOT-its

Arguments for the pool version of BOT-its and RBOT-its algorithms. Pool limit will be enforced when one of these parameters is set by the user.

--max-pool-size: Maximum pool size of tuples
--max-mem-GB: Maximum memory in GB consumed by the pool of tuples

(P)RBOT-its

Arguments for the refine version of BOT-its and PBOT-its. Will be activated when at least one of the timeouts is provided by the user

--maxsat-timeout

Overall timeout in seconds for each call to MaxSAT in refine

--intermediate-maxsat-timeout

Timeout for MaxSAT between solutions in seconds

--max-window-size

Maximum window size in literals for MaxSAT after which the refine procedure is triggered

Default: 7500000

--min-free-tests

Minimum number of ‘free’ tests to start refining

Default: 10

Parallel P(R)BOT-its

Arguments for the parallel pool version of P(R)BOT-its. Activated when ‘parallel_num_pools’ is not None.

--parallel-num-pools

Number of pools per parallel process. If != None, activate parallel P(R)BOT-its

--max-parallel-jobs

Maximum number of parallel jobs

Default: 48

--out-folder

Output folder where partial test suites will be stored

Default: .

--submitter-script

Submitter script (required if using the parallel mode)

--filter-tuples

Each worker will try to filter covered tuples by previous workers

Default: False

--progressive

Master will submit jobs progressively until reaching ‘max_parallel_jobs’

Default: False

--start-tupl

Start tuple (only used by parallel master)

--end-tupl

End tuple (only used by parallel master)

--ws-csv-folder

Folder with csv files to warmstart the current worker (only used by parallel master)

maxsat-its

MaxSAT-its algorithm

ctlog maxsat-its [-h] [--verbose {CRITICAL,ERROR,WARNING,INFO,DEBUG}] [--in-format {acts,extacts}] [--strength STRENGTH] [--out-file OUT_FILE]
                 [--out-format {csv,acts}] [--skip-validation] [--all-missing-tuples] [--soft-param-tupls] [--filter-allowed]
                 [--maxsat-timeout MAXSAT_TIMEOUT] [--intermediate-maxsat-timeout INTERMEDIATE_MAXSAT_TIMEOUT] [--seed SEED]
                 in_file

Named Arguments

--verbose

Possible choices: CRITICAL, ERROR, WARNING, INFO, DEBUG

CTLOG verbosity level

Default: “INFO”

Input SUT

Input System Under Test (SUT) parameters

in_file

Input SUT in one of the supported formats: acts, extacts.

--in-format

Possible choices: acts, extacts

Format of the input SUT. If None, inferred from the extension of the input file.

--strength, -t

Desired strength for the MCAC

Default: 2

Output MCAC

Output Mixed Covering Array with Constraints (MCAC) parameters

--out-file

Output file where the MCAC will be written. If None -> stdout

--out-format

Possible choices: csv, acts

Output MCAC format

Default: “csv”

--skip-validation

Skips the validation of the generated MCAC (validation can consume huge amounts of time for strengths > 3)

Default: False

--all-missing-tuples

Print the number of missing tuples during the validation

Default: False

Algorithm

Parameters of the algorithm

--soft-param-tupls

Use a variable for each parameter tuple for soft clauses (i.e. less soft constraints)

Default: False

--filter-allowed

Filter forbidden tuples in the encoding

Default: False

--maxsat-timeout

Overall timeout in seconds for each call to MaxSAT in refine

--intermediate-maxsat-timeout

Timeout for MaxSAT between solutions in seconds

Default: 10

--seed, -s

Default: 1

CALOT

CALOT algorithm with the CCX encoding

ctlog CALOT [-h] [--verbose {CRITICAL,ERROR,WARNING,INFO,DEBUG}] [--in-format {acts,extacts}] [--strength STRENGTH] [--out-file OUT_FILE]
            [--out-format {csv,acts}] [--skip-validation] [--all-missing-tuples] --ub UB [--seed SEED] [--timeout TIMEOUT]
            [--solver {Glucose41,Cadical,Cadical152}]
            in_file

Named Arguments

--verbose

Possible choices: CRITICAL, ERROR, WARNING, INFO, DEBUG

CTLOG verbosity level

Default: “INFO”

Input SUT

Input System Under Test (SUT) parameters

in_file

Input SUT in one of the supported formats: acts, extacts.

--in-format

Possible choices: acts, extacts

Format of the input SUT. If None, inferred from the extension of the input file.

--strength, -t

Desired strength for the MCAC

Default: 2

Output MCAC

Output Mixed Covering Array with Constraints (MCAC) parameters

--out-file

Output file where the MCAC will be written. If None -> stdout

--out-format

Possible choices: csv, acts

Output MCAC format

Default: “csv”

--skip-validation

Skips the validation of the generated MCAC (validation can consume huge amounts of time for strengths > 3)

Default: False

--all-missing-tuples

Print the number of missing tuples during the validation

Default: False

Algorithm

Parameters of the algorithm

--ub

Upper bound on the number of tests of the MCAC

--seed

Random Number Generator seed

Default: 1

--timeout

Timeout after which the MaxSAT solver will be interrupted

Default: 10000

--solver

Possible choices: Glucose41, Cadical, Cadical152

SAT solver in OptiLog used by CALOT

Default: “Glucose41”

maxsat-mcac

MCAC generation through MaxSAT

ctlog maxsat-mcac [-h] [--verbose {CRITICAL,ERROR,WARNING,INFO,DEBUG}] [--in-format {acts,extacts}] [--strength STRENGTH] [--out-file OUT_FILE]
                  [--out-format {csv,acts}] [--skip-validation] [--all-missing-tuples] --ub UB [--msat-path MSAT_PATH] [--seed SEED] [--timeout TIMEOUT]
                  [--enc {ccx,cx,ccx-a1}]
                  in_file

Named Arguments

--verbose

Possible choices: CRITICAL, ERROR, WARNING, INFO, DEBUG

CTLOG verbosity level

Default: “INFO”

Input SUT

Input System Under Test (SUT) parameters

in_file

Input SUT in one of the supported formats: acts, extacts.

--in-format

Possible choices: acts, extacts

Format of the input SUT. If None, inferred from the extension of the input file.

--strength, -t

Desired strength for the MCAC

Default: 2

Output MCAC

Output Mixed Covering Array with Constraints (MCAC) parameters

--out-file

Output file where the MCAC will be written. If None -> stdout

--out-format

Possible choices: csv, acts

Output MCAC format

Default: “csv”

--skip-validation

Skips the validation of the generated MCAC (validation can consume huge amounts of time for strengths > 3)

Default: False

--all-missing-tuples

Print the number of missing tuples during the validation

Default: False

Algorithm

Parameters of the algorithm

--ub

Upper bound on the number of tests of the MCAC

--msat-path

Path to the MaxSAT solver that will be used to solve the encoding. If None use our Python Linear MaxSAT solver

--seed

Random Number Generator seed

Default: 1

--timeout

Timeout after which the MaxSAT solver will be interrupted

Default: 10000

--enc

Possible choices: ccx, cx, ccx-a1

Encoding to be used. Described in ‘Incomplete MaxSAT approaches for Combinatorial Testing’

Default: “ccx”

SUT-G

Implementation of the SUT-G Combinatorial Testing Generator

ctlog SUT-G [-h] [--verbose {CRITICAL,ERROR,WARNING,INFO,DEBUG}] [--ext-glob EXT_GLOB] [--timeout TIMEOUT] [--max-confl MAX_CONFL] [--min-confl MIN_CONFL]
            [--n-params N_PARAMS] [--n-seeds N_SEEDS] [--inc-assumps INC_ASSUMPS] [--dec-assumps DEC_ASSUMPS] [--max-tries MAX_TRIES] [--n-jobs N_JOBS] --out
            OUT [--seed SEED]
            input

Positional Arguments

input: Path to the input cnf instance. If a folder is given, all the instances with ‘ext’ are considered.

Named Arguments

--verbose

Possible choices: CRITICAL, ERROR, WARNING, INFO, DEBUG

CTLOG verbosity level

Default: “INFO”

--ext-glob

Extension glob for the instances if ‘input’ is a folder.

--timeout

Timeout for the SAT solver in seconds

Default: 30

--max-confl

Maximum number of conflicts for the final instance

Default: 10000

--min-confl

Minimum number of conflicts for the final instance

Default: 5000

--n-params

Number of parameters of the generated SUT

Default: 30

--n-seeds

Number of seeds to compute an average on the number of conflicts

Default: 5

--inc-assumps

Number of variables to increse the assumptions

Default: 10

--dec-assumps

Number of variables to decrease the assumptions

Default: 5

--max-tries

Maximum number of tries to reach the desired conflicts’ range

Default: 100

--n-jobs

Number of parallel jobs (only valid when processing a folder)

Default: 1

--out

--seed

Random seed

Default: 1