BlackBox

The examples presented so far show how a model implemented in Python or with Python bindings can be used. Here, we present how we can use the BlackBox module of OptiLog to calibrate any model that can be executed using a CLI, even models that are compiled and we do not have the source code nor bindings.

For simplicity, we will consider that we have a (discrete) SIR model available through:

$ ./sir ${instance} --beta ...
MSE: XXXXX

The parameters that it accepts are:

  • –beta: \(\beta\) in the ODEs.

  • –delta: \(\delta\) in the ODEs.

  • –n: the effective population.

  • –initial-i: the initial population in the Infected compartment.

  • –initial-r: the initial population in the Recovered compartment.

Model implementation

The first step is to define the BlackBox that will represent the external program, by creating a subclass of SystemBlackBox in model.py, defining the parameters that the program accepts in the config class attribute, and how to call the program in the constructor.

...
from optilog.blackbox import *
from optilog.running import ParsingInfo
from optilog.tuning import *

class ExternalSir(SystemBlackBox):
    config = {
        "n": Int(70_000, 500_000, default=70_000),
        "initial-i": Int(1, 1_000, default=40),
        "initial-r": Int(0, 1_000, default=4),
        "beta": Real(0.1, 1.0, default=0.7),
        "delta": Real(0.01, 1.0, default=0.1),
    }

    def __init__(self, *args, **kwargs):
        _model = (Path(__file__).parent / "sir").resolve()
        super().__init__(
            arguments=[_model, SystemBlackBox.Instance],
            *args,
            parsing_info=self.get_output_parser(),
            **kwargs,
        )
    ...

In the constructor, when calling the super().__init__ method we have the parsing_info parameter set. This will instruct the BlackBox to parse the output of the external program and create attributes after its execution. For this example, we define a filter in the parser that will read the line containing the MSE of the model, and set it as a cost attribute after execution.

class ExternalSir(SystemBlackBox):
    ...
    @staticmethod
    def get_output_parser():
        parser = ParsingInfo()
        parser.add_filter(
            "cost", r"MSE: ([+-]?(?:\d+(?:\.\d*)?|\.\d+)(?:[eE][+-]?\d+)?)",
            cast_to=float)
        return parser
    ...

Finally, we implement the function that will format the parameters correctly for this program:

class ExternalSir(SystemBlackBox):
    ...
    def format_config(self, args):
        other = ""
        for k, v in self.configured.items():
            other += f" --{k} {v}"
        args = args + shlex.split(other)
        return args

Once we have the BlackBox defined, we could execute the model by using:

sir = ExternalSir()
sir.run("myData.dat")
print("The cost for the default parameters is", sir.cost)

To be calibrated, we can define the BlackBox instance as a parameter of the model function (which have the @ac decorator). Instead of defining all the parameters that can be configured in this BlackBox another time, we set the BlackBox type as CfgObj(ExternalSir). The model function is as simple as:

...
@ac
def model(datafile, sir: CfgObj(ExternalSir)):
    sir.run(datafile)
    return sir.cost

Finally, the entrypoint for this example does not need to load the dataset, so it can pass it directly to the model as:

...
def entrypoint(data, seed):
    cost = model(data)
    print(f"Result: {cost}")

Preparing the model to be run using docker

Example of output

Once the calibration process finishes, the software will output the best configuration found during the calibration.

The best parameters are listed for each configurable function. Note that, as the sir object was internally calibrated, the parameters of this object are indicated as: sir>parameter. Also, the configuration JSON is reported as it can be used to inject the parameters using OptiLog.

[...]
Tuning process finished.
========================
Best parameters:
- model(sir>n): 141718
- model(sir>initial_i): 802
- model(sir>initial_r): 847
- model(sir>beta): 0.24870931153434647
- model(sir>delta): 0.01311400823831338
========================
Best Config JSON:
{'_0_0__sir__n': 141718, '_1_0__sir__initial_i': 802, [...]}