Running ======= To run the calibration process Docker_ is used. This allows to create an isolated environment that can be run in any machine that has Docker_ (ideal for cloud computing tasks). By default, Docker_ will run the `entrypoint `_. This is already implemented, so when you start the docker container it will run the calibration process by default. The provided project can be executed using .. code-block:: docker run ${image_name} where :code:`${image_name}` is the name that was defined: - if the **Makefile** was used, the name in this file - otherwise, the image name that was specified when running the :code:`docker build` command. If nothing is specified, the calibration process will start. It will use all the files in the data folder, so if you have copied multiple data files you have to manually specify which one should be used. .. note:: It is common to specify a name for the specific container, to be identified later. You can specify the name of the container to be started with .. code-block:: docker run --name ${myname} ${image_name} ... More options can be found at the `official documentation `_. To change the default behaviour, you can pass parameters to :code:`docker run`. To see all the available options run :code:`docker run --rm ${image_name} --help` [#]_. The *entrypoint* accepts the following commands: - configure - run .. _run-docker: docker run configure ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This is the command that you will usually run. It executes the entire calibration process. You have to specify the data file to use. Note that all the datafiles wil be located at */epidemic/project/data/...*, so if you want to calibrate the project using the dataset **myCountry.csv** run: .. code-block:: bash docker run ${image_name} configure /epidemic/project/data/myCountry.csv You can specify the following optional parameters: - slots: The number of processes that will be used to calibrate the model. It should match the number of cores in the machine you plan to use. - verbose: Select between *critical*, *error*, *warning*, *info*, *debug*. This is the verbosity of PyDGGA. For example, to run the same dataset as before, but using a machine with 48 cores and setting the verbosity to *error* run: .. code-block:: bash docker run ${image_name} configure /epidemic/project/data/myCountry.csv --verbose error --slots 48 docker run run ~~~~~~~~~~~~~~~~~~~~~~ Sometimes to verify if the model is working correctly, you might be interested in running the model inside the container without executing the entire calibration process. For this purpose, the *entrypoint* also accepts the command :code:`run`. You can run a single execution of the model for an instance and a random seed by executing: .. code-block:: bash docker run ${image_name} run /epidemic/project/data/${instance} ${seed} This will execute the *entrypoint* function in the file **model.py**. .. _running-cloud: Run in the cloud ~~~~~~~~~~~~~~~~ In order to run the calibration process in the cloud, you have two options: 1. If you have access to the underlying machine that will run the container, upload the template folder (using SSH, FTP...) and then run the same steps on the remote machine as if it was a local machine, as explained above. 2. If you have access to a service that can run Docker images, you must first upload your image to a `registry `_ that can be accessed by this service. .. note:: The 2nd method also works if you have access to the machine, and instead of uploading the template and building the image on the remote machine, you can upload the image built locally and pull it from the remote machine. Here we explain how you can upload your image to a registry. Common registries are: - `Docker hub (the official Docker registry) `_ - `ECR (from Amazon Web Services) `_ - `Azure Container Registry (from Microsoft) `_ - `Self hosted repository `_ Here we present a guide for the first two repositories, as well as a basic guide on how to run an image in Amazon EC2 virtual machines. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Upload your image to Dockerhub ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The first step is to create a user at ``_. With a user, you can go to ``_ to start creating your repository. A repository will contain all the versions for an image. Give your repository a descriptive name, usually it is the same as the one given for the image. Once the repository is created, you have to name the image accordingly. This can be done before creating the image (i.e. when building the image specifying the name), or after by using :code:`docker tag`. The name will be composed by your user, the name, and the tag: :code:`myUser/myImage:myTag`, although the :code:`:myTag` part is optional, and if not specified the tag will be *latest*. .. note:: The tag should be specified if you plan to publish/use/keep multiple versions of your image If you have created your image before and you did not set the name for this repository, it usually will be :code:`myImage:myTag`. You can set another name for the image by using: .. code-block:: bash docker image tag myImage:myTag myUser/myImage:myTag .. note:: This will not delete the new name for the image. You will have listed two images with different name but pointing to the same image. Once you have your image correctly named (you can check if it is listed in :code:`docker image ls`), you can push it to Docker hub by using: .. code-block:: bash docker push myUser/myImage:myTag More info on the push command can be found `here `_. .. note:: If you create a private repository, you have to run :code:`docker login` before being able to push or access the image from the current machine. ~~~~~~~~~~~~~~~~~~~~~~~~ Upload your image to ECR ~~~~~~~~~~~~~~~~~~~~~~~~ The process to upload your image to ECR is similar to the one used to push to Docker hub. The main difference is that the names of the images should contain an indication that the registry used is ECR. First, you have to create a repository for your image. You can either use the web interface available at ``_ or use the `AWS cli tool (recomended) `_. To create the repository in the default registry using the CLI tool use: .. code-block:: bash aws ecr create-repository \ --repository-name myImage \ --image-scanning-configuration scanOnPush=true \ --region ${aws_region} Then, name the image accordingly: .. code-block:: bash docker tag myImage:myTag ${aws_account_id}.dkr.ecr.${aws_region}.amazonaws.com/myImage:myTag and push it (previously login in to have permissions): .. code-block:: bash aws ecr get-login-password --region ${aws_region} | docker login \ --username AWS --password-stdin \ ${aws_account_id}.dkr.ecr.${aws_region}.amazonaws.com docker push ${aws_account_id}.dkr.ecr.${aws_region}.amazonaws.com/myImage:myTag More details can be found at `ECR (AWS) `_ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Run in an EC2 virtual machine ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To calibrate the model in a virtual machine in the cloud you can spawn a EC2 instance `using the AWS CLI tool `_ or the `web interface `_. When prompted for the instance AMI, choose *Amazon Linux 2022*, which already has Docker installed. Otherwise, select another AMI and install Docker manually. Once the machine is running and Docker is available in the machine, you can: Build the image in the virtual machine Copy the entire template folder to the virtual machine, for example using :code:`scp` Pull an already built image that is available in a registry Connect to the remote machine, and then pull the image from your registry with .. code-block:: # From dockerhub docker pull myUser/myImage:myTag # or from ECR docker pull ${aws_account_id}.dkr.ecr.${aws_region}.amazonaws.com/myImage:myTag .. note:: Pulling an image is optional, as :code:`docker run` will pull images that are not found locally Then, you can simply run the calibration process as explained in :ref:`run-docker`. .. rubric:: Footnotes .. [#] The :code:`--rm` flag will delete automatically the container once it finishes. Do not use it if you plan to read the logs or pull some files from the container later on. Here, as we only consult the help we can remove it safely.