apptainer run --bind /share/groupname/userid/my_inpudata:/mnt/datafiles mycode.sifYou are highly recommended to go through details of Apptainer Bind Paths and Mounts topic.
module load apptainer
Currently, you can only run Apptainer jobs in the sif queue. To run Apptainer On HPC, you create an LSF job script, say submit.sh, similar to the following:
#!/bin/bash #BSUB -W 15 #BSUB -n 2 #BSUB -q sif #BSUB -R span[hosts=1] #BSUB -o out.%J #BSUB -e err.%J module load apptainer apptainer run mycode.sif
The job can be submitted to LSF by the command
bsub < submit.sh
apptainer pull lolcow.sif docker://sylabsio/lolcowto build an Apptainer container lolcow.sif from that Docker image, and then run the Apptainer container. You may also use the command apptainer build lolcow.sif docker://sylabsio/lolcow to build the Apptainer container. The apptainer pull command actually runs an apptainer build behind the scenes, which translates Docker oci format to Apptainer sif format. Actually, you can also run a Docker container directly with the command apptainer run docker://sylabsio/lolcow. However, that is not recommended. The reason is that Docker Hub has limits on anonymous access to its API. Every time you use a docker:// URI to run (pull, etc.) a container, Apptainer will make requests to Docker Hub in order to check whether the container has been modified there. On a shared system like NCSU HPC cluster, and when many Apptainer container jobs run in parallel, that can quickly exhaust the Docker Hub API limits. It is recommended that you pull a Docker image form the Docker Hub for one time, and then run jobs using the local Apptainer container.
apptainer pull python3.7.sif docker://quay.io/bitnami/python:3.7Pulling images from Quay.io may need authentication.
apptainer pull pytorch.sif docker://nvcr.io/nvidia/pytorch:21.09-py3
apptainer pull alpine.sif docker://ghcr.io/containerd/alpine:latest
docker save 5a15b484bc65 -o mycode.tarto create a tar archive file from the image. Then, you can upload the tar file to HPC and use the command
apptainer build mycode.sif docker-archive:mycode.tarto build an Apptainer container.
The MPI in the container must be compatible with the MPI available the host. The MPI that both Apptainer supports and the HPC cluster supports is Open MPI. Thus, the MPI you use to compile your MPI application in your Apptainer container needs to be Open MPI. HPC cluster has various versions of Open MPI. When you run your containerized MPI app, in which the MPI used is Open MPI, you can use a version of Open MPI on HPC that is compatible with the version of the Open MPI used in your Apptainer container.
Assume your Open MPI based Apptainer container is mpibcast.sif and the MPI based app in the container is /opt/mpibcast. Then, the command in your LSF job script to run your containerized MPI app is:
mpirun apptainer exec --bind /share/${GROUP}/${USER}/tmp:/share mpibcast.sif /opt/mpibcastThe reason why we need to bind the "/share" directory in the container to a your own tmp directory under the "/share" on HPC is, the Open MPI in the container needs to create some temporary files in "/share" directory in the container. If you don't do the bind, then "/share" in the container by default is mapped to "/share" on HPC. However, you don't have write permission for the directory, and that will cause run time problem. Thus, you need to bind the "/share" directory in the container to a directory on HPC for which you have write permission. Of course, in your LSF job script, you should have already loaded the Open MPI module and the Apptainer module before you do the "mpirun apptainer exec ......" command. Below is a sample LSF job script:
#!/bin/bash #BSUB -W 15 #BSUB -n 6 #BSUB -q sif #BSUB -R span[ptile=2] #BSUB -o out.%J #BSUB -e err.%J module load openmpi-gcc/openmpi4.1.0-gcc9.3.0 module load apptainer mpirun apptainer exec --bind /share/${GROUP}/${USER}/tmp:/share mpibcast.sif /opt/mpibcast
You can find an example of MPI application installed in an Apptainer container in the "Some Examples" section below. It contains source code for the MPI application, definition file for the Apptainer container, and the LSF job script for running the container. It also contains information about how to create the Apptainer container. For further information, please refer to Apptainer and MPI applications in Apptainer User Guide.
module load apptainer
apptainer pull lolcow.sif docker://godlovedc/lolcow
#!/bin/bash #BSUB -W 15 #BSUB -n 2 #BSUB -q sif #BSUB -R span[hosts=1] #BSUB -o out.%J #BSUB -e err.%J module load apptainer apptainer run lolcow.sif
bsub < submit.sh
Below are the steps that you can follow to create the container and run the container.
This MPI C code is a simple example of MPI_Bcast. Initially, the int variable aNumber is initialized to value 17 on process 0 and is initialized to value −1 on all other processes. The code prints out the initial value on MPI each process. Then, the code calls MPI_Bcast to broadcast the value of aNumber to all other processes. Finally, the code prints out the final value on each process. The aNumber on all the processes should have the final value of 17.
The definition file selects the OS for the container, sets up needed building and running time environment variables, installs system packages that are needed for installing and using Open MPI, installs Open MPI, and finally compiles the MPI source code mpibcast.c using the Open MPI in the container.
sudo yum install apptainer
sudo apptainer build mpitest.sif mpitest.def
The LSF job script first loads the Open MPI module and the Apptainer module on HPC, and then does the following command to run the container:
mpirun apptainer exec --bind /share/${GROUP}/${USER}/tmp:/share mpibcast.sif /opt/mpibcast
Last modified: April 05 2024 18:43:00.