LAMMPS

LAMMPS stands for Large-scale Atomic/Molecular Massively Parallel Simulator and is copywrited by Sandia (DOE). See the On-line Manual for LAMMPS documentation. Here we provide sample files, give instructions for users to install their own version, and give instructions how to run it on the CPUs and GPUs with the staff supported version of LAMMPS on Henry2 HPC.

External Links:
LAMMPS website
LAMMPS Tutorials

Typically, it's not a good idea to run lammps from your /home directory, as its output can easily overflow your quota.



The new 29Sep21u3 staff-supported version of LAMMPS runs on CPUs and the newer rtx2080 GPUs (NVIDIA):


The 7Aug19 staff-supported version of LAMMPS runs on CPUs and the newer rtx2080 GPUs (NVIDIA):


The 22Aug2018 staff-supported version of LAMMPS runs on CPUs and most of the 60 GPUs (NVIDIA):
- The 4 newest GPUs, four (4) RTX 2080Ti NVIDIA
- The 48 older GPUs, M2070Q + M2090
- The 3 older GPUs, 3 M2070 (1 GPU per node, instead of 2)


Table of Contents - Select for a Particular Topic
• LAMMPS with GPUS
• LAMMPS with CPUS
• How to install and maintain your own LAMMPS version - with conda.
• How to install and maintain your own LAMMPS version - with makefile.
• List of LAMMPS packages in the staff supported version.


• LAMMPS for GPUs

To test LAMMPS on the latest 29Sep21u3 version for GPUs (interactively) do

 
bsub -Is -n 2 -W 10 -q gpu -R "select[rtx2080]" -gpu "num=2:mode=shared:mps=yes" tcsh
module load lammps
mpirun lmp_mpi -sf gpu < /usr/local/apps/lammps/lammps-22Aug2018/examples/accelerate/in.lj

To test LAMMPS version 29Sep21u3 for GPUs in batch mode, create "testGPU.bsub" with the following lines (mps=yes helps multi-processor jobs run faster concurrently on newer GPUs - rtx2080, gtx1080, k20m):

#! /bin/tcsh
#BSUB -n 2
#BSUB -W 20
#BSUB -q gpu
#BSUB -R "select[rtx2080]"
#BSUB -gpu "num=2:mode=shared:mps=yes"
#BSUB -o out.%J
#BSUB -e err.%J
module load lammps
mpirun lmp_mpi -sf gpu < /usr/local/apps/lammps/lammps-7Aug19/examples/accelerate/in.lj

To test LAMMPS version 7Aug19 for GPUs in batch mode, create "testGPU.bsub" with the following lines (mps=yes helps multi-processor jobs run faster concurrently on newer GPUs - rtx2080, gtx1080, k20m):

#! /bin/tcsh
#BSUB -n 2
#BSUB -W 20
#BSUB -q gpu
#BSUB -R "select[rtx2080]"
#BSUB -gpu "num=1:mode=shared:mps=yes"
#BSUB -o out.%J
#BSUB -e err.%J
module load PrgEnv-intel/2018.2.199
module load cuda/10.1
mpirun /usr/local/apps/lammps/lammps-7Aug19/src/lmp_gpu-rtx2080 -sf gpu < /usr/local/apps/lammps/lammps-7Aug19/examples/accelerate/in.lj

To test lammps-22Aug2018 in batch mode, create "testGPU.bsub" with the following lines:

#! /bin/tcsh
#BSUB -n 2
#BSUB -W 20
#BSUB -q gpu
#BSUB -R "select[m2070q || m2090]"
#BSUB -gpu "num=1:mode=shared:mps=no"
#BSUB -o out.%J
#BSUB -e err.%J
module load PrgEnv-intel/2016.0.109
source /usr/local/apps/cuda/cuda-8.0/cuda.csh
mpirun /usr/local/apps/lammps/lammps-22Aug2018/src/lmp_gpu -sf gpu < /usr/local/apps/lammps/lammps-22Aug2018/examples/accelerate/in.lj


How to select a specfic GPU:
• lshosts | grep gpu | more
This shows a list of the hosts with gpus. The far right column shows the gpu model as names that could be used in the "-R select" command.
For example, to select the newest RTX2080 GPU, use
#BSUB -R "select[rtx2080]"
To use any of the 6 newer gpus (the RTX2080 or the GTX1080), use:
#BSUB -R "select[rtx2080 || gtx1080]"
To use the P100 GPU for applications that require double precision (eg. ab-initio applications), use:
#BSUB -R "select[p100]"

• ngpus>1
Most of the GPU nodes have 2 GPUs per node, but some have 1.
To select only the nodes that have 2 or more GPUs per node (all of them except 3 nodes)
#BSUB -R "select[ngpus>1....]"

• To select a specfic hostname, -m (not recommended)
To select a specific hostname, use:
#BSUB -m n2e6-1
• To avoid a specific hostname, hname!= (not recommended)
To avoid a specific hostname, use:
#BSUB -R "(hname != n2e6-3)"....


• LAMMPS for CPUs
To test LAMMPS version 29Sep21u3 for CPUs (interactively) do

bsub -Is -n 2 -W 10 tcsh
module load lammps
mpirun lmp_mpi < /usr/local/apps/lammps/lammps-7Aug19/examples/accelerate/in.lj

To do the same thing in batch mode, create "testCPU.bsub" with the following lines:

#! /bin/tcsh
#BSUB -n 2
#BSUB -W 10
#BSUB -o out.%J
#BSUB -e err.%J
module load lammps
mpirun lmp_mpi < /usr/local/apps/lammps/lammps-7Aug19/examples/accelerate/in.lj

Run this batch job with:

bsub < test.bsub

• How to install and maintain your own LAMMPS version - with conda.

Users are encouraged to install and maintain their own version of LAMMPS in:

/usr/local/usrapps/projectid

Where "projectid" is a directory name given to the project by the HPC administrator; a request must be made (an alternative is to install it in /home/username, which takes up about 310 MB out of 1000 MB of /home/username space).

These two links give an overview of the installation process via conda:
Initialize conda environment on Henry2
Install via conda
Go to /usr/local/usrapps/projectid. Then initialize conda with:

module load conda
conda init tcsh
[log out then back in again]

Then do


conda config --add channels conda-forge
conda create --prefix /usr/local/usrapps/[your_path]/my_lammps_env
conda activate /usr/local/usrapps/[your_path]/my_lammps_env
conda install lammps

When installation is complete, you can test it with a batch job "test.bsub", that looks like:

#! /bin/tcsh
#BSUB -n 2
#BSUB -W 10 
#BSUB -o out.%J
#BSUB -e err.%J
conda activate /usr/local/usrapps/[your_path]/my_lammps_env
mpirun lmp_mpi < /usr/local/apps/lammps/lammps-7Aug19/examples/accelerate/in.lj

Run this batch job with:

bsub < test.bsub

The output should be similar to: out.37038
• How to install and maintain your own LAMMPS version - with makefile.

Users are encouraged to install and maintain their own version of LAMMPS in:

/usr/local/usrapps/projectid

Where "projectid" is a directory name given to the project by the HPC administrator (a request must be made). You can download LAMMPS in several ways, but the git or svn way allows you to easily update. The downside of these ways is that the documentation is not downloaded. Go to /usr/local/usrapps/projectid. Then

git clone -b stable https://github.com/lammps/lammps.git 
/usr/local/usrapps/projectid

Weeks later, you can check for updates with

git checkout stable
git pull

In /usr/local/usrapps/projectid, gunzip and untar the file.
Copy /usr/local/apps/lammps/lammps-5Jun19/src/MAKE/MINE/Makefile.cpu to /usr/local/usrapps/projectid/lammps-5Jun19/src/MAKE/MINE/Makefile.cpu
Note that in Makefile.cpu, the line with CCFLAGS has

CCFLAGS = -g -O3 -restrict -xSSSE3 -axSSE4.2,AVX,CORE-AVX-I,CORE-AVX2

By using using this, a "fat" binary is created. That will allow the executable to run on the whole range of cpus available on the HPC.

Now compile: You have a choice of compilers
The lammps-5Jun19 distribution was tested to compile with
intel2018 (up to -O3 optimization)
The lammps16Mar18 distribution was tested to compile with
intel2016 (up to -O3 optimization)
intel2018 (up to -O3 optimization)
PGI-Porland (up to -O1 optimization)
Invoke the compiler environments with either


module load PrgEnv-intel/2018.2.199

or

module load PrgEnv-intel/2016.0.109

or

module load PrgEnv-pgi/18.4

Then go to directory /usr/local/usrapps/projectid/lammpsxxx/src
Then compile it with "make -j 2 cpu"
Then test it in a way similar to "LAMMPS for CPUs" above.
For the GPU package, ../lib/gpu/libgpu.a must be created - follow instructions at the end of the ../lib/gpu/README file. In order for NVIDIA's mps service to work on the newer GPUs (rtx2080, gtx1080, k20m), use -DCUDA_PROXY in the CUDR_CPP variable in the ../lib/gpu/Makefile.linux file.
• List of packages that are installed in the staff supported version.

The list of installed packages (as of 06/17/2022) in 29Sep21u3's version:
make pi
Installed YES: package ASPHERE
Installed YES: package BODY
Installed YES: package CLASS2
Installed YES: package COLLOID
Installed YES: package COMPRESS
Installed YES: package CORESHELL
Installed YES: package DIPOLE
Installed YES: package GPU
Installed YES: package GRANULAR
Installed YES: package INTEL
Installed YES: package KSPACE
Installed YES: package MANYBODY
Installed YES: package MC
Installed YES: package MISC
Installed YES: package MOLECULE
Installed YES: package MPIIO
Installed YES: package OPT
Installed YES: package PERI
Installed YES: package PLUMED
Installed YES: package PYTHON
Installed YES: package QEQ
Installed YES: package QMMM
Installed YES: package REAXFF
Installed YES: package REPLICA
Installed YES: package RIGID
Installed YES: package SHOCK
Installed YES: package SPIN
Installed YES: package SRD

Last modified: June 17 2022 15:09:57.