LAMMPS

LAMMPS stands for Large-scale Atomic/Molecular Massively Parallel Simulator and is copywrited by Sandia (DOE). See the On-line Manual for LAMMPS documentation, which is linked below. Here we give instructions how to run it on the CPUs and GPUs with the staff supported version on the Hazel HPC. Also given are instructions for users to install their own version.

External Links:
LAMMPS website
LAMMPS Tutorials
LAMMPS benchmarks


Typically, it's not a good idea to run lammps from your /home directory, as its output can easily overflow your quota.

2022Jun23, the latest staff-supported version of LAMMPS, runs on CPUs and the new a30 GPUs (NVIDIA):
The modulefiles for the newest installation can be seen with

module avail lammps/2022Jun23

They are:
lammps/2022Jun23/intel/gpu
lammps/2022Jun23/intel/cpu
lammps/2022Jun23/gnu/gpu
lammps/2022Jun23/gnu/cpu


The new default modulefile is:
lammps/2022Jun23/intel/cpu
It was
lammps/29Sep21u3

Older versions:
The 29Sep21u3 staff-supported version of LAMMPS runs on CPUs and the rtx2080 GPUs (NVIDIA):
The 7Aug19 staff-supported version of LAMMPS runs on CPUs and the rtx2080 GPUs (NVIDIA):
The 22Aug2018 staff-supported version of LAMMPS runs on CPUs and most of the older GPUs (NVIDIA), including the four (4) RTX 2080Ti NVIDIA


Table of Contents - Select for a Particular Topic
• LAMMPS with GPUS
• LAMMPS with CPUS
• LAMMPS Hazel benchmarks
• How to install and maintain your own LAMMPS version - with conda.
• How to install and maintain your own LAMMPS version - with makefile.
• List of LAMMPS packages in the staff supported version.


• LAMMPS for GPUs

To test LAMMPS on the latest 2022Jun23 version for GPUs (interactively) do

 
bsub -Is -n 16 -R "span[hosts=1]" -W 5 -q gpu -R "select[a30]" -gpu "num=2:mode=shared:mps=yes" bash
module load lammps/2022Jun23/intel/gpu
mpirun lmp_hazel -sf gpu -in /usr/local/apps/lammps/lammps2022Jun23intelgpu/bench/in.lj.512k

To test LAMMPS version 2022Jun23 for GPUs in batch mode, create "testGPU.bsub" with the following lines (mps=yes helps multi-processor jobs run faster concurrently on newer GPUs - a30, rtx2080, gtx1080, k20m):

#!/bin/bash
#BSUB -n 16
#BSUB -R "span[hosts=1]"
#BSUB -W 5
#BSUB -q gpu
#BSUB -R "select[a30]"
#BSUB -gpu "num=2:mode=shared:mps=yes"
#BSUB -o out.%J
#BSUB -e err.%J
module load lammps/2022Jun23/intel/gpu
mpirun lmp_hazel -sf gpu -in /usr/local/apps/lammps/lammps2022Jun23intelgpu/bench/in.lj.512k

and run it with:

bsub < testGPU.bsub


How to select a specfic GPU:
• lshosts | grep gpu | more
This shows a list of the hosts with gpus. The far right column shows the gpu model as names that could be used in the "-R select" command.
For example, to select the newest A30 GPU, use
#BSUB -R "select[a30]"
To use any of the 6 oldwer gpus (the RTX2080 or the GTX1080), use:
#BSUB -R "select[rtx2080 || gtx1080]"
To use the P100 GPU for applications that require double precision (eg. ab-initio applications), use:
#BSUB -R "select[p100]"
Most of the GPU nodes have 2 GPUs per node, but the rtx2080 has 4.

• To select a specfic hostname, -m (not recommended)
To select a specific hostname, use:
#BSUB -m "c041n01"
• To avoid a specific hostname, hname!= (not recommended)
To avoid a specific hostname, use:
#BSUB -R "(hname != c014n01)"....


• LAMMPS for CPUs
To test LAMMPS version 2022Jun23 for CPUs (interactively) do

bsub -Is -n 16 -R "span[hosts=1]" -W 10 bash
module load lammps/2022Jun23/intel/cpu
mpirun lmp_hazel < /usr/local/apps/lammps/lammps2022Jun23intelcpu/bench/in.lj.512k

To do the same thing in batch mode, create "testCPU.bsub" with the following lines:

#! /bin/bash
#BSUB -n 16
#BSUB -R "span[hosts=1]"
#BSUB -W 5
#BSUB -o out.%J
#BSUB -e err.%J
module load lammps/2022Jun23/intel/cpu
mpirun lmp_hazel < /usr/local/apps/lammps/lammps2022Jun23intelcpu/in.lj.512k

Run this batch job with:

bsub < testCPU.bsub

• LAMMPS Hazel benchmarks
Running the examples above on Hazel should give timings that meet or exceed the benchmarks shown on LAMMPS' website
lj model, Gold6226 node,
Loop time of 2.13852 on 16 procs for 100 steps with 512000 atoms
lj model, a30 2x GPU node,
Loop time of 0.283579 on 16 procs for 100 steps with 512000 atoms

• How to install and maintain your own LAMMPS version - with conda.

Users are encouraged to install and maintain their own version of LAMMPS in:

/usr/local/usrapps/projectid

Where "projectid" is a directory name given to the project by the HPC administrator; a request must be made (an alternative is to install it in /home/username, which takes up about 310 MB out of 1000 MB of /home/username space).

These two links give an overview of the installation process via conda:
Initialize conda environment on Henry2
Install via conda
Go to /usr/local/usrapps/projectid. Then initialize conda with:

module load conda
conda init tcsh
[log out then back in again]

Then do


conda config --add channels conda-forge
conda create --prefix /usr/local/usrapps/[your_path]/my_lammps_env
conda activate /usr/local/usrapps/[your_path]/my_lammps_env
conda install lammps

When installation is complete, you can test it with a batch job "test.bsub", that looks like:

#! /bin/tcsh
#BSUB -n 2
#BSUB -W 10 
#BSUB -o out.%J
#BSUB -e err.%J
conda activate /usr/local/usrapps/[your_path]/my_lammps_env
mpirun lmp_mpi < /usr/local/apps/lammps/lammps-7Aug19/examples/accelerate/in.lj

Run this batch job with:

bsub < test.bsub

The output should be similar to: out.37038
• How to install and maintain your own LAMMPS version - with makefile.

Users are encouraged to install and maintain their own version of LAMMPS in:

/usr/local/usrapps/projectid

Where "projectid" is a directory name given to the project by the HPC administrator (a request must be made). You can download LAMMPS in several ways, but the git or svn way allows you to easily update. The downside of these ways is that the documentation is not downloaded. Go to /usr/local/usrapps/projectid. Then

git clone -b stable https://github.com/lammps/lammps.git 
/usr/local/usrapps/projectid

Weeks later, you can check for updates with

git checkout stable
git pull

In /usr/local/usrapps/projectid, gunzip and untar the file.
Copy /usr/local/apps/lammps/lammps-5Jun19/src/MAKE/MINE/Makefile.cpu to /usr/local/usrapps/projectid/lammps-5Jun19/src/MAKE/MINE/Makefile.cpu
Note that in Makefile.cpu, the line with CCFLAGS has

CCFLAGS = -g -O3 -restrict -xSSSE3 -axSSE4.2,AVX,CORE-AVX-I,CORE-AVX2

By using using this, a "fat" binary is created. That will allow the executable to run on the whole range of cpus available on the HPC.

Now compile: You have a choice of compilers; intel, gnu, or PGI.
The latest version was compiled with

module load PrgEnv-intel/2022.1.0

or


module load openmpi-gcc/openmpi4.1.0-gcc10.2.0

and

module load cuda/12.0

After invoking the compiler, go to directory /usr/local/usrapps/projectid/lammpsxxx/src
Then compile it with "make -j 2 cpu"
Then test it in a way similar to "LAMMPS for CPUs" above.
For the GPU package, ../lib/gpu/libgpu.a must be created - follow instructions at the end of the ../lib/gpu/README file. In order for NVIDIA's mps service to work on the newer GPUs (a30, rtx2080, gtx1080, k20m), use -DCUDA_PROXY in the CUDR_CPP variable in the ../lib/gpu/Makefile.linux file.
• List of packages that are installed in the staff supported version.

The list of installed packages (as of 2023/03/13) in 2022Jun23's version:
make yes-most
make yes-gpu
make pi
nstalled YES: package ASPHERE
Installed YES: package BOCS
Installed YES: package BODY
Installed YES: package BPM
Installed YES: package BROWNIAN
Installed YES: package CG-DNA
Installed YES: package CG-SDK
Installed YES: package CLASS2
Installed YES: package COLLOID
Installed YES: package CORESHELL
Installed YES: package DIELECTRIC
Installed YES: package DIFFRACTION
Installed YES: package DIPOLE
Installed YES: package DPD-BASIC
Installed YES: package DPD-MESO
Installed YES: package DPD-REACT
Installed YES: package DPD-SMOOTH
Installed YES: package DRUDE
Installed YES: package EFF
Installed YES: package EXTRA-COMPUTE
Installed YES: package EXTRA-DUMP
Installed YES: package EXTRA-FIX
Installed YES: package EXTRA-MOLECULE
Installed YES: package EXTRA-PAIR
Installed YES: package FEP
Installed YES: package GPU
Installed YES: package GRANULAR
Installed YES: package INTERLAYER
Installed YES: package KSPACE
Installed YES: package MANYBODY
Installed YES: package MC
Installed YES: package MEAM
Installed YES: package MISC
Installed YES: package ML-IAP
Installed YES: package ML-SNAP
Installed YES: package MOFFF
Installed YES: package MOLECULE
Installed YES: package OPENMP
Installed YES: package OPT
Installed YES: package ORIENT
Installed YES: package PERI
Installed YES: package PHONON
Installed YES: package PLUGIN
Installed YES: package QEQ
Installed YES: package REACTION
Installed YES: package REAXFF
Installed YES: package REPLICA
Installed YES: package RIGID
Installed YES: package SHOCK
Installed YES: package SPH
Installed YES: package SPIN
Installed YES: package SRD
Installed YES: package UEF
Installed YES: package YAFF

Last modified: March 13 2023 13:14:54.