Application Script Examples
GROMACS Applications
GROningen MAchine for Chemical Simulations (GROMACS) is a
molecular dynamics
package mainly designed for simulations of
proteins
,
lipids
, and
nucleic acids
. It was originally developed in the Biophysical Chemistry department of
University of Groningen
, and is now maintained by contributors in universities and research centres worldwide. GROMACS is one of the fastest and most popular software packages available, and can run on
central processing units
(CPUs) and
graphics processing units
(GPUs).
Step 1: Log in to ParamShakti with your login credentials.
ssh
your_name@paramshakti.iitkgp.ac.in
Step 2: Load Gromacs software.
Check for availability of the software using “module avail gromacs”. It shows the list of available
gromacs software versions on ParamShakti. Then, load the gromacs software using “module load
gromacs_version” (For example, “module load apps/gromacs/2020/cpu”).
Step 3: Gromacs Input File
a)create a directory in your scratch folder and navigate to that directory.
“mkdir /scratch/$USER/gromacs; cd /scratch/$USER/gromacs”
b)Input file can be download to /scratch/$USER/ from
“wget
https://ftp.gromacs.org/pub/benchmarks/water_GMX50_bare.tar.gz”
“tar xzf water_GMX50_bare.tar.gz”
c) Navigate to directory 3072 ”cd water-cut1.0_GMX50_bare/3072”
you will see 4 input files “conf.gro pme.mdp rf.mdp topol.top”
d) Generate the "topol.tpr" binary file using the command "module load apps/gromacs/2020/cpu;gmx_mpi grompp -f pme.mdp -c conf.gro -p topol.top"
you will see 4 input files “conf.gro pme.mdp rf.mdp topol.top”
The mdp option can be used is pme with 50000 steps
pme.mdp
title = dppc
cpp = /lib/cpp
integrator = md
nsteps = 50000 ; !autoset
nstlist = 10
nstfout = 0
nstxout = 0
nstvout = 0
nstxtcout = 0
nstlog = 0
dt = 0.002
constraints = all-bonds
nstenergy = 0
ns_type = grid
coulombtype = PME ; !autoset
rlist = 1.0 ; !autoset
rvdw = 1.0 ; !autoset
rcoulomb = 1.0 ; !autoset
tcoupl = v-rescale
tc_grps = system
tau_t = 0.1
ref_t = 300
freezegrps = ; !autoset
freezedim = ; !autoset
fourier_spacing = 0.125 ; !autoset
nstcalcenergy = 100 ; !autogen
cutoff-scheme = verlet ; !autogen
topol.top
#include "oplsaa.ff/forcefield.itp"
[ moleculetype ]
; molname nrexcl
SOL 2
[ atoms ]
; nr type resnr residue atom cgnr charge mass
1 opls_116 1 SOL OW 1 -0.8476
2 opls_117 1 SOL HW1 2 0.4238
3 opls_117 1 SOL HW2 3 0.4238
#ifndef FLEXIBLE
[ settles ]
; OW funct doh dhh
1 1 0.1 0.16330
[ exclusions ]
1 2 3
2 1 3
3 1 2
#else
[ bonds ]
; i j funct length force.c.
1 2 1 0.1 345000 0.1 345000
1 3 1 0.1 345000 0.1 345000
[ angles ]
; i j k funct angle force.c.
2 1 3 1 109.47 383 109.47 383
#endif
[ system ]
Water
[ molecules ]
SOL 256000
Step 4: Copy the slurm script given below and save it as “submit.sh”. Now, submit the job using the command “sbatch submit.sh”. On successful submission, you can see the job status using “squeue -u $USER”.
MPI-CPU Script
This will initiate 4 MPI Process with 4 threads each .
#!/bin/bash
#SBATCH -J job_name # name of the job
#SBATCH -p medium # name of the partition: available options "large, medium or shared"
#SBATCH -n 4 # no of processes or tasks
#SBATCH --cpus-per-task=4 # no of threads per process or task
#SBATCH -t 01:00:00 # walltime in HH:MM:SS, Max value 72:00:00
#list of modules you want to use, for example
module load apps/gromacs/2020/cpu
#name of the executable
exe="gmx_mpi"
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
#run the application
mpirun -bootstrap slurm -n $SLURM_NTASKS $exe mdrun -s topol.tpr # specify the application command-line options, if any, after $exe
GPU Script
This will initiate 4 MPI Process with 6 threads(4 Worker threads and 2 communication threads) in a single GPU device .
#!/bin/bash
#SBATCH -J job_name # name of the job
#SBATCH -p gpu # name of the partition: available options "gpu"
#SBATCH -n 4 # no of processes or tasks
#SBATCH --gres=gpu:1 # request gpu card: it should be either 1 or 2
#SBATCH --cpus-per-task=4 # no of threads per process or task
#SBATCH -t 01:00:00 # walltime in HH:MM:SS, max value 72:00:00
#list of modules you want to use, for example
module load apps/gromacs/2020/gpu_cuda_10.1
#name of the executable
exe="gmx_mpi"
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
#run the application
mpirun -bootstrap slurm -n $SLURM_NTASKS $exe mdrun -s topol.tpr # specify the application command-line options, if any, after $exe
Step 5: Gromacs output files
After the job starts running, you can see the output files related to Gromacs in the same directory. For example,
[xxxxxxx@login01 3072]$ cat slurm-112741.out
==========================================
SLURM_CLUSTER_NAME = param-shakti
SLURM_JOB_ACCOUNT = xxxxxxx
SLURM_JOB_ID = 112741
SLURM_JOB_NAME = job_name
SLURM_JOB_NODELIST = cn098
SLURM_JOB_USER = xxxxxxx
SLURM_JOB_UID = 6034
SLURM_JOB_PARTITION = medium
SLURM_TASK_PID = 40413
SLURM_SUBMIT_DIR = /scratch/xxxxxxx/gromacs/water-cut1.0_GMX50_bare/3072
SLURM_CPUS_ON_NODE = 16
SLURM_NTASKS = 4
SLURM_TASK_PID = 40413
==========================================
:-) GROMACS - gmx mdrun, 2020-UNCHECKED (-:
GROMACS is written by:
Emile Apol Rossen Apostolov Paul Bauer Herman J.C. Berendsen
Par Bjelkmar Christian Blau Viacheslav Bolnykh Kevin Boyd
Aldert van Buuren Rudi van Drunen Anton Feenstra Alan Gray
Gerrit Groenhof Anca Hamuraru Vincent Hindriksen M. Eric Irrgang
Aleksei Iupinov Christoph Junghans Joe Jordan Dimitrios Karkoulis
Peter Kasson Jiri Kraus Carsten Kutzner Per Larsson
Justin A. Lemkul Viveca Lindahl Magnus Lundborg Erik Marklund
Pascal Merz Pieter Meulenhoff Teemu Murtola Szilard Pall
Sander Pronk Roland Schulz Michael Shirts Alexey Shvetsov
Alfons Sijbers Peter Tieleman Jon Vincent Teemu Virolainen
Christian Wennberg Maarten Wolf Artem Zhmurov
and the project leaders:
Mark Abraham, Berk Hess, Erik Lindahl, and David van der Spoel
Copyright (c) 1991-2000, University of Groningen, The Netherlands.
Copyright (c) 2001-2019, The GROMACS development team at
Uppsala University, Stockholm University and
the Royal Institute of Technology, Sweden.
check out http://www.gromacs.org for more information.
GROMACS is free software; you can redistribute it and/or modify it
under the terms of the GNU Lesser General Public License
as published by the Free Software Foundation; either version 2.1
of the License, or (at your option) any later version.
GROMACS: gmx mdrun, version 2020-UNCHECKED
Executable: /home/apps/gromacs_2020/bin/gmx_mpi
Data prefix: /home/apps/gromacs_2020
Working dir: /scratch/xxxxxx/gromacs/water-cut1.0_GMX50_bare/3072
Command line:
gmx_mpi mdrun -s topol.tpr
Compiled SIMD: AVX2_256, but for this host/run AVX_512 might be better (see
log).
Reading file topol.tpr, VERSION 2020-UNCHECKED (single precision)
Changing nstlist from 10 to 40, rlist from 1 to 1.099
Using 4 MPI processes
Non-default thread affinity set, disabling internal thread affinity
Using 4 OpenMP threads per MPI process
starting mdrun 'Water'
5000 steps, 10.0 ps.
Writing final coordinates.
Dynamic load balancing report:
DLB was off during the run due to low measured imbalance.
Average load imbalance: 0.5%.
The balanceable part of the MD step is 60%, load imbalance is computed from this.
Part of the total run time spent waiting due to load imbalance: 0.3%.
Core t (s) Wall t (s) (%)
Time: 19057.053 1191.067 1600.0
(ns/day) (hour/ns)
Performance: 0.726 33.079
GROMACS reminds you: "The Microsecond is Within Reach" (P.J. Van Maaren)
[xxxxxx@login01 3072]$
LAMMPS applications
LAMMPS is an acronym for Large-scale Atomic/ Molecular Massively Parallel Simulator. This is extensively used in the fields of Material Science, Physics, Chemistry and may others. More information about LAMMPS may please be found at
https://lammps.sandia.gov.
Step 1: Log in to ParamShakti with your login credentials.
ssh
your_name@paramshakti.iitkgp.ac.in
Step 2: Load LAMMPS software.
Check for availability of the software using “module avail lammps”. It shows the list of available
lammps software versions on ParamShakti. Then, load the lammps software using “module load
lammps_version” (For example, “module load apps/lammps/12.12.2018/intel”).
Step 3: LAMMPS Input file
Create a new directory and copy the LAMMPS input file given below and save the file as “in.lj”.
# 3d Lennard-Jones melt
variable x index 1
variable y index 1
variable z index 1
variable xx equal 64*$x
variable yy equal 64*$y
variable zz equal 64*$z
units lj
atom_style atomic
lattice fcc 0.8442
region box block 0 ${xx} 0 ${yy} 0 ${zz}
create_box 1 box
create_atoms 1 box
mass 1 1.0
velocity all create 1.44 87287 loop geom
pair_style lj/cut 2.5
pair_coeff 1 1 1.0 1.0 2.5
neighbor 0.3 bin
neigh_modify delay 0 every 20 check no
fix 1 all nve
run 1000000
Step 4: Copy the slurm script given below and save it as “submit.sh”. Now, submit the job using the command “sbatch submit.sh”. On successful submission, you can see the job status using “squeue -u $USER”.
MPI-CPU Script
This will initiate 4 MPI Process with 1 threads(1 worker thread) each .
#!/bin/bash
#SBATCH -J job_name # name of the job
#SBATCH -p medium # name of the partition: available options "large, medium or shared "
#SBATCH -n 4 # no of processes or tasks
#SBATCH --cpus-per-task=1 # no of threads per process or task
#SBATCH -t 01:00:00 # walltime in HH:MM:SS, max value 72:00:00
#list of modules you want to use, for example
module load apps/lammps/7Aug19/intel
#name of the executable
exe="lmp_mpi"
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
#run the application
mpirun -bootstrap slurm -n $SLURM_NTASKS $exe -in in.lj # specify the application command-line options, if any, after $exe
GPU Script
This will initiate 4 MPI Process in a single GPU device .
#!/bin/bash
#SBATCH -J job_name # name of the job
#SBATCH -p gpu # name of the partition: available options "gpu"
#SBATCH -n 4 # no of processes or tasks
#SBATCH --gres=gpu:1 # request gpu card: it should be either 1 or 2
#SBATCH --cpus-per-task=1 # no of threads per process or task
#SBATCH -t 01:00:00 # walltime in HH:MM:SS, max value 72:00:00
#list of modules you want to use, for example
module load apps/lammps/29Oct2020/cuda10.2/gpu
#name of the executable
exe="lmp_gpu"
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
#run the application
mpirun -bootstrap slurm -n $SLURM_NTASKS $exe -sf gpu -pk gpu 1 -in in.lj # specify the application command-line options, if any, after $exe
Step 5: LAMMPS output files
After the job starts running, you can see the output files related to LAMMPS in the same directory. For example, see log.lammps file
LAMMPS (22 Aug 2018)
using 1 OpenMP thread(s) per MPI task
Lattice spacing in x,y,z = 1.6796 1.6796 1.6796
Created orthogonal box = (0 0 0) to (107.494 107.494 107.494)
8 by 10 by 16 MPI processor grid
Created 1048576 atoms
Time spent = 0.048476 secs
Neighbor list info ...
update every 20 steps, delay 0 steps, check no
max neighbors/atom: 2000, page size: 100000
master list distance cutoff = 2.8
ghost atom cutoff = 2.8
binsize = 1.4, bins = 77 77 77
1 neighbor lists, perpetual/occasional/extra = 1 0 0
(1) pair lj/cut, perpetual
attributes: half, newton on
pair build: half/bin/atomonly/newton
stencil: half/bin/3d/newton
bin: standard
Setting up Verlet run ...
Unit style : lj
Current step : 0
Time step : 0.005
Per MPI rank memory allocation (min/avg/max) = 2.699 | 2.703 | 2.708 Mbytes
Step Temp E_pair E_mol TotEng Press
0 1.44 -6.7733681 0 -4.6133701 -5.0196704
1000000 0.65695755 -5.7125359 0 -4.7271005 0.48799127
Loop time of 723.716 on 1280 procs for 1000000 steps with 1048576 atoms
Performance: 596918.946 tau/day, 1381.757 timesteps/s
99.5% CPU use with 1280 MPI tasks x 1 OpenMP threads
MPI task timing breakdown:
Section | min time | avg time | max time |%varavg| %total
---------------------------------------------------------------
Pair | 424.38 | 435.47 | 461.05 | 26.2 | 60.17
Neigh | 59.782 | 60.365 | 62.991 | 3.9 | 8.34
Comm | 193.24 | 219.39 | 231.11 | 38.5 | 30.31
Output | 0.00013494 | 0.00085223 | 0.0088639 | 0.0 | 0.00
Modify | 6.4813 | 6.6462 | 7.541 | 5.6 | 0.92
Other | | 1.841 | | | 0.25
Nlocal: 819.2 ave 845 max 786 min
Histogram: 3 2 34 115 256 372 315 137 33 13
Nghost: 2417.97 ave 2468 max 2369 min
Histogram: 8 31 81 216 314 327 202 76 22 3
Neighs: 30698 ave 32432 max 28796 min
Histogram: 4 16 47 194 306 325 245 103 34 6
Total # of neighbors = 39293494
Ave neighs/atom = 37.4732
Neighbor list builds = 50000
Dangerous builds not checked
Total wall time: 0:12:03
NAMD Application
Nano Scale Molecular Dynamics (NAMD) software for molecular dynamics simulation is designed for high-performance simulations of Large Macro Molecular system on parallel computers. This software also makes use of GPGPUs. NAMD was developed by the Theoretical and Computational Biophysics Group in the Beckman Institute for Advanced Science and Technology at the University of Illinois at Urbana-Champaign. More information about NAMD may please be found at
http://www.ks.uiuc.edu/Research/namd/
Step 1: Log in to ParamShakti with your login credentials.
ssh
your_name@paramshakti.iitkgp.ac.in
Step 2: Load namd software.
Check for availability of the software using “module avail namd”. It shows the list of available
namd software versions on ParamShakti. Then, load the namd software using “module load
namd_version” (For example, “module load apps/namd/2.13/impi2019v5/cpu”).
Step 3: Namd Input file
a)create a directory in your scratch folder and navigate to that
directory.
“mkdir /scratch/$USER/namd; cd /scratch/$USER/namd”
b)Input file can be download to /scratch/$USER/ from
“wget http://www.ks.uiuc.edu/Research/namd/utilities/stmv.tar.gz”
“tar xzf stmv.tar.gz”
mv stmv/* .
c)Edit stmv.namd and change value of numsteps e.g. to 10000
stmv.namd
#############################################################
## ADJUSTABLE PARAMETERS ##
#############################################################
structure stmv.psf
coordinates stmv.pdb
#############################################################
## SIMULATION PARAMETERS ##
#############################################################
# Input
paraTypeCharmm on
parameters par_all27_prot_na.inp
temperature 298
# Force-Field Parameters
exclude scaled1-4
1-4scaling 1.0
cutoff 12.
switching on
switchdist 10.
pairlistdist 13.5
# Integrator Parameters
timestep 1.0
nonbondedFreq 1
fullElectFrequency 4
stepspercycle 20
# Constant Temperature Control
langevin on ;# do langevin dynamics
langevinDamping 5 ;# damping coefficient (gamma) of 5/ps
langevinTemp 298
langevinHydrogen off ;# don't couple langevin bath to hydrogens
# Constant Pressure Control (variable volume)
useGroupPressure yes ;# needed for rigidBonds
useFlexibleCell no
useConstantArea no
langevinPiston on
langevinPistonTarget 1.01325 ;# in bar -> 1 atm
langevinPistonPeriod 100.
langevinPistonDecay 50.
langevinPistonTemp 298
cellBasisVector1 216.832 0. 0.
cellBasisVector2 0. 216.832 0.
cellBasisVector3 0. 0 216.832
cellOrigin 0. 0. 0.
PME on
PMEGridSizeX 216
PMEGridSizeY 216
PMEGridSizeZ 216
# Output
outputName /usr/tmp/stmv-output
outputEnergies 20
outputTiming 20
numsteps 10000
Step 4: Copy the slurm script given below and save it as “submit.sh”. Now, submit the job using the command “sbatch submit.sh”. On successful submission, you can see the job status using “squeue -u $USER”. It will initiate 2 MPI Process, each with 5 threads (4 worker threads and 1 communication thread).
MPI-CPU Script
#!/bin/bash
#SBATCH -J job_name # name of the job
#SBATCH -p medium # name of the partition: available options "large, medium or shared "
#SBATCH -n 2 # no of processes or tasks
#SBATCH --cpus-per-task=4 # no of threads per process or task
#SBATCH -t 01:00:00 # walltime in HH:MM:SS, Max value 72:00:00
#list of modules you want to use, for example
module load apps/namd/2.13/impi2019v5/cpu
#name of the executable
exe="namd2"
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
#run the application
mpirun -bootstrap slurm -n $SLURM_NTASKS $exe +ppn $OMP_NUM_THREADS +idlepoll ./stmv.namd # specify the application command-line options, if any, after $exe
GPU Script
This will initiate 2 MPI Process with 9 threads each (6 worker threads per process + addition 3 communication threads)
#!/bin/bash
#SBATCH -J job_name # name of the job
#SBATCH -p gpu # name of the partition: available options "gpu"
#SBATCH -n 2 # no of processes or tasks (Max 2 for this script)
#SBATCH --gres=gpu:2 # request gpu card: it should be either 1 or 2
#SBATCH --cpus-per-task=6 # no of threads per process or task
#SBATCH -t 01:00:00 # walltime in HH:MM:SS, max value 72:00:00
#list of modules you want to use, for example
module load apps/namd/2.13/impi2019v5/cuda
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
#name of the executable
exe="namd2"
#run the application
mpirun -bootstrap slurm -n $SLURM_NTASKS $exe +ppn $OMP_NUM_THREADS +idlepoll ./stmv.namd +devices 0,1 # specify the application command-line options, if any, after $exe
Step 5: NAMD output files
After the job starts running, you can see the output files related to Namd in the same directory. For example,
TIMING: 10000 CPU: 140.503, 0.0137648/step Wall: 142.867, 0.0139643/step, 0 hours remaining, 2153.882812 MB of memory in use.
ETITLE: TS BOND ANGLE DIHED IMPRP ELECT VDW BOUNDARY MISC KINETIC TOTAL TEMP POTENTIAL TOTAL3 TEMPAVG PRESSURE GPRESSURE VOLUME PRESSAVG GPRESSAVG
ENERGY: 10000 367211.7587 279345.1935 82007.0777 5091.8001 -4523649.5745 385248.5094 0.0000 0.0000 945987.1898 -2458758.0453 297.5373 -3404745.2351 -2450238.2196 297.4597 -56.1661 -17.8061 10187151.2683 17.4129 18.8925
WRITING EXTENDED SYSTEM TO OUTPUT FILE AT STEP 10000
WRITING COORDINATES TO OUTPUT FILE AT STEP 10000
The last position output (seq=-2) takes 0.012 seconds, 2232.031 MB of memory in use
WRITING VELOCITIES TO OUTPUT FILE AT STEP 10000
The last velocity output (seq=-2) takes 0.012 seconds, 2232.105 MB of memory in use
====================================================
WallClock: 168.633499 CPUTime: 165.723175 Memory: 2232.113281 MB
[Partition 0][Node 0] End of program
real 2m55.139s
user 96m16.215s
sys 9m7.399s