High Performance Computing Facility

Job Submission

Serial

Script for a Single Core Serial Job

#!/bin/bash
#SBATCH -J job_name                # name of the job
#SBATCH -p shared                  # name of the partition: available options "shared"
#SBATCH -n 1                       # no of processes
#SBATCH -t 01:00:00                # walltime in HH:MM:SS, Max value 72:00:00

#list of modules you want to use, for example
#module load apps/lammps/12.12.2018/intel

#name of the executable
exe=name_executable
#run the application
$exe                               # specify the application command-line options, if any, after $exe

OpenMP

Script for a Single Node OpenMP Job

#!/bin/bash
#SBATCH -J job_name              # name of the job
#SBATCH -p shared                # name of the partition: available options "shared"
#SBATCH -N 1                     # no of nodes
#SBATCH -n 1                     # no of processes or tasks
#SBATCH --cpus-per-task=10       # no of threads per process or task
#SBATCH -t 01:00:00              # walltime in HH:MM:SS, Max value 72:00:00

#list of modules you want to use, for example
#module load apps/lammps/12.12.2018/intel

#name of the executable
exe=name_executable

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

#run the application
$exe                           # specify the application command-line options, if any, after $exe

MPI

Script for a Multinode MPI Job

#!/bin/bash
#SBATCH -J job_name                # name of the job
#SBATCH -p medium                  # name of the partition: available options "large,medium"
#SBATCH -N 2                       # no of nodes
#SBATCH --ntasks-per-node=40       # ntasks per node
#SBATCH -t 01:00:00                # walltime in HH:MM:SS, Max value 72:00:00 for medium, and 168:00:00 for large

#list of modules you want to use, for example
#module load apps/lammps/12.12.2018/intel

#name of the executable
exe=name_executable

#run the application
mpirun -bootstrap slurm -n $SLURM_NTASKS $exe      # specify the application command-line options, if any, after $exe

Please try to use No. of MPI Tasks (i.e $SLURM_NTASKS) multiple of 40 to avoid Node sharing for different users and to increase the overall utilization of cluster.

MPI+OpenMP (Hybrid)

Script for a Multinode MPI+OpenMP (Hybrid) Job

#!/bin/bash
#SBATCH -J job_name                # name of the job
#SBATCH -p medium                  # name of the partition: available options "medium, large"
#SBATCH -N 2                       # no of nodes
#SBATCH --ntasks-per-node=40       # ntasks for node
#SBATCH --cpus-per-task=1          # number of threads per task - used for OpenMP
#SBATCH -t 01:00:00                # walltime in HH:MM:SS, Max value 72:00:00 for medium, and 168:00:00 for large

#list of modules you want to use, for example
#module load apps/lammps/12.12.2018/intel

#name of the executable
exe=name_executable

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

#run the application
mpirun -bootstrap slurm -n $SLURM_NTASKS $exe    # specify the application command-line options, if any, after $exe

General Procedure to Prepare the Batch Script and Submit the Job

PARAM Shakti provides a well-defined software environment to compile and run applications using Environment Modules and Spack. These tools ensure that the correct versions of compilers, libraries, and applications are available during a user session and within SLURM batch jobs.

Using Environment Modules

module avail                                      # This command lists all the available modules
module load intel/2018.0.1.163                    # This will load the intel compilers into your environment
module unload intel/2018.0.1.163                  # This will remove all environment setting related to intel-2018 compiler loaded previously
module show intel/2018.0.1.163                    # Displays information about intel module, including environment changes

Using Spack

Spack is an alternative and more flexible package manager that allows users to install and manage multiple versions of compilers, libraries, and applications with customized build options. Users can either use pre-installed Spack packages or build their own software stacks. Required packages must be loaded in the environment before launching the job.

spack list                    # Lists all available packages
spack find                    # Lists all installed packages
spack install <package> # Installs the specified package
spack load <package>    # Loads the installed package
spack unload <package>  # Unloads the package
spack spec <package>    # Displays dependency and build configuration

A simple Slurm job script (This is a sample script to demonstrate different parameters)

#!/bin/bash
#SBATCH -J lammps-mpi                             #Job name(--job-name)
#SBATCH -o %j-out-myjob                           #Name of stdout output file(--output)
#SBATCH -e %j-err-myjob                           #Name of stderr error file(--error)
#SBATCH -p shared                                 #Queue (--partition) name ; available options "shared,medium,large or gpu"
#SBATCH -n 3                                      #Total Number of mpi tasks (--ntasks .should be 1 for serial)
#SBATCH -c 2                                      #(--cpus-per-task) Number of Threads
#SBATCH -t=00:10:00                               # specifies walltime(--time maximum duration)of run
#SBATCH --mem=23000                               # specifies Minimum 23000MB Memory required per node in MB if no unit specified. It is optional.
#SBATCH --mem-per-cpu=6000M                       # specifies Minimum 6000MB memory required per allocated CPU. It is optional.
#SBATCH --mail-user=abc@iitkgp.ac.in              # user's email ID where job status info will be sent
#SBATCH --mail-type=ALL                           # Send Mail for all type of event regarding the job

 # Thread Reservation Section
if [ -n "$SLURM_CPUS_PER_TASK" ]; then
             omp_threads=$SLURM_CPUS_PER_TASK
           else
             omp_threads=1
fi

# Load Application/Compiler/Library modules, for example
# module load apps/lammps/7Aug19/intel

# name of the executable
exe=name_executable

export OMP_NUM_THREADS=$omp_threads

# Executables along with right Parameters and right input files
mpirun -bootstrap slurm -n $SLURM_NTASKS $exe

Parameters Used in SLURM Job Script

The job flags are used with sbatch command. The syntax for the SLURM directive in a script is #sbatch <flag>. Some of the flags are used with the srun and >salloc commands. To Know more about the parameters refer Man Page man sbatch.

Resource	Flag Syntax	Description
partition	`--partition=partition_name` (or) `-p`	Partition is a queue for jobs.
time	`--time=01:00:00` (or) `-t`	Time limit for the job.
Total No. of MPI tasks	`--ntasks=8` (or) `-n 8`	Corresponds to the total MPI tasks required by the job.
Number of threads per MPI task	`--cpus-per-task=4` (or) `-c 4`	Corresponds to the cores per MPI task.
GPU	`--gres=gpu:N` where `N=1` or `2`	Request use of GPUs on compute nodes.
error file	`--error=err.out`	Job error messages will be written to this file.
job name	`--job-name="lammps"`	Name of job.
output file	`--output=lammps.out`	Name of file for stdout.
email address	`--mail-user=username@iitkgp.ac.in`	User's email address.
memory	`--mem=2300`	Memory required per node in MB.

Submit the Job

Once the required modules or Spack packages are loaded, users can prepare a SLURM batch script specifying the resource requirements (partition, number of nodes, CPUs, memory, GPUs, etc.) and submit the job using sbatch. Ensuring the correct software environment is set within the batch script is essential for successful and reproducible job execution.

$ sbatch slurm-job.sh
  Submitted batch job 106

$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
      150 standard   simple  user1   R   0:31  1 atom01

Sample batch scripts available at /home/iitkgp/slurm-scripts. For more information on slurm usage, refer to SLURM Documentation

Running Interactive Jobs

In general, the jobs can be run in an interactive manner or in batch mode. You can run an interactive job as follows:

The following command asks for a single core on one hour with default amount of memory.

$ srun --nodes=1 --ntasks-per-node=1 --time=01:00:00 --pty bash -i