Essentials

PARAM Shakti Usage Policy

1. Users are required to acknowledge the use of PS in all publications, presentations, thesis, webpages, etc., by including the following or a similar statement:
   Acknowledgement Template Copy

   We acknowledge National Supercomputing Mission (NSM) for providing computing resources of 'PARAM Shakti' at IIT Kharagpur, which is implemented by C-DAC and supported by the Ministry of Electronics and Information Technology (MeitY) and Department of Science and Technology (DST), Government of India.

   Users are also requested to inform the PS administration of any such outcome for annual reports/documentation/uploading on the website.
2. All announcements about PS will be made in relevant local newsgroups and/or through the website (www.hpc.iitkgp.ac.in). Users are encouraged to check such announcements regularly.
3. PS resources are to be used for academic research purposes only.
4. Project Guide/Adviser must have a user account on PS before requesting user accounts for his/her group members.
5. Users should not share their password with anyone. Using the same account by multiple users is not allowed. Users should monitor their PS usage to detect unauthorized use.
6. Users must not engage in hacking, reverse-engineering, or other activities that may constitute infringement of intellectual property rights. Users should not engage in activities that may bring disrepute to the facility or the Institute.
7. Users should ensure that all software, tools, libraries, and materials they use comply with applicable license terms and conditions. PS administration is not responsible for unauthorized licenses or usage.
8. All jobs must be submitted using the Slurm scheduler. Login nodes are meant for simple tasks only — all CPU and/or memory intensive tasks must be run on compute nodes. Running resource-intensive jobs on login nodes is strictly prohibited; in such cases, the user account will be deactivated immediately.
9. Temporary scratch storage is provided to all users according to the allocated disk quota. Scratch directories are subject to automatic deletion without notice when their time limit expires. Currently, scratch directories/files have a default time limit of 30 days. Data is held in this temporary storage at the user's own risk.
10. Each user will have a default disk quota of up to 50 GB in their home space and up to 2 TB (soft limit) in scratch space.
11. The user is the owner and hence responsible for all data copied and generated using PS. PS administration is not responsible for the same. Users should ensure the required backup and protection of data. PS administration is not responsible for account compromise, data theft, data publication claims, etc.
12. Users' data stored on the scratch directory on PS is not backed up or archived by the system administration. The PS administration is also not responsible for restoring damaged or lost files. Backing up and archiving data is the responsibility of the user and their Project Guide/Adviser.
13. Inactive Accounts: Accounts (except for faculty) that are not active for more than 90 days will be temporarily locked. To unlock the account before 180 days, the user should send a request to the PS administration.
14. Expired or Non-Renewed Accounts: The account will be locked and, after 90 days, the user account along with its data will be deleted. PS administration will not be responsible for any data loss after account deletion.
15. Although PS administration will take utmost care in maintaining the resources, there are no guarantees that resources or services will be available at all times, that they will suit every purpose, or that data will never be lost or corrupted. System administration cannot be held responsible for data loss in such events.
16. Access to PS may be unavailable at some times due to scheduled maintenance, upgrades, etc. Access may also be unavailable without notification if hardware, software, or other issues necessitate immediate maintenance. Notifications of all scheduled and unscheduled maintenance will be updated on the website (www.hpc.iitkgp.ac.in) or in the relevant local newsgroups.
17. Faculty/Advisers are required to inform PS administration about the removal of a user account for any student leaving the campus permanently.
18. Any unauthorized usage of PS will result in immediate account suspension and may be subject to appropriate disciplinary action as decided by the competent authority of the Institute.

Follow these practices to make efficient use of PARAM Shakti and ensure a good experience for all users.

1. Do NOT run any job longer than a few minutes on the login nodes. The login node is for compilation only — always run computational jobs on compute nodes via the Slurm scheduler.
2. Refer to the Quick Start Guide on the Paramshakti website — it is a good starting point for new users.
3. Check the Slurm sample scripts at /home/iitkgp/slurm-scripts for job submission examples.
4. Use the same compiler to compile all parts/modules/library-dependencies of an application. Mixing compilers (e.g., pgcc + icc) may cause linking or execution issues.
5. Choose appropriate compiler flags (e.g., -O3) to improve performance substantially (verify output accuracy). Refer to compiler documentation, online resources, or man pages.
6. Modules/libraries used for execution should be the same as those used for compilation. Specify them in the job submission script.
7. Be aware of the disk space utilized by your job(s). Estimate storage requirements before submitting multiple jobs.
8. Submit jobs only from $SCRATCH. Back up results/summaries to $HOME.
9. $SCRATCH is NOT backed up! Download all your data to your Desktop/Laptop.
10. Before installing any software in your home directory, ensure it is from a reliable and safe source. Ransomware is on the rise!
11. Do not use spaces when creating directories and files.
12. Inform PARAM Shakti support when you notice something unusual — e.g., unexpected slowdowns, missing or corrupted files.
13. Do not submit jobs to the gpu or gpu-low partition unless your code strictly requires GPU-based computing.
14. Provide the --time (wall-time) parameter judiciously in your job submission script, as it significantly affects queue waiting time.

Important Points

• Service Unit (SU): One hour of wall time on either one CPU core or one GPU card.
• Funds transferred to PARAM Shakti computing time are non-refundable.
• Unused core hours expire after 180 days from the recharge/top-up date.
• Band change is allowed: PIs can recharge anytime; additional SUs will be appended to the project account.

CPU Charges

(Applicable for Users of IIT Kharagpur)

GPU Charges

(Applicable for Users of IIT Kharagpur)

External Users (Other than IIT Kharagpur)

(Applicable for Users outside IIT Kharagpur)

Modes of Payment

Usage charges are accepted under the SRIC project “PARAM Shakti Usage Charge” (Code: PSU).

1. To pay from your SRIC project or FDF, send an email to Deepayan Maiti (deepayan@eoffice.iitkgp.ac.in), SRIC, mentioning your project code and amount, with a copy to hpc@iitkgp.ac.in. Use the following template:
   New Message Copy

   Todeepayan@eoffice.iitkgp.ac.in

   Cchpc@iitkgp.ac.in

   SubjectPARAM Shakti Usage Charge (PSU)

   Name:
   Employee Code:
   An amount of Rs. __________ (Rupees **amount in words** Only) may be
   credited to the PARAM Shakti Usage Charge (PSU) for the use of
   computational resources of PARAM Shakti by my group.
   The amount may be debited from my SRIC project ______ (Code)/FDF.

2. Upon receiving confirmation from SRIC, the HPC team will contact you regarding the breakdown of the total amount into CPU and GPU. SRIC will provide a receipt. A usage certificate for submission to funding agencies will also be provided.

How to Submit Jobs in High Priority Queues

By default, only the Faculty/PI user account can use the high priority queue against available SU. Access can be granted to specific users under their account using the acct-manage command.

For chargeable partitions, add the following options to your Slurm batch script:

#SBATCH -A "FAC_projectcode"
#SBATCH -q "FAC_projectcode"

Example: If dcd is the Faculty/PI user account and c3d is the SRIC project code, to use the medium partition:

#SBATCH -p medium
#SBATCH -A dcd_c3d
#SBATCH -q dcd_c3d

High-Priority users should verify job priority using the sprio command. If the QoS is not receiving a higher priority, resubmit with the correct account and notify HPC Support.

How to Check SU Balance

Use the acct-manage command and select option p in the main menu to view project details and available balance. High priority account users can also run hpabalance to view their latest account balance.

Calculate Service Units (SU) from an allocated budget, or estimate the cost based on SU usage. Results update instantly as you type.

This guide covers configuring MATLAB (from the cluster itself, or from your own desktop) to run serial and parallel jobs on PARAM Shakti and the DGX AI Cluster, submitting interactive and batch jobs, and debugging failed jobs.

Initial Configuration — Running MATLAB on the Cluster

After logging into the cluster, configure MATLAB to run parallel jobs on the cluster by calling the shell script configCluster.sh.

$ module load matlab[/VERSION]
$ configCluster.sh

Initial Configuration — Running MATLAB on the Desktop

This setup is for job submission when MATLAB is installed on your own machine and jobs run remotely on the cluster. It needs to be done once per cluster, per MATLAB version installed on your machine.

1. Start MATLAB and run userpath to find your MATLAB user folder.
2. Download the MATLAB plugin scripts and extract the ZIP contents into that folder.
3. Create a new cluster profile by running configCluster and selecting a cluster from the prompt.

configCluster
[1] ai-cluster
[2] param-shakti
Select a cluster [1-2]:

Configuring Jobs

Before submitting a job, scheduler flags such as partition, account, e-mail, GPUs, memory, and walltime can be set via AdditionalProperties. None of these are required.

% Get a handle to the cluster
c = parcluster;

% REQUIRED — specify the partition
c.AdditionalProperties.Partition = 'partition-name';

% OPTIONAL
c.AdditionalProperties.AccountName = 'account-name';        % account
c.AdditionalProperties.Constraint = 'feature-name';         % constraint
c.AdditionalProperties.EmailAddress = 'user-id@iitkgp.ac.in'; % email on job status
c.AdditionalProperties.GPUsPerNode = 1;                     % GPUs per node (default: 0)
c.AdditionalProperties.GPUCard = 'gpu-card';                % specific GPU card
c.AdditionalProperties.MemPerCPU = '6GB';                   % memory per core (default: 4GB)
c.AdditionalProperties.ProcsPerNode = 4;                    % cores per node
c.AdditionalProperties.QoS = 'qos-name';                    % quality of service
c.AdditionalProperties.RequireExclusiveNode = true;         % node exclusivity (default: false)
c.AdditionalProperties.Reservation = 'reservation-name';    % reservation
c.AdditionalProperties.WallTime = '1-05:30';                % wall time (D-HH:MM)

% Persist changes between MATLAB sessions
c.saveProfile

% View the current configuration
c.AdditionalProperties

% Unset a value when no longer needed
c.AdditionalProperties.EmailAddress = '';
c.AdditionalProperties.RequireExclusiveNode = false;

Interactive Jobs

To run an interactive pool job on the cluster, continue to use parpool as before — the pool now runs across multiple nodes on the cluster instead of a local host.

% Get a handle to the cluster
c = parcluster;

% Open a pool of 64 workers on the cluster
pool = c.parpool(64);

% Run a parfor over 1000 iterations
parfor idx = 1:1000
    a(idx) = rand;
end

% Delete the pool when it's no longer needed
pool.delete

Independent Batch Jobs

Use the batch command to submit asynchronous jobs to the cluster. It returns a job object used to access the submitted job’s output — see the MATLAB documentation for more on batch.

% Get a handle to the cluster
c = parcluster;

% Submit a job to query where MATLAB is running on the cluster
job = c.batch(@pwd, 1, {}, 'CurrentFolder', '.');

% Query job state, then fetch results once finished
job.State
job.fetchOutputs{1}

% Delete the job once results are no longer needed
job.delete

To retrieve running or completed jobs, call parcluster again — its Jobs array lists everything queued, running, finished, or failed.

c = parcluster;
jobs = c.Jobs

% Get a handle to the second job in the list
job2 = c.Jobs(2);

% Fetch all results from the second job in the list
job2.fetchOutputs{:}

% Alternate: load results if the job was a script instead of a function
job2.load

Parallel Batch Jobs

The batch command also supports parallel workflows. Save this example separately as parallel_example.m:

function [sim_t, A] = parallel_example(iter)
    if nargin == 0
        iter = 8;
    end
    disp('Start sim')
    A = nan(iter,1);
    t0 = tic;
    parfor idx = 1:iter
        A(idx) = idx;
        pause(2)
        idx
    end
    sim_t = toc(t0);
    disp('Sim completed')
    save RESULTS A
end

This time, specify a Pool argument when calling batch:

% Get a handle to the cluster
c = parcluster;

% Submit a batch pool job using 4 workers for 16 simulations
job = c.batch(@parallel_example, 1, {16}, 'CurrentFolder', '.', 'Pool', 4);

% View current job status, then fetch results once finished
job.State
job.fetchOutputs{1}

ans =

    8.8872

Run the same simulation with a larger pool, and keep track of the job ID to retrieve results later:

% Submit a batch pool job using 8 workers for 16 simulations
job = c.batch(@parallel_example, 1, {16}, 'CurrentFolder', '.', 'Pool', 8);

% Get the job ID, then clear job from workspace (as though MATLAB exited)
id = job.ID
id =

    4

clear job

With a handle to the cluster, findJob searches for the job with the given ID:

% Get a handle to the cluster, then find the old job
c = parcluster;
job = c.findJob('ID', 4);

% Retrieve state, then fetch the results
job.State
ans =

    finished

job.fetchOutputs{1};
ans =

    4.7270

Debugging

If a job produces an error, call getDebugLog to view the error log file — specify the task for an independent job, or just the job object for a Pool job.

% Independent job — specify the task
c.getDebugLog(job.Tasks)

% Pool job — specify only the job object
c.getDebugLog(job)

If the cluster admin needs the scheduler ID of a job (e.g. its Slurm job ID) to help troubleshoot, retrieve it with getTaskSchedulerIDs:

job.getTaskSchedulerIDs()
ans =

    25539

Helper Functions

To Learn More

• Parallel Computing Overview
• Parallel Computing Documentation
• Parallel Computing Coding Examples
• Parallel Computing Tutorials
• Parallel Computing Videos
• Parallel Computing Webinars

Search these topics on mathworks.com/products/parallel-computing, or use the resources below.

These workshops cover SLURM job submission, shared/distributed-memory parallel programming (OpenMP, MPI), Intel code-optimization tools, and GPU computing with CUDA and OpenACC — recorded and made available for self-paced learning.

Supercomputing for PARAM Shakti Users (23–27 Nov 2020)

A five-day virtual workshop jointly organized by the Centre for Development of Advanced Computing (C-DAC), Pune, and IIT Kharagpur. Download the full agenda (PDF).

Day 1–2 — SLURM & Job Submission

Introduction to SLURM, batch scripts, and the job submission procedure, with sample batch scripts for serial, parallel, and hybrid (MPI+OpenMP) jobs and an explanation of each script’s flags.

• Slides: PARAM Shakti Overview, SLURM, Job Submission
• Video: Day 1–2 recording

Day 3 — OpenMP & MPI

Shared-memory parallelism with OpenMP, and distributed-memory parallelism with MPI (point-to-point and collective communication).

• Slides: OpenMP, MPI Point-to-Point, MPI Collective, Parallel Matrix Multiplication
• Video: Day 3 recording

Day 4 — Intel Tools & Optimization

Intel code-optimization tools, plus a demonstration of the Intel AI Portfolio and Python.

• Slides: Intel Threading Advisor, Intel Vectorization Advisor, Intel VTune Amplifier, OpenVINO, Intel DAAL, Vectorization for C++

Day 5 — GPU Computing (CUDA / OpenACC)

GPU computing with NVIDIA CUDA and OpenACC, including mixed-precision training, accelerated data science, and AI on HPC.

• Slides: GPU Computing, Mixed Precision & TensorRT, AI on HPC, Accelerated Data Science, OpenACC
• Video: Day 5 recording

HPC Software Workshop (09 Dec 2022)

A virtual workshop on HPC software jointly organized by C-DAC Bengaluru and IIT Kharagpur. Download the agenda (PDF) or watch the recording.

• CAPC-AutoPar
• C-DAC HPC Profiler v0.9
• HPC Dashboard v5.0
• ParaDE — IDE for HPC
• Parallel Computing (NSM Meet)
• Programming with OpenMP
• PU Architecture
• Sumegha Cloud Lab Kit
• Suparikshan — Monitoring & Management

THINK PARALLEL Training (30 Jan – 03 Feb 2023)

A one-week in-person training, "THINK PARALLEL: Parallel Programming for Engineers & Scientists," organized by C-DAC Bengaluru at C-DAC Knowledge Park. Download the brochure (PDF).