New: Tutorials & Self-Learning is now live — a step-by-step Parallel MATLAB guide and C-DAC HPC workshop resources. Explore Tutorials   •   The HPC ERP System is currently available for user testing. We invite all faculty members, researchers, and students to participate in the testing process. HPC User Access Portal   •   Research Outputs: Please update your publications, theses, patents, and other research outputs carried out using Param Shakti and the GPU Cluster. Click here to submit/update.
Notice

DGX AI Cluster

A state-of-the-art AI and High-Performance Computing facility designed to support large-scale deep learning, artificial intelligence, scientific computing, and data-intensive research.

Overview

The DGX AI Cluster is IIT Kharagpur's dedicated infrastructure for large-scale deep learning, artificial intelligence, and data-intensive research, built on NVIDIA's DGX H100 platform with a high-speed InfiniBand interconnect and a dedicated parallel file system.

5 × DGX H100
Compute Nodes
40
NVIDIA H100 GPUs
400 Gbps
Quantum InfiniBand
1.1 PiB
Lustre Parallel File System
Proxmox HA
Management Cluster
Slurm
Workload Manager

AI Cluster

Infrastructure Overview

Computing Resources

  • Five NVIDIA DGX H100 systems with 8 GPUs each (40 total H100 GPUs)
  • Nodes dgx1–dgx4 grouped under the dgx_all partition
  • Node dgx5 dedicated to the dgx_ccds partition

Storage & Networking

  • 1.1 PiB Lustre parallel file system
  • NVIDIA Quantum InfiniBand at 400 Gbps bandwidth
  • 1.5 TB home directory quota per user
  • ~1 TB RAID storage per assigned DGX node

Management

  • Three Intel Xeon Gold 6530 servers providing High Availability via Proxmox
  • Slurm workload manager with QoS policies
Get Started

Ready to Run AI Workloads on the DGX Cluster?

Apply for an HPC account and access dedicated NVIDIA H100 GPU resources at IIT Kharagpur.

HPC Account Quick Start Guide

Cluster

System Architecture

Users reach the cluster through the Internet to the Master/Login nodes, which bridge to the IB Switch (and onward to shared Storage). The IB Switch buses a direct high-speed Data/InfiniBand path to all five DGX nodes, while the Master/Login nodes separately drive a BMC (out-of-band management) network and a secondary Eth management network, each routed through its own switch to every DGX node.

DGX AI cluster architecture — User/Internet reach the Master/Login nodes, which bridge to the IB Switch and onward to Storage; the IB Switch buses Data/InfiniBand directly to all 5 DGX nodes, while Master/Login separately drives BMC and Eth management networks to each DGX node through their own switches. Master/LoginNodes IB Switch Storage User Internet DGX1 DGX2 DGX3 DGX4 DGX5 BMC Switch Eth Switch Data / InfiniBand BMC (out-of-band) Eth (management)

Access

Access & QoS Policies

Access Information

ssh <username>@gpucluster.iitkgp.ac.in
  • dgx_all Nodes dgx1–dgx4
  • dgx_ccds Node dgx5

Storage & Quota

  • 1.5 TB quota at /home/<username>
  • ~1 TB RAID storage per assigned DGX node at /dgx<node>/<username>
  • Multi-node jobs should store datasets in /home/<username>, not node-local storage
  • Check quota usage with the myquota command

Data Lifecycle Policy

  • Data in /home/<username> is automatically deleted after 30 days of inactivity
  • Users are responsible for maintaining their own backups — the cluster administration is not liable for data loss
  • Transfer data to your local system upon job completion to avoid auto-deletion

QoS Limitations

QoS Max Jobs/User Max CPUs GPUs (Min–Max) Max Wall Time
gpu2 3 56 1–2 72 hrs (3 days)
gpu4 1 56 3–4 48 hrs (2 days)
gpu6 1 112 5–6 24 hrs (1 day)
gpu8 1 112 7–8 12 hours

Job Submission Guidelines

  • Select the appropriate partition — dgx_all (DGX1–DGX4) or dgx_ccds (DGX5)
  • Specify node count with #SBATCH --nodes=<N>
  • Specify GPUs per node with #SBATCH --gres=gpu:<count>
  • Match task/CPU configuration to your application's actual requirements
  • For multi-node jobs, store datasets in /home/<username>, not node-local storage

Sample Batch Script

#!/bin/bash
#SBATCH --job-name=job_name                      # Job name
#SBATCH --output=/home/<username>/output_%j.txt  # Standard output
#SBATCH --error=/home/<username>/error_%j.txt    # Standard error
#SBATCH --partition=dgx_all                       # Partition assigned to your account
#SBATCH --qos=gpu2                                # QoS (must match the number of gpu)
#SBATCH --nodes=1                                 # Number of nodes (increase for multi-node jobs)
#SBATCH --ntasks-per-node=2                       # Number of tasks per node (max 8)
#SBATCH --gres=gpu:2                              # GPUs per node (max 8)
#SBATCH --time=24:00:00                           # Adjust wall time according to the selected QoS

# Navigate to the working directory (path for the data stored in any dgx[1-5] node, e.g. dgx1)
cd /dgx1/<username>/<path>

# Load necessary modules (optional)
module load cuda/12.4
module load python/3.10

# Activate your Python environment
source /home/<username>/miniconda3/bin/activate <env>

# Execute the training script
python train_model.py

Common Errors to Avoid

Mismatch Between --gres=gpu and --qos (Incorrect GPU Count)

#!/bin/bash
#SBATCH --job-name=job1
#SBATCH --output=/home/<username>/output_%j.txt
#SBATCH --error=/home/<username>/error_%j.txt
#SBATCH --partition=dgx_all
#SBATCH --qos=gpu2              # QoS allows max 2 GPUs
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=2
#SBATCH --gres=gpu:4            # Requesting 4 GPUs, but gpu2 allows max 2
#SBATCH --time=24:00:00

cd /dgx1/<username>/<path>
source /home/<username>/miniconda3/bin/activate <env>
python train_model.py
  • The gpu2 QoS only allows up to 2 GPUs, but the script requests 4 GPUs (--gres=gpu:4)
  • Slurm will reject this job due to exceeding the allowed GPU count
  • Fix: change --gres=gpu:4 to --gres=gpu:2

Wall Time Exceeds QoS Limit

#!/bin/bash
#SBATCH --job-name=job2
#SBATCH --output=/home/<username>/output_%j.txt
#SBATCH --error=/home/<username>/error_%j.txt
#SBATCH --partition=dgx_all
#SBATCH --qos=gpu4              # gpu4 has max wall time of 48 hours
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=2
#SBATCH --gres=gpu:3
#SBATCH --time=72:00:00         # Exceeding max wall time of 48 hours for gpu4

cd /dgx1/<username>/<path>
source /home/<username>/miniconda3/bin/activate <env>
python train_model.py
  • The gpu4 QoS has a maximum wall time of 48 hours, but the script requests 72 hours (--time=72:00:00)
  • The job will be rejected or truncated to the max allowed wall time
  • Fix: reduce --time to 48:00:00 or select an appropriate QoS

Insufficient GPUs for Selected QoS

#!/bin/bash
#SBATCH --job-name=job4
#SBATCH --output=/home/<username>/output_%j.txt
#SBATCH --error=/home/<username>/error_%j.txt
#SBATCH --partition=dgx_all
#SBATCH --qos=gpu6              # gpu6 requires min 5 GPUs
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=2
#SBATCH --gres=gpu:2            # Only 4 (2x2) GPUs requested, but gpu6 requires at least 5
#SBATCH --time=24:00:00

cd /home/<username>/<path>
source /home/<username>/miniconda3/bin/activate <env>
python train_model.py
  • The job requests 2 GPUs per node on 2 nodes, resulting in a total of 4 GPUs
  • The selected gpu6 QoS requires at least 5 GPUs, so the job doesn't meet the minimum
  • Fix: increase --gres=gpu to satisfy the QoS requirement or select a lower QoS (e.g. gpu4)

Submitting a Multi-Node Job From a Compute Node Instead of the Login Node

sbatch /dgx1/<username>/train_script.sh
  • Jobs should be submitted from the login node, not from a compute node
  • Submission from dgx1 (or any dgx node) will fail because Slurm requires multi-node job submission from the login node
cd ~
sbatch train_script.sh

Monitoring Commands

Command Description
sinfo View partition and node availability
squeue View queued and running jobs
scontrol show job <id> View detailed job information
myquota Check home directory quota usage
df -h Check filesystem disk usage
module avail List available environment modules
module list List currently loaded modules
nvidia-smi View GPU utilization and status

Users with CPU-intensive workloads or those requiring less than 16GB GPU memory per GPU should use the PARAM Shakti HPC Cluster instead, to optimize resource allocation.

Institute Intercom: 84604
shaktisupport@iitkgp.ac.in
Ticketing Tool

Need Help or Have Questions?

Our HPC support team is here to assist. Open a ticket or reach us by email.

Open a Ticket