System Hardware Specifications

• Total number of nodes: 454 (12 + 384 + 22 + 36)
  1. Master nodes: 2
  2. Login nodes: 8
  3. Service/Management nodes: 2
  4. CPU nodes: 384
  5. High Memory nodes: 36
  6. GPU accelerated nodes: 22

• Master Nodes: 2

  PARAM Shakti is an aggregation of a large number of computers connected through networks. The basic purpose of the master node is to manage and monitor each of the constituent component of PARAM Shakti from a system’s perspective. This involves operations like monitoring the health of the components, the load on the components, the utilization of various sub-components of the computers in PARAM Shakti.

  Master Nodes  : 2
  2* Intel Xeon SKL G-6148 Cores =40, 2.4 GHz	Total Cores = 80 cores
  Memory= 384 GB	Total Memory  = 768 GB
  HDD = 900 GB

  Login Nodes: 8

  Login nodes are typically used for administrative tasks such as editing, writing scripts, transferring files, managing your jobs and the like. You will always get connected to one of the login nodes. From the login nodes you can get connected to a compute node and execute and interactive job or submit batch jobs through the batch system (SLURM) to run your jobs on compute nodes. For ALL users PARAM Shakti login nodes are the entry points and hence are shared. By default, there will be a limit on the CPU time that can be used on a login node by a user and there is a limit/user on the memory as well. If any of these are exceeded, the job will get terminated. 

  Login Nodes  : 8
  2* Intel Xeon SKL G-6148 Cores = 40, 2.4 GHz	Total Cores = 320 cores
  Memory= 384 GB	Total Memory = 3,072 GB
  HDD = 900 GB

  Service/Management Nodes: 2

  Typically, the purpose of the service node is to provide Job Scheduling Services and other services to the cluster.

  Service/Management Nodes  : 2
  2* Intel Xeon SKL G-6148 Cores = 40, 2.4 GHz	Total Cores = 240 cores
  Memory= 384 GB	Total Memory= 2,304 GB
  HDD = 900 GB

  CPU Compute Nodes: 384

  CPU nodes are indeed the work horses of PARAM Shakti. All the CPU intensive activities are carried on these nodes. Users can access these nodes from the login node to run interactive or batch jobs. Some of the nodes have higher memory, which can be exploited by users in the aforementioned way.

  CPU only Compute Nodes  : 384
  2* Intel Xeon SKL G-6148 Cores = 40, 2.4GHz	Total Cores = 15,360 cores
  Memory= 192 GB, DDR4 2666 MHz	Total Memory=73,728 GB
  SSD = 480 GB (local scratch) per node

   

  High Memory nodes: 36

  Some compute nodes may feature a particular specification to be used for a particular job, or stage in your workflow.

  These are High Memory nodes that provide users to run their memory intensive jobs.

   

  CPU only Compute Nodes with High memory  : 36

  2* Intel Xeon SKL G-6148 Cores = 40, 2.4GHz                                                      Total Cores = 1440 cores

  Memory= 768 GB, DDR4 2666 MHz                       Total Memory=27,648 GB

  SSD = 480 GB (local scratch) per node

  GPU Compute Nodes: 22

  GPU compute nodes are the nodes that have CPU cores along with accelerators cards. For some applications GPUs get markedly high performance. For exploiting these, one has to make use of special libraries which map computations on the Graphical Processing Units (Typically one has to make use of CUDA or OpencCL).

  GPU Compute Nodes : 22
  2* Intel Xeon SKL G-6148 per node CPU Cores = 40, 2.4GHz	Total CPU Cores = 880
  Memory= 192 GB, DDR4 2666 MHz	Total Memory= 4,224 GB
  SSD= 480 GB (local scratch) per node
  2*nVidia V100 per node GPU Cores per node= 2*5120= 10240  GPU Tensor Cores per node=2*640= 1280
  GPU Memory = 16 GB HBM2 per nVidia V100

  Storage

  • Based on Lustre parallel file system 
  • Total useable capacity 2.1 PiB primary storage and Archival storage 500 TiB 
  • Throughput 50 GB/s

• Operating System

  • Operating system on PARAM Shakti is Linux – CentOS 7.6