Kohinoor 1

IntroductionKohinoor 1 is the first High Performance Computing (HPC) cluster installed at TIFR-TCIS Hyderabad. This cluster is composed of 17 nodes, in which one is a head node and all other are execution nodes. This cluster is a heterogenous cluster which is composed of 12 CPU nodes and nodes and 4 CPU nodes with 2 Fermi M 2090 GPUs per node. The cluster nodes are connected using Infiniband HBAs through a 36 port completely non blocking Mellanox QDR Interconnect infiniband (IB) switch. The cluster is managed by the open source batch scheduler “Open Grid Engine” software for job scheduling and load balancing. The head-node allows user logins for job submission in cluster. The cluster has a local Network Attached Storage (NAS) of 40 TB across the nodes through IB switch, which is used for computational runs and a 36 TB NAS is attached to the head node of the cluster for the purpose of archiving and post processing of data.

VendorOEM – IBM (Supplied and installed by M/s. Locuz Enterprises, Hyderabad)

Kohinoor 1 Overview

  1. Master node
    • IBM System x3650 M4 2U rack model
    • 8-core Dual CPU Intel Xeon E5-2665 2.4GHz 20MB cache
    • 64GB ECC DDR3 1600MHz LP RDIMM RAM
    • 900GB 2.5in 10K RPM 6Gbps SAS
    • Emulex 8GB FC single-port HBA
    • Mellanox Infiniband HCA
  2. Compute nodes (CPU only nodes) [12 Nos.]
    • IBM System x iDataPlex dx360 M4 2U rack model
    • 8-core Dual CPU Intel Xeon E5-2665 2.4GHz 20MB cache
    • 32 GB ECC DDR3 1333MHz LP RDIMM RAM
    • 500GB 7.2K RPM 6Gbps SATA Hard drives
    • Mellanox Infiniband HCA
  3. Compute nodes with 2 GPUs on each node [4 Nos.]
    • IBM System x iDataPlex dx360 M4 2U rack model
    • 8-core Dual CPU Intel Xeon E5-2665 2.4GHz 20MB cache
    • 32 GB ECC DDR3 1333MHz LP RDIMM RAM
    • 500GB 7.2K RPM 6Gbps NL SAS Hard drives
    • NVIDIA Tesla M2090 (Fermi Architecture)
    • Mellanox Infiniband HCA
  4. NAS Storage (Compute space)
    • IBM System Storage DS3512 Express Single Controller 2U rack model
    • 3TB 3.5in 7.2K RPM NL SAS
    • 8Gb FC 4 Port Daughter Card
    • 50 TB storage capacity
    • NFS share over Infiniband
  5. NAS Storage (Archival & post processing)
    • Supermicro 2U storage server with 12 hot pluggable Hard drive bays
    • 12 x 4 TB 3.5” 7.2 K RPM SATA drives configured with RAID 5
    • NFS shared 1 Gpbs ethernet
  6. Networking & Interconnect
    • Primary compute nodes communication network is through a completely nonblocking interconnect of 36 port Mellanox QDR IB switch
    • Secondary communication network for cluster management is through a 24 port Gigabit Ethernet switch
  7. System Software
    • Operating System – CentOS 5.8
    • Clustering tool – Rocks Cluster Distribution 5.5
    • Job Scheduler – Open Grid Engine
  8. Libraries
    • GNU compiler collection
    • Intel Compilers non commercial edition
    • CUDA 5.0
    • openMPI 1.4.5
  9. Application software/Libraries
    • LAMMPS, NAMD, Quantum Expresso, HOOMD, GROMACS, SIESTA, GERRIS, Simpson, FFTW, CPMD, ABINIT, RUMD, VASP, etc.,

TCIS-Kohinoor1 Cluster Document