As of December 31st, 2013, the upgrade of the discovery cluster from Red Hat Enterprise Linux (RHEL) 5.10 to CentOS 6.5 will be complete. All users that have not requested migration will be migrated that morning. This post contains information about the changes to the discovery cluster.

Upgraded Discovery Cluster

From your command line prompt, you should not notice too many differences in using the new cluster. Here are some changes you should be aware of:

  • With the upgrade to CentOS 6.5 there is a new GNU suite of compilers at version 4.4.7.

    • If you have compiled your code on the older cluster, you should consider recompiling on the new cluster.

  • There are new versions of most of the applications, libraries and compilers installed on discovery.

  • The system version of java is now at 2.6.x and the default module version is 2.7.x.

  • The system version of perl is now at 5.10.x and the module version is at 5.16.x.

  • The system version of PHP is now at 5.3.3.

Note: Always review the login message for the latest news on the cluster.

New Test Nodes

There are three new 16 core test nodes in this cluster, named x01, x02 and x03. Use the tnodeload command to see which one is most available before ssh’ing to one.

Naming of Compute Nodes

The compute nodes are divided into different cells where each cell has the same hardware configuration including the processor type, number of cores and amount of memory. Often users want to select a particular cell when they run their jobs. To make it easier to tell which cell your job is running in, we have renamed the compute nodes so the name now indicates the letter of the cell where the nodes resides. Commands like pbsmon and tnodeload reflect the naming changes to the nodes and are more user informative. The six cells a – f are described on the new discovery website and in the table below:

Cell

Chipset

Cores

RAM

Node Count

cella

amd 2.7Ghz

8

32Gb

33

cellb

intel 2.3Ghz

8

32Gb

16

cellc

amd 2.4Ghz

16

64Gb

27

celld

amd 3.0Ghz

16

64Gb

39

celle

amd 3.1Ghz

16

64Gb

23

cellf

amd 2.8Ghz

48

192Gb

8

Customizing Your Environment With Modules

You can use the module command to load specific versions of the applications, libraries, compilers and other programs that are installed on discovery. In the upgraded cluster we are loading fewer modules by default so that your user environment will be less complicated. You may find that the versions of the applications that you were loading are not available on the upgraded cluster. If this is the case, use the module avail command to see what versions are installed. Use the command module load application/version to load the application for your current login session or module initadd application/version to have it loaded automatically each time you log in. To remove the unwanted modules from your environment, issue the the command module initrm application/version. The discovery website contains additional information and examples of how to use the module command.

If you find that there are applications that you used on the old discovery that are not installed on the upgraded discovery, send email to research.computing@dartmouth.edu and we will install it for you.

Information for MPI Users

If you run programs that use MPI, the command to start your jobs on the upgraded cluster has changed. Previously, we used the mpiexec program that is developed at the Ohio Supercomputer Center. Unfortunately, this program doesn’t work correctly with the version of the torque job manager that is running on the upgraded cluster so you will need to use the mpirun command that is part of MPICH and MVAPICH. Here are the commands that you should use to start up your MPI programs when you submit your job to the queue:

For MPICH programs:

mpirun -np # -machinefile $PBS_NODEFILE program_name [programs args]

For MVAPICH programs that run on infiniband nodes:

mpirun_rsh -rsh -np # -hostfile $PBS_NODEFILE program_name [programs args]

See the Parallel Programming page on the discovery website for more information.

If you have questions about the upgraded cluster, please send email to research.computing@dartmouth.edu.

The Discovery Support Team