As of December 31st, 2013, the upgrade of the discovery cluster from Red Hat Enterprise Linux (RHEL) 5.10 to CentOS 6.5 will be complete. All users that have not requested migration will be migrated that morning. This post contains information about the changes to the discovery cluster.

Upgraded Discovery Cluster

From your command line prompt, you should not notice too many differences in using the new cluster. Here are some changes you should be aware of:

  • With the upgrade to CentOS 6.5 there is a new GNU suite of compilers at version 4.4.7.

    • If you have compiled your code on the older cluster, you should consider recompiling on the new cluster.

  • There are new versions of most of the applications, libraries and compilers installed on discovery.

  • The system version of java is now at 2.6.x and the default module version is 2.7.x.

  • The system version of perl is now at 5.10.x and the module version is at 5.16.x.

  • The system version of PHP is now at 5.3.3.

Note: Always review the login message for the latest news on the cluster.

New Test Nodes

There are three new 16 core test nodes in this cluster, named x01, x02 and x03. Use the tnodeload command to see which one is most available before ssh’ing to one.

Naming of Compute Nodes

The compute nodes are divided into different cells where each cell has the same hardware configuration including the processor type, number of cores and amount of memory. Often users want to select a particular cell when they run their jobs. To make it easier to tell which cell your job is running in, we have renamed the compute nodes so the name now indicates the letter of the cell where the nodes resides. Commands like pbsmon and tnodeload reflect the naming changes to the nodes and are more user informative. The six cells a – f are described on the new discovery website and in the table below:





Node Count


amd 2.7Ghz





intel 2.3Ghz





amd 2.4Ghz





amd 3.0Ghz





amd 3.1Ghz





amd 2.8Ghz




Customizing Your Environment With Modules

You can use the module command to load specific versions of the applications, libraries, compilers and other programs that are installed on discovery. In the upgraded cluster we are loading fewer modules by default so that your user environment will be less complicated. You may find that the versions of the applications that you were loading are not available on the upgraded cluster. If this is the case, use the module avail command to see what versions are installed. Use the command module load application/version to load the application for your current login session or module initadd application/version to have it loaded automatically each time you log in. To remove the unwanted modules from your environment, issue the the command module initrm application/version. The discovery website contains additional information and examples of how to use the module command.

If you find that there are applications that you used on the old discovery that are not installed on the upgraded discovery, send email to and we will install it for you.

Information for MPI Users

If you run programs that use MPI, the command to start your jobs on the upgraded cluster has changed. Previously, we used the mpiexec program that is developed at the Ohio Supercomputer Center. Unfortunately, this program doesn’t work correctly with the version of the torque job manager that is running on the upgraded cluster so you will need to use the mpirun command that is part of MPICH and MVAPICH. Here are the commands that you should use to start up your MPI programs when you submit your job to the queue:

For MPICH programs:

mpirun -np # -machinefile $PBS_NODEFILE program_name [programs args]

For MVAPICH programs that run on infiniband nodes:

mpirun_rsh -rsh -np # -hostfile $PBS_NODEFILE program_name [programs args]

See the Parallel Programming page on the discovery website for more information.

If you have questions about the upgraded cluster, please send email to

The Discovery Support Team