There are several Moab and Torque commands you can use to monitor the status of your submitted job, the job scheduler, and the job
queue.

  • The commands you will use reside in /opt/moab/active/bin/ and /opt/torque/active/bin/ (both in your PATH).
  • The torque and moab commands have manual pages (e. g., type “man qsub” at the command line).

You should read our scheduler policies to inform yourself of limitations we apply to help ensure efficient use of cluster resources which is also fairly distributed among users. (qr)

To display the jobs currently in the queue running or waiting to be run issue the Moab command: showq

  • The output is frequently hundreds of lines long. Pipe it to more to display the information one screen at a time.
    • showq -u username (myjobs)
    • qshow

A PBS command used to display the jobs currently in the queue waiting to be run: qstat (myqstat)

The output from the qstat command provides the following information:

  • Job ID (job_identifier)
  • Name (submitted PBS script name)
  • User (submit user name)
  • Time Use (running time)
  • S (job status: “R” means running, “Q” means queued)
  • Queue (queue name)

To display the status of a specific job that you have submitted, include the job identifier number.

  • For example:  qstat 28671

To delete a job that you have submitted to the queue: qdel <jobID>

To query the cluster node status and list some basic information  about each of the nodes: pbsnodes -a (pbsmon)

To get more information about your job status and estimated start/complete times:

  • checkjob -v <jobID> will give you information on the availability of nodes which match your request.
  • showstart <jobID> will give you an estimated start time for you job, which can change for a variety of reasons, such as, if more jobs with higher priority are submitted, or if the node the job is assigned to goes down.
  • If your job cannot run because you’ve requested resources that don’t exist, this command will state it cannot determine a start time.

The following are ways one can determine how many cores they should request based on memory usage.

  • Checking the memory usage of a running job:
    • First log onto the node your job is running on.  Using one of the test nodes is recommended.
    • You can use the Linux commands ps -x  to find the Linux process ID <PID> of your job.
    • Then use the Linux pmap command: pmap <PID>
    • The last line of the output gives the total memory usage of the running process.
    • For more information read the online manual pages:
      • man ps
      • man pmap
  • Checking the memory usage of a completed job:
    • After logging onto the cluster, rsh to knorr where the job logs are stored.
    • Then use: tracejob  -almn <number-of-days-ago> <jobID>
    • Also you can use the qhist command to look at past jobs.