Jobstats is a free and open-source job monitoring platform designed for CPU and GPU clusters that use the Slurm workload manager.
The jobstats command provides users with a job efficiency report:
$ jobstats 39798795
================================================================================
Slurm Job Statistics
================================================================================
Job ID: 39798795
User/Account: aturing/math
Job Name: sys_logic_ordinals
State: COMPLETED
Nodes: 2
CPU Cores: 48
CPU Memory: 256GB (5.3GB per CPU-core)
GPUs: 4
QOS/Partition: della-gpu/gpu
Cluster: della
Start Time: Fri Mar 4, 2022 at 1:56 AM
Run Time: 18:41:56
Time Limit: 4-00:00:00
Overall Utilization
================================================================================
CPU utilization [||||| 10%]
CPU memory usage [||| 6%]
GPU utilization [|||||||||||||||||||||||||||||||||| 68%]
GPU memory usage [||||||||||||||||||||||||||||||||| 66%]
────────────────────────────────────────────────────────────────────────────
SM util. [||||||||||||||||||||||| 48%]
OCC util. [|||||||||| 20%]
TC util. [||||||||||||||| 30%]
INT util. [||| 6%]
FP16 Max util. [ 0%]
FP32 Avg util. [|| 4%]
FP64 Max util. [ 0%]
DRAM BW util. [|||||||||||||||||||| 40%]
Detailed Utilization
================================================================================
CPU utilization per node (CPU time used/run time)
della-i14g2: 1-21:41:20/18-16:46:24 (efficiency=10.2%)
della-i14g3: 1-18:48:55/18-16:46:24 (efficiency=9.5%)
Total used/runtime: 3-16:30:16/37-09:32:48, efficiency=9.9%
CPU memory usage per node - used/allocated
della-i14g2: 7.9GB/128.0GB (335.5MB/5.3GB per core of 24)
della-i14g3: 7.8GB/128.0GB (334.6MB/5.3GB per core of 24)
Total used/allocated: 15.7GB/256.0GB (335.1MB/5.3GB per core of 48)
GPU utilization per node
della-i14g2 (GPU 0): 65.7%
della-i14g2 (GPU 1): 64.5%
della-i14g3 (GPU 0): 72.9%
della-i14g3 (GPU 1): 67.5%
GPU memory usage per node - maximum used/total
della-i14g2 (GPU 0): 26.5GB/40.0GB (66.2%)
della-i14g2 (GPU 1): 26.5GB/40.0GB (66.2%)
della-i14g3 (GPU 0): 26.5GB/40.0GB (66.2%)
della-i14g3 (GPU 1): 26.5GB/40.0GB (66.2%)
Detailed GPU Metrics
------------------------------------------------------
SM | OCC | TC | INT | FP32 | DRAM BW | Power | Temp
----+-----+-----+------+------+---------+-------+-----
della-i14g2 (GPU 0) 48% | 20% | 30% | 6.3% | 4.1% | 40% | 680W | 66°C
della-i14g2 (GPU 1) 48% | 20% | 30% | 6.2% | 4.1% | 40% | 673W | 54°C
della-i14g3 (GPU 0) 48% | 20% | 30% | 6.2% | 4.1% | 40% | 671W | 59°C
della-i14g3 (GPU 1) 48% | 20% | 30% | 6.3% | 4.1% | 40% | 678W | 65°C
------------------------------------------------------
-------------------------------------------------
PCIe Recv | PCIe Sent | NVLink Recv | NVLink Sent
----------+-----------+-------------+------------
della-i14g2 (GPU 0) 80 MB/s | 13 MB/s | 285 MB/s | 347 MB/s
della-i14g2 (GPU 1) 80 MB/s | 13 MB/s | 287 MB/s | 346 MB/s
della-i14g3 (GPU 0) 81 MB/s | 13 MB/s | 287 MB/s | 346 MB/s
della-i14g3 (GPU 1) 81 MB/s | 13 MB/s | 287 MB/s | 346 MB/s
-------------------------------------------------
https://princetonuniversity.github.io/jobstats/setup/detailed_gpu_metrics/
Notes
================================================================================
* This job only used 6% of the 256GB of total allocated CPU memory. For
future jobs, please allocate less memory by using a Slurm directive such
as --mem-per-cpu=1G or --mem=10G. This will reduce your queue times and
make the resources available to other jobs. For more info:
https://researchcomputing.princeton.edu/support/knowledge-base/memory
* This job only needed 19% of the requested time which was 4-00:00:00. For
future jobs, please request less time by modifying the --time Slurm
directive. This will lower your queue times and allow the Slurm job
scheduler to work more effectively for all users. For more info:
https://researchcomputing.princeton.edu/support/knowledge-base/slurm
* See the URL below for various job metrics plotted as a function of time:
https://mydella.princeton.edu/pun/sys/jobstats/39798795
Subscribe to the Jobstats mailing list to receive updates and other news.
- 15 Nov 2026: The Jobstats team will be available at Supercomputing 2026 for any discussions (see contact below)
- 5 Oct 2026: Jobstats version 1.1.0 was released (see CHANGELOG)
- 2 Sep 2026: Detailed GPU metrics added to main branch
- 29 Jul 2026: PEARC 2026: "Getting the Most Out of Your GPUs" (Slides, Paper)
- 23 Apr 2026: Presentation on Jobstats at NCAR (see slides)
- 16 Nov 2025: The Jobstats team will be available at Supercomputing 2025 for any discussions (see contact below)
- 28 Sep 2025: Subscribe to the Jobstats mailing list
- 27 Aug 2025: Job summary statistics can now be stored in a MariaDB database instead of Slurm DB (contribution by Anish Chanda of Iowa State Univ.)
- 26 Jul 2025: Open OnDemand helper app now supports direct routing to Grafana dashboard for a given job
- 24 Jul 2025: Jobstats wins "Best Poster Award" at PEARC 2025
- 01 Jun 2025: Job Defense Shield is available for sending email alerts, generating reports, and cancelling jobs with 0% GPU utilization
Begin with What is Jobstats? in the documentation.
- American Museum of Natural History
- Brown University - Center for Computation and Visualization
- Clemson University - Research Computing and Data
- Deakin University - Applied Artificial Intelligence Initiative
- Free University of Berlin - High-Performance Computing
- George Mason University - Office of Research Computing
- Harvard University - Research Computing
- Johns Hopkins University - Advanced Research Computing
- Iowa State University - High Performance Computing
- Massachusetts Institute of Technology - Research Computing
- Monash University - e-Research
- Northwestern University - Research Computing and Data Services
- NAVER LABS Europe
- Pacific Northwest National Laboratory
- Princeton University - Computer Science Department
- Princeton University - Research Computing
- University of Queensland - Research Computing Centre
- University of Virginia - Research Computing
- Yale University - Center for Research Computing
- and many more
One can create a GitHub issue or write to the developers: Josko Plazonic (plazonic@princeton.edu) and Jonathan Halverson (halverson@princeton.edu)