- cluster-users - Engineering Mailing Lists

CoE HPC News, June 29 2023: AC maintenance
by Yelle, Robert Brian 30 Jun '23

30 Jun '23

HPCC users, Maintenance was performed on the KEC chillers earlier today. The KEC server room temperature is still quite warm since the maintenance, but is expected to cool down slowly over time. The cluster resources are currently offline until the server room temperature returns to normal. Jobs currently running in the cluster will continue, but in the event that the temperature rises, I may need to perform an emergency shutdown of parts or most of the cluster, which may result in terminating those running jobs. I will post updates of the cluster status in the link below: https://it.engineering.oregonstate.edu/hpc/hpc-cluster-status-and-news<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fit.engine…> Rob Yelle HPC Manager

1 0

CoE HPC News, June 5 2023: Summer break maintenance
by Yelle, Robert Brian 22 Jun '23

22 Jun '23

HPCC users, Please check out the latest HPC cluster news regarding upcoming maintenance below. Summer break maintenance The cluster will undergo its regularly scheduled quarterly maintenance during Summer break, June 19-21. The following activities will be performed: Operating system updates Slurm upgrade and configuration changes BIOS and firmware updates as needed Nvidia/CUDA driver updates Infiniband updates The entire cluster will be offline starting Monday afternoon the 19th at 1pm, and until approximately Wednesday the 21st at 3pm. Jobs scheduled to run into this offline period will remain pending with the message “ReqNodeNotAvail, Reserved for maintenance”. If you wish for your Slurm job to start and finish before the offline period begins, you will need to adjust your time accordingly. For the latest cluster news and status updates, check out the link below: https://it.engineering.oregonstate.edu/hpc/hpc-cluster-status-and-news<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fit.engine…> Robert Yelle HPC Manager

1 2

CoE HPC News, Apr 25 2023: AC maintenance, new DGX limits
by Yelle, Robert Brian 25 Apr '23

25 Apr '23

HPCC users, First, thanks to all of you who participated in the HPC usage survey, this was greatly appreciated. The results help us to better understand how the cluster is currently being used and what future trainings might be helpful or of interest. AC maintenance Last Saturday night, some compute nodes were shut down due to datacenter temperatures rising to unsafe levels. This resulted in the termination of some jobs. Cooling has since been partly restored, but this was a short term fix and some additional but urgent maintenance is needed which requires a temporary shutdown of the chillers. This will take place starting early tomorrow (Wednesday) morning. To mitigate the heat load, the cluster will run at reduced capacity during the AC maintenance. It is likely that no additional jobs will be accepted during this time until the maintenance has completed, which should be tomorrow afternoon at the earliest. I expect the cluster will be restored to full capacity by late Thursday morning. New GPU limits on DGX systems Due to the continued increase in the number of users of the DGX systems, the demand for GPUs on the DGX systems lately has been at an all time high! The previous GPU limit of 8 is no longer sustainable at present without some additional modifications. Effective immediately, the cumulative GPU limit on the DGX systems has been reduced to 4 until further notice. For those of you who have deadlines coming up, please contact me directly and we’ll try to work something out. Cheers, Rob Yelle HPC Manager

1 0

Survey of cluster usage and training interest
by Yelle, Robert Brian 14 Apr '23

14 Apr '23

Cluster users, We are conducting a survey of our users to better understand how the cluster is being used. When you get a chance, please take a few moments to briefly answer the questions below, and reply directly to me and NOT to the list. Your feedback would be greatly appreciated. 1) How would you describe your cluster usage? a) primarily CPU based b) primarily GPU based c) mix of CPU and GPU 2. What application(s) do you run on the cluster (e.g. Matlab, Ansys, Star-CCM+, VASP, custom, etc.)? 3. What programming language(s) do you use or program in (e.g. Python, C, Fortran, R, etc.)? 4. What libraries, platforms and APIs do you primarily use on the cluster? (e.g. MPI, MKL, BLAS, LAPACK, CUDA, list your key Python or R packages, etc.). Also, Nvidia will offer a series of trainings in the near future on using GPUs on DGX systems. Please answer the following if you are interested in these trainings: 5. Are you interested in migrating from CPU based code to GPU? 6. Are you interested in an introductory workshop on GPU computing? 7. Are you interested in a workshop on accelerating your Python GPU code? 8. Are you interested in a workshop on profiling your GPU code? Thanks! Rob Yelle HPC Manager

5 5

CoE HPC News, Mar 10 2023: IB Upgrade, Spring Maintenance Week
by Yelle, Robert Brian 31 Mar '23

31 Mar '23

HPCC users, Please check out the latest HPC cluster news below. Infiniband Upgrade We are upgrading our high-speed infiniband backbone of our cluster! All new servers added to the cluster will be have up to 200Gb/s bandwidth. Springbreak Maintenance The cluster will undergo its regularly scheduled quarterly maintenance during Springbreak, March 27-31. The following activities will be performed: Operating system updates BIOS and firmware updates as needed Nvidia/cuda driver updates Infiniband infrastructure upgrade Due to the infiniband upgrade, we anticipate an extended offline period, which will start Monday afternoon the 27th at 1pm, and run through Thursday afternoon the 30th. Jobs scheduled to run into this offline period will remain pending with the message “ReqNodeNotAvail, Reserved for maintenance”. If you wish for your Slurm job to start and finish before the offline period begins, you will need to adjust your time accordingly. For the latest cluster news and status updates, check out the link below: https://it.engineering.oregonstate.edu/hpc/hpc-cluster-status-and-news<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fit.engine…> Have a nice weekend! Robert Yelle HPC Manager

1 2

CoE HPC News, Feb 28 2023: DGX queue status and limits
by Yelle, Robert Brian 28 Feb '23

28 Feb '23

HPCC users, The cluster will be operating at reduced capacity while cooling problems are being addressed. The DGX nodes are currently running at full capacity and at full utilization. I received a number of email lately regarding the status of the DGX queues, puzzling pending messages, and scheduling or related issues. Many of the pending messages on the DGX queues appear to be incorrect and are likely due to a bug in the scheduler. The main reason why users jobs to these queues remain pending is because there are currently no available resources. To address this and other scheduling issues that have come up, I plan to update the scheduler tomorrow morning around 9am. Currently running jobs should not be impacted by the update. Also, in order to reduce the load and allow more users access, I plan to temporarily reduce the cumulative per-user limits on the dgx and dgx2 queues to 4 GPUs and 24 CPUs. If you have a single job that exceeds this limit, please contact me. Rob Yelle HPC Manager

1 0

Dgx systems
by Yelle, Robert Brian 09 Feb '23

09 Feb '23

Cluster users, Multiple users have reported problems using the GPUs on dgx2-1. There do not appear to be any jobs utilizing the GPUs on this node at this time, so it has been taken offline for urgent maintenance, and will be rebooted tomorrow (Thursday) morning along with the scheduled rolling maintenance and reboots of dgx2-2 and dgx2-4. The dgx partitions should be back to full capacity by noon tomorrow. Thanks to everyone who reported these problems. Cheers, Rob Yelle HPC Manager

1 0

CoE HPC News, December 6, 2022: Christmas break maintenance
by Yelle, Robert Brian 14 Dec '22

14 Dec '22

Cluster users, This is your friendly reminder that the cluster will undergo its regularly scheduled maintenance starting next Monday the 12th. The following maintenance activities will be performed: Operating system updates Slurm update and configuration changes Nvidia driver updates Infiniband driver updates BIOS and firmware updates as needed Miscellaneous hardware maintenance as needed The entire cluster will be offline starting Tuesday the 13th at 8am, and will remain offline until approximately Wednesday the 14th at 3pm. Jobs scheduled to run into that offline period will remain pending with the message “ReqNodeNotAvail, Reserved for maintenance”. If you wish for your Slurm job to start and finish before the maintenance period begins, you will need to adjust your time limit accordingly, e.g. to change to 2 days do: scontrol update job {jobid} TimeLimit=2-00:00:00 Alternatively, you can cancel your pending job, adjust your walltime (using the —-time option) and resubmit. Rob Yelle HPC Manager

1 1

CoE HPC Newsletter, Nov 21 Edition: DGX queue change
by Yelle, Robert Brian 21 Nov '22

21 Nov '22

Cluster users, Please take note of DGX queue change and upcoming cluster maintenance below. DGX queue change The load on the DGX systems has sustained a load of 90-100% utilization for the last several weeks, making it difficult for many to obtain resources from the dgx and dgx2 partitions. To help reduce the load and make GPU resources more readily available on the DGX nodes without further reducing the overall GPU limits, the max GPU per job limit for the dgx partition has been reduced from 4 to 3. So if you need 4 GPUs or more per job, please use the dgx2 partition instead. Jobs requesting 4 GPUs from the dgx partition will result in the “QOSMaxGRESPerJob” error message. Winter maintenance This is an early heads-up that the cluster will undergo its regular quarterly maintenance after finals week, December 12-16. To see the latest updates on the cluster, check out the link below: https://it.engineering.oregonstate.edu/hpc/hpc-cluster-status-and-news Cheers, Rob Yelle HPC Manager

1 0

CoE HPC News, November 10 2022 Edition: Cooling failure
by Yelle, Robert Brian 10 Nov '22

10 Nov '22

HPCC users, Early this morning there was a cooling failure in the KEC datacenter which allowed temperatures to climb to unsafe levels, resulting in the automatic shutdown of all DGX-2 nodes and thus the termination of all jobs running on these nodes. Cooling has been restored to safe temperatures, and all of the DGX-2 nodes are back online. I know many of you have deadlines coming up, so may want to check the status of your jobs and resubmit as needed. Rob Yelle HPC Manager

1 0