- cluster-users - Engineering Mailing Lists

CoE HPC News, December 21 2023: Update on Winter maintenance and EL8/EL9 upgrade
by Yelle, Robert Brian 21 Dec '23

21 Dec '23

HPCC users, Update on maintenance activities and EL8/EL9 upgrades The cluster is back online but at reduced capacity while maintenance and upgrades proceed on the rest of the cluster. Additional resources will continue to be restored as maintenance and/or upgrades to EL8 or EL9 are completed. Also, due to a large number of research groups facing deadlines in the coming weeks, the upgrade period will be extended and several systems will remain on EL7 through February 5. If you have any questions or concerns, let me know. For up-to-date status on the cluster, check out the link below: https://it.engineering.oregonstate.edu/hpc/hpc-cluster-status-and-news<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fit.engine…> Happy Holidays! Rob Yelle HPC Manager

1 0

CoE HPC News, December 1 2023: Winter maintenance, EL8/EL9 upgrade, DGX H100 status
by Yelle, Robert Brian 01 Dec '23

01 Dec '23

HPCC users, Winter maintenance/OS upgrades The cluster will operate at reduced capacity throughout the month of December through mid-January while cluster nodes are upgraded to either EL8 or EL9 linux. I will post a tentative upgrade schedule sometime next week, and will coordinate with partition group owners to help minimize the impact of these upgrades. In addition to the OS upgrades, regular maintenance activities will be performed during this time, e.g.: OnDemand HPC portal updates Slurm update and configuration changes Nvidia driver updates Infiniband updates BIOS and firmware updates as needed Miscellaneous hardware maintenance as needed The entire cluster will be offline after Finals week, starting Monday, December 18th at 1pm, and will remain offline until approximately Wednesday the 20th at 3pm. Jobs scheduled to run into that offline period will remain pending with the message “ReqNodeNotAvail, Reserved for maintenance”. If you wish for your Slurm job to start and finish before the offline period begins, you will need to adjust your time limit accordingly, e.g. to change to 2 days do: scontrol update job {jobid} TimeLimit=2-00:00:00 Alternatively, you can cancel your pending job, adjust your walltime (using the --time option) and resubmit. Status of Nvidia DGX H100 systems I want to thank everyone who has given me feedback regarding the new DGX systems. A GPU issue came up which required that I open a ticket with Nvidia to troubleshoot. The nodes were then taken offline, then reinstalled and reconfigured. The GPU issue has now been addressed, so pytorch and other CUDA applications should work. Also, the automount problem appears to be resolved, so users can access directories in /nfs/hpc/*. I anticipate that the HPC portal/vncserver will be available for these hosts sometime next week, so stay tuned. However, ssh access is still not available. The dgxh partition is back online and the testing period resumed while other issues continue to be worked out. Feel free to give them another shot, and please continue to report any further issues to me. If you have any questions or concerns, let me know. For up-to-date status on the cluster, check out the link below: https://it.engineering.oregonstate.edu/hpc/hpc-cluster-status-and-news<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fit.engine…> Have a nice weekend, Rob Yelle HPC Manager

1 0

COE HPC cluster down
by Yelle, Robert Brian 30 Oct '23

30 Oct '23

Cluster users, The datacenter experienced a cooling failure this morning which required an emergency shutdown of the entire cluster. Facilities is working on the issue, and we hope to bring the cluster back online sometime tomorrow. Rob Yelle HPC Manager

1 1

CoE HPC News, October 24 2023: New DGX systems and partition changes
by Yelle, Robert Brian 24 Oct '23

24 Oct '23

HPCC users, Please check out the latest news below regarding the COE HPC cluster: New Nvidia DGX H100 systems are online! Many of you may have received the announcement last week that three new Nvidia DGX H100 systems have been added to the COE HPC cluster. Each DGX H100 system has 112 CPUs, 8 Nvidia H100 GPUs, and 2TB RAM! These systems are now available in a new partition called “dgxh”. This partition is in a testing period for at least this week to resolve existing issues and smooth out any other kinks that crop up. During this testing period, the current resource limits are 2 GPUs and 32 CPUs per user, and the time limit is 24 hours. Be advised that the DGX H100 systems are running RHEL9 based linux, which is different than the RHEL7 based systems currently used by the rest of the cluster. Also, these systems are not yet available through the HPC portal or through ssh. Give them a try and let me know of any issues you encounter. DGX partition change We currently have separate “dgx” and “dgx2” partitions for the our DGX-2 systems, depending on how many GPUs are needed. For various reasons that have come up over time, it is no longer advantageous to have separate partitions for these systems, so these will again be merged into a single “dgx2” partition. The “dgx” partition will be phased out or possibly re-purposed, so please use “dgx2” instead. The resource limits will remain at 4 GPUs and 32 CPUs on the dgx2 partition for now, but may increase again in the future. Submit-a offline this week Just a reminder that Submit-a will be offline until next Monday the 30th for maintenance. Until then, please use submit-b or submit-c. If you have any questions or concerns, let me know. For up-to-date status on the cluster, check out the link below: https://it.engineering.oregonstate.edu/hpc/hpc-cluster-status-and-news<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fit.engine…> Cheers, Rob Yelle HPC Manager

1 0

CoE HPC News, October 9 2023: Training and update reminder
by Yelle, Robert Brian 09 Oct '23

09 Oct '23

HPCC users, Here are some updates and reminders from last week’s newsletter, especially for the benefit of the many new users who have recently been added to the cluster users list. HPC training reminders Attention new cluster users! I am offering my “Intro to HPC” workshops every Wednesday at 3pm and Thursday at 4pm from October 11 through November 9. This workshop is designed to help new users become acquainted with and start using the cluster. To book a workshop session, click on the “Intro to HPC training session” link on the site below: https://it.engineering.oregonstate.edu/hpc<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fit.engine…> AI and ML series of trainings from MarkIII starting next week! For anyone who is interested, Mark III is offering a free five part series of AI and ML trainings starting next Tuesday October 17 and running through November 14: Tuesday, October 17 at 11am: Intro to AI and Machine Learning: The basics, tutorial and lab Tuesday, October 24 at 11am: Intro to Deep Learning: An introduction to Neural Networks Tuesday, October 31 at 11am: Intro to Datasets Tuesday, November 7 at 11am: Intro to Computer Vision and Image Analytics Tuesday, November 14 at 11am: Getting Started with Containers and AI If you are interested in more details and want to register for any of the MarkIII trainings listed above, please check out the “OSU AI Series” on the right hand side of the HPC web site above. Also note the HPC resources from Nvidia also displayed on the right hand side. Updates on cluster updates The updates of the cluster compute and GPU nodes should be completed by the end of the day Tuesday October 10. The submit nodes will be taken offline and scheduled to be updated on the following dates: Submit-c: Monday, October 16 at 9am Submit-b: Monday, October 23 at 9am Submit-a: Monday, October 30 at 9am Please plan accordingly. Srun jobs still running on these hosts when they are scheduled to be updated will be terminated. Jobs submitted via sbatch or the HPC portal will not be affected. If you have any questions or concerns, let me know. For up-to-date status on the cluster, check out the link below: https://it.engineering.oregonstate.edu/hpc/hpc-cluster-status-and-news<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fit.engine…> Cheers, Rob Yelle HPC Manager

2 1

CoE HPC News, October 4 2023: Updates, upgrades and trainings
by Yelle, Robert Brian 04 Oct '23

04 Oct '23

HPCC users, Here is the latest news regarding the COE HPC cluster: OS and GPU driver updates The cluster is currently running at reduced capacity while OS updates are being rolled out. In addition, some users have reported that they cannot run their GPU codes on some GPU nodes, which appears to be due to the older drivers on these nodes. New GPU drivers are available, so the GPU nodes are scheduled to be offline next week in a staggered fashion (some Monday the 9th and some Tuesday the 10th) so that the new drivers can be installed. If you need GPU resources, please schedule your jobs so that they can complete before these offline periods, otherwise they will remain pending with the message “Required Node not available” or “ReqNodeNotAvail”. If that happens, you may be able to get in by reducing your time requirement using the “--time” or “-t" option in srun or sbatch. OS upgrade Many of you are aware that many COE linux servers have been or are being upgraded to Enterprise Linux 9 (EL9). This will also happen on the HPC cluster over the course of this Fall quarter. More details to come. Intro to HPC workshop and other HPC training resources I am offering my “Intro to HPC” workshops every Wednesday at 3pm and Thursday at 4pm from October 11 through November 9. This workshop is designed to help new users become acquainted with and start using the cluster. To book a workshop session, click on the “Intro to HPC training session” link on the site below: https://it.engineering.oregonstate.edu/hpc<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fit.engine…> In addition to the workshop, other HPC resources and training offerings from Nvidia and Mark III are displayed on the right hand side of the HPC web site. Mark III is offering a series of AI and ML trainings every Tuesday starting October 17 through November 14. I encourage you to check them out and sign up for them if you are interested. For up-to-date status on the cluster, check out the link below: https://it.engineering.oregonstate.edu/hpc/hpc-cluster-status-and-news<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fit.engine…> Cheers, Rob Yelle HPC Manager

1 0

CoE HPC News, August 11 2023: Emergency shut down and upcoming heat wave
by Yelle, Robert Brian 11 Aug '23

11 Aug '23

HPCC users, Sometime after 8pm last night, there was a systems control failure that caused the chillers and pumps to stop working. This led to a rapid increase in the server room temperature, causing it to go to unsafe levels for the compute servers. The entire cluster had to be shut down to protect these servers. Facilities was notified, and they were able fix the problem and restore cooling to the server room, and they will continue to evaluate the cooling situation. In the meantime the cluster will be brought online later this morning, but at reduced capacity over the weekend and through next week to lessen the heat load during the upcoming heat wave. I know that some of you have deadlines fast approaching, and we will do what we can to accommodate you. For up-to-date status on the cluster, check out the link below: https://it.engineering.oregonstate.edu/hpc/hpc-cluster-status-and-news<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fit.engine…> Rob Yelle HPC Manager

1 0

CoE HPC News, July 17 2023: Future heat waves, new queue and queue limits
by Yelle, Robert Brian 17 Jul '23

17 Jul '23

HPCC users, Here is the latest news regarding the COE HPC cluster: Datacenter cooling issues Over the weekend parts of the cluster had to be shut down due to cooling issues arising from the previous heat wave. The cluster is now operating at full capacity, but be advised that in the future when there is a heat advisory, the cluster will operate at reduced capacity during the weekends, and during the weekdays when necessary. DGX partition limits DGX limits have been lowered again back to 4 GPUs due to high demand. The solution I had in place that was designed to provide greater flexibility did not work out as planned, so I am working on a different solution to this which may involve significant changes to the DGX queues and policies. If anyone has any feedback or suggestions on what they would like to see, feel free to email me. Share partition limits The CPU limit on the share partition is currently capped at 32 CPUs per node. New GPU partition “ampere” A new GPU queue called “ampere” consisting of A40 GPUs has been created and is currently open to all COE users. This queue is in a trial phase for the summer. At present you can reserve up to 2 GPUs and 16 CPUs total which is subject to change based on availability and demand. Give it a shot and if you have any feedback, let me know. More changes… More changes affecting the cluster are on the way this summer, so stay tuned! For up-to-date status on the cluster, check out the link below: https://it.engineering.oregonstate.edu/hpc/hpc-cluster-status-and-news<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fit.engine…> Cheers, Rob Yelle HPC Manager

1 0

CoE HPC News, July 6 2023: Submit-a relocation
by Yelle, Robert Brian 06 Jul '23

06 Jul '23

Cluster users, The cluster is online and near full capacity, including all submit nodes and HPC portal access. Submit-a relocation This morning, submit-b and submit-c were brought down to be relocated. Early next Thursday morning the 13th, I plan to relocate submit-a. This will require that I power off submit-a, which will terminate any srun sessions running on it. For the next week, I will be draining submit-a, which means ssh access will be restricted to existing sessions, so please use submit-b or submit-c for new login and srun sessions. Jobs submitted through sbatch on this node or through the HPC portal will not be affected. For up-to-date status on the cluster, check out the link below: https://it.engineering.oregonstate.edu/hpc/hpc-cluster-status-and-news<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fit.engine…> If you have any questions or concerns, let me know. Rob Yelle HPC Manager

1 0

CoE HPC News, June 30 2023: More AC maintenance
by Yelle, Robert Brian 30 Jun '23

30 Jun '23

Cluster users, The cluster is back online near full capacity. Here is more news regarding the COE HPC cluster: More AC maintenance Another round of AC maintenance is scheduled for next Thursday July 6 which will impact the KEC server rooms. To mitigate the heat load, the cluster will operate at reduced capacity again starting next Wednesday evening the 5th, until the server room temperature returns to a safe operating level, which should be the night of the 6th based on this past round of AC maintenance. Early Thursday morning during the AC maintenance, I plan to relocate two submit servers, submit-b and submit-c. This will require that I power them off which will terminate any srun sessions running on them. Jobs submitted through sbatch on these nodes or through the HPC portal will be unaffected. In addition, the Slurm scheduler will temporarily be offline early Thursday morning. This will prevent new submissions (which will be limited anyways) or queries on job and node status, but running jobs will not be affected. I anticipate that Slurm will be back online by 10am and the two submit nodes by 11am. For up-to-date status on the cluster, check out the link below: https://it.engineering.oregonstate.edu/hpc/hpc-cluster-status-and-news<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fit.engine…> If you have any questions or concerns, let me know. Have a great weekend! Rob Yelle HPC Manager

1 0