October 2023 - cluster-users - Engineering Mailing Lists

COE HPC cluster down
by Yelle, Robert Brian 30 Oct '23

30 Oct '23

Cluster users, The datacenter experienced a cooling failure this morning which required an emergency shutdown of the entire cluster. Facilities is working on the issue, and we hope to bring the cluster back online sometime tomorrow. Rob Yelle HPC Manager

1 1

CoE HPC News, October 24 2023: New DGX systems and partition changes
by Yelle, Robert Brian 24 Oct '23

24 Oct '23

HPCC users, Please check out the latest news below regarding the COE HPC cluster: New Nvidia DGX H100 systems are online! Many of you may have received the announcement last week that three new Nvidia DGX H100 systems have been added to the COE HPC cluster. Each DGX H100 system has 112 CPUs, 8 Nvidia H100 GPUs, and 2TB RAM! These systems are now available in a new partition called “dgxh”. This partition is in a testing period for at least this week to resolve existing issues and smooth out any other kinks that crop up. During this testing period, the current resource limits are 2 GPUs and 32 CPUs per user, and the time limit is 24 hours. Be advised that the DGX H100 systems are running RHEL9 based linux, which is different than the RHEL7 based systems currently used by the rest of the cluster. Also, these systems are not yet available through the HPC portal or through ssh. Give them a try and let me know of any issues you encounter. DGX partition change We currently have separate “dgx” and “dgx2” partitions for the our DGX-2 systems, depending on how many GPUs are needed. For various reasons that have come up over time, it is no longer advantageous to have separate partitions for these systems, so these will again be merged into a single “dgx2” partition. The “dgx” partition will be phased out or possibly re-purposed, so please use “dgx2” instead. The resource limits will remain at 4 GPUs and 32 CPUs on the dgx2 partition for now, but may increase again in the future. Submit-a offline this week Just a reminder that Submit-a will be offline until next Monday the 30th for maintenance. Until then, please use submit-b or submit-c. If you have any questions or concerns, let me know. For up-to-date status on the cluster, check out the link below: https://it.engineering.oregonstate.edu/hpc/hpc-cluster-status-and-news<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fit.engine…> Cheers, Rob Yelle HPC Manager

1 0

CoE HPC News, October 9 2023: Training and update reminder
by Yelle, Robert Brian 09 Oct '23

09 Oct '23

HPCC users, Here are some updates and reminders from last week’s newsletter, especially for the benefit of the many new users who have recently been added to the cluster users list. HPC training reminders Attention new cluster users! I am offering my “Intro to HPC” workshops every Wednesday at 3pm and Thursday at 4pm from October 11 through November 9. This workshop is designed to help new users become acquainted with and start using the cluster. To book a workshop session, click on the “Intro to HPC training session” link on the site below: https://it.engineering.oregonstate.edu/hpc<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fit.engine…> AI and ML series of trainings from MarkIII starting next week! For anyone who is interested, Mark III is offering a free five part series of AI and ML trainings starting next Tuesday October 17 and running through November 14: Tuesday, October 17 at 11am: Intro to AI and Machine Learning: The basics, tutorial and lab Tuesday, October 24 at 11am: Intro to Deep Learning: An introduction to Neural Networks Tuesday, October 31 at 11am: Intro to Datasets Tuesday, November 7 at 11am: Intro to Computer Vision and Image Analytics Tuesday, November 14 at 11am: Getting Started with Containers and AI If you are interested in more details and want to register for any of the MarkIII trainings listed above, please check out the “OSU AI Series” on the right hand side of the HPC web site above. Also note the HPC resources from Nvidia also displayed on the right hand side. Updates on cluster updates The updates of the cluster compute and GPU nodes should be completed by the end of the day Tuesday October 10. The submit nodes will be taken offline and scheduled to be updated on the following dates: Submit-c: Monday, October 16 at 9am Submit-b: Monday, October 23 at 9am Submit-a: Monday, October 30 at 9am Please plan accordingly. Srun jobs still running on these hosts when they are scheduled to be updated will be terminated. Jobs submitted via sbatch or the HPC portal will not be affected. If you have any questions or concerns, let me know. For up-to-date status on the cluster, check out the link below: https://it.engineering.oregonstate.edu/hpc/hpc-cluster-status-and-news<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fit.engine…> Cheers, Rob Yelle HPC Manager

2 1

CoE HPC News, October 4 2023: Updates, upgrades and trainings
by Yelle, Robert Brian 04 Oct '23

04 Oct '23

HPCC users, Here is the latest news regarding the COE HPC cluster: OS and GPU driver updates The cluster is currently running at reduced capacity while OS updates are being rolled out. In addition, some users have reported that they cannot run their GPU codes on some GPU nodes, which appears to be due to the older drivers on these nodes. New GPU drivers are available, so the GPU nodes are scheduled to be offline next week in a staggered fashion (some Monday the 9th and some Tuesday the 10th) so that the new drivers can be installed. If you need GPU resources, please schedule your jobs so that they can complete before these offline periods, otherwise they will remain pending with the message “Required Node not available” or “ReqNodeNotAvail”. If that happens, you may be able to get in by reducing your time requirement using the “--time” or “-t" option in srun or sbatch. OS upgrade Many of you are aware that many COE linux servers have been or are being upgraded to Enterprise Linux 9 (EL9). This will also happen on the HPC cluster over the course of this Fall quarter. More details to come. Intro to HPC workshop and other HPC training resources I am offering my “Intro to HPC” workshops every Wednesday at 3pm and Thursday at 4pm from October 11 through November 9. This workshop is designed to help new users become acquainted with and start using the cluster. To book a workshop session, click on the “Intro to HPC training session” link on the site below: https://it.engineering.oregonstate.edu/hpc<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fit.engine…> In addition to the workshop, other HPC resources and training offerings from Nvidia and Mark III are displayed on the right hand side of the HPC web site. Mark III is offering a series of AI and ML trainings every Tuesday starting October 17 through November 14. I encourage you to check them out and sign up for them if you are interested. For up-to-date status on the cluster, check out the link below: https://it.engineering.oregonstate.edu/hpc/hpc-cluster-status-and-news<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fit.engine…> Cheers, Rob Yelle HPC Manager

1 0