- cluster-users - Engineering Mailing Lists

CoE HPC News, March 12 2025: Email notification, Springbreak maintenance
by Yelle, Robert Brian 12 Mar '25

12 Mar '25

Cluster users, See below the latest news regarding the COE HPC cluster. Email notification Email notification for Slurm jobs was not functional for a while after the head node migration in December, but it is working now. New partition limits This is a reminder that the new partition or queue limits are posted and updated under “Summary of partition limits” in the Slurm Howto below: https://it.engineering.oregonstate.edu/hpc/slurm-howto<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fit.engine…> Springbreak maintenance The next cluster maintenance is scheduled for the week of March 24. The maintenance activities planned for this time include: Slurm update and queue+configuration changes Operating system image updates Nvidia GPU, infiniband, and storage driver updates BIOS and firmware updates as needed Miscellaneous hardware maintenance as needed The entire cluster will be offline starting Monday, March 24 at 12pm, and will remain offline until approximately Wednesday the 26th at 4pm. Jobs scheduled to run into that offline period will remain pending with the message “ReqNodeNotAvail, Reserved for maintenance”. If you wish for your Slurm job to start and finish before the offline period begins, you will need to adjust your time limit accordingly, e.g. if there is less than 3 days until the maintenance begins, then to change the time limit of your pending job to 2 days do: scontrol update job {jobid} TimeLimit=2-00:00:00 Alternatively, you can cancel your pending job, adjust your walltime (using the --time option) and resubmit. Job purge/Queue changes Any pending jobs remaining during the maintenance period may be purged while new queueing changes are implemented. If you have any questions or concerns, let me know. For up-to-date status on the cluster, check out the link below: https://it.engineering.oregonstate.edu/hpc/hpc-cluster-status-and-news<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fit.engine…> Rob Yelle HPC Manager

1 0

CoE HPC News, February 14 2025: Stuck jobs, queue changes, published features
by Yelle, Robert Brian 14 Feb '25

14 Feb '25

Cluster users, Happy Valentines Day! See below the latest news regarding the COE HPC cluster. Jobs stuck in queue? I have been getting a lot of complaints lately of jobs being stuck in the queue. The most likely reason is that the queue is busy (see Job queuing changes below), but sometimes a job may be stuck for some other reason, so I encourage users to continue to reach out to me if they feel their job is stuck in the queue, just make sure you leave your job in the queue for me to troubleshoot. This will allow me to see what is holding the job up and possibly make adjustments to push it through. Job queuing changes Many of you have noticed that several queues have been very busy lately. In fact, the cluster has reached an all time high lately in terms of percent utilization of resources overall. In particular, the dgxh and dgx2 partitions have been at or near 100% utilization for much of the last two weeks. It is fantastic that the COE cluster has been such a useful resource for so many. The down side is that the wait times have been very long in many cases, and I am receiving complaints almost every day about queue wait times. Also, in many cases cluster resources (CPUs and GPUs) are not being efficiently used – in other words, they have been reserved but are idle for whatever reason, and to date there is no reliable mechanism to address this. To address the high demand and other queuing challenges, I am considering and evaluating a number of major changes to certain partitions. In the past I normally just reduced the resource limits, and for some partitions like dgxh I kept the time limit at 2 days for faster job turnover, but those one-dimensional approaches are starting to become too restrictive. Another approach I have started to implement is a relatively new feature in Slurm, one that limits the number of resource-run-limits one user can have active at a time. For instance, the dgxh partition formerly had a limit of 2 GPUs with a separate, independent time limit of 2 days. However, since the dgxh partition is in such high demand, I have set a new limit on this partition that allows only 1 GPU to be used for 2 days, or 2 GPUs at a time to be used for only 1 day, i.e. a 2 GPU-day or 48-GPU-hour limit. The flip side of this limit is that it is now possible to schedule a larger number of resources but for a shorter period of time, e.g. 4 GPUs for 12 hours, or 8 GPUs for 6 hours. I have applied similar limits to the dgx2, ampere, gpu, and share partitions. The updated resource limits are posted under “Summary of partition limits” in the Slurm-howto link below: https://it.engineering.oregonstate.edu/hpc/slurm-howto<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fit.engine…> The goal of implementing these new limits is to try reducing wait times while keeping resource limits flexible, and these limits will be adjusted to accommodate needs and demand. The new limits may now result in some jobs pending with “Max*RunMinsPerUser” messages. If you already have jobs submitted with the older limits, you may leave them in the queue for now and I’ll try to push through what I can over the next week. More changes may be implemented and announced in the near future. If you have any questions, feedback or suggestions regarding the queue changes, let me know. Published list of features, “ib” is now obsolete The “ib” feature had been used by those users running MPI jobs. Now the entire cluster has infiniband, so the “ib” feature or “constraint=ib” directive in Slurm is no longer required and has been removed. To see an updated list of features, including OS and GPU features, check out the “Summary of partitions” in the link below: https://it.engineering.oregonstate.edu/hpc/slurm-howto<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fit.engine…> Stay tuned for more news and updates. For up-to-date status on the cluster, check out the link below: https://it.engineering.oregonstate.edu/hpc/hpc-cluster-status-and-news<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fit.engine…> Rob Yelle HPC Manager

1 0

CoE HPC News, January 21, 2025: Upcoming Slurm outage, noVNC errors, new DGX
by Yelle, Robert Brian 21 Jan '25

21 Jan '25

Cluster users, Happy New Year! See below the latest news regarding the COE HPC cluster. Brief Slurm outage The new head node will be offline for brief maintenance this Thursday morning, January 23, between 8-10am. The Slurm scheduler will be down so no new jobs will be queued during this time, but currently running or pending jobs will not be affected. NoVNC zlib error Since the OnDemand HPC Portal upgrade in December, a number of users have received the following error on the portal: noVNC encountered an error: Uncaught Error: Incomplete zlib block The error message is contained in a large red box, and usually occurs when trying to select text. Unfortunately you cannot get rid of the box so you need to re-launch your session to work around it. To avoid getting this error message, make sure you enable compression before launching your interactive app. At present there is no way for me to set the minimum compression level, so this must be done by the user – just don’t set it to 0. New Nvidia DGX H200 server The College of Engineering recently purchased a new DGX H200 server, and this has now been added to the dgxh partition. To request an H200 GPU on the dgxh partition, you can use the “--constraint=h200” Slurm option, or add “h200” as a feature on the HPC portal. Coming soon Some of you may have noticed that the Multi-instance GPU (MIG) feature of the H100 GPU has been enabled on some of the dgxh nodes. The MIG feature will allow more efficient use of the H100 GPUs, and will shorten waiting times. This is currently in testing phase, and a couple of issues have come up during testing, but it is expected to be rolled out to all dgxh nodes once these issues are resolved. The next cluster-wide maintenance is scheduled for Springbreak, March 24-28. Rob Yelle HPC Manager

1 0

DGXH nodes offline this Thursday
by Yelle, Robert Brian 31 Dec '24

31 Dec '24

Cluster users, The dgxh nodes will be offline for maintenance this Thursday morning, January 2nd. They should return to service later that day. Rob Yelle HPC Manager

1 0

CoE HPC News, December 2 2024: Christmas break maintenance
by Yelle, Robert Brian 20 Dec '24

20 Dec '24

Cluster users, The next cluster maintenance is scheduled for the week of December 16. The maintenance activities planned for this time include: Head node upgrade OnDemand HPC portal upgrade Slurm upgrade and configuration changes Operating system image updates Nvidia GPU and infiniband driver updates BIOS and firmware updates as needed Miscellaneous hardware maintenance as needed The entire cluster will be offline starting Monday, December 16 at 1pm, and will remain offline until approximately Wednesday the 18th at 4pm. Jobs scheduled to run into that offline period will remain pending with the message “ReqNodeNotAvail, Reserved for maintenance”. If you wish for your Slurm job to start and finish before the offline period begins, you will need to adjust your time limit accordingly, e.g. to change to 2 days do: scontrol update job {jobid} TimeLimit=2-00:00:00 Alternatively, you can cancel your pending job, adjust your walltime (using the --time option) and resubmit. If you have any questions or concerns, let me know. For up-to-date status on the cluster, check out the link below: https://it.engineering.oregonstate.edu/hpc/hpc-cluster-status-and-news<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fit.engine…> Rob Yelle HPC Manager

1 1

Fall 2024 AI/ML Education series by MarkIII and Nvidia
by Yelle, Robert Brian 30 Oct '24

30 Oct '24

Cluster users, For those of you interested in or working on AI/ML, please see the upcoming training series below: Monday November 4: Intro to Machine Learning and AI Tuesday November 19: Intro to Deep Learning Tuesday December 3: Intro to LLMs Tuesday December 10: Intro to Omniverse and Digital Twins To register for one or more of these trainings, please check out the link below: https://trending.markiiisys.com/osu-aiseries-fall2024<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftrending.…> Cheers, Rob Yelle HPC Manager

1 0

AI Training Opportunity
by Yelle, Robert Brian 22 Oct '24

22 Oct '24

Cluster users, I’m forwarding an AI training opportunity described below: Microway is hosting an NVIDIA training session this Thursday at 9am on using RAG to augment chatbots. Register to attend or get the recording: https://www.microway.com/events/building-intelligent-chatbots-using-rag/<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.micro…> Cheers, Rob Yelle HPC Manager

1 0

FW: Kelley Engineering Center - High Performance Computer Update
by Yelle, Robert Brian 25 Sep '24

25 Sep '24

Cluster users, Just in case this didn’t post earlier, please see message below from Wade Marcum. Rob Yelle HPC Manager From: Marcum, Wade <Wade.Marcum(a)oregonstate.edu> Date: Wednesday, September 25, 2024 at 9:40 AM To: cluster-users(a)engr.orst.edu <cluster-users(a)engr.oregonstate.edu> Cc: Marcum, Wade <Wade.Marcum(a)oregonstate.edu> Subject: Kelley Engineering Center - High Performance Computer Update Dear Cluster Users- We continue to invest in our High-Performance Computer (HPC) to support our users and the world-class research and scholarship that results. To this end, we have developed a policy (following best-practices) that articulates the approach we will be implementing moving forward - a 'condo-model'. This policy is intended to provide clarity and transparency for all users on hardware purchase, usage, and lifecycle. This will, over time, impact all existing servers installed in the KEC HPC, however, it should not impact your ability to utilize our community HPC in an adverse way. I encourage you please review the policy posted at the following link: https://it.engineering.oregonstate.edu/kec-server-room-equipment-policy<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fit.engine…> Best Regards, Wade

1 0

CoE HPC News, September 10 2024: Fall maintenance
by Yelle, Robert Brian 24 Sep '24

24 Sep '24

Cluster users, The next cluster maintenance is scheduled for the week of September 23. The maintenance activities planned for this time include: Operating system updates OnDemand HPC portal upgrade Slurm update and configuration changes Nvidia driver updates BIOS and firmware updates as needed Miscellaneous hardware maintenance as needed The entire cluster will be offline starting Monday, September 23 at 8am, and will remain offline until approximately Tuesday the 24th at 4pm. Jobs scheduled to run into that offline period will remain pending with the message “ReqNodeNotAvail, Reserved for maintenance”. If you wish for your Slurm job to start and finish before the offline period begins, you will need to adjust your time limit accordingly, e.g. to change to 2 days do: scontrol update job {jobid} TimeLimit=2-00:00:00 Alternatively, you can cancel your pending job, adjust your walltime (using the --time option) and resubmit. If you have any questions or concerns, let me know. For up-to-date status on the cluster, check out the link below: https://it.engineering.oregonstate.edu/hpc/hpc-cluster-status-and-news<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fit.engine…> Rob Yelle HPC Manager

1 2

Cluster offline due to storage
by Yelle, Robert Brian 24 Jul '24

24 Jul '24

HPC users, The cluster is offline again, this time due to a failure of the HPC share storage. A number of users noticed issues with their jobs or python environments, and accessing their files or directories. I have opened up a ticket with the vendor on this issue and have been working with them over the course of the day but no resolution yet. It is not clear what caused the issue or when it will be resolved. I’ll update the list when I have more answers. Rob Yelle HPC Manager

1 1