- cluster-users - Engineering Mailing Lists

HPC Cluster offline
by Yelle, Robert Brian 18 Jul '24

18 Jul '24

Cluster users, The HPC cluster is currently down. The word is that earlier this afternoon there was some fire alarm testing that triggered the shutdown of the KEC datacenters, which also impacted networking. Power and networking has been restored, and cooling is currently being restored, so I will start bringing the cluster back online shortly. Thank you for your patience while the cluster resources are being restored. Rob Yelle HPC Manager

1 1

CoE HPC News, June 27 2024: Current status and various reminders
by Yelle, Robert Brian 27 Jun '24

27 Jun '24

Cluster users, Survey If you haven’t had a chance yet, please take a few minutes to complete the survey below, your feedback is important. https://oregonstate.qualtrics.com/jfe/form/SV_290Wnkkv7IFqSW2<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Foregonsta…> Cluster status Most of the cluster has been upgraded and is back online. The following hosts are still offline until further notice: submit-a (please use submit-b or submit-c instead) dgx2-[3,4] dgxs-[1-3] cn-h-[5-8] The HPC portal is back to a “mostly working” state, but some interactive applications do not work well on the portal at present. The portal is still being worked on. In the meantime, if your desired application does not work well, try using the Advanced COEHPC Desktop and launch the application from there. Submit node hostkeys When you access the upgraded nodes, you may be met with the following message: "host key for submit-b.hpc.eng.oregonstate.edu has changed and you have requested strict checking. Host key verification failed." Or something similar. To address this, please remove your old host keys as follows, e.g.: ssh-keygen -R submit-b.hpc.engr.oregonstate.edu Repeat this for all submit nodes, and for any other HPC hosts that you need ssh access for. After that, try connecting again and accept the new host keys and you should be set. New HPC storage All HPC share data has been migrated to our new DDN storage appliance, still located on /nfs/hpc/share, and all upgraded nodes are now using this storage. This directory is available on the HPC cluster only and is not visible on the Flips (access.engr.oregonstate.edu) I will mount the old share on /mnt/share on the submit nodes for a week or so to give people a chance to check the current HPC shares against the old one, and to copy files or directories that may be missing. If anyone has any questions or problems, let me know. For up-to-date status on the cluster, check out the link below: https://it.engineering.oregonstate.edu/hpc/hpc-cluster-status-and-news<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fit.engine…> Rob Yelle HPC Manager

1 0

CoE HPC News, June 10 2024: Summer maintenance reminder
by Yelle, Robert Brian 20 Jun '24

20 Jun '24

Cluster users, This is a reminder that Summer maintenance is scheduled to start next week (after Finals week), and will go on until completed. During this time I plan to complete the migration of the remaining cluster nodes to the EL8 and EL9 based operating systems. In addition to the OS upgrades, regular maintenance activities will be performed during this time, e.g.: OnDemand HPC portal upgrade Slurm upgrade and configuration changes Nvidia driver updates BIOS and firmware updates as needed Miscellaneous hardware maintenance as needed The entire cluster will be offline starting Monday, June 17 at 1pm, and will remain offline until approximately Wednesday the 19th at 4pm. Jobs scheduled to run into that offline period will remain pending with the message “ReqNodeNotAvail, Reserved for maintenance”. If you wish for your Slurm job to start and finish before the offline period begins, you will need to adjust your time limit accordingly, e.g. to change to 2 days do: scontrol update job {jobid} TimeLimit=2-00:00:00 Alternatively, you can cancel your pending job, adjust your walltime (using the --time option) and resubmit. If you have any questions or concerns, let me know. For up-to-date status on the cluster, check out the link below: https://it.engineering.oregonstate.edu/hpc/hpc-cluster-status-and-news<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fit.engine…> Rob Yelle HPC Manager

1 2

CoE HPC News, May 6 2024: DGXH partition, submit-c and a host of issues
by Yelle, Robert Brian 06 May '24

06 May '24

Cluster users, See below the latest news on the COE HPC cluster. Update on DGXH partition The dgxh partition has been in very high demand lately, running almost steadily at or near 100% for the last few weeks, and wait times for this resource have been as long as a week or more. I am continually working on improving this situation so that everyone gets a fair chance to use these resources, and in a way that meets their need. When your allocation is ready, make sure you use it effectively and don’t waste it. This is a limited resource that is primarily for production runs. We have other resources that can be used for testing and debugging. Sessions that are found running idle or unused for a period of time may be terminated, so use this resource wisely. VSCode issues A number of our users use VSCode to ssh to the cluster. This has an unfortunate side effect of leaving defunct processes running on the submit nodes which can eventually prevent the user from logging in again. This problem is occurring with increasing regularity, so to address this, VSCode processes will be automatically cleaned up on a periodic basis. We advise that you do not use “srun” through VScode as it may be terminated with the VSCode process. Submit-c is online Submit-c has been upgraded to EL8 and is back online for both ssh and portal access. Also, upgraded versions of VScode should work on this node. If anyone has any problems with submit-c, let me know. HPC Portal issues Occasionally when one logs in through the HPC portal, they are met with a “Bad Request” error. One way to work around this problem is to choose a different submit node, e.g. if you get a “Bad Request” on submit-a, then try submit-b or submit-c. Alternatively, try deleting your browser cache and cookies, then try again. Some users have recently encountered a problem being locked out of their Desktop sessions on EL9 systems. This issue should now be resolved and users should no longer be locked out, but to prevent the constant lock out I recommend users disable or change their screensaver settings. Apptainer/Singularity issues It was recently reported that Apptainer is not working on the EL9 systems. This is still being worked on. In the meantime I recommend using Apptainer on the EL7 systems where possible. Summer Maintenance The next cluster maintenance period is scheduled for the week of June 17, after finals week. The cluster will be offline for part of that week, so please plan accordingly. If you have any questions or concerns, let me know. For up-to-date status on the cluster, check out the link below: https://it.engineering.oregonstate.edu/hpc/hpc-cluster-status-and-news<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fit.engine…> Rob Yelle HPC Manager

1 0

Question about Queuing on HPC Cluster
by Fakhry, Mahmoud 16 Apr '24

16 Apr '24

Hello Robert, I was wondering if there is an issue with the HPC Cluster currently since I have had some nodes sitting in the queue for a while. Or maybe it has to do with the settings I’m using? List modules to load: anaconda Queue/Partitionclass gpu Number of hours: 24 Number of cores: 8 Number of GPUs: 1 Memory per node (optional): 32g Thanks, Mahmoud

1 0

CoE HPC News, April 9 2024: DGX-2 update, GPU Day and AI Training Series Reminders
by Yelle, Robert Brian 09 Apr '24

09 Apr '24

HPCC users, Update on DGX-2 nodes Last week a number of users reported a problem with the Nvidia GPU driver on the DGX-2 systems after the upgrade to EL9, so the newly upgraded DGX-2 systems had to be taken offline. A ticket was opened with Nvidia last week to troubleshoot the issue. Earlier this morning Nvidia spotted a potential problem, and a fix is currently being rolled out to the DGX-2 nodes. Most of the DGX-2 systems should be back online by this evening. For up-to-date status on the cluster, including the DGX nodes, check out the link below: https://it.engineering.oregonstate.edu/hpc/hpc-cluster-status-and-news<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fit.engine…> AI Week and GPU Day Reminder Just a reminder that this is AI Week. Tomorrow (Wednesday April 10) is GPU Day, where Nvidia and Mark III will team up to “bring you an action-packed day of learning about what GPUs are, how they can help your research, and how to optimize them for your workloads”. To register for GPU day and other AI Week events, please check out the link below: https://dri.oregonstate.edu/ai-week<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdri.orego…> MarkIII AI Training Series Reminder Another reminder that Mark III is offering a seven part series on AI and ML trainings every Tuesday at 11am starting next week, April 16 and running through May 28. The training topics are listed below: April 16 - Intro to Machine Learning and AI: The Basics, A Tutorial, and Lab April 23 - Intro to Deep Learning: An Introduction to Neural Networks April 30 - Introduction to Datasets May 7 - Introduction to Large Language Models May 14 - Getting Started with Containers and the software stack around AI + How to get started working with OSU HPC Services May 21 - Intro to Omniverse & Digital Twins May 28 - Intro to Isaac Sim and AI in Robotics I encourage you to check them out and sign up for them using the link below if you are interested: https://trending.markiiisys.com/osu-aiseries-2024<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftrending.…> If you have any questions or concerns, let me know. Rob Yelle HPC Manager

1 0

CoE HPC News, April 1 2024: EL8/EL9 update, AI Week, Workshops and Trainings
by Yelle, Robert Brian 02 Apr '24

02 Apr '24

HPCC users, Welcome to the start of Spring quarter 2024! See below the latest news on the CoE HPC cluster. Update on EL8/EL9 upgrade I was not able to complete the upgrade of the rest of the cluster to EL8 and EL9 during the Springbreak maintenance week. The upgrade process has been bumpy, and at this time only nodes from the dgx2, dgxs and ampere partitions have been migrated to EL9. The ampere partition is available now, and the dgx2 and dgxs partitions should be available by tomorrow (Wednesday) morning. Most of the rest of the cluster, including submit-a and submit-b, is still running EL7. I will continue working on the rest of the cluster over the course of this term, and will try to post on the HPC status page the approximate dates when certain nodes or partitions will be upgraded. I will work with affected research groups on scheduling upgrade windows. AI Week! As announced through various sources, next week (April 8-12) is AI Week. Wednesday April 10 is GPU Day, where Nvidia and Mark III will team up to “bring you an action-packed day of learning about what GPUs are, how they can help your research, and how to optimize them for your workloads”. For more information, and to register for GPU day and other AI Week events, please check out the link below: https://dri.oregonstate.edu/ai-week<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdri.orego…> Intro to HPC workshop I am offering my “Intro to HPC” workshops again this quarter, approximately once per week starting next Tuesday the 9th. This workshop is designed to help new users become acquainted with and start using the cluster. Due to a problem with my Bookings, people were unable to book a meeting during April. This has now been fixed, so if you are interested in attending this workshop this month, please register for a date using the link below: https://outlook.office365.com/owa/calendar/HPCworkshop@OregonStateUniversit…<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Foutlook.o…> MarkIII AI training series Mark III is offering a seven part series on AI and ML trainings every Tuesday at 11am from April 16 through May 28: April 16 - Intro to Machine Learning and AI: The Basics, A Tutorial, and Lab April 23 - Intro to Deep Learning: An Introduction to Neural Networks April 30 - Introduction to Datasets May 7 - Introduction to Large Language Models May 14 - Getting Started with Containers and the software stack around AI + How to get started working with OSU HPC Services May 21 - Intro to Omniverse & Digital Twins May 28 - Intro to Isaac Sim and AI in Robotics I encourage you to check them out and sign up for them using the link below if you are interested: https://trending.markiiisys.com/osu-aiseries-2024<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftrending.…> If you have any questions or concerns, let me know. For up-to-date status on the cluster, check out the link below: https://it.engineering.oregonstate.edu/hpc/hpc-cluster-status-and-news<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fit.engine…> Rob Yelle HPC Manager

1 0

CoE HPC News, March 14 2024: Cluster outages, spring maintenance, broken software
by Yelle, Robert Brian 26 Mar '24

26 Mar '24

HPCC users, See below the latest news on the CoE HPC cluster. Unplanned cluster outage Yesterday afternoon I made a change in the queuing system which did not affect the queuing server itself, but it did impact all of the client nodes. I frequently make changes to the queuing system, however this particular change unexpectedly caused client daemons to crash, which unfortunately and unexpectedly caused several jobs to terminate prematurely with the “NODE FAIL” message. For those of you wondering why your job or session was terminated yesterday, this is likely the reason, and I apologize for the inconvenience this outage has caused. Springbreak Maintenance/OS upgrades The next cluster maintenance is scheduled for Springbreak, March 25-29. During this week I plan to complete the migration of the remaining cluster nodes to the EL8 and EL9 based operating systems. In addition to the OS upgrades, regular maintenance activities will be performed during this time, e.g.: OnDemand HPC portal upgrade Slurm update and configuration changes Nvidia driver updates Infiniband updates BIOS and firmware updates as needed Miscellaneous hardware maintenance as needed The entire cluster will be offline after Finals week, starting Monday, March 25 at 12pm, and will remain offline until approximately Wednesday the 27th at 5pm. Jobs scheduled to run into that offline period will remain pending with the message “ReqNodeNotAvail, Reserved for maintenance”. If you wish for your Slurm job to start and finish before the offline period begins, you will need to adjust your time limit accordingly, e.g. to change to 2 days do: scontrol update job {jobid} TimeLimit=2-00:00:00 Alternatively, you can cancel your pending job, adjust your walltime (using the --time option) and resubmit. Tentative planned cooling outage A cooling outage impacting the HPC cluster has tentatively been scheduled for Friday April 5 (time window TBD). The cluster will run at reduced capacity that day until the planned AC maintenance has completed. Dealing with broken software on EL8/EL9 As the upgrades have progressed, a number of users have determined that some software which had worked on EL7 no longer works on EL9. In some cases, this has been addressed by installing new or compatible packages on the EL9 hosts. Also, it is currently possible to avoid an EL8/EL9 host by requesting the “el7” feature, either through the HPC portal or by adding the “--constraint=el7” option to your srun or sbatch options. However, keep in mind that this solution is only for the short term, since the entire cluster will eventually be running EL8/EL9, so in these cases the longer term solution is to rebuild the software on the EL9 (or EL8) host. I appreciate everyone’s feedback and patience regarding the upgrade, and I encourage everyone to continue reporting any issues that they encounter with the upgrade. Status of Nvidia DGX H100 systems The dgxh partition will remain in testing phase until the week of April 15. If you have any questions or concerns, let me know. For up-to-date status on the cluster, check out the link below: https://it.engineering.oregonstate.edu/hpc/hpc-cluster-status-and-news<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fit.engine…> Have a nice weekend, Rob Yelle HPC Manager

1 1

FW: Free GenAI/LLM Virtual Workshop Series for Higher-Ed
by Yelle, Robert Brian 12 Feb '24

12 Feb '24

Cluster users, Please see announcement below regarding Free GenAI/LLM workshop series, which starts this Thursday the 15th: Free GenAI/LLM Virtual Workshop Series for Higher Ed (NALA)<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fevents.nv…> Join NVIDIA’s Deep Learning Institute (DLI) this February for a series of free, virtual instructor-led workshops providing hands-on experience with GPU-accelerated servers in the cloud to complete end-to-end projects in the areas of Generative AI and Large Language Models (LLMs). Each of these workshops are led by a DLI Certified Instructor and offer an opportunity to earn an industry-recognized certificate of competency based on assessments to support your career growth. This GenAI/LLM workshop series follows the larger DLI Learning Path for GenAI and LLMs<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnvdam.wid…>. North America Workshops (taught in English): * Fundamentals of Deep Learning * February 15, 2024 @ 8:00 a.m.-4:00 p.m. PST * Generative AI With Diffusion Models * February 15, 2024 @ 8:00 a.m.-4:00 p.m. PST * Fundamentals of Deep Learning * February 16, 2024 @ 8:00 a.m.-4:00 p.m. PST * Generative AI With Diffusion Models * February 16, 2024 @ 8:00 a.m.-4:00 p.m. PST * Building Transformer Based Natural Language Processing Applications * February 22, 2024 @ 8:00 a.m.-4:00 p.m. PST * Building Conversational AI Applications * February 23, 2024 @ 8:00 a.m.-4:00 p.m. PST * Model Parallelism: Building and Deploying Large Neural Networks * February 28, 2024 @ 8:00 a.m.-4:00 p.m. PST * Data Parallelism: How to Train Deep Learning Models on Multiple GPUs * February 29, 2024 @ 8:00 a.m.-4:00 p.m. PST Rob Yelle HPC Manager

1 0

CoE HPC News, February 2 2024: Updates on EL8/EL9 migration, EL9 fixes, VSCode
by Yelle, Robert Brian 02 Feb '24

02 Feb '24

HPCC users, See below the latest news on the CoE HPC cluster. Busy queues The demand for resources, especially GPUs, has been very high lately, due to upcoming deadlines. This is resulting in much longer wait times than usual, so please be patient. If you have been waiting for over a day for your resources, let me know and leave your job in the queue as I may be able to work it in or at least determine what is holding it up. EL8/EL9 upgrades The cluster upgrade to EL8 and EL9 has been progressing slowly as various bugs are being worked through. Progress is expected to pick up the week of February 12, as submit nodes and more compute nodes from the “share”, “dgx2” and “dgxs” partition will be migrated. These partitions will not be migrated all at once, but each week one or more nodes from each partition will be upgraded until the migration is completed. At this point it is anticipated that the migration will be completed by the end of March. At present the DGX nodes “dgx2-4” and “dgxs-3” have been upgraded to EL9 with support for Cuda 12.2, and are now available through the dgx2 and dgxs partitions. Note that if you do not wish to land on an EL8 or EL9 node at this time, then you may request the “el7” feature, either through the HPC portal or by adding the “--constraint=el7” option to your srun or sbatch options. For those of you who would prefer to land on an EL9 node, for instance, because of updated dependencies and software support, or the availability of more recent versions of Cuda, then you may request the “el9” feature through the portal or use the “--constraint=el9” option in srun or sbatch. Please continue to report any issues that you encounter with the cluster as the upgrade progresses. Portal and ssh fixes on Nvidia DGX H100 and other EL9 systems The HPC portal and ssh are now working properly on the dgxh and other EL9 systems. The dgxh partition is currently available on the Advanced Desktop (Xfce desktop only!) and Jupyter Server apps. Note that with the Advanced Desktop on EL9 systems, you should disable the screen lock or you risk being locked out of your session and you’ll have to start over. To do this, click on “Applications” on the upper left hand corner, then select “Settings”, then “Xfce screensaver”, then the “Lock Screen” tab, and turn off the “Enable Lock Screen”. The dgxh partition will remain in testing phase until further notice while other issues are being addressed. VSCode issues Some users have recently reported a problem using VSCode on the HPC submit nodes. It appears that the latest version or update of VSCode no longer supports EL7, so you may not be able to run it on the submit nodes. The submit nodes will be upgraded to EL8 later this month through early March. Until then, my recommendation is to use an older version of VSCode if possible, or use the Flip servers until the submit nodes are upgraded. Springbreak Maintenance The next cluster maintenance is scheduled for Springbreak, March 25-29. If you have any questions or concerns, let me know. For up-to-date status on the cluster, check out the link below: https://it.engineering.oregonstate.edu/hpc/hpc-cluster-status-and-news<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fit.engine…> Have a nice weekend, Rob Yelle HPC Manager

1 0