- cluster-users - Engineering Mailing Lists

CoE HPC News, October 14 2022 Edition: Workshop, HPC Portal apps
by Yelle, Robert Brian 14 Oct '22

14 Oct '22

HPCC users, Please check out the latest HPC cluster news below. Intro to HPC Workshop I will be holding an HPC workshop over Zoom, covering the basics of using the CoE HPC cluster next week at the following date and times: Wednesday, October 19 @3pm Thursday, October 20 @4pm If you are interested in attending this workshop, please let me know which session works best for you. New STAR-CCM+ app STAR-CCM+ has been added to the list of interactive apps on the HPC portal. If anyone has any problems using it, let me know. You can check out the HPC portal here: https://ondemand.hpc.engr.oregonstate.edu<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fondemand.…> Updated Jupyter Server app Previously the Jupyter Server app allowed you to activate your own python virtual environment and conda installation, but could not activate an environment created using conda. The Jupyter Server app on the HPC portal has now been further improved to allow your own conda environment. If anybody has any problems using this app, let me know. New Software installs Here is a list of recent software installs: Python 3.10 GCC 12.2 Mathematica 13.1 Matlab 2022b with Parallel Server. If anybody is interested in using Matlab Parallel Server, please contact me. The software listed above can be accessed through the modules system, type “module avail” to see what is available. For the latest cluster news and status updates, check out the link below: https://it.engineering.oregonstate.edu/hpc/hpc-cluster-status-and-news<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fit.engine…> Have a nice weekend! Rob Yelle HPC Manager

1 0

CoE HPC News, Sept 1 2022 Edition: Fall maintenance
by Yelle, Robert Brian 14 Sep '22

14 Sep '22

HPCC users, Here is the latest news on the CoE HPC cluster: Fall maintenance week: The HPC Cluster will undergo its regularly scheduled quarterly maintenance September 12-16. The following maintenance activities will be performed: Operating system updates BIOS and firmware updates as needed Slurm scheduler upgrade Nvidia driver and CUDA updates Miscellaneous hardware maintenance as needed The entire cluster will be offline starting Tuesday the 13th at 8am, and will remain offline until approximately Wednesday the 14th at 3pm. Jobs scheduled to run into that offline period will remain pending with the message “ReqNodeNotAvail, Reserved for maintenance”. If you wish for your Slurm job to start and finish before the maintenance period begins, you will need to adjust your time limit accordingly, e.g. to change to 2 days do: scontrol update job {jobid} TimeLimit=2-00:00:00 Alternatively, you can cancel your pending job, adjust your walltime (using the --time option) and resubmit. I will send out a reminder next week, and another announcement when the cluster is back online. For cluster news and status updates, check out the link below: https://it.engineering.oregonstate.edu/hpc/hpc-cluster-status-and-news<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fit.engine…> Cheers, Rob Yelle HPC Manager

1 2

CoE HPC News, August 8 2022 Edition: Cooling outage, HPC portal apps, DGX queue reminders
by Yelle, Robert Brian 08 Aug '22

08 Aug '22

HPCC users, Please see the latest cluster news below. Datacenter cooling outage Most of the HPC cluster underwent an emergency shutdown last night due to a cooling failure in the KEC datacenter as temperatures had reached critical levels. Unfortunately many jobs or interactive sessions were terminated as a result of the emergency shutdown. Most of the cooling has been restored, and HPC resources are slowly being brought back online to a level that can be accommodated by the available cooling. New HPC portal apps New interactive apps have been added to the HPC portal: Matlab Mathematica R Studio Ansys Workbench (for approved Ansys users only) If you use any of these applications, check these out and let me know if you have any trouble using them. You can check out the HPC portal here: https://ondemand.hpc.engr.oregonstate.edu DGX queue change reminder This is a reminder that the DGX partitions have been redefined as follows: If you need 4 GPUs or less, please use the “dgx” partition. If you need 4 GPUs or more, please use the “dgx2” partition. If your jobs to the dgx/dgx2 partitions are pending with “QOSMinGRES” or “QOSMaxGRES”, or if your jobs are rejected for those reasons, that means you need to change the partition as noted above. For the latest cluster news and status updates, check out the link below: https://it.engineering.oregonstate.edu/hpc/hpc-cluster-status-and-news<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fit.engine…> Cheers, Rob Yelle HPC Manager

1 0

CoE HPC News, August 1 2022 Edition: DGX queue and limit changes
by Yelle, Robert Brian 01 Aug '22

01 Aug '22

HPCC users, Please check out the latest HPC cluster news below. DGX partition changes Two changes are being introduced to the DGX systems this month to improve job scheduling and flexibility. First, the “dgx” and “dgx2” partitions are being redefined. Starting tomorrow morning, the “dgx” partition will be used for smaller GPU workloads, i.e. 4 GPUs or less, and 12 CPUs or less, whereas the “dgx2” partition will be for larger GPU workloads, i.e. 4 GPUs or more, and 12 CPUs or more. For those of you who normally request fewer than 4 GPUs at a time under the “dgx2” partition, please change to the “dgx” partition. As some of you are aware, the larger the resource request, the longer the wait, and this change was recommended by the vendor as a way to improve the scheduling of larger workloads on the DGX systems. New DGX limits The second change involves the DGX resource limits. The GPU and CPU limits for the DGX systems sometimes change based on overall load and resource availability. Lately we have settled to a limit of 8 GPUs and 32 CPUs in use at a time per user. This limit will temporarily be lifted in lieu of new limits based on cumulative GPU and CPU running times. This means that if GPUs are available, then more GPUs than the normal limit can be used by each user at a time, though for a shorter period (e.g. 16 GPUs for one day). This is to improve job flexibility while also maximizing use of resources, to allow users to run more jobs at a time, or to run larger single calculations or experiments. The new limits will be activated starting this week, and may require a lot of adjustment at first to optimize the load on the dgx partitions. These limits will be posted on the HPC status page once activated. If you have any questions about, or experience issues due to the new limits, let me know. Jupyter Notebook app The Jupyter Notebook app on the HPC Portal is being replaced by the new Jupyter Server app, and will no longer appear in the list of interactive apps. To follow status updates on the news mentioned above, check out the link below: https://it.engineering.oregonstate.edu/hpc/hpc-cluster-status-and-news<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fit.engine…> Cheers, Rob Yelle HPC Manager

1 0

CoE HPC News, July 12 2022 Edition: HPC Portal Maintenance, Jupyter Lab and more
by Yelle, Robert Brian 12 Jul '22

12 Jul '22

HPCC users, Please check out the latest HPC cluster news below. HPC Portal maintenance The HPC Portal will be offline next Monday morning the 19th for maintenance. During this time, the Web and Portal packages will be updated, and to replace expiring SSL certificates. Jobs running through the portal at the time should not be interrupted, but they will not be accessible until the portal is back online. The portal is expected to be back online by noon on the 19th. New Jupyter Server app An improved Jupyter Server app has been added to the list of Interative Apps on the HPC Portal. This app gives you the option to use Jupyter Lab, and allows you to specify your own Python or Conda environment to run Jupyter, if desired. This app is in beta testing phase and will eventually replace the existing Jupyter Notebook app. Preempt Partition The “preempt” partition has undergone a lengthy testing period, and is being used with increasing frequency and bears mentioning. The preempt partition is available to all users, but it is not the same as the share partition. It is a low priority partition which can give you access to unused resources, but that access can be cancelled or “preempted” by a higher priority request or job. The preempt partition can be useful for those who have jobs that are restartable or checkpointed, or for short jobs or requests, i.e. where there is low risk or mitigated loss to interruption. It is not recommended to use this partition for long jobs that are not checkpointed. New Software installs Here is a list of recent software installs: GCC 12.1, 11.3, 9.5 Mathematica 13.0 Matlab 2021b-r3 with Parallel Server. If anybody is interested in using Matlab Parallel Server, please contact me. Anaconda 2022.05, will become the new default version Cuda 11.7 have been installed, and Nvidia drivers have recently been upgraded on most GPU nodes to support the latest Cuda version. Drivers for Cuda 11.5+ have not yet been released for the DGX systems. The software listed above can be accessed through the modules system, type “module avail” to see what is available. For the latest cluster news and status updates, check out the link below: https://it.engineering.oregonstate.edu/hpc/hpc-cluster-status-and-news Cheers, Rob Yelle HPC Manager

1 0

CoE HPC News, June 15 2022 Edition: Summer maintenance
by Yelle, Robert Brian 15 Jun '22

15 Jun '22

HPCC users, The CoE HPC cluster is back online, but at limited availability while maintenance continues on the cluster. Additional resources will become available in time as maintenance progresses. Some partitions are still down, but all partitions should be back up by tomorrow afternoon. Cheers, Rob Yelle HPC Manager

1 0

CoE HPC News, June 6 2022 Edition: Summer maintenance
by Yelle, Robert Brian 06 Jun '22

06 Jun '22

HPCC users, Here is the latest news on the CoE HPC cluster: Summer break maintenance week: The HPC Cluster will undergo its regularly scheduled quarterly maintenance next week (June 13-17) The following maintenance activities will be performed: Operating system updates BIOS and firmware updates as needed Nvidia driver and CUDA updates Slurm configuration changes HPC Portal updates and configuration changes Miscellaneous hardware maintenance as needed The entire cluster will be offline starting Tuesday the 14th at 8am, and will remain offline until approximately Wednesday the 15th at 3pm. Jobs scheduled to run into that offline period will remain pending with the message “ReqNodeNotAvail, Reserved for maintenance”. If you wish for your Slurm job to start and finish before the maintenance period begins, you will need to adjust your time limit accordingly, e.g. to change to 2 days do: scontrol update job {jobid} TimeLimit=2-00:00:00 Alternatively, you can cancel your pending job, adjust your walltime (using the --time option) and resubmit. I will send out another announcement when the cluster is back online. After the offline period, maintenance will continue on various parts of the cluster throughout the week. For cluster news and status updates, check out the link below: https://it.engineering.oregonstate.edu/hpc/hpc-cluster-status-and-news Cheers, Rob Yelle HPC Manager

1 0

CoE HPC Cluster down
by Yelle, Robert Brian 16 May '22

16 May '22

HPCC users, The HPC cluster has been brought down due to a failure in the KEC cooling system. Facilities is working on the problem. The cluster will not be brought back up until the problem has been resolved and cooling restored to the datacenter. It is not yet known when that will happen, but I’ve been informed that tomorrow is the soonest that the cluster can come back up. I will keep you all updated and let you know when the cluster is back online. You may also check the cluster status in the web site below: https://it.engineering.oregonstate.edu/hpc/hpc-cluster-status-and-news Rob Yelle HPC Manager

1 1

CoE HPC News, March 14 2022 Edition: Spring maintenance
by Yelle, Robert Brian 23 Mar '22

23 Mar '22

HPCC users, Here is the latest news on the CoE HPC cluster: Spring break maintenance week: The HPC Cluster will undergo its regularly scheduled quarterly maintenance next week (March 21-25) The following maintenance activities will be performed: Operating system updates BIOS and firmware updates as needed Nvidia driver and CUDA updates Slurm configuration changes Open OnDemand configuration changes Power redistribution for various compute nodes Miscellaneous hardware maintenance as needed The entire cluster will be offline starting Tuesday the 22nd at 8am, and will remain offline until approximately Wednesday the 8th at 3pm. Jobs scheduled to run into that offline period will remain pending with the message “ReqNodeNotAvail, Reserved for maintenance”. If you wish for your Slurm job to start and finish before the maintenance period begins, you will need to adjust your time limit accordingly, e.g. to change to 2 days do: scontrol update job {jobid} TimeLimit=2-00:00:00 Alternatively, you can cancel your pending job, adjust your walltime (using the —-time option) and resubmit. I will send out another announcement when the cluster is back online. After the offline period, maintenance will continue on various parts of the cluster throughout the week. For cluster news and status updates, check out the link below: https://it.engineering.oregonstate.edu/hpc/hpc-cluster-status-and-news Cheers, Rob Yelle HPC Manager

1 2

CoE HPC News, January 24, 2022 edition: Slurm maintenance and tools
by Yelle, Robert Brian 24 Jan '22

24 Jan '22

HPCC users, Here is the latest news on the CoE HPC cluster: 1. Queuing system maintenance this week. The partitions or queues will be brought offline every morning this week between 8 and 9am, starting tomorrow (Tuesday the 25th). This means that new or pending jobs will not start during that one hour window, so that troubleshooting and maintenance can be performed on the Slurm queuing environment. Jobs currently running on the cluster during this time will not be affected. The submit nodes will still be accessible, and data still be available. The queues will resume by 9am. Please plan accordingly. I apologize for any inconvenience this may cause. If you have any questions or concerns about this, let me know. 2. New commands/scripts for Slurm. At the request of our users, some additional, potentially useful commands have been added to the Slurm module to assist our users, e.g.: “nodestat {partition}” will show the status and resources used on each node on that partition, e.g., how many CPUs, GPUs and RAM are on each node, and how much are currently in use. “showjob {jobid}” will provide information on a currently running or pending job. This can be used to obtain the estimated start time for a pending job, if available. “tracejob -S YYYY-MM-DD” will provide information on jobs started since the date YYYY-MM-DD. “sql” gives an alternate, longer listing format of job listing than the default “squeue” command. “squ” only lists jobs owned by user, using the default “squeue” format Try them out, and let me know how they might be improved or if there are other details that you would like to see and I will look into it. Cheers, Rob Yelle HPC Manager

1 0