- cluster-users - Engineering Mailing Lists

CoE HPC News, November 23 2021 Edition: upcoming Winter maintenance
by Yelle, Robert Brian 09 Dec '21

09 Dec '21

HPCC users, Here is the latest news on the CoE HPC cluster: Winter maintenance: The HPC Cluster will undergo its regularly scheduled quarterly maintenance after Finals week (December 13-17) The following maintenance activities will be performed: Operating system updates Slurm update and configuration changes Nvidia driver updates (including DGX systems!) Infiniband driver updates BIOS and firmware updates as needed Miscellaneous hardware maintenance as needed The entire cluster will be offline starting Tuesday the 14th at 8am, and will remain offline until approximately Wednesday the 15th at 1pm. Jobs scheduled to run into that offline period will remain pending with the message “ReqNodeNotAvail, Reserved for maintenance”. If you wish for your Slurm job to start and finish before the maintenance period begins, you will need to adjust your time limit accordingly, e.g. to change to 2 days do: scontrol update job {jobid} TimeLimit=2-00:00:00 Alternatively, you can cancel your pending job, adjust your walltime (using the —-time option) and resubmit. I will send out a reminder of the upcoming maintenance during Finals week. Operating System upgrade on DGX servers delayed: Nvidia has recently released drivers for Cuda 11.4 on the DGX systems running Centos 7. These drivers will be installed during the maintenance scheduled for after Finals. Due to Nvidia’s continued support for EL7, the previously planned upgrade to an EL8 compatible operating system on the DGX systems has been postponed until further notice. If you have any questions or concerns, let me know. Rob Yelle HPC Manager

1 1

Re: [cluster-users] CoE HPC News, August 30 2021 Edition: Fall maintenance, OS upgrade
by Yelle, Robert Brian 08 Sep '21

08 Sep '21

HPCC Users, The HPC cluster is back online with limited availability. Parts of the cluster are still offline for maintenance, and additional nodes will be brought back online during the course of the week after their maintenance has completed. Stay tuned for more announcements in the upcoming weeks. Rob Yelle HPC Manager From: Yelle, Robert Brian <robert.yelle(a)oregonstate.edu> Date: Monday, August 30, 2021 at 3:56 PM To: cluster-users(a)engr.orst.edu<mailto:cluster-users@engr.orst.edu> <cluster-users(a)engr.orst.edu<mailto:cluster-users@engr.orst.edu>> Subject: CoE HPC News, August 30 2021 Edition: Fall maintenance, OS upgrade HPCC users, Here is the latest news on the CoE HPC cluster: Fall maintenance: The HPC Cluster will undergo its regularly scheduled quarterly maintenance next week (September 7-10) The following maintenance activities will be performed: OS updates BIOS and firmware updates as needed Slurm update and configuration changes Power redistribution for various compute nodes Miscellaneous hardware maintenance as needed The entire cluster will be offline starting Tuesday the 7th at 8am, and will remain offline until approximately Wednesday the 8th at 1pm. Jobs scheduled to run into that offline period will remain pending with the message “ReqNodeNotAvail, Reserved for maintenance”. If you wish for your Slurm job to start and finish before the maintenance period begins, you will need to adjust your time limit accordingly, e.g. to change to 2 days do: scontrol update job {jobid} TimeLimit=2-00:00:00 Alternatively, you can cancel your pending job, adjust your walltime (using the —-time option) and resubmit. I will send out another announcement when the cluster is back online. After the offline period, maintenance will continue on various parts of the cluster throughout the week. Operating System upgrade: Nvidia says they will probably not support Cuda 11.2 or higher on DGX nodes running the RHEL 7 or Centos 7 operating systems. To ensure that our users will be able to use Cuda 11.2 or higher on these systems, we plan on upgrading all the DGX nodes to Centos 8 or some other RHEL 8 based system. This is a major change and will take some time to test and deploy the new image(s), and test applications on these images. We will probably start with the DGXS nodes and then roll out the new OS image to the DGX-2 nodes, and eventually to the entire cluster as things stabilize on the new OS. Stay tuned for more details as this develops. If you have any questions or concerns, let me know. Rob Yelle HPC Manager

1 0

CoE HPC News, August 30 2021 Edition: Fall maintenance, OS upgrade
by Yelle, Robert Brian 30 Aug '21

30 Aug '21

HPCC users, Here is the latest news on the CoE HPC cluster: Fall maintenance: The HPC Cluster will undergo its regularly scheduled quarterly maintenance next week (September 7-10) The following maintenance activities will be performed: OS updates BIOS and firmware updates as needed Slurm update and configuration changes Power redistribution for various compute nodes Miscellaneous hardware maintenance as needed The entire cluster will be offline starting Tuesday the 7th at 8am, and will remain offline until approximately Wednesday the 8th at 1pm. Jobs scheduled to run into that offline period will remain pending with the message “ReqNodeNotAvail, Reserved for maintenance”. If you wish for your Slurm job to start and finish before the maintenance period begins, you will need to adjust your time limit accordingly, e.g. to change to 2 days do: scontrol update job {jobid} TimeLimit=2-00:00:00 Alternatively, you can cancel your pending job, adjust your walltime (using the —-time option) and resubmit. I will send out another announcement when the cluster is back online. After the offline period, maintenance will continue on various parts of the cluster throughout the week. Operating System upgrade: Nvidia says they will probably not support Cuda 11.2 or higher on DGX nodes running the RHEL 7 or Centos 7 operating systems. To ensure that our users will be able to use Cuda 11.2 or higher on these systems, we plan on upgrading all the DGX nodes to Centos 8 or some other RHEL 8 based system. This is a major change and will take some time to test and deploy the new image(s), and test applications on these images. We will probably start with the DGXS nodes and then roll out the new OS image to the DGX-2 nodes, and eventually to the entire cluster as things stabilize on the new OS. Stay tuned for more details as this develops. If you have any questions or concerns, let me know. Rob Yelle HPC Manager

1 0

Call for Intro to HPC workshop
by Yelle, Robert Brian 02 Jul '21

02 Jul '21

HPCC users, I am offering a brief introductory workshop to the HPC cluster over Zoom sometime next week. If you are interested in attending this, let me know your availability and please provide any details on what you would like to learn from this session. Thanks, Rob Yelle HPC Manager

1 0

CoE HPC News, June 2 edition: Summerbreak maintenance
by Yelle, Robert Brian 16 Jun '21

16 Jun '21

HPCC users, The HPC Cluster will undergo its regularly scheduled quarterly maintenance after Finals Week, during Summerbreak (June 14-18). The following maintenance activities will be performed: OS updates BIOS and firmware updates Infiniband updates Slurm scheduler configuration changes Power redistribution for various compute nodes Miscellaneous hardware maintenance as needed Maintenance activities will begin the Monday after Finals week, and the entire cluster will be offline starting Tuesday the 15th at 8am, and will remain offline until Wednesday the 16th at 2pm. After that, maintenance will continue on parts of the cluster throughout the week. Jobs found running on the Univa scheduler by Tuesday morning will need to be terminated, and Slurm jobs scheduled to run into that offline period will remain pending with the message “ReqNodeNotAvail, Reserved for maintenance”. If you wish for your Slurm job to start and finish before the maintenance period begins, you will need to adjust your time limit accordingly, e.g. to change to 2 days do: scontrol update job {jobid} TimeLimit=2-00:00:00 Alternatively, you can cancel your pending job, adjust your walltime using the --time option, and resubmit. If you have any questions or concerns, let me know. Rob Yelle HPC Manager

1 2

CoE HPC News, May 12 Edition: Urgent updates
by Yelle, Robert Brian 13 May '21

13 May '21

HPCC users, Three of the DGX2 systems (dgx2-1, dgx2-2 and dgx2-5) are currently offline for urgent firmware updates. They should be available again later this week. Also, the Slurm scheduler will be offline briefly tomorrow morning (May 13) between 7 and 9am for an urgent update. The cluster will remain online and submit nodes will still be accessible, but Slurm commands issued may fail and no new jobs will be started during this time. Currently running jobs should not be impacted by this update. Rob Yelle HPC Manager

1 1

CoE HPC News, March 15 edition: Springbreak maintenance
by Yelle, Robert Brian 26 Mar '21

26 Mar '21

HPCC users, Here is the latest news on the CoE HPC cluster: Springbreak maintenance: The HPC Cluster will undergo its regularly scheduled quarterly maintenance after Finals Week, during Springbreak (March 22-26). The following maintenance activities will be performed: OS updates BIOS and firmware updates Infiniband updates Slurm update and configuration changes Power redistribution for various compute nodes Miscellaneous hardware maintenance as needed Maintenance activities will begin the Monday after Finals week, and the entire cluster will be offline starting Tuesday the 23th at 8am, and will remain offline until Wednesday the 24th at 4pm. After that, maintenance will continue on parts of the cluster throughout the week. Jobs found running on the Univa cluster by Tuesday morning will need to be terminated, and Slurm jobs scheduled to run into that offline period will remain pending with the message “ReqNodeNotAvail, Reserved for maintenance”. If you wish for your Slurm job to start and finish) before the maintenance period begins, you will need to adjust your time limit accordingly, e.g. to change to 2 days do: scontrol update job {jobid} TimeLimit=2-00:00:00 Alternatively, you can cancel your pending job, adjust your walltime (using the —-time option) and resubmit. If you have any questions or concerns, let me know. Rob Yelle HPC Manager

1 2