Thanks Rick,
Have any of our endpoints changed?
--Tom
Thomas G. Dietterich, Distinguished Professor Voice: 541-737-5559
School of Electrical Engineering FAX: 541-737-1300
and Computer Science URL: eecs.oregonstate.edu/~tgd
US Mail: 1148 Kelley Engineering Center
Office: 2067 Kelley Engineering Center
Oregon State Univ., Corvallis, OR 97331-5501
From: Rainqc-jobman <rainqc-jobman-bounces@engr.oregonstate.edu>
On Behalf Of Rick Hagenaars - CITG
Sent: Wednesday, March 15, 2023 11:15
To: RainQC Job Manager <rainqc-jobman@engr.oregonstate.edu>
Subject: Re: [Rainqc-jobman] RainQC Job Manager daily report 2023-03-11
[This email originated from outside of OSU. Use caution with links and attachments.]
[This email originated from outside of OSU. Use caution with links and attachments.]
Hi all,
After spending 50 hours trying to migrate the RabbitMQ into the new "Messages for RabbitMQ" service on IBM Cloud we decided to rather use an external managed RabbitMQ cluster and everything worked within 1 hour. Systems should now
be operational again and the regular QA/QC is now catching up with the data from the last couple of days.
I expect from tomorrow on everything should be running normal again (at least until we need to start migrating CF applications end of May).
Kind regards,
Rick Hagenaars
TAHMO Project, Faculty CiTG, TU Delft, Stevinweg 1, 2628CN, Delft The Netherlands
T (M) +31(0)645833496
E-mail:
h.f.hagenaars@tudelft.nl or rhagenaars@tahmo.org
From: Slater, Michael <slater@oregonstate.edu>
Sent: Tuesday, March 14, 2023 3:50:04 AM
To: Rick Hagenaars - CITG
Subject: RE: [Rainqc-jobman] RainQC Job Manager daily report 2023-03-11
Hi Rick,
Thanks for letting me know about the outages. At worst, we’ll just re-add some scoring jobs and run them again. The last few days have been rather puzzling to
me, as my own data completeness estimates showed that station data was in the system, but RainQC seemed to be having troubles reading any data after a certain date. That might be the difference between reading raw data vs. controlled data, though.
Best wishes for the migrations and other updates!
Cheers,
Michael
From: Rainqc-jobman <rainqc-jobman-bounces@engr.oregonstate.edu>
On Behalf Of Rick Hagenaars - CITG
Sent: Sunday, March 12, 2023 1:03 AM
To: RainQC Job Manager <rainqc-jobman@engr.oregonstate.edu>
Subject: Re: [Rainqc-jobman] RainQC Job Manager daily report 2023-03-11
[This email originated from outside of OSU. Use caution with links and attachments.]
[This email originated from outside of OSU. Use caution with links and attachments.]
Hi Michael,
There’s indeed been some substantial outages, last week due to METER and this week because we needed to do migrations for the database. The compose services will no longer be operational after next Wednesday on IBM cloud. For sensordx we
already changed the Postgres last November so it shouldn’t get affected.
There might be some more downtime on Monday since the rabbitmq migration wasn’t successful last week.
Kind regards,
Rick Hagenaars
On 11 Mar 2023, at 10:28, Slater, Michael <slater@oregonstate.edu> wrote: