The TAHMO data API looks like it’s in a failure mode of some sort. Lots of calls failed. —ms ________________________________ From: Rainqc-jobman <rainqc-jobman-bounces@engr.orst.edu> on behalf of Dietterich, Thomas <tgd@oregonstate.edu> Sent: Monday, June 27, 2022 10:17:13 PM To: RainQC Job Manager <rainqc-jobman@engr.orst.edu> Subject: Re: [Rainqc-jobman] RainQC Job Manager daily report 2022-06-28 Wow, a bad day for JobMan. 0 successes! Only 93 stations had 100% data… --Tom Thomas G. Dietterich, Distinguished Professor Voice: 541-737-5559 School of Electrical Engineering FAX: 541-737-1300 and Computer Science URL: eecs.oregonstate.edu/~tgd US Mail: 1148 Kelley Engineering Center Office: 2067 Kelley Engineering Center Oregon State Univ., Corvallis, OR 97331-5501 From: Rainqc-jobman <rainqc-jobman-bounces@engr.orst.edu> On Behalf Of slater@oregonstate.edu Sent: Monday, June 27, 2022 21:48 To: rainqc-jobman@engr.orst.edu Subject: [Rainqc-jobman] RainQC Job Manager daily report 2022-06-28 Current UTC date: 2022-06-28 ----------------------------------------------------------- Daily Model Data Completeness Check: data completeness 50% | complete models: 131 of 222 (59.01%) data completeness 60% | complete models: 127 of 222 (57.21%) data completeness 70% | complete models: 124 of 222 (55.86%) data completeness 75% | complete models: 124 of 222 (55.86%) data completeness 80% | complete models: 124 of 222 (55.86%) data completeness 85% | complete models: 122 of 222 (54.95%) data completeness 90% | complete models: 122 of 222 (54.95%) data completeness 95% | complete models: 96 of 222 (43.24%) data completeness 100% | complete models: 93 of 222 (41.89%) ----------------------------------------------------------- Processed daily jobs for UTC date: 2022-06-27 Start time: 2022-06-28T04:44:37+00:00 End time : 2022-06-28T04:48:14+00:00 Elapsed time HH:MM:SS: 0:03:37 --------------------- Before job processing table stats: Active job table record count: 788 Active job table 'failure' count: 90 Active job table 'success' count: 115 Active job table 'success' count for today's jobs (2022-06-27): 0 Job history table record count: 63192 Scoring job record table record count: 286 --------------------- After job processing table stats: Active job table record count: 805 Active job table 'failure' count: 17 Active job table 'success' count: 0 Active job table 'success' count for today's jobs (2022-06-27): 0 Job history table record count: 63397 Scoring job record table record count: 287 -----------------------------------------------------------

After running "-process", we can run "-info" and it will tell us what happened. 2022-06-27 23:33:37.488 | INFO : Listing table info 2022-06-27 23:33:37.668 | INFO : Active job table record count: 805 2022-06-27 23:33:37.668 | INFO : Active job table 'failure' count: 90 2022-06-27 23:33:37.670 | INFO : Active job table 'success' count: 150 2022-06-27 23:33:37.671 | INFO : Active job table 'success' count for today's jobs (2022-06-27): 115 2022-06-27 23:33:37.672 | INFO : Job history table record count: 63397 2022-06-27 23:33:37.672 | INFO : Scoring job record table record count: 287 So it looks like after the network and data API came back, we were able to score 115 models which isn't too bad. We also picked up 35 jobs from preceding days. And we can see that no new jobs were added, e.g., the row counts have not changed from what we saw earlier: Job history table record count: 63397 Scoring job record table record count: 287 From: Rainqc-jobman <rainqc-jobman-bounces@engr.orst.edu> On Behalf Of Slater, Michael Sent: Monday, June 27, 2022 10:50 PM To: RainQC Job Manager <rainqc-jobman@engr.orst.edu> Subject: Re: [Rainqc-jobman] RainQC Job Manager daily report 2022-06-28 Actually, a network connection went down for some reason and the job manager aborted. This is a good example of why the job manager command "-process" exists. This doesn't add the day's jobs to the job table again, it just processes what's already in the job table. However, it doesn't produce its own email report. We might add a report as an (optional) feature. --ms From: Rainqc-jobman <rainqc-jobman-bounces@engr.orst.edu<mailto:rainqc-jobman-bounces@engr.orst.edu>> On Behalf Of Slater, Michael Sent: Monday, June 27, 2022 10:19 PM To: RainQC Job Manager <rainqc-jobman@engr.orst.edu<mailto:rainqc-jobman@engr.orst.edu>> Subject: Re: [Rainqc-jobman] RainQC Job Manager daily report 2022-06-28 The TAHMO data API looks like it's in a failure mode of some sort. Lots of calls failed. -ms ________________________________ From: Rainqc-jobman <rainqc-jobman-bounces@engr.orst.edu<mailto:rainqc-jobman-bounces@engr.orst.edu>> on behalf of Dietterich, Thomas <tgd@oregonstate.edu<mailto:tgd@oregonstate.edu>> Sent: Monday, June 27, 2022 10:17:13 PM To: RainQC Job Manager <rainqc-jobman@engr.orst.edu<mailto:rainqc-jobman@engr.orst.edu>> Subject: Re: [Rainqc-jobman] RainQC Job Manager daily report 2022-06-28 Wow, a bad day for JobMan. 0 successes! Only 93 stations had 100% data... --Tom Thomas G. Dietterich, Distinguished Professor Voice: 541-737-5559 School of Electrical Engineering FAX: 541-737-1300 and Computer Science URL: eecs.oregonstate.edu/~tgd US Mail: 1148 Kelley Engineering Center Office: 2067 Kelley Engineering Center Oregon State Univ., Corvallis, OR 97331-5501 From: Rainqc-jobman <rainqc-jobman-bounces@engr.orst.edu<mailto:rainqc-jobman-bounces@engr.orst.edu>> On Behalf Of slater@oregonstate.edu<mailto:slater@oregonstate.edu> Sent: Monday, June 27, 2022 21:48 To: rainqc-jobman@engr.orst.edu<mailto:rainqc-jobman@engr.orst.edu> Subject: [Rainqc-jobman] RainQC Job Manager daily report 2022-06-28 Current UTC date: 2022-06-28 ----------------------------------------------------------- Daily Model Data Completeness Check: data completeness 50% | complete models: 131 of 222 (59.01%) data completeness 60% | complete models: 127 of 222 (57.21%) data completeness 70% | complete models: 124 of 222 (55.86%) data completeness 75% | complete models: 124 of 222 (55.86%) data completeness 80% | complete models: 124 of 222 (55.86%) data completeness 85% | complete models: 122 of 222 (54.95%) data completeness 90% | complete models: 122 of 222 (54.95%) data completeness 95% | complete models: 96 of 222 (43.24%) data completeness 100% | complete models: 93 of 222 (41.89%) ----------------------------------------------------------- Processed daily jobs for UTC date: 2022-06-27 Start time: 2022-06-28T04:44:37+00:00 End time : 2022-06-28T04:48:14+00:00 Elapsed time HH:MM:SS: 0:03:37 --------------------- Before job processing table stats: Active job table record count: 788 Active job table 'failure' count: 90 Active job table 'success' count: 115 Active job table 'success' count for today's jobs (2022-06-27): 0 Job history table record count: 63192 Scoring job record table record count: 286 --------------------- After job processing table stats: Active job table record count: 805 Active job table 'failure' count: 17 Active job table 'success' count: 0 Active job table 'success' count for today's jobs (2022-06-27): 0 Job history table record count: 63397 Scoring job record table record count: 287 -----------------------------------------------------------