After running “—process”, we can run “—info” and it will tell us what happened.
2022-06-27 23:33:37.488 | INFO : Listing table info
2022-06-27 23:33:37.668 | INFO : Active job table record count: 805
2022-06-27 23:33:37.668 | INFO : Active job table 'failure' count: 90
2022-06-27 23:33:37.670 | INFO : Active job table 'success' count: 150
2022-06-27 23:33:37.671 | INFO : Active job table 'success' count for today's jobs (2022-06-27): 115
2022-06-27 23:33:37.672 | INFO : Job history table record count: 63397
2022-06-27 23:33:37.672 | INFO : Scoring job record table record count: 287
So it looks like after the network and data API came back, we were able to score 115 models which isn’t too bad. We also picked up 35 jobs from preceding days.
And we can see that no new jobs were added, e.g., the row counts have not changed from what we saw earlier:
Job history table record count: 63397
Scoring job record table record count: 287
From: Rainqc-jobman <rainqc-jobman-bounces@engr.orst.edu>
On Behalf Of Slater, Michael
Sent: Monday, June 27, 2022 10:50 PM
To: RainQC Job Manager <rainqc-jobman@engr.orst.edu>
Subject: Re: [Rainqc-jobman] RainQC Job Manager daily report 2022-06-28
Actually, a network connection went down for some reason and the job manager aborted. This is a good example of why the job manager command “—process” exists.
This doesn’t add the day’s jobs to the job table again, it just processes what’s already in the job table. However, it doesn’t produce its own email report. We might add a report as an (optional) feature. --ms
From: Rainqc-jobman <rainqc-jobman-bounces@engr.orst.edu>
On Behalf Of Slater, Michael
Sent: Monday, June 27, 2022 10:19 PM
To: RainQC Job Manager <rainqc-jobman@engr.orst.edu>
Subject: Re: [Rainqc-jobman] RainQC Job Manager daily report 2022-06-28
The TAHMO data API looks like it’s in a failure mode of some sort. Lots of calls failed. —ms
From: Rainqc-jobman <rainqc-jobman-bounces@engr.orst.edu>
on behalf of Dietterich, Thomas <tgd@oregonstate.edu>
Sent: Monday, June 27, 2022 10:17:13 PM
To: RainQC Job Manager <rainqc-jobman@engr.orst.edu>
Subject: Re: [Rainqc-jobman] RainQC Job Manager daily report 2022-06-28
Wow, a bad day for JobMan. 0 successes! Only 93 stations had 100% data…
--Tom
Thomas G. Dietterich, Distinguished Professor Voice: 541-737-5559
School of Electrical Engineering FAX: 541-737-1300
and Computer Science URL: eecs.oregonstate.edu/~tgd
US Mail: 1148 Kelley Engineering Center
Office: 2067 Kelley Engineering Center
Oregon State Univ., Corvallis, OR 97331-5501
From: Rainqc-jobman <rainqc-jobman-bounces@engr.orst.edu>
On Behalf Of slater@oregonstate.edu
Sent: Monday, June 27, 2022 21:48
To: rainqc-jobman@engr.orst.edu
Subject: [Rainqc-jobman] RainQC Job Manager daily report 2022-06-28
Current UTC date: 2022-06-28
-----------------------------------------------------------
Daily Model Data Completeness Check:
data completeness 50% | complete models: 131 of 222 (59.01%)
data completeness 60% | complete models: 127 of 222 (57.21%)
data completeness 70% | complete models: 124 of 222 (55.86%)
data completeness 75% | complete models: 124 of 222 (55.86%)
data completeness 80% | complete models: 124 of 222 (55.86%)
data completeness 85% | complete models: 122 of 222 (54.95%)
data completeness 90% | complete models: 122 of 222 (54.95%)
data completeness 95% | complete models: 96 of 222 (43.24%)
data completeness 100% | complete models: 93 of 222 (41.89%)
-----------------------------------------------------------
Processed daily jobs for UTC date: 2022-06-27
Start time: 2022-06-28T04:44:37+00:00
End time : 2022-06-28T04:48:14+00:00
Elapsed time HH:MM:SS: 0:03:37
---------------------
Before job processing table stats:
Active job table record count: 788
Active job table 'failure' count: 90
Active job table 'success' count: 115
Active job table 'success' count for today's jobs (2022-06-27): 0
Job history table record count: 63192
Scoring job record table record count: 286
---------------------
After job processing table stats:
Active job table record count: 805
Active job table 'failure' count: 17
Active job table 'success' count: 0
Active job table 'success' count for today's jobs (2022-06-27): 0
Job history table record count: 63397
Scoring job record table record count: 287
-----------------------------------------------------------