The low number of successful calls is not all that uncommon for a Friday night. I don’t have the statistics at hand, but as I recall things, Fridays are (one of?) our worst days for data completeness. Let’s hope for a strong Saturday performance with a lot of previous jobs successes. Thankfully, this is also not all that uncommon.

The job manager hand-off process will continue MWF 8AM next week. Monday’s goal is to get the team running the job manager locally, e.g., use foreground mode and the various commands to get jobs into tables, process the jobs, examine the job movement across the active and history tables, etc., without worrying about updating the system of record (via the ‘reportquality’ endpoint). This requires me to push some small changes to the GitLab (my machine is ahead of the repo in a few small areas) so that the job manager can be run with the flag to NOT update TAHMO.

Michael

From: slater@oregonstate.edu <slater@oregonstate.edu>
Sent: Friday, July 8, 2022 9:49 PM
To: rainqc-jobman@ENGR.ORST.EDU
Cc: Slater, Michael <slater@oregonstate.edu>
Subject: RainQC Job Manager daily report 2022-07-09

Current UTC date: 2022-07-09
-----------------------------------------------------------
Daily Model Data Completeness Check:
data completeness    50% | complete models: 132 of 222 (59.46%)
data completeness    60% | complete models: 121 of 222 (54.50%)
data completeness    70% | complete models: 116 of 222 (52.25%)
data completeness    75% | complete models: 116 of 222 (52.25%)
data completeness    80% | complete models: 115 of 222 (51.80%)
data completeness    85% | complete models: 110 of 222 (49.55%)
data completeness    90% | complete models: 110 of 222 (49.55%)
data completeness    95% | complete models:  76 of 222 (34.23%)
data completeness   100% | complete models:  67 of 222 (30.18%)
-----------------------------------------------------------
Processed daily jobs for UTC date: 2022-07-08
Start time: 2022-07-09T04:19:49+00:00
End time  : 2022-07-09T04:49:18+00:00
Elapsed time HH:MM:SS: 0:29:29
---------------------
Before job processing table stats:
Active job table record count: 788
Active job table 'failure' count: 88
Active job table 'success' count: 117
Active job table 'success' count for today's jobs (2022-07-08): 0
Job history table record count: 65634
Scoring job record table record count: 297
---------------------
After job processing table stats:
Active job table record count: 805
Active job table 'failure' count: 88
Active job table 'success' count: 102
Active job table 'success' count for today's jobs (2022-07-08): 77
Job history table record count: 65839
Scoring job record table record count: 298
-----------------------------------------------------------