I won’t make it to this week's meeting, 9/28. Jobs continue to run fine against the new cloud PostgreSQL.

From last week's discussion on the duplicate models that we occasionally saw in the daily job reports, I still need to make an update to the job manager to filter out old versions of models from the RainQC model list.

Also from last week, I added a log warning if the reporting flag is set to false in the config file. I also changed the way we handle the SSL mode for PostgreSQL. The new way adds a field to the database section(s) in the configuration file. This cleans up the code changes a bit. I'm still aiming for a dry run of the updated job manager with the Kenyan team this Friday, if the day still works (let me know please). Michael ________________________________ From: slater@oregonstate.edu <slater@oregonstate.edu> Sent: Wednesday, September 28, 2022 12:36:15 AM To: rainqc-jobman@ENGR.ORST.EDU <rainqc-jobman@ENGR.ORST.EDU> Cc: Slater, Michael <slater@oregonstate.edu> Subject: RainQC Job Manager daily report 2022-09-28 Current UTC date: 2022-09-28 -> scoring models for previous day --------------------------------------------------------------------------------- Daily Model Data Completeness Check: data completeness 50% | complete models: 160 of 251 (63.75%) data completeness 60% | complete models: 156 of 251 (62.15%) data completeness 70% | complete models: 156 of 251 (62.15%) data completeness 75% | complete models: 156 of 251 (62.15%) data completeness 80% | complete models: 156 of 251 (62.15%) data completeness 85% | complete models: 156 of 251 (62.15%) data completeness 90% | complete models: 153 of 251 (60.96%) data completeness 95% | complete models: 130 of 251 (51.79%) data completeness 100% | complete models: 120 of 251 (47.81%) ---------------------------- station status | total: 286, delayed: 107, offline 24h: 32, offline week: 28 | battery, min: 0, max: 100, mean: 62.73, std dev: 28.22 | battery, common values: [(100, 209), (0, 34), (90, 4), (99, 3), (97, 3)] | battery <= mean, common countries: [('GH', 18), ('ML', 8), ('KE', 6), ('TG', 5), ('TZ', 3)] ---------------------------- 55 LOW DATA (< 0.9) and 43 NO DATA weather stations impacted 94 RainQC models LOW/NO data station impact on models: [('TA00016', 7), ('TA00157', 6), ('TA00078', 6), ('TA00677', 5), ('TA00126', 5), ('TA00652', 4), ('TA00335', 4), ('TA00691', 4), ('TA00680', 4), ('TA00356', 4), ('TA00124', 3), ('TA00044', 3), ('TA00041', 3), ('TA00043', 3), ('TA00050', 3), ('TA00422', 3), ('TA00180', 3), ('TA00533', 3), ('TA00655', 3), ('TA00276', 2), ('TA00308', 2), ('TA00257', 2), ('TA00487', 2), ('TA00203', 2), ('TA00262', 2), ('TA00265', 2), ('TA00393', 2), ('TA00271', 2), ('TA00307', 2), ('TA00339', 2), ('TA00471', 2), ('TA00466', 2), ('TA00664', 2), ('TA00014', 1), ('TA00031', 1), ('TA00039', 1), ('TA00092', 1), ('TA00128', 1), ('TA00254', 1), ('TA00201', 1), ('TA00219', 1), ('TA00236', 1), ('TA00286', 1), ('TA00279', 1), ('TA00392', 1), ('TA00277', 1), ('TA00290', 1), ('TA00334', 1), ('TA00336', 1), ('TA00350', 1), ('TA00362', 1), ('TA00333', 1), ('TA00382', 1), ('TA00434', 1), ('TA00493', 1), ('TA00702', 1)] ----------------------------------------------------------- Processed daily jobs for UTC date: 2022-09-27 Start time: 2022-09-28T05:50:47+00:00 End time : 2022-09-28T07:36:14+00:00 Elapsed time HH:MM:SS: 1:45:27 --------------------- Before job processing job table stats: Total 'success' count: 154 Total 'failure' count: 117 Total record count: 1692 Job history table record count: 3077 Scoring job record table record count: 19 --------------------- After job processing job table stats: Total 'success' count: 127 (flag=2 count: 4) (flag=1 abstentions: 2) | 'success' count for 2022-09-27: 111 (flag=2 count: 4) | 'success' count for 2022-09-26: 16 (flag=2 count: 0) | 'success' count for 2022-09-25: 0 (flag=2 count: 0) | 'success' count for 2022-09-24: 0 (flag=2 count: 0) | 'success' count for 2022-09-23: 0 (flag=2 count: 0) | 'success' count for 2022-09-22: 0 (flag=2 count: 0) | 'success' count for 2022-09-21: 0 (flag=2 count: 0) Anomalies (flag=2): TA00006 2022-09-27 | score: 1058.068 (thresh: 72.125) -- 'pr' mm t: 0.034 [58.759] TA00121 2022-09-27 | score: 167.299 (thresh: 140.856) -- 'pr' mm t: 0.408 [26.777] TA00397 2022-09-27 | score: 435.240 (thresh: 90.000) -- 'pr' mm t: 1.275 [27.408] TA00567 2022-09-27 | score: 115.986 (thresh: 90.000) -- 'pr' mm t: 0.000 [0.493, 0.646, 4.199, 6.928] -------- Total 'failure' count: 822 Total record count: 1672 Job history table record count: 3348 Scoring job record table record count: 20 -----------------------------------------------------------