Strange day for RainQC, with 40%+ model completeness, but zero successful model scoring calls to RainQC. As far as I can tell, all the TAHMO data fetches performed by the RainQC server failed. A sample 'message' that was passed back from RainQC server to the job manager: {'message': 'the GET from http://tahmogohub:8080 failed. URL: http://tahmogohub:8080/v2/stationaccessrights?filter=code%21eq%21TA00116%26%..., err: Get http://tahmogohub:8080/v2/stationaccessrights?filter=code%21eq%21TA00116%26%...: net/http: request canceled (Client.Timeout exceeded while awaiting headers)'}" In a quick glance at the log, there were many messages similar to the above. We may have some sort of data access rights issue going on with the RainQC server. Michael From: slater@oregonstate.edu <slater@oregonstate.edu> Sent: Friday, November 17, 2023 12:13 AM To: rainqc-jobman@ENGR.ORST.EDU Cc: Slater, Michael <slater@oregonstate.edu> Subject: RainQC Job Manager daily report 2023-11-17 Current UTC date: 2023-11-17 -> scoring models for previous day: 2023-11-16 --------------------------------------------------------------------------------- Daily Model Data Completeness Check: data completeness 50% | complete models: 137 of 273 (50.18%) data completeness 60% | complete models: 134 of 273 (49.08%) data completeness 70% | complete models: 130 of 273 (47.62%) data completeness 75% | complete models: 130 of 273 (47.62%) data completeness 80% | complete models: 123 of 273 (45.05%) data completeness 85% | complete models: 116 of 273 (42.49%) data completeness 90% | complete models: 112 of 273 (41.03%) data completeness 95% | complete models: 111 of 273 (40.66%) data completeness 100% | complete models: 110 of 273 (40.29%) ---------------------------- station status | total: 313, delayed: 130, offline 24h: 77, offline week: 71 | battery, min: 0, max: 100, mean: 69.07, std dev: 27.99 | battery, common values: [(100, 194), (0, 78), (76, 4), (96, 3), (97, 3)] | battery <= mean, common countries: [('KE', 21), ('GH', 16), ('UG', 11), ('ML', 5), ('RW', 5)] ---------------------------- 92 LOW DATA (< 0.9) and 75 NO DATA weather stations impacted 161 RainQC models LOW/NO data station impact on models: [('TA00127', 8), ('TA00715', 8), ('TA00568', 8), ('TA00196', 7), ('TA00719', 7), ('TA00185', 7), ('TA00414', 6), ('TA00073', 6), ('TA00621', 5), ('TA00320', 5), ('TA00231', 4), ('TA00190', 4), ('TA00530', 4), ('TA00565', 4), ('TA00543', 4), ('TA00003', 3), ('TA00035', 3), ('TA00036', 3), ('TA00222', 3), ('TA00041', 3), ('TA00267', 3), ('TA00178', 3), ('TA00066', 3), ('TA00094', 3), ('TA00115', 3), ('TA00126', 3), ('TA00484', 3), ('TA00217', 3), ('TA00482', 3), ('TA00385', 3), ('TA00436', 3), ('TA00430', 3), ('TA00308', 2), ('TA00050', 2), ('TA00302', 2), ('TA00072', 2), ('TA00118', 2), ('TA00165', 2), ('TA00148', 2), ('TA00155', 2), ('TA00487', 2), ('TA00210', 2), ('TA00232', 2), ('TA00271', 2), ('TA00309', 2), ('TA00691', 2), ('TA00339', 2), ('TA00364', 2), ('TA00373', 2), ('TA00451', 2), ('TA00462', 2), ('TA00471', 2), ('TA00014', 1), ('TA00031', 1), ('TA00044', 1), ('TA00070', 1), ('TA00078', 1), ('TA00091', 1), ('TA00095', 1), ('TA00123', 1), ('TA00157', 1), ('TA00212', 1), ('TA00219', 1), ('TA00229', 1), ('TA00237', 1), ('TA00251', 1), ('TA00689', 1), ('TA00269', 1), ('TA00276', 1), ('TA00287', 1), ('TA00290', 1), ('TA00336', 1), ('TA00343', 1), ('TA00350', 1), ('TA00362', 1), ('TA00369', 1), ('TA00382', 1), ('TA00389', 1), ('TA00422', 1), ('TA00432', 1), ('TA00454', 1), ('TA00493', 1), ('TA00528', 1), ('TA00529', 1), ('TA00533', 1), ('TA00535', 1), ('TA00586', 1), ('TA00588', 1), ('TA00652', 1), ('TA00655', 1), ('TA00677', 1), ('TA00702', 1)] ----------------------------------------------------------- Processed daily jobs for UTC date: 2023-11-16 Start time: 2023-11-17T06:41:32+00:00 End time : 2023-11-17T08:13:15+00:00 Elapsed time HH:MM:SS: 1:31:44 --------------------- Before job processing job table stats: Total 'success' count: 116 Total 'failure' count: 162 Total record count: 1266 Job history table record count: 113478 Scoring job record table record count: 434 --------------------- After job processing job table stats: Total 'success' count: 0 (flag=2 count: 0) (flag 2->1 downgrades: 0) | 'success' count for 2023-11-16: 0 (flag=2 count: 0) | 'success' count for 2023-11-15: 0 (flag=2 count: 0) | 'success' count for 2023-11-14: 0 (flag=2 count: 0) | 'success' count for 2023-11-13: 0 (flag=2 count: 0) | 'success' count for 2023-11-12: 0 (flag=2 count: 0) | 'success' count for 2023-11-11: 0 (flag=2 count: 0) | 'success' count for 2023-11-10: 0 (flag=2 count: 0) Anomalies (flag=2): -------- Total 'failure' count: 163 Total record count: 1261 Job history table record count: 113756 Scoring job record table record count: 435 -----------------------------------------------------------