Strange day for RainQC, with 40%+ model completeness, but zero successful model scoring calls to RainQC.  As far as I can tell, all the TAHMO data fetches performed by the RainQC server failed. A sample ‘message’ that was passed back from RainQC server to the job manager:

{'message': 'the GET from http://tahmogohub:8080 failed. URL: http://tahmogohub:8080/v2/stationaccessrights?filter=code%21eq%21TA00116%26%26label%21eq%21SensorDX, err: Get http://tahmogohub:8080/v2/stationaccessrights?filter=code%21eq%21TA00116%26%26label%21eq%21SensorDX: net/http: request canceled (Client.Timeout exceeded while awaiting headers)'}"

In a quick glance at the log, there were many messages similar to the above. We may have some sort of data access rights issue going on with the RainQC server.

Michael

From: slater@oregonstate.edu <slater@oregonstate.edu>
Sent: Friday, November 17, 2023 12:13 AM
To: rainqc-jobman@ENGR.ORST.EDU
Cc: Slater, Michael <slater@oregonstate.edu>
Subject: RainQC Job Manager daily report 2023-11-17

Current UTC date: 2023-11-17 -> scoring models for previous day: 2023-11-16
---------------------------------------------------------------------------------
Daily Model Data Completeness Check:
data completeness    50% | complete models: 137 of 273 (50.18%)
data completeness    60% | complete models: 134 of 273 (49.08%)
data completeness    70% | complete models: 130 of 273 (47.62%)
data completeness    75% | complete models: 130 of 273 (47.62%)
data completeness    80% | complete models: 123 of 273 (45.05%)
data completeness    85% | complete models: 116 of 273 (42.49%)
data completeness    90% | complete models: 112 of 273 (41.03%)
data completeness    95% | complete models: 111 of 273 (40.66%)
data completeness   100% | complete models: 110 of 273 (40.29%)
----------------------------
station status | total: 313, delayed: 130, offline 24h: 77, offline week: 71
 | battery, min: 0, max: 100, mean: 69.07, std dev: 27.99
 | battery, common values: [(100, 194), (0, 78), (76, 4), (96, 3), (97, 3)]
 | battery <= mean, common countries: [('KE', 21), ('GH', 16), ('UG', 11), ('ML', 5), ('RW', 5)]
----------------------------
92 LOW DATA (< 0.9) and 75 NO DATA weather stations impacted 161 RainQC models
LOW/NO data station impact on models: [('TA00127', 8), ('TA00715', 8), ('TA00568', 8), ('TA00196', 7), ('TA00719', 7), ('TA00185', 7), ('TA00414', 6), ('TA00073', 6), ('TA00621', 5), ('TA00320', 5), ('TA00231', 4), ('TA00190', 4), ('TA00530', 4), ('TA00565', 4), ('TA00543', 4), ('TA00003', 3), ('TA00035', 3), ('TA00036', 3), ('TA00222', 3), ('TA00041', 3), ('TA00267', 3), ('TA00178', 3), ('TA00066', 3), ('TA00094', 3), ('TA00115', 3), ('TA00126', 3), ('TA00484', 3), ('TA00217', 3), ('TA00482', 3), ('TA00385', 3), ('TA00436', 3), ('TA00430', 3), ('TA00308', 2), ('TA00050', 2), ('TA00302', 2), ('TA00072', 2), ('TA00118', 2), ('TA00165', 2), ('TA00148', 2), ('TA00155', 2), ('TA00487', 2), ('TA00210', 2), ('TA00232', 2), ('TA00271', 2), ('TA00309', 2), ('TA00691', 2), ('TA00339', 2), ('TA00364', 2), ('TA00373', 2), ('TA00451', 2), ('TA00462', 2), ('TA00471', 2), ('TA00014', 1), ('TA00031', 1), ('TA00044', 1), ('TA00070', 1), ('TA00078', 1), ('TA00091', 1), ('TA00095', 1), ('TA00123', 1), ('TA00157', 1), ('TA00212', 1), ('TA00219', 1), ('TA00229', 1), ('TA00237', 1), ('TA00251', 1), ('TA00689', 1), ('TA00269', 1), ('TA00276', 1), ('TA00287', 1), ('TA00290', 1), ('TA00336', 1), ('TA00343', 1), ('TA00350', 1), ('TA00362', 1), ('TA00369', 1), ('TA00382', 1), ('TA00389', 1), ('TA00422', 1), ('TA00432', 1), ('TA00454', 1), ('TA00493', 1), ('TA00528', 1), ('TA00529', 1), ('TA00533', 1), ('TA00535', 1), ('TA00586', 1), ('TA00588', 1), ('TA00652', 1), ('TA00655', 1), ('TA00677', 1), ('TA00702', 1)]
-----------------------------------------------------------
Processed daily jobs for UTC date: 2023-11-16
Start time: 2023-11-17T06:41:32+00:00
End time  : 2023-11-17T08:13:15+00:00
Elapsed time HH:MM:SS: 1:31:44
---------------------
Before job processing job table stats:
Total 'success' count: 116
Total 'failure' count: 162
Total record count: 1266
Job history table record count: 113478
Scoring job record table record count: 434
---------------------
After job processing job table stats:
Total 'success' count:                0 (flag=2 count:   0) (flag 2->1 downgrades:   0)
 | 'success' count for 2023-11-16:    0 (flag=2 count:   0)
 | 'success' count for 2023-11-15:    0 (flag=2 count:   0)
 | 'success' count for 2023-11-14:    0 (flag=2 count:   0)
 | 'success' count for 2023-11-13:    0 (flag=2 count:   0)
 | 'success' count for 2023-11-12:    0 (flag=2 count:   0)
 | 'success' count for 2023-11-11:    0 (flag=2 count:   0)
 | 'success' count for 2023-11-10:    0 (flag=2 count:   0)
Anomalies (flag=2):
--------
Total 'failure' count: 163
Total record count: 1261
Job history table record count: 113756
Scoring job record table record count: 435
-----------------------------------------------------------