Again, similar to yesterday, RainQC server successfully scored zero stations for the daily jobs. The job manager pulled the usual 273 models:
2023-03-11 22:11:26.340 | INFO : RainQC raw model count: 273, parsed model count: 273
The data is present, as seen from our data completeness estimate:
2023-03-11 22:30:21.720 | INFO : Model completeness:
2023-03-11 22:30:21.721 | INFO : data completeness 50% | complete models: 174 of 273 (63.74%)
2023-03-11 22:30:21.721 | INFO : data completeness 60% | complete models: 173 of 273 (63.37%)
2023-03-11 22:30:21.722 | INFO : data completeness 70% | complete models: 170 of 273 (62.27%)
2023-03-11 22:30:21.722 | INFO : data completeness 75% | complete models: 170 of 273 (62.27%)
2023-03-11 22:30:21.728 | INFO : data completeness 80% | complete models: 170 of 273 (62.27%)
2023-03-11 22:30:21.730 | INFO : data completeness 85% | complete models: 169 of 273 (61.90%)
2023-03-11 22:30:21.731 | INFO : data completeness 90% | complete models: 169 of 273 (61.90%)
2023-03-11 22:30:21.731 | INFO : data completeness 95% | complete models: 166 of 273 (60.81%)
2023-03-11 22:30:21.732 | INFO : data completeness 100% | complete models: 164 of 273 (60.07%)
However, we see return values like the following example:
2023-03-11 23:16:02.319 | INFO : --- [1233] --- getting model | name: TA00310, id: 63799410a70bcec7dc859eca, start: 2023-03-11, end: 2023-03-11
2023-03-11 23:16:04.141 | INFO : Processing score | station: TA00310
2023-03-11 23:16:04.143 | WARNING : process_job_table | response problem, skipping flag update. state: error_scoring, response: {'model_id': '63799410a70bcec7dc859eca',
'neighbors': ['TA00110', 'TA00124'], 'qc_app_version': 1.2, 'results': [{'message': 'Target station has no data for the given date', 'period_end': '2023-03-11T23:59:59Z', 'period_start': '2023-03-11T00:00:00Z', 'qc_flag': 0, 'qc_flag_threshold': 63.717, 'qc_flag_threshold_method':
'Threshold is obtained from evaluation metric', 'qc_method': 'MixLinearResidual with 2 neighbors)', 'qc_variable': 'precipitation', 'score': None, 'sensorcode': None}], 'status': 'fail', 'target_station': 'TA00310', 'weather_variable': 'precipitation'}
While the data is there for nominally 60% of the models, the RainQC server does not appear to be fetching it successfully.
Does anyone know what is going on?
Thanks,
Michael
From: slater@oregonstate.edu <slater@oregonstate.edu>
Sent: Saturday, March 11, 2023 11:16 PM
To: rainqc-jobman@ENGR.ORST.EDU
Cc: Slater, Michael <slater@oregonstate.edu>
Subject: RainQC Job Manager daily report 2023-03-12
Current UTC date: 2023-03-12 -> scoring models for previous day: 2023-03-11
---------------------------------------------------------------------------------
Daily Model Data Completeness Check:
data completeness 50% | complete models: 174 of 273 (63.74%)
data completeness 60% | complete models: 173 of 273 (63.37%)
data completeness 70% | complete models: 170 of 273 (62.27%)
data completeness 75% | complete models: 170 of 273 (62.27%)
data completeness 80% | complete models: 170 of 273 (62.27%)
data completeness 85% | complete models: 169 of 273 (61.90%)
data completeness 90% | complete models: 169 of 273 (61.90%)
data completeness 95% | complete models: 166 of 273 (60.81%)
data completeness 100% | complete models: 164 of 273 (60.07%)
----------------------------
station status | total: 313, delayed: 118, offline 24h: 46, offline week: 42
| battery, min: 0, max: 100, mean: 66.39, std dev: 29.81
| battery, common values: [(100, 220), (0, 50), (97, 3), (8, 2), (77, 2)]
| battery <= mean, common countries: [('GH', 17), ('UG', 13), ('KE', 8), ('ML', 4), ('ZM', 4)]
----------------------------
67 LOW DATA (< 0.9) and 55 NO DATA weather stations impacted 104 RainQC models
LOW/NO data station impact on models: [('TA00199', 7), ('TA00076', 6), ('TA00231', 4), ('TA00530', 4), ('TA00700', 4), ('TA00011', 3), ('TA00173', 3), ('TA00032', 3), ('TA00035', 3), ('TA00036', 3), ('TA00267', 3), ('TA00066', 3), ('TA00094', 3), ('TA00126', 3), ('TA00217', 3), ('TA00482', 3), ('TA00542', 3), ('TA00308', 2), ('TA00043', 2), ('TA00050', 2), ('TA00102', 2), ('TA00136', 2), ('TA00164', 2), ('TA00487', 2), ('TA00684', 2), ('TA00210', 2), ('TA00223', 2), ('TA00232', 2), ('TA00262', 2), ('TA00271', 2), ('TA00314', 2), ('TA00691', 2), ('TA00339', 2), ('TA00373', 2), ('TA00398', 2), ('TA00451', 2), ('TA00462', 2), ('TA00471', 2), ('TA00278', 1), ('TA00014', 1), ('TA00031', 1), ('TA00044', 1), ('TA00095', 1), ('TA00157', 1), ('TA00201', 1), ('TA00212', 1), ('TA00219', 1), ('TA00229', 1), ('TA00237', 1), ('TA00260', 1), ('TA00276', 1), ('TA00287', 1), ('TA00290', 1), ('TA00336', 1), ('TA00350', 1), ('TA00362', 1), ('TA00369', 1), ('TA00422', 1), ('TA00432', 1), ('TA00493', 1), ('TA00524', 1), ('TA00533', 1), ('TA00535', 1), ('TA00652', 1), ('TA00655', 1), ('TA00677', 1), ('TA00702', 1)]
-----------------------------------------------------------
Processed daily jobs for UTC date: 2023-03-11
Start time: 2023-03-12T06:30:21+00:00
End time : 2023-03-12T07:16:12+00:00
Elapsed time HH:MM:SS: 0:45:51
---------------------
Before job processing job table stats:
Total 'success' count: 11
Total 'failure' count: 134
Total record count: 1107
Job history table record count: 45387
Scoring job record table record count: 183
---------------------
After job processing job table stats:
Total 'success' count: 0 (flag=2 count: 0) (flag 2->1 downgrades: 0)
| 'success' count for 2023-03-11: 0 (flag=2 count: 0)
| 'success' count for 2023-03-10: 0 (flag=2 count: 0)
| 'success' count for 2023-03-09: 0 (flag=2 count: 0)
| 'success' count for 2023-03-08: 0 (flag=2 count: 0)
| 'success' count for 2023-03-07: 0 (flag=2 count: 0)
| 'success' count for 2023-03-06: 0 (flag=2 count: 0)
| 'success' count for 2023-03-05: 0 (flag=2 count: 0)
Anomalies (flag=2):
--------
Total 'failure' count: 134
Total record count: 1235
Job history table record count: 45532
Scoring job record table record count: 184
-----------------------------------------------------------