Again, similar to yesterday, RainQC server successfully scored zero stations for the daily jobs. The job manager pulled the usual 273 models: 2023-03-11 22:11:26.340 | INFO : RainQC raw model count: 273, parsed model count: 273 The data is present, as seen from our data completeness estimate: 2023-03-11 22:30:21.720 | INFO : Model completeness: 2023-03-11 22:30:21.721 | INFO : data completeness 50% | complete models: 174 of 273 (63.74%) 2023-03-11 22:30:21.721 | INFO : data completeness 60% | complete models: 173 of 273 (63.37%) 2023-03-11 22:30:21.722 | INFO : data completeness 70% | complete models: 170 of 273 (62.27%) 2023-03-11 22:30:21.722 | INFO : data completeness 75% | complete models: 170 of 273 (62.27%) 2023-03-11 22:30:21.728 | INFO : data completeness 80% | complete models: 170 of 273 (62.27%) 2023-03-11 22:30:21.730 | INFO : data completeness 85% | complete models: 169 of 273 (61.90%) 2023-03-11 22:30:21.731 | INFO : data completeness 90% | complete models: 169 of 273 (61.90%) 2023-03-11 22:30:21.731 | INFO : data completeness 95% | complete models: 166 of 273 (60.81%) 2023-03-11 22:30:21.732 | INFO : data completeness 100% | complete models: 164 of 273 (60.07%) However, we see return values like the following example: 2023-03-11 23:16:02.319 | INFO : --- [1233] --- getting model | name: TA00310, id: 63799410a70bcec7dc859eca, start: 2023-03-11, end: 2023-03-11 2023-03-11 23:16:04.141 | INFO : Processing score | station: TA00310 2023-03-11 23:16:04.143 | WARNING : process_job_table | response problem, skipping flag update. state: error_scoring, response: {'model_id': '63799410a70bcec7dc859eca', 'neighbors': ['TA00110', 'TA00124'], 'qc_app_version': 1.2, 'results': [{'message': 'Target station has no data for the given date', 'period_end': '2023-03-11T23:59:59Z', 'period_start': '2023-03-11T00:00:00Z', 'qc_flag': 0, 'qc_flag_threshold': 63.717, 'qc_flag_threshold_method': 'Threshold is obtained from evaluation metric', 'qc_method': 'MixLinearResidual with 2 neighbors)', 'qc_variable': 'precipitation', 'score': None, 'sensorcode': None}], 'status': 'fail', 'target_station': 'TA00310', 'weather_variable': 'precipitation'} While the data is there for nominally 60% of the models, the RainQC server does not appear to be fetching it successfully. Does anyone know what is going on? Thanks, Michael From: slater@oregonstate.edu <slater@oregonstate.edu> Sent: Saturday, March 11, 2023 11:16 PM To: rainqc-jobman@ENGR.ORST.EDU Cc: Slater, Michael <slater@oregonstate.edu> Subject: RainQC Job Manager daily report 2023-03-12 Current UTC date: 2023-03-12 -> scoring models for previous day: 2023-03-11 --------------------------------------------------------------------------------- Daily Model Data Completeness Check: data completeness 50% | complete models: 174 of 273 (63.74%) data completeness 60% | complete models: 173 of 273 (63.37%) data completeness 70% | complete models: 170 of 273 (62.27%) data completeness 75% | complete models: 170 of 273 (62.27%) data completeness 80% | complete models: 170 of 273 (62.27%) data completeness 85% | complete models: 169 of 273 (61.90%) data completeness 90% | complete models: 169 of 273 (61.90%) data completeness 95% | complete models: 166 of 273 (60.81%) data completeness 100% | complete models: 164 of 273 (60.07%) ---------------------------- station status | total: 313, delayed: 118, offline 24h: 46, offline week: 42 | battery, min: 0, max: 100, mean: 66.39, std dev: 29.81 | battery, common values: [(100, 220), (0, 50), (97, 3), (8, 2), (77, 2)] | battery <= mean, common countries: [('GH', 17), ('UG', 13), ('KE', 8), ('ML', 4), ('ZM', 4)] ---------------------------- 67 LOW DATA (< 0.9) and 55 NO DATA weather stations impacted 104 RainQC models LOW/NO data station impact on models: [('TA00199', 7), ('TA00076', 6), ('TA00231', 4), ('TA00530', 4), ('TA00700', 4), ('TA00011', 3), ('TA00173', 3), ('TA00032', 3), ('TA00035', 3), ('TA00036', 3), ('TA00267', 3), ('TA00066', 3), ('TA00094', 3), ('TA00126', 3), ('TA00217', 3), ('TA00482', 3), ('TA00542', 3), ('TA00308', 2), ('TA00043', 2), ('TA00050', 2), ('TA00102', 2), ('TA00136', 2), ('TA00164', 2), ('TA00487', 2), ('TA00684', 2), ('TA00210', 2), ('TA00223', 2), ('TA00232', 2), ('TA00262', 2), ('TA00271', 2), ('TA00314', 2), ('TA00691', 2), ('TA00339', 2), ('TA00373', 2), ('TA00398', 2), ('TA00451', 2), ('TA00462', 2), ('TA00471', 2), ('TA00278', 1), ('TA00014', 1), ('TA00031', 1), ('TA00044', 1), ('TA00095', 1), ('TA00157', 1), ('TA00201', 1), ('TA00212', 1), ('TA00219', 1), ('TA00229', 1), ('TA00237', 1), ('TA00260', 1), ('TA00276', 1), ('TA00287', 1), ('TA00290', 1), ('TA00336', 1), ('TA00350', 1), ('TA00362', 1), ('TA00369', 1), ('TA00422', 1), ('TA00432', 1), ('TA00493', 1), ('TA00524', 1), ('TA00533', 1), ('TA00535', 1), ('TA00652', 1), ('TA00655', 1), ('TA00677', 1), ('TA00702', 1)] ----------------------------------------------------------- Processed daily jobs for UTC date: 2023-03-11 Start time: 2023-03-12T06:30:21+00:00 End time : 2023-03-12T07:16:12+00:00 Elapsed time HH:MM:SS: 0:45:51 --------------------- Before job processing job table stats: Total 'success' count: 11 Total 'failure' count: 134 Total record count: 1107 Job history table record count: 45387 Scoring job record table record count: 183 --------------------- After job processing job table stats: Total 'success' count: 0 (flag=2 count: 0) (flag 2->1 downgrades: 0) | 'success' count for 2023-03-11: 0 (flag=2 count: 0) | 'success' count for 2023-03-10: 0 (flag=2 count: 0) | 'success' count for 2023-03-09: 0 (flag=2 count: 0) | 'success' count for 2023-03-08: 0 (flag=2 count: 0) | 'success' count for 2023-03-07: 0 (flag=2 count: 0) | 'success' count for 2023-03-06: 0 (flag=2 count: 0) | 'success' count for 2023-03-05: 0 (flag=2 count: 0) Anomalies (flag=2): -------- Total 'failure' count: 134 Total record count: 1235 Job history table record count: 45532 Scoring job record table record count: 184 -----------------------------------------------------------