Re: [Rainqc-jobman] RainQC Job Manager daily report 2022-08-10

Hi everyone, For listing the anomalies, is there a preference for sorting by date+station or sorting by station+date? What we do now is sort by date+station: Anomalies (flag=2): TA00682 2022-08-03 | score: 308.398 (thresh: 261.338) -- 'pr' mm t: 0.000 [17.142, 23.176] TA00686 2022-08-03 | score: 182.706 (thresh: 90.000) -- 'pr' mm t: 0.255 [0.0, 2.227, 17.142, 23.176] TA00148 2022-08-04 | score: 426.326 (thresh: 90.000) -- 'pr' mm t: 0.034 [22.4] TA00364 2022-08-04 | score: 190.747 (thresh: 79.321) -- 'pr' mm t: 1.531 [23.226] TA00398 2022-08-04 | score: 1418.571 (thresh: 90.000) -- 'pr' mm t: 0.068 [38.349] TA00682 2022-08-04 | score: 3087.313 (thresh: 261.338) -- 'pr' mm t: 0.000 [49.777, 85.636] TA00160 2022-08-08 | score: 691.851 (thresh: 266.803) -- 'pr' mm t: 1.769 [0.0, 33.714] TA00686 2022-08-08 | score: 215.214 (thresh: 90.000) -- 'pr' mm t: 0.000 [0.0, 1.769, 3.809, 33.714] TA00056 2022-08-09 | score: 218.996 (thresh: 146.763) -- 'pr' mm t: 0.000 [0.0, 0.017, 20.2] TA00196 2022-08-09 | score: 112.270 (thresh: 89.761) -- 'pr' mm t: 0.000 [0.0, 1.785, 20.2] However, if we sort by the station and then the date, IMO it's easier to see multiple anomalies for the same station: Anomalies (flag=2): TA00056 2022-08-09 | score: 218.996 (thresh: 146.763) -- 'pr' mm t: 0.000 [0.0, 0.017, 20.2] TA00148 2022-08-04 | score: 426.326 (thresh: 90.000) -- 'pr' mm t: 0.034 [22.4] TA00160 2022-08-08 | score: 691.851 (thresh: 266.803) -- 'pr' mm t: 1.769 [0.0, 33.714] TA00196 2022-08-09 | score: 112.270 (thresh: 89.761) -- 'pr' mm t: 0.000 [0.0, 1.785, 20.2] TA00364 2022-08-04 | score: 190.747 (thresh: 79.321) -- 'pr' mm t: 1.531 [23.226] TA00398 2022-08-04 | score: 1418.571 (thresh: 90.000) -- 'pr' mm t: 0.068 [38.349] TA00682 2022-08-03 | score: 308.398 (thresh: 261.338) -- 'pr' mm t: 0.000 [17.142, 23.176] TA00682 2022-08-04 | score: 3087.313 (thresh: 261.338) -- 'pr' mm t: 0.000 [49.777, 85.636] TA00686 2022-08-03 | score: 182.706 (thresh: 90.000) -- 'pr' mm t: 0.255 [0.0, 2.227, 17.142, 23.176] TA00686 2022-08-08 | score: 215.214 (thresh: 90.000) -- 'pr' mm t: 0.000 [0.0, 1.769, 3.809, 33.714] I suppose we could start making a CSV file and attaching it to the daily report, so those so inclined can sort things however they want :-) Any thoughts? Thanks, Michael From: slater@oregonstate.edu <slater@oregonstate.edu> Sent: Tuesday, August 9, 2022 11:18 PM To: rainqc-jobman@ENGR.ORST.EDU Cc: Slater, Michael <slater@oregonstate.edu> Subject: RainQC Job Manager daily report 2022-08-10 Current UTC date: 2022-08-10 -> scoring models for previous day --------------------------------------------------------------------------------- Daily Model Data Completeness Check: data completeness 50% | complete models: 74 of 222 (33.33%) data completeness 60% | complete models: 71 of 222 (31.98%) data completeness 70% | complete models: 66 of 222 (29.73%) data completeness 75% | complete models: 66 of 222 (29.73%) data completeness 80% | complete models: 66 of 222 (29.73%) data completeness 85% | complete models: 66 of 222 (29.73%) data completeness 90% | complete models: 66 of 222 (29.73%) data completeness 95% | complete models: 66 of 222 (29.73%) data completeness 100% | complete models: 66 of 222 (29.73%) ---------------------------- station status | total: 271, delayed: 129, offline 24h: 103, offline week: 35 | battery, min: 0, max: 100, mean: 57.83, std dev: 27.1 | battery, common values: [(100, 160), (0, 43), (74, 4), (65, 3), (53, 3)] | battery <= mean, common countries: [('GH', 19), ('KE', 15), ('ML', 8), ('MW', 7), ('TG', 6)] ---------------------------- 121 LOW DATA (< 0.9) and 108 NO DATA weather stations impacted 156 RainQC models LOW/NO data station impact on models: [('TA00570', 10), ('TA00127', 9), ('TA00568', 9), ('TA00447', 7), ('TA00199', 7), ('TA00078', 6), ('TA00198', 6), ('TA00677', 5), ('TA00126', 5), ('TA00160', 5), ('TA00301', 5), ('TA00370', 5), ('TA00652', 4), ('TA00586', 4), ('TA00140', 4), ('TA00074', 4), ('TA00335', 4), ('TA00391', 4), ('TA00691', 4), ('TA00680', 4), ('TA00270', 4), ('TA00537', 4), ('TA00044', 3), ('TA00041', 3), ('TA00043', 3), ('TA00050', 3), ('TA00045', 3), ('TA00178', 3), ('TA00089', 3), ('TA00091', 3), ('TA00422', 3), ('TA00170', 3), ('TA00251', 3), ('TA00180', 3), ('TA00495', 3), ('TA00319', 3), ('TA00530', 3), ('TA00533', 3), ('TA00536', 3), ('TA00655', 3), ('TA00276', 2), ('TA00308', 2), ('TA00171', 2), ('TA00101', 2), ('TA00102', 2), ('TA00257', 2), ('TA00256', 2), ('TA00118', 2), ('TA00313', 2), ('TA00165', 2), ('TA00148', 2), ('TA00467', 2), ('TA00155', 2), ('TA00168', 2), ('TA00684', 2), ('TA00289', 2), ('TA00203', 2), ('TA00300', 2), ('TA00210', 2), ('TA00262', 2), ('TA00265', 2), ('TA00267', 2), ('TA00393', 2), ('TA00271', 2), ('TA00494', 2), ('TA00385', 2), ('TA00307', 2), ('TA00339', 2), ('TA00361', 2), ('TA00367', 2), ('TA00368', 2), ('TA00398', 2), ('TA00471', 2), ('TA00466', 2), ('TA00547', 2), ('TA00664', 2), ('TA00014', 1), ('TA00031', 1), ('TA00039', 1), ('TA00282', 1), ('TA00092', 1), ('TA00109', 1), ('TA00123', 1), ('TA00128', 1), ('TA00136', 1), ('TA00254', 1), ('TA00179', 1), ('TA00201', 1), ('TA00219', 1), ('TA00033', 1), ('TA00236', 1), ('TA00287', 1), ('TA00286', 1), ('TA00237', 1), ('TA00279', 1), ('TA00392', 1), ('TA00277', 1), ('TA00290', 1), ('TA00336', 1), ('TA00350', 1), ('TA00351', 1), ('TA00322', 1), ('TA00362', 1), ('TA00375', 1), ('TA00333', 1), ('TA00373', 1), ('TA00382', 1), ('TA00383', 1), ('TA00255', 1), ('TA00231', 1), ('TA00482', 1), ('TA00036', 1), ('TA00493', 1), ('TA00543', 1), ('TA00565', 1), ('TA00542', 1), ('TA00602', 1), ('TA00639', 1), ('TA00641', 1), ('TA00638', 1), ('TA00702', 1)] ----------------------------------------------------------- Processed daily jobs for UTC date: 2022-08-09 Start time: 2022-08-10T04:57:14+00:00 End time : 2022-08-10T06:18:26+00:00 Elapsed time HH:MM:SS: 1:21:12 --------------------- Before job processing job table stats: Total 'success' count: 60 Total 'failure' count: 115 Total record count: 1145 Job history table record count: 72483 Scoring job record table record count: 334 --------------------- After job processing job table stats: Total 'success' count: 248 (flag=2 count: 10) (flag=1 abstentions: 2) | 'success' count for 2022-08-09: 66 (flag=2 count: 2) | 'success' count for 2022-08-08: 13 (flag=2 count: 2) | 'success' count for 2022-08-07: 24 (flag=2 count: 0) | 'success' count for 2022-08-06: 32 (flag=2 count: 0) | 'success' count for 2022-08-05: 35 (flag=2 count: 0) | 'success' count for 2022-08-04: 39 (flag=2 count: 4) | 'success' count for 2022-08-03: 39 (flag=2 count: 2) Anomalies (flag=2): TA00682 2022-08-03 | score: 308.398 (thresh: 261.338) -- 'pr' mm t: 0.000 [17.142, 23.176] TA00686 2022-08-03 | score: 182.706 (thresh: 90.000) -- 'pr' mm t: 0.255 [0.0, 2.227, 17.142, 23.176] TA00148 2022-08-04 | score: 426.326 (thresh: 90.000) -- 'pr' mm t: 0.034 [22.4] TA00364 2022-08-04 | score: 190.747 (thresh: 79.321) -- 'pr' mm t: 1.531 [23.226] TA00398 2022-08-04 | score: 1418.571 (thresh: 90.000) -- 'pr' mm t: 0.068 [38.349] TA00682 2022-08-04 | score: 3087.313 (thresh: 261.338) -- 'pr' mm t: 0.000 [49.777, 85.636] TA00160 2022-08-08 | score: 691.851 (thresh: 266.803) -- 'pr' mm t: 1.769 [0.0, 33.714] TA00686 2022-08-08 | score: 215.214 (thresh: 90.000) -- 'pr' mm t: 0.000 [0.0, 1.769, 3.809, 33.714] TA00056 2022-08-09 | score: 218.996 (thresh: 146.763) -- 'pr' mm t: 0.000 [0.0, 0.017, 20.2] TA00196 2022-08-09 | score: 112.270 (thresh: 89.761) -- 'pr' mm t: 0.000 [0.0, 1.785, 20.2] -------- Total 'failure' count: 115 Total record count: 1192 Job history table record count: 72658 Scoring job record table record count: 335 -----------------------------------------------------------