Hi everyone,
For listing the anomalies, is there a preference for sorting by date+station or sorting by station+date? What we do now is sort by date+station:
Anomalies (flag=2):
TA00682 2022-08-03 | score: 308.398 (thresh: 261.338) -- 'pr' mm t: 0.000 [17.142, 23.176]
TA00686 2022-08-03 | score: 182.706 (thresh: 90.000) -- 'pr' mm t: 0.255 [0.0, 2.227, 17.142, 23.176]
TA00148 2022-08-04 | score: 426.326 (thresh: 90.000) -- 'pr' mm t: 0.034 [22.4]
TA00364 2022-08-04 | score: 190.747 (thresh: 79.321) -- 'pr' mm t: 1.531 [23.226]
TA00398 2022-08-04 | score: 1418.571 (thresh: 90.000) -- 'pr' mm t: 0.068 [38.349]
TA00682 2022-08-04 | score: 3087.313 (thresh: 261.338) -- 'pr' mm t: 0.000 [49.777, 85.636]
TA00160 2022-08-08 | score: 691.851 (thresh: 266.803) -- 'pr' mm t: 1.769 [0.0, 33.714]
TA00686 2022-08-08 | score: 215.214 (thresh: 90.000) -- 'pr' mm t: 0.000 [0.0, 1.769, 3.809, 33.714]
TA00056 2022-08-09 | score: 218.996 (thresh: 146.763) -- 'pr' mm t: 0.000 [0.0, 0.017, 20.2]
TA00196 2022-08-09 | score: 112.270 (thresh: 89.761) -- 'pr' mm t: 0.000 [0.0, 1.785, 20.2]
However, if we sort by the station and then the date, IMO it’s easier to see multiple anomalies for the same station:
Anomalies (flag=2):
TA00056 2022-08-09 | score: 218.996 (thresh: 146.763) -- 'pr' mm t: 0.000 [0.0, 0.017, 20.2]
TA00148 2022-08-04 | score: 426.326 (thresh: 90.000) -- 'pr' mm t: 0.034 [22.4]
TA00160 2022-08-08 | score: 691.851 (thresh: 266.803) -- 'pr' mm t: 1.769 [0.0, 33.714]
TA00196 2022-08-09 | score: 112.270 (thresh: 89.761) -- 'pr' mm t: 0.000 [0.0, 1.785, 20.2]
TA00364 2022-08-04 | score: 190.747 (thresh: 79.321) -- 'pr' mm t: 1.531 [23.226]
TA00398 2022-08-04 | score: 1418.571 (thresh: 90.000) -- 'pr' mm t: 0.068 [38.349]
TA00682 2022-08-03 | score: 308.398 (thresh: 261.338) -- 'pr' mm t: 0.000 [17.142, 23.176]
TA00682 2022-08-04 | score: 3087.313 (thresh: 261.338) -- 'pr' mm t: 0.000 [49.777, 85.636]
TA00686 2022-08-03 | score: 182.706 (thresh: 90.000) -- 'pr' mm t: 0.255 [0.0, 2.227, 17.142, 23.176]
TA00686 2022-08-08 | score: 215.214 (thresh: 90.000) -- 'pr' mm t: 0.000 [0.0, 1.769, 3.809, 33.714]
I suppose we could start making a CSV file and attaching it to the daily report, so those so inclined can sort things however they want :-)
Any thoughts?
Thanks,
Michael
From: slater@oregonstate.edu <slater@oregonstate.edu>
Sent: Tuesday, August 9, 2022 11:18 PM
To: rainqc-jobman@ENGR.ORST.EDU
Cc: Slater, Michael <slater@oregonstate.edu>
Subject: RainQC Job Manager daily report 2022-08-10
Current UTC date: 2022-08-10 -> scoring models for previous day
---------------------------------------------------------------------------------
Daily Model Data Completeness Check:
data completeness 50% | complete models: 74 of 222 (33.33%)
data completeness 60% | complete models: 71 of 222 (31.98%)
data completeness 70% | complete models: 66 of 222 (29.73%)
data completeness 75% | complete models: 66 of 222 (29.73%)
data completeness 80% | complete models: 66 of 222 (29.73%)
data completeness 85% | complete models: 66 of 222 (29.73%)
data completeness 90% | complete models: 66 of 222 (29.73%)
data completeness 95% | complete models: 66 of 222 (29.73%)
data completeness 100% | complete models: 66 of 222 (29.73%)
----------------------------
station status | total: 271, delayed: 129, offline 24h: 103, offline week: 35
| battery, min: 0, max: 100, mean: 57.83, std dev: 27.1
| battery, common values: [(100, 160), (0, 43), (74, 4), (65, 3), (53, 3)]
| battery <= mean, common countries: [('GH', 19), ('KE', 15), ('ML', 8), ('MW', 7), ('TG', 6)]
----------------------------
121 LOW DATA (< 0.9) and 108 NO DATA weather stations impacted 156 RainQC models
LOW/NO data station impact on models: [('TA00570', 10), ('TA00127', 9), ('TA00568', 9), ('TA00447', 7), ('TA00199', 7), ('TA00078', 6), ('TA00198', 6), ('TA00677', 5), ('TA00126', 5), ('TA00160', 5), ('TA00301', 5), ('TA00370', 5), ('TA00652', 4), ('TA00586', 4), ('TA00140', 4), ('TA00074', 4), ('TA00335', 4), ('TA00391', 4), ('TA00691', 4), ('TA00680', 4), ('TA00270', 4), ('TA00537', 4), ('TA00044', 3), ('TA00041', 3), ('TA00043', 3), ('TA00050', 3), ('TA00045', 3), ('TA00178', 3), ('TA00089', 3), ('TA00091', 3), ('TA00422', 3), ('TA00170', 3), ('TA00251', 3), ('TA00180', 3), ('TA00495', 3), ('TA00319', 3), ('TA00530', 3), ('TA00533', 3), ('TA00536', 3), ('TA00655', 3), ('TA00276', 2), ('TA00308', 2), ('TA00171', 2), ('TA00101', 2), ('TA00102', 2), ('TA00257', 2), ('TA00256', 2), ('TA00118', 2), ('TA00313', 2), ('TA00165', 2), ('TA00148', 2), ('TA00467', 2), ('TA00155', 2), ('TA00168', 2), ('TA00684', 2), ('TA00289', 2), ('TA00203', 2), ('TA00300', 2), ('TA00210', 2), ('TA00262', 2), ('TA00265', 2), ('TA00267', 2), ('TA00393', 2), ('TA00271', 2), ('TA00494', 2), ('TA00385', 2), ('TA00307', 2), ('TA00339', 2), ('TA00361', 2), ('TA00367', 2), ('TA00368', 2), ('TA00398', 2), ('TA00471', 2), ('TA00466', 2), ('TA00547', 2), ('TA00664', 2), ('TA00014', 1), ('TA00031', 1), ('TA00039', 1), ('TA00282', 1), ('TA00092', 1), ('TA00109', 1), ('TA00123', 1), ('TA00128', 1), ('TA00136', 1), ('TA00254', 1), ('TA00179', 1), ('TA00201', 1), ('TA00219', 1), ('TA00033', 1), ('TA00236', 1), ('TA00287', 1), ('TA00286', 1), ('TA00237', 1), ('TA00279', 1), ('TA00392', 1), ('TA00277', 1), ('TA00290', 1), ('TA00336', 1), ('TA00350', 1), ('TA00351', 1), ('TA00322', 1), ('TA00362', 1), ('TA00375', 1), ('TA00333', 1), ('TA00373', 1), ('TA00382', 1), ('TA00383', 1), ('TA00255', 1), ('TA00231', 1), ('TA00482', 1), ('TA00036', 1), ('TA00493', 1), ('TA00543', 1), ('TA00565', 1), ('TA00542', 1), ('TA00602', 1), ('TA00639', 1), ('TA00641', 1), ('TA00638', 1), ('TA00702', 1)]
-----------------------------------------------------------
Processed daily jobs for UTC date: 2022-08-09
Start time: 2022-08-10T04:57:14+00:00
End time : 2022-08-10T06:18:26+00:00
Elapsed time HH:MM:SS: 1:21:12
---------------------
Before job processing job table stats:
Total 'success' count: 60
Total 'failure' count: 115
Total record count: 1145
Job history table record count: 72483
Scoring job record table record count: 334
---------------------
After job processing job table stats:
Total 'success' count: 248 (flag=2 count: 10) (flag=1 abstentions: 2)
| 'success' count for 2022-08-09: 66 (flag=2 count: 2)
| 'success' count for 2022-08-08: 13 (flag=2 count: 2)
| 'success' count for 2022-08-07: 24 (flag=2 count: 0)
| 'success' count for 2022-08-06: 32 (flag=2 count: 0)
| 'success' count for 2022-08-05: 35 (flag=2 count: 0)
| 'success' count for 2022-08-04: 39 (flag=2 count: 4)
| 'success' count for 2022-08-03: 39 (flag=2 count: 2)
Anomalies (flag=2):
TA00682 2022-08-03 | score: 308.398 (thresh: 261.338) -- 'pr' mm t: 0.000 [17.142, 23.176]
TA00686 2022-08-03 | score: 182.706 (thresh: 90.000) -- 'pr' mm t: 0.255 [0.0, 2.227, 17.142, 23.176]
TA00148 2022-08-04 | score: 426.326 (thresh: 90.000) -- 'pr' mm t: 0.034 [22.4]
TA00364 2022-08-04 | score: 190.747 (thresh: 79.321) -- 'pr' mm t: 1.531 [23.226]
TA00398 2022-08-04 | score: 1418.571 (thresh: 90.000) -- 'pr' mm t: 0.068 [38.349]
TA00682 2022-08-04 | score: 3087.313 (thresh: 261.338) -- 'pr' mm t: 0.000 [49.777, 85.636]
TA00160 2022-08-08 | score: 691.851 (thresh: 266.803) -- 'pr' mm t: 1.769 [0.0, 33.714]
TA00686 2022-08-08 | score: 215.214 (thresh: 90.000) -- 'pr' mm t: 0.000 [0.0, 1.769, 3.809, 33.714]
TA00056 2022-08-09 | score: 218.996 (thresh: 146.763) -- 'pr' mm t: 0.000 [0.0, 0.017, 20.2]
TA00196 2022-08-09 | score: 112.270 (thresh: 89.761) -- 'pr' mm t: 0.000 [0.0, 1.785, 20.2]
--------
Total 'failure' count: 115
Total record count: 1192
Job history table record count: 72658
Scoring job record table record count: 335
-----------------------------------------------------------