Preview of the updated daily report, with neighbor distances from the target. The report gets wider, due to the added distances. The below output is what we would
have seen for yesterday’s jobs. Neighbors are sorted from close -> far.
Any thoughts?
Michael
----
2022-11-30 07:13:46.910 | INFO : Anomalies (flag=2):
2022-11-30 07:13:46.911 | INFO : TA00028 2022-11-29 | score: 64.484 (thresh: 4.314) -- 'pr' t: 0.153 mm [(25.512 mm, 27 km), (10.346 mm, 54 km)]
2022-11-30 07:13:46.911 | INFO : TA00061 2022-11-29 | score: 369.874 (thresh: 80.285) -- 'pr' t: 0.000 mm [(36.827 mm, 23 km), (0.051 mm, 34 km), (21.935 mm,
35 km)]
2022-11-30 07:13:46.911 | INFO : TA00076 2022-11-29 | score: 144.803 (thresh: 114.899) -- 'pr' t: 21.935 mm [(36.827 mm, 12 km), (0.051 mm, 24 km), (0.0 mm,
35 km)]
2022-11-30 07:13:46.912 | INFO : TA00310 2022-11-28 | score: 80.930 (thresh: 63.717) -- 'pr' t: 0.000 mm [(3.808 mm, 63 km), (8.592 mm, 67 km)]
2022-11-30 07:13:46.912 | INFO : TA00323 2022-11-29 | score: 172.668 (thresh: 150.073) -- 'pr' t: 0.017 mm [(11.196 mm, 21 km), (10.046 mm, 45 km), (8.894 mm,
104 km)]
2022-11-30 07:13:46.912 | INFO : TA00329 2022-11-29 | score: 228.403 (thresh: 186.163) -- 'pr' t: 0.000 mm [(10.046 mm, 15 km), (4.676 mm, 25 km), (11.196 mm,
37 km), (0.017 mm, 59 km), (8.894 mm, 79 km)]
2022-11-30 07:13:46.913 | INFO : TA00366 2022-11-29 | score: 663.414 (thresh: 61.132) -- 'pr' t: 0.000 mm [(24.514 mm, 99 km)]
2022-11-30 07:13:46.913 | INFO : TA00451 2022-11-26 | score: 159.022 (thresh: 100.920) -- 'pr' t: 0.000 mm [(1.872 mm, 22 km), (12.28 mm, 40 km)]
2022-11-30 07:13:46.913 | INFO : TA00592 2022-11-29 | score: 226.496 (thresh: 90.635) -- 'pr' t: 0.068 mm [(23.297 mm, 80 km)]
From: slater@oregonstate.edu <slater@oregonstate.edu>
Sent: Tuesday, November 29, 2022 11:06 PM
To: rainqc-jobman@ENGR.ORST.EDU
Cc: Slater, Michael <slater@oregonstate.edu>
Subject: RainQC Job Manager daily report 2022-11-30
Current UTC date: 2022-11-30 -> scoring models for previous day
---------------------------------------------------------------------------------
Daily Model Data Completeness Check:
data completeness 50% | complete models: 190 of 273 (69.60%)
data completeness 60% | complete models: 190 of 273 (69.60%)
data completeness 70% | complete models: 190 of 273 (69.60%)
data completeness 75% | complete models: 190 of 273 (69.60%)
data completeness 80% | complete models: 190 of 273 (69.60%)
data completeness 85% | complete models: 190 of 273 (69.60%)
data completeness 90% | complete models: 183 of 273 (67.03%)
data completeness 95% | complete models: 181 of 273 (66.30%)
data completeness 100% | complete models: 174 of 273 (63.74%)
----------------------------
station status | total: 313, delayed: 107, offline 24h: 39, offline week: 31
| battery, min: 0, max: 100, mean: 71.76, std dev: 22.86
| battery, common values: [(100, 217), (0, 34), (86, 3), (77, 3), (82, 3)]
| battery <= mean, common countries: [('GH', 13), ('KE', 9), ('UG', 6), ('TZ', 5), ('TG', 4)]
----------------------------
56 LOW DATA (< 0.9) and 47 NO DATA weather stations impacted 90 RainQC models
LOW/NO data station impact on models: [('TA00570', 9), ('TA00074', 7), ('TA00185', 7), ('TA00068', 6), ('TA00530', 4), ('TA00565', 4), ('TA00543', 4), ('TA00011', 3), ('TA00222', 3), ('TA00267', 3), ('TA00126', 3), ('TA00154', 3), ('TA00542', 3), ('TA00308', 2), ('TA00043', 2), ('TA00050', 2), ('TA00155', 2), ('TA00487', 2), ('TA00684', 2), ('TA00262', 2), ('TA00271', 2), ('TA00314', 2), ('TA00691', 2), ('TA00339', 2), ('TA00451', 2), ('TA00471', 2), ('TA00014', 1), ('TA00031', 1), ('TA00044', 1), ('TA00070', 1), ('TA00091', 1), ('TA00095', 1), ('TA00201', 1), ('TA00212', 1), ('TA00219', 1), ('TA00229', 1), ('TA00257', 1), ('TA00392', 1), ('TA00276', 1), ('TA00287', 1), ('TA00290', 1), ('TA00336', 1), ('TA00350', 1), ('TA00362', 1), ('TA00369', 1), ('TA00389', 1), ('TA00422', 1), ('TA00432', 1), ('TA00470', 1), ('TA00493', 1), ('TA00533', 1), ('TA00535', 1), ('TA00652', 1), ('TA00655', 1), ('TA00677', 1), ('TA00702', 1)]
-----------------------------------------------------------
Processed daily jobs for UTC date: 2022-11-29
Start time: 2022-11-30T06:23:57+00:00
End time : 2022-11-30T07:06:18+00:00
Elapsed time HH:MM:SS: 0:42:21
---------------------
Before job processing job table stats:
Total 'success' count: 174
Total 'failure' count: 99
Total record count: 864
Job history table record count: 17784
Scoring job record table record count: 80
---------------------
After job processing job table stats:
Total 'success' count: 181 (flag=2 count: 9) (flag=1 abstentions: 5)
| 'success' count for 2022-11-29: 169 (flag=2 count: 7)
| 'success' count for 2022-11-28: 6 (flag=2 count: 1)
| 'success' count for 2022-11-27: 0 (flag=2 count: 0)
| 'success' count for 2022-11-26: 2 (flag=2 count: 1)
| 'success' count for 2022-11-25: 0 (flag=2 count: 0)
| 'success' count for 2022-11-24: 0 (flag=2 count: 0)
| 'success' count for 2022-11-23: 4 (flag=2 count: 0)
Anomalies (flag=2):
TA00028 2022-11-29 | score: 64.484 (thresh: 4.314) -- 'pr' mm t: 0.153 [10.346, 25.512]
TA00061 2022-11-29 | score: 369.874 (thresh: 80.285) -- 'pr' mm t: 0.000 [0.051, 21.935, 36.827]
TA00076 2022-11-29 | score: 144.803 (thresh: 114.899) -- 'pr' mm t: 21.935 [0.0, 0.051, 36.827]
TA00310 2022-11-28 | score: 80.930 (thresh: 63.717) -- 'pr' mm t: 0.000 [3.808, 8.592]
TA00323 2022-11-29 | score: 172.668 (thresh: 150.073) -- 'pr' mm t: 0.017 [8.894, 10.046, 11.196]
TA00329 2022-11-29 | score: 228.403 (thresh: 186.163) -- 'pr' mm t: 0.000 [0.017, 4.676, 8.894, 10.046, 11.196]
TA00366 2022-11-29 | score: 663.414 (thresh: 61.132) -- 'pr' mm t: 0.000 [24.514]
TA00451 2022-11-26 | score: 159.022 (thresh: 100.920) -- 'pr' mm t: 0.000 [1.872, 12.28]
TA00592 2022-11-29 | score: 226.496 (thresh: 90.635) -- 'pr' mm t: 0.068 [23.297]
--------
Total 'failure' count: 86
Total record count: 864
Job history table record count: 18057
Scoring job record table record count: 81
-----------------------------------------------------------