July 2022 - Rainqc-jobman - Engineering Mailing Lists

low/no data station impact
by Slater, Michael 14 Jul '22

14 Jul '22

I've added a simple count of the impacted models for low/no data weather stations. It'll add a line similar to the below to the log/daily message. This is a list of tuples, in order of highest impact to least, where each tuple is composed of a problematic weather station and the count of models it impacts (as either the target or a model or a neighbor in a model). 2022-07-14 01:53:46.222 | INFO : LOW/NO data station impact on models: [('TA00568', 9), ('TA00078', 6), ('TA00069', 6), ('TA00677', 5), ('TA00283', 5), ('TA00126', 5), ('TA00652', 4), ('TA00586', 4), ('TA00335', 4), ('TA00537', 4), ('TA00044', 3), ('TA00041', 3), ('TA00043', 3), ('TA00050', 3), ('TA00178', 3), ('TA00091', 3), ('TA00180', 3), ('TA00530', 3), ('TA00533', 3), ('TA00655', 3), ('TA00276', 2), ('TA00308', 2), ('TA00171', 2), ('TA00257', 2), ('TA00155', 2), ('TA00203', 2), ('TA00262', 2), ('TA00265', 2), ('TA00307', 2), ('TA00471', 2), ('TA00466', 2), ('TA00547', 2), ('TA00664', 2), ('TA00014', 1), ('TA00031', 1), ('TA00039', 1), ('TA00092', 1), ('TA00128', 1), ('TA00254', 1), ('TA00219', 1), ('TA00236', 1), ('TA00277', 1), ('TA00336', 1), ('TA00351', 1), ('TA00322', 1), ('TA00362', 1), ('TA00333', 1), ('TA00382', 1), ('TA00383', 1), ('TA00639', 1), ('TA00702', 1)] For a particular problematic station, say TA00568, we can inspect the log to see its impact (as both a target and a neighbor): 2022-07-14 01:53:46.069 | INFO : TA00568 is LOW DATA with non-null data percentage: 47.22% 2022-07-14 01:53:46.150 | INFO : Model for target TA00173 (k=3) is data incomplete as NEIGHBOR(s) {'TA00568'} is/are NO/LOW DATA. 2022-07-14 01:53:46.151 | INFO : Model for target TA00183 (k=5) is data incomplete as NEIGHBOR(s) {'TA00568'} is/are NO/LOW DATA. 2022-07-14 01:53:46.151 | INFO : Model for target TA00184 (k=5) is data incomplete as NEIGHBOR(s) {'TA00568'} is/are NO/LOW DATA. 2022-07-14 01:53:46.152 | INFO : Model for target TA00185 (k=5) is data incomplete as NEIGHBOR(s) {'TA00568'} is/are NO/LOW DATA. 2022-07-14 01:53:46.199 | INFO : Model for target TA00568 is data incomplete as TARGET is LOW DATA. 2022-07-14 01:53:46.202 | INFO : Model for target TA00569 (k=5) is data incomplete as NEIGHBOR(s) {'TA00568'} is/are NO/LOW DATA. 2022-07-14 01:53:46.203 | INFO : Model for target TA00570 (k=5) is data incomplete as NEIGHBOR(s) {'TA00568'} is/are NO/LOW DATA. 2022-07-14 01:53:46.217 | INFO : Model for target TA00674 (k=5) is data incomplete as NEIGHBOR(s) {'TA00568'} is/are NO/LOW DATA. 2022-07-14 01:53:46.219 | INFO : Model for target TA00680 (k=5) is data incomplete as NEIGHBOR(s) {'TA00568'} is/are NO/LOW DATA. We could distill the logging output into something more concise if we wanted to put more information into the daily status message regarding the problematic weather stations and their impact on scoring. --ms

1 0

RainQC Job Manager daily report 2022-07-14
by slater＠oregonstate.edu 14 Jul '22

14 Jul '22

1 0

RainQC Job Manager daily report 2022-07-13
by slater＠oregonstate.edu 13 Jul '22

13 Jul '22

1 0

RainQC Job Manager daily report 2022-07-13
by slater＠oregonstate.edu 13 Jul '22

13 Jul '22

1 0

RainQC Job Manager daily report 2022-07-13
by slater＠oregonstate.edu 13 Jul '22

13 Jul '22

1 0

RainQC Job Manager daily report 2022-07-12
by slater＠oregonstate.edu 12 Jul '22

12 Jul '22

1 0

RainQC Job Manager daily report 2022-07-11
by slater＠oregonstate.edu 11 Jul '22

11 Jul '22

1 0

RainQC Job Manager daily report 2022-07-11
by slater＠oregonstate.edu 11 Jul '22

11 Jul '22

1 0

RainQC Job Manager daily report 2022-07-11
by slater＠oregonstate.edu 11 Jul '22

11 Jul '22

1 0

RainQC Job Manager daily report 2022-07-10
by slater＠oregonstate.edu 10 Jul '22

10 Jul '22

1 0