January 2023 - Rainqc-jobman - Engineering Mailing Lists

Action Items and Next Steps for the RainQC system
by Dietterich, Thomas 12 Jan '23

12 Jan '23

We had a short meeting today. Our next meeting is on 25 January; Tema will lead the meeting. Here is a list of our current issues and action items: 1. CloudFoundry. IBM is terminating support for CloudFoundry. Rick, what will replace CloudFoundry and what do we need to change in our system? 1. Ground Truth Clogging Dataset. We want to create a labeled data set that assigns a label of "good", "clogged", or "other-failure" to each rain gauge on a daily basis. We will use this dataset to develop and test new QC methods for precipitation * Rick: can you export the quality objects in json format for the period from January 2017 through October 2022? * Austin and Victoria will analyze the exported JSON for each station and day. For each block of time where a station has been flagged, they will look at the maximum precipitation that was reported. If this is less than epsilon (for some threshold, maybe 5mm?) for the entire block of time, they will mark the rain gauge as "clogged" during that time. If the maximum reported precipitation is greater than epsilon, they will mark the rain gauge as "other failure". Finally, for all days where the QC object is ok, they will mark the rain gauge as "good". The output should be a CSV file with columns for station id, sensor id (to handle stations with multiple rain gauges), date, reported precipitation, and the assigned class label {good, clogged, other-failure}. JSON format would be fine also; whichever you prefer. I assume we define a day as running from 0:00-23:59:59 GMT. 1. Hand-off of the job manager. Michael will meet with Austin and Tema to finalize the hand-off of responsibility for monitoring the job manager. This will be delayed until the floods in California are over. 1. Collective Classification. Our current approach flags each station individually. A flaw in this approach is that if a station is bad, we tend to flag not only the bad station but also many of its neighbors. The idea of collective classification is to jointly analyze the stations. We start by obtaining the anomaly scores for all of the stations computed in the usual way. Let Open := the set of all stations for which we have scores and Done := {} * Repeat i. Let s be the station in Open with the highest anomaly score. ii. If the score is below the anomaly threshold then mark all remaining stations in Open as "ok" and exit iii. Else Flag s as "broken", remove it from Open, and add it to Done iv. Recompute the anomaly scores of all neighbors of s (i.e., all stations that use s as a neighbor). In the current system, we have no way of doing this, so we can just flag all of them as "ok", remove them from Open and add them to Done. * The action item is to implement collective classification. This is not currently assigned to anyone. Any volunteers to implement this? 1. GP Clogging Model. Implement and test the algorithm proposed by Ciira and Tom for detecting clogged sensors. This item is assigned to Ciira and Tom. 1. Multi-day Detection. Implement and test a version of the current algorithm that sums the precipitation for K consecutive days. We suspect that K of 5-8 days might give much more accurate results. This will involve modifying the training code and the scoring code. Every day, this code will score the sum of the precipitation for the K most recent days. Unfortunately, this will also increase the problems caused by missing data, because if any data is missing from any of those K days (for the target station or any of its neighbors), then we will not be able to score the station. This item is not assigned to anyone. Any volunteers to implement this? I will be traveling in India from January 22 through February 19. I will therefore miss the next two meetings (25 January and 8 February). Tema will run the meetings in my absence. --Tom Thomas G. Dietterich, Distinguished Professor Voice: 541-737-5559 School of Electrical Engineering FAX: 541-737-1300 and Computer Science URL: eecs.oregonstate.edu/~tgd US Mail: 1148 Kelley Engineering Center Office: 2067 Kelley Engineering Center Oregon State Univ., Corvallis, OR 97331-5501

2 2

january 2023 / first 10 days
by Slater, Michael 12 Jan '23

12 Jan '23

Here's the RainQC anomalies 10 days into the new year: Weekly Anomalies (flag=2) for January 2023: -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00028 2023-01-03 | score: 64.358 (thresh: 4.314) -- 'pr' t: 2.125 mm n: (0.901 mm, 27 km), (8.964 mm, 54 km) TA00028 2023-01-06 | score: 4.401 (thresh: 4.314) -- 'pr' t: 0.000 mm n: (0.0 mm, 27 km), (6.071 mm, 54 km) TA00028 2023-01-08 | score: 111.119 (thresh: 4.314) -- 'pr' t: 31.798 mm n: (0.884 mm, 27 km), (0.136 mm, 54 km) TA00028 2023-01-09 | score: 95.561 (thresh: 4.314) -- 'pr' t: 8.024 mm n: (1.53 mm, 27 km), (26.054 mm, 54 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00061 2023-01-11 | score: 131.624 (thresh: 80.285) -- 'pr' t: 0.000 mm n: (0.034 mm, 23 km), (0.085 mm, 34 km), (11.017 mm, 35 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00073 2023-01-06 | score: 76.961 (thresh: 62.936) -- 'pr' t: 0.000 mm n: (6.071 mm, 37 km), (0.0 mm, 57 km), (0.0 mm, 58 km), (0.068 mm, 61 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00076 2023-01-08 | score: 173.446 (thresh: 114.899) -- 'pr' t: 0.000 mm n: (7.587 mm, 12 km), (0.561 mm, 24 km), (5.677 mm, 35 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00171 2023-01-09 | score: 143.889 (thresh: 137.760) -- 'pr' t: 0.136 mm n: (18.445 mm, 26 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00172 2023-01-03 | score: 222.657 (thresh: 130.022) -- 'pr' t: 18.714 mm n: (15.589 mm, 48 km), (30.7 mm, 75 km), (0.901 mm, 87 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00179 2023-01-01 | score: 370.661 (thresh: 154.777) -- 'pr' t: 0.000 mm n: (0.833 mm, 13 km), (0.051 mm, 16 km), (30.692 mm, 19 km) TA00179 2023-01-05 | score: 216.129 (thresh: 154.777) -- 'pr' t: 14.800 mm n: (3.045 mm, 13 km), (22.129 mm, 16 km), (17.33 mm, 19 km) TA00179 2023-01-06 | score: 167.713 (thresh: 154.777) -- 'pr' t: 0.000 mm n: (10.65 mm, 13 km), (2.958 mm, 16 km), (0.493 mm, 19 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00190 2023-01-03 | score: 497.670 (thresh: 181.186) -- 'pr' t: 0.408 mm n: (19.788 mm, 8 km), (12.421 mm, 26 km), (30.7 mm, 31 km), (0.731 mm, 38 km), (15.441 mm, 39 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00210 2023-01-04 | score: 100.740 (thresh: 76.739) -- 'pr' t: 0.000 mm n: (5.465 mm, 76 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00272 2023-01-08 | score: 126.407 (thresh: 86.379) -- 'pr' t: 0.000 mm n: (7.23 mm, 15 km) TA00272 2023-01-09 | score: 97.570 (thresh: 86.379) -- 'pr' t: 9.800 mm n: (23.229 mm, 15 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00323 2023-01-03 | score: 631.569 (thresh: 150.073) -- 'pr' t: 2.805 mm n: (21.101 mm, 21 km), (0.272 mm, 45 km), (70.471 mm, 104 km) TA00323 2023-01-08 | score: 170.500 (thresh: 150.073) -- 'pr' t: 20.043 mm n: (21.911 mm, 21 km), (31.691 mm, 45 km), (1.02 mm, 104 km) TA00323 2023-01-09 | score: 2986.136 (thresh: 150.073) -- 'pr' t: 6.328 mm n: (105.19 mm, 21 km), (7.499 mm, 45 km), (6.701 mm, 104 km) TA00323 2023-01-10 | score: 1048.503 (thresh: 150.073) -- 'pr' t: 0.374 mm n: (61.182 mm, 21 km), (0.374 mm, 45 km), (0.119 mm, 104 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00327 2023-01-01 | score: 61.393 (thresh: 60.333) -- 'pr' t: 0.000 mm n: (0.238 mm, 76 km), (11.337 mm, 91 km), (1.785 mm, 102 km), (1.225 mm, 104 km) TA00327 2023-01-08 | score: 292.294 (thresh: 60.333) -- 'pr' t: 1.020 mm n: (31.691 mm, 76 km), (21.911 mm, 91 km), (18.859 mm, 102 km), (20.043 mm, 104 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00329 2023-01-07 | score: 272.060 (thresh: 186.163) -- 'pr' t: 0.119 mm n: (11.879 mm, 15 km), (44.32 mm, 25 km), (15.509 mm, 37 km), (11.838 mm, 59 km), (3.997 mm, 79 km) TA00329 2023-01-08 | score: 450.844 (thresh: 186.163) -- 'pr' t: 0.187 mm n: (31.691 mm, 15 km), (18.859 mm, 25 km), (21.911 mm, 37 km), (20.043 mm, 59 km), (1.02 mm, 79 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00332 2023-01-09 | score: 488.983 (thresh: 302.184) -- 'pr' t: 9.691 mm n: (0.119 mm, 25 km), (7.499 mm, 26 km), (105.19 mm, 31 km), (6.701 mm, 102 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00361 2023-01-08 | score: 1087.403 (thresh: 188.050) -- 'pr' t: 1.904 mm n: (65.62 mm, 37 km), (16.917 mm, 73 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00366 2023-01-01 | score: 617.459 (thresh: 61.132) -- 'pr' t: 0.000 mm n: (23.273 mm, 99 km) TA00366 2023-01-02 | score: 569.585 (thresh: 61.132) -- 'pr' t: 0.000 mm n: (21.93 mm, 99 km) TA00366 2023-01-03 | score: 663.414 (thresh: 61.132) -- 'pr' t: 0.000 mm n: (24.514 mm, 99 km) TA00366 2023-01-04 | score: 663.414 (thresh: 61.132) -- 'pr' t: 0.000 mm n: (24.514 mm, 99 km) TA00366 2023-01-05 | score: 663.414 (thresh: 61.132) -- 'pr' t: 0.000 mm n: (24.514 mm, 99 km) TA00366 2023-01-06 | score: 663.414 (thresh: 61.132) -- 'pr' t: 0.000 mm n: (24.514 mm, 99 km) TA00366 2023-01-07 | score: 663.414 (thresh: 61.132) -- 'pr' t: 0.000 mm n: (24.514 mm, 99 km) TA00366 2023-01-08 | score: 663.414 (thresh: 61.132) -- 'pr' t: 0.000 mm n: (24.514 mm, 99 km) TA00366 2023-01-09 | score: 663.414 (thresh: 61.132) -- 'pr' t: 0.000 mm n: (24.514 mm, 99 km) TA00366 2023-01-10 | score: 663.414 (thresh: 61.132) -- 'pr' t: 0.000 mm n: (24.514 mm, 99 km) TA00366 2023-01-11 | score: 663.414 (thresh: 61.132) -- 'pr' t: 0.000 mm n: (24.514 mm, 99 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00368 2023-01-02 | score: 412.123 (thresh: 177.364) -- 'pr' t: 0.034 mm n: (29.701 mm, 51 km), (0.255 mm, 73 km) TA00368 2023-01-08 | score: 1159.429 (thresh: 177.364) -- 'pr' t: 16.917 mm n: (65.62 mm, 51 km), (1.904 mm, 73 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00373 2023-01-05 | score: 197.340 (thresh: 123.608) -- 'pr' t: 0.255 mm n: (14.8 mm, 13 km), (27.8 mm, 14 km), (17.33 mm, 18 km), (3.045 mm, 24 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00451 2023-01-08 | score: 179.221 (thresh: 100.920) -- 'pr' t: 0.000 mm n: (0.0 mm, 22 km), (15.893 mm, 40 km) TA00451 2023-01-10 | score: 133.592 (thresh: 100.920) -- 'pr' t: 0.170 mm n: (1.514 mm, 22 km), (28.678 mm, 40 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00592 2023-01-07 | score: 190.074 (thresh: 90.635) -- 'pr' t: 0.000 mm n: (8.587 mm, 80 km) TA00592 2023-01-08 | score: 183.061 (thresh: 90.635) -- 'pr' t: 0.000 mm n: (8.102 mm, 80 km) TA00592 2023-01-09 | score: 218.614 (thresh: 90.635) -- 'pr' t: 0.000 mm n: (10.474 mm, 80 km) TA00592 2023-01-10 | score: 597.275 (thresh: 90.635) -- 'pr' t: 0.000 mm n: (28.711 mm, 80 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00607 2023-01-03 | score: 982.514 (thresh: 51.001) -- 'pr' t: 0.000 mm n: (45.679 mm, 35 km) TA00607 2023-01-06 | score: 301.404 (thresh: 51.001) -- 'pr' t: 0.034 mm n: (30.345 mm, 35 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00641 2023-01-09 | score: 343.759 (thresh: 310.876) -- 'pr' t: 0.034 mm n: (31.338 mm, 60 km), (3.791 mm, 95 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00679 2023-01-03 | score: 132.628 (thresh: 107.015) -- 'pr' t: 0.493 mm n: (17.885 mm, 5 km), (7.956 mm, 7 km), (2.346 mm, 11 km), (19.051 mm, 15 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00719 2023-01-03 | score: 126.212 (thresh: 100.242) -- 'pr' t: 19.788 mm n: (12.421 mm, 25 km), (30.7 mm, 29 km), (14.512 mm, 42 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- Monthly Anomaly Count: 51

1 0

RainQC Job Manager daily report 2023-01-12
by slater＠oregonstate.edu 12 Jan '23

12 Jan '23

1 0

RainQC Job Manager daily report 2023-01-11
by slater＠oregonstate.edu 11 Jan '23

11 Jan '23

1 0

RainQC Job Manager daily report 2023-01-10
by slater＠oregonstate.edu 10 Jan '23

10 Jan '23

1 0

RainQC Job Manager daily report 2023-01-09
by slater＠oregonstate.edu 09 Jan '23

09 Jan '23

1 0

RainQC Job Manager daily report 2023-01-08
by slater＠oregonstate.edu 08 Jan '23

08 Jan '23

1 0

RainQC Job Manager daily report 2023-01-07
by slater＠oregonstate.edu 07 Jan '23

07 Jan '23

1 0

RainQC Job Manager daily report 2023-01-06
by slater＠oregonstate.edu 06 Jan '23

06 Jan '23

1 0

RainQC Job Manager daily report 2023-01-05
by slater＠oregonstate.edu 05 Jan '23

05 Jan '23

1 0