Here are the flag=2 anomalies for February 2024. Only 73 anomalies compared to the 153 total anomalies of January 2024. Data completeness wasn't very good for February, but more importantly, the RainQC API success rate seems to be low. There were quite a few days where the model count at 100% data completeness was significantly higher than the number of stations we were able to score via RainQC. There seems to be a higher than usual number of scoring retries that ultimately end in failure due to HTTP error 502, bad gateway. For instance, from tonight's jobs: Giving up rainqc_score_request(...) after 5 tries (requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://sensordx.tahmo.org/api/score/TA00586?utcStartDate=2024-03-04+00%3A00%3A00&utcEndDate=2024-03-04+23%3A59%3A59&save=True&variable=precipitation&id=6379e7c7a70bcec7dc859f13<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsensordx.tahmo.org%2Fapi%2Fscore%2FTA00586%3FutcStartDate%3D2024-03-04%2B00%253A00%253A00%26utcEndDate%3D2024-03-04%2B23%253A59%253A59%26save%3DTrue%26variable%3Dprecipitation%26id%3D6379e7c7a70bcec7dc859f13&data=05%7C02%7Crainqc-jobman%40engr.oregonstate.edu%7Cf009c11f5a8d4dd5a69f08dc3db3d00e%7Cce6d05e13c5e4d6287a84c4a2713c113%7C0%7C0%7C638453089689573575%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=R%2Bsere%2FEkoYsPc5MNwB5pZVeCxImkySIJQ%2BAmV4%2FvDg%3D&reserved=0>) https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/502<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdeveloper.mozilla.org%2Fen-US%2Fdocs%2FWeb%2FHTTP%2FStatus%2F502&data=05%7C02%7Crainqc-jobman%40engr.oregonstate.edu%7Cf009c11f5a8d4dd5a69f08dc3db3d00e%7Cce6d05e13c5e4d6287a84c4a2713c113%7C0%7C0%7C638453089689573575%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=PncXWG8mWKwpl9wLlsVRtRPds6aTgNq33DNCAqHvFxU%3D&reserved=0> Keep in mind, there is backoff in place, so increasing randomized delays after retries. While the number of 502 errors was high, it was still lower than the difference between the model count at 100% data completeness and how many models we were able to score via RainQC. We might be seeing the combined effects of API failures between the Job Manager and RainQC and API failures between RainQC and various TAHMO APIs, e.g., without retry logic in RainQC there will be cases where a model/station will be data incomplete due to API failure, not due to actual data completeness. ---- Anomalies (flag=2) for February 2024: -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00004 2024-02-13 | score: 121.990 (thresh: 110.285) -- 'pr' t: 0.000 mm n: (5.251 mm, 25 km), (0.187 mm, 25 km) TA00004 2024-02-14 | score: 338.149 (thresh: 110.285) -- 'pr' t: 0.000 mm n: (24.979 mm, 25 km), (2.212 mm, 25 km) TA00004 2024-02-16 | score: 120.711 (thresh: 110.285) -- 'pr' t: 0.000 mm n: (0.0 mm, 25 km), (5.238 mm, 25 km) TA00004 2024-02-24 | score: 472.160 (thresh: 110.285) -- 'pr' t: 0.000 mm n: (36.966 mm, 25 km), (0.136 mm, 25 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00005 2024-02-17 | score: 261.131 (thresh: 123.300) -- 'pr' t: 0.000 mm n: (0.408 mm, 25 km), (0.748 mm, 37 km), (21.574 mm, 44 km) TA00005 2024-02-21 | score: 137.910 (thresh: 123.300) -- 'pr' t: 0.000 mm n: (0.0 mm, 25 km), (0.0 mm, 37 km), (10.187 mm, 44 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00025 2024-02-27 | score: 1065.078 (thresh: 221.074) -- 'pr' t: 34.757 mm n: (62.08 mm, 5 km), (14.246 mm, 12 km), (1.173 mm, 17 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00026 2024-02-13 | score: 469.140 (thresh: 78.247) -- 'pr' t: 0.000 mm n: (30.836 mm, 34 km), (0.0 mm, 49 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00028 2024-02-05 | score: 27.403 (thresh: 4.314) -- 'pr' t: 0.153 mm n: (0.0 mm, 27 km), (6.024 mm, 54 km) TA00028 2024-02-08 | score: 118.147 (thresh: 4.314) -- 'pr' t: 28.632 mm n: (4.969 mm, 27 km), (0.136 mm, 54 km) TA00028 2024-02-09 | score: 48.669 (thresh: 4.314) -- 'pr' t: 0.136 mm n: (18.712 mm, 27 km), (0.068 mm, 54 km) TA00028 2024-02-10 | score: 41.265 (thresh: 4.314) -- 'pr' t: 0.119 mm n: (13.029 mm, 27 km), (8.828 mm, 54 km) TA00028 2024-02-13 | score: 30.052 (thresh: 4.314) -- 'pr' t: 0.153 mm n: (0.0 mm, 27 km), (17.829 mm, 54 km) TA00028 2024-02-27 | score: 106.790 (thresh: 4.314) -- 'pr' t: 14.106 mm n: (6.614 mm, 27 km), (0.034 mm, 54 km) TA00028 2024-02-28 | score: 147.452 (thresh: 4.314) -- 'pr' t: 10.876 mm n: (22.34 mm, 27 km), (19.223 mm, 54 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00042 2024-02-28 | score: 241.317 (thresh: 54.610) -- 'pr' t: 0.000 mm n: (17.992 mm, 117 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00045 2024-02-15 | score: 758.507 (thresh: 90.312) -- 'pr' t: 0.000 mm n: (0.0 mm, 35 km), (45.905 mm, 49 km) TA00045 2024-02-17 | score: 107.168 (thresh: 90.312) -- 'pr' t: 0.000 mm n: (0.306 mm, 35 km), (6.836 mm, 49 km) TA00045 2024-02-23 | score: 552.309 (thresh: 90.312) -- 'pr' t: 0.000 mm n: (0.0 mm, 35 km), (36.756 mm, 49 km) TA00045 2024-02-24 | score: 634.252 (thresh: 90.312) -- 'pr' t: 0.000 mm n: (1.105 mm, 35 km), (40.06 mm, 49 km) TA00045 2024-02-28 | score: 134.625 (thresh: 90.312) -- 'pr' t: 0.000 mm n: (0.0 mm, 35 km), (9.809 mm, 49 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00046 2024-02-28 | score: 108.457 (thresh: 83.136) -- 'pr' t: 39.617 mm n: (17.992 mm, 13 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00047 2024-02-21 | score: 165.352 (thresh: 61.920) -- 'pr' t: 0.034 mm n: (27.336 mm, 117 km) TA00047 2024-02-24 | score: 95.630 (thresh: 61.920) -- 'pr' t: 0.000 mm n: (6.085 mm, 117 km) TA00047 2024-02-29 | score: 408.265 (thresh: 61.920) -- 'pr' t: 0.017 mm n: (43.209 mm, 117 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00064 2024-02-27 | score: 355.274 (thresh: 232.296) -- 'pr' t: 0.034 mm n: (8.041 mm, 21 km), (35.646 mm, 28 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00065 2024-02-26 | score: 1288.896 (thresh: 256.913) -- 'pr' t: 12.224 mm n: (62.133 mm, 31 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00073 2024-02-05 | score: 79.705 (thresh: 62.936) -- 'pr' t: 0.000 mm n: (6.024 mm, 37 km), (0.0 mm, 57 km), (0.714 mm, 58 km), (0.0 mm, 61 km) TA00073 2024-02-13 | score: 143.018 (thresh: 62.936) -- 'pr' t: 0.000 mm n: (17.829 mm, 37 km), (0.0 mm, 57 km), (0.0 mm, 58 km), (0.068 mm, 61 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00120 2024-02-15 | score: 42.251 (thresh: 35.716) -- 'pr' t: 25.338 mm n: (0.0 mm, 75 km) TA00120 2024-02-22 | score: 53.881 (thresh: 35.716) -- 'pr' t: 63.650 mm n: (0.204 mm, 75 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00124 2024-02-17 | score: 130.957 (thresh: 83.974) -- 'pr' t: 6.390 mm n: (38.948 mm, 49 km) TA00124 2024-02-29 | score: 120.070 (thresh: 83.974) -- 'pr' t: 0.017 mm n: (22.357 mm, 49 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00178 2024-02-13 | score: 104.370 (thresh: 102.337) -- 'pr' t: 0.000 mm n: (5.219 mm, 26 km), (0.136 mm, 31 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00187 2024-02-27 | score: 153.721 (thresh: 103.141) -- 'pr' t: 14.408 mm n: (39.854 mm, 26 km), (0.136 mm, 79 km), (17.864 mm, 81 km), (1.921 mm, 88 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00242 2024-02-12 | score: 438.420 (thresh: 205.965) -- 'pr' t: 0.017 mm n: (23.4 mm, 19 km), (16.336 mm, 23 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00272 2024-02-29 | score: 172.781 (thresh: 86.379) -- 'pr' t: 0.000 mm n: (10.234 mm, 15 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00281 2024-02-27 | score: 1771.301 (thresh: 23.820) -- 'pr' t: 5.168 mm n: (117.495 mm, 89 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00303 2024-02-29 | score: 92.893 (thresh: 84.602) -- 'pr' t: 5.421 mm n: (4.896 mm, 30 km), (37.148 mm, 37 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00310 2024-02-17 | score: 309.678 (thresh: 63.717) -- 'pr' t: 0.000 mm n: (38.948 mm, 63 km), (6.39 mm, 67 km) TA00310 2024-02-28 | score: 75.210 (thresh: 63.717) -- 'pr' t: 0.068 mm n: (18.212 mm, 63 km), (18.35 mm, 67 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00321 2024-02-06 | score: 864.514 (thresh: 281.327) -- 'pr' t: 5.769 mm n: (55.132 mm, 15 km), (3.893 mm, 24 km), (32.612 mm, 26 km), (2.604 mm, 45 km), (0.918 mm, 76 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00323 2024-02-02 | score: 220.752 (thresh: 150.073) -- 'pr' t: 0.000 mm n: (10.404 mm, 21 km), (0.34 mm, 45 km), (1.649 mm, 104 km) TA00323 2024-02-26 | score: 211.614 (thresh: 150.073) -- 'pr' t: 10.522 mm n: (13.742 mm, 21 km), (30.541 mm, 45 km), (24.877 mm, 104 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00327 2024-02-05 | score: 107.743 (thresh: 60.333) -- 'pr' t: 0.000 mm n: (6.612 mm, 76 km), (0.0 mm, 91 km), (18.243 mm, 102 km), (0.0 mm, 104 km) TA00327 2024-02-11 | score: 113.780 (thresh: 60.333) -- 'pr' t: 0.000 mm n: (6.722 mm, 76 km), (4.168 mm, 91 km), (12.663 mm, 102 km), (1.496 mm, 104 km) TA00327 2024-02-12 | score: 134.231 (thresh: 60.333) -- 'pr' t: 0.000 mm n: (12.694 mm, 76 km), (5.814 mm, 91 km), (0.238 mm, 102 km), (0.51 mm, 104 km) TA00327 2024-02-16 | score: 68.449 (thresh: 60.333) -- 'pr' t: 0.000 mm n: (0.0 mm, 76 km), (0.0 mm, 91 km), (20.336 mm, 102 km), (0.0 mm, 104 km) TA00327 2024-02-24 | score: 157.531 (thresh: 60.333) -- 'pr' t: 0.000 mm n: (0.0 mm, 76 km), (15.25 mm, 91 km), (4.032 mm, 102 km), (14.476 mm, 104 km) TA00327 2024-02-25 | score: 126.034 (thresh: 60.333) -- 'pr' t: 0.000 mm n: (0.731 mm, 76 km), (0.0 mm, 91 km), (5.969 mm, 102 km), (10.562 mm, 104 km) TA00327 2024-02-26 | score: 175.167 (thresh: 60.333) -- 'pr' t: 24.877 mm n: (30.541 mm, 76 km), (13.742 mm, 91 km), (68.088 mm, 102 km), (10.522 mm, 104 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00329 2024-02-26 | score: 736.562 (thresh: 186.163) -- 'pr' t: 1.156 mm n: (30.541 mm, 15 km), (68.088 mm, 25 km), (13.742 mm, 37 km), (10.522 mm, 59 km), (24.877 mm, 79 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00354 2024-02-28 | score: 57.170 (thresh: 12.399) -- 'pr' t: 0.357 mm n: (21.267 mm, 39 km), (5.289 mm, 76 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00367 2024-02-11 | score: 518.896 (thresh: 267.378) -- 'pr' t: 34.332 mm n: (74.418 mm, 37 km), (0.187 mm, 51 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00368 2024-02-11 | score: 2121.180 (thresh: 177.364) -- 'pr' t: 0.187 mm n: (34.332 mm, 51 km), (74.418 mm, 73 km) TA00368 2024-02-27 | score: 331.297 (thresh: 177.364) -- 'pr' t: 0.034 mm n: (12.736 mm, 51 km), (21.248 mm, 73 km) TA00368 2024-02-29 | score: 819.593 (thresh: 177.364) -- 'pr' t: 0.051 mm n: (31.004 mm, 51 km), (24.113 mm, 73 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00391 2024-02-22 | score: 259.797 (thresh: 112.951) -- 'pr' t: 0.000 mm n: (11.289 mm, 11 km), (10.765 mm, 12 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00470 2024-02-18 | score: 78.760 (thresh: 74.917) -- 'pr' t: 6.361 mm n: (4.4 mm, 16 km), (26.189 mm, 26 km), (1.853 mm, 47 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00586 2024-02-10 | score: 300.454 (thresh: 192.496) -- 'pr' t: 3.433 mm n: (33.241 mm, 9 km), (31.25 mm, 10 km), (9.049 mm, 79 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00588 2024-02-19 | score: 97.299 (thresh: 85.418) -- 'pr' t: 0.000 mm n: (0.0 mm, 13 km), (7.311 mm, 43 km) TA00588 2024-02-23 | score: 312.406 (thresh: 85.418) -- 'pr' t: 47.981 mm n: (34.682 mm, 13 km), (37.565 mm, 43 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00594 2024-02-02 | score: 149.715 (thresh: 105.729) -- 'pr' t: 0.000 mm n: (13.804 mm, 80 km) TA00594 2024-02-03 | score: 106.245 (thresh: 105.729) -- 'pr' t: 0.000 mm n: (10.343 mm, 80 km) TA00594 2024-02-09 | score: 160.948 (thresh: 105.729) -- 'pr' t: 0.000 mm n: (14.613 mm, 80 km) TA00594 2024-02-11 | score: 136.465 (thresh: 105.729) -- 'pr' t: 0.255 mm n: (26.889 mm, 80 km) TA00594 2024-02-26 | score: 114.768 (thresh: 105.729) -- 'pr' t: 0.391 mm n: (25.917 mm, 80 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00602 2024-02-07 | score: 139.822 (thresh: 54.427) -- 'pr' t: 0.119 mm n: (18.513 mm, 69 km) TA00602 2024-02-11 | score: 588.433 (thresh: 54.427) -- 'pr' t: 15.989 mm n: (46.216 mm, 69 km) TA00602 2024-02-24 | score: 129.338 (thresh: 54.427) -- 'pr' t: 8.373 mm n: (24.588 mm, 69 km) TA00602 2024-02-27 | score: 398.321 (thresh: 54.427) -- 'pr' t: 27.473 mm n: (40.209 mm, 69 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00607 2024-02-01 | score: 63.080 (thresh: 51.001) -- 'pr' t: 0.085 mm n: (14.349 mm, 35 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- TA00685 2024-02-23 | score: 669.945 (thresh: 154.195) -- 'pr' t: 1.020 mm n: (37.565 mm, 36 km), (34.682 mm, 66 km) -------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------- Monthly Anomaly Count: 73