December 2023 - Rainqc-jobman - Engineering Mailing Lists

RainQC Job Manager daily report 2023-12-31
by RainQC Job Manager 31 Dec '23

31 Dec '23

1 0

RainQC Job Manager daily report 2023-12-30
by RainQC Job Manager 30 Dec '23

30 Dec '23

1 0

RainQC Job Manager daily report 2023-12-29
by RainQC Job Manager 29 Dec '23

29 Dec '23

1 0

RainQC Job Manager daily report 2023-12-28
by RainQC Job Manager 28 Dec '23

28 Dec '23

1 0

RainQC Job Manager daily report 2023-12-27
by RainQC Job Manager 27 Dec '23

27 Dec '23

1 0

RainQC Job Manager daily report 2023-12-26
by RainQC Job Manager 26 Dec '23

26 Dec '23

1 0

RainQC Job Manager daily report 2023-12-25
by RainQC Job Manager 25 Dec '23

25 Dec '23

1 0

RainQC Job Manager daily report 2023-12-24
by RainQC Job Manager 24 Dec '23

24 Dec '23

1 0

Re: [Rainqc-jobman] Missing RainQC Job Manager daily report 2023-12-22
by Slater, Michael 23 Dec '23

23 Dec '23

There isn't a normal daily report for today. The job manager encountered an unusual exception, possibly due to concurrent access to the job database that interfered with the currently active transaction in the job manager (the same exception that was encountered ~10 days ago). I was able to manually process the active jobs, so whatever the problem was, it appears to have been transient. There were 7 reported anomalies for the jobs today. 2023-12-22 23:34:06.300 | INFO : --- [1055/1255] --- getting model | name: TA00493, id: 6379ac2ba70bcec7dc859edc, start: 2023-12-22, end: 2023-12-22 2023-12-22 23:34:29.115 | ERROR : Exception in process_jobs_for_day: current transaction is aborted, commands ignored until end of transaction block Here's the data completeness info from the log file and the status of the active job table after manual processing. 2023-12-22 22:50:57.895 | INFO : Model completeness: 2023-12-22 22:50:57.897 | INFO : data completeness 50% | complete models: 131 of 273 (47.99%) 2023-12-22 22:50:57.906 | INFO : data completeness 60% | complete models: 131 of 273 (47.99%) 2023-12-22 22:50:57.907 | INFO : data completeness 70% | complete models: 131 of 273 (47.99%) 2023-12-22 22:50:57.908 | INFO : data completeness 75% | complete models: 131 of 273 (47.99%) 2023-12-22 22:50:57.909 | INFO : data completeness 80% | complete models: 128 of 273 (46.89%) 2023-12-22 22:50:57.909 | INFO : data completeness 85% | complete models: 127 of 273 (46.52%) 2023-12-22 22:50:57.910 | INFO : data completeness 90% | complete models: 120 of 273 (43.96%) 2023-12-22 22:50:57.911 | INFO : data completeness 95% | complete models: 120 of 273 (43.96%) 2023-12-22 22:50:57.911 | INFO : data completeness 100% | complete models: 120 of 273 (43.96%) LOW/NO data station impact on models: [('TA00127', 8), ('TA00568', 8), ('TA00196', 7), ('TA00061', 7), ('TA00719', 7), ('TA00185', 7), ('TA00124', 6), ('TA00414', 6), ('TA00621', 5), ('TA00320', 5), ('TA00587', 5), ('TA00537', 5), ('TA00067', 4), ('TA00231', 4), ('TA00530', 4), ('TA00565', 4), ('TA00543', 4), ('TA00700', 4), ('TA00035', 3), ('TA00036', 3), ('TA00222', 3), ('TA00041', 3), ('TA00267', 3), ('TA00178', 3), ('TA00066', 3), ('TA00115', 3), ('TA00126', 3), ('TA00217', 3), ('TA00482', 3), ('TA00385', 3), ('TA00289', 3), ('TA00436', 3), ('TA00430', 3), ('TA00542', 3), ('TA00308', 2), ('TA00050', 2), ('TA00072', 2), ('TA00118', 2), ('TA00136', 2), ('TA00165', 2), ('TA00148', 2), ('TA00155', 2), ('TA00487', 2), ('TA00210', 2), ('TA00223', 2), ('TA00232', 2), ('TA00262', 2), ('TA00271', 2), ('TA00691', 2), ('TA00339', 2), ('TA00373', 2), ('TA00451', 2), ('TA00462', 2), ('TA00471', 2), ('TA00014', 1), ('TA00031', 1), ('TA00044', 1), ('TA00078', 1), ('TA00091', 1), ('TA00095', 1), ('TA00123', 1), ('TA00212', 1), ('TA00219', 1), ('TA00229', 1), ('TA00237', 1), ('TA00251', 1), ('TA00269', 1), ('TA00290', 1), ('TA00336', 1), ('TA00343', 1), ('TA00344', 1), ('TA00350', 1), ('TA00362', 1), ('TA00369', 1), ('TA00382', 1), ('TA00389', 1), ('TA00422', 1), ('TA00432', 1), ('TA00493', 1), ('TA00528', 1), ('TA00529', 1), ('TA00533', 1), ('TA00535', 1), ('TA00547', 1), ('TA00588', 1), ('TA00639', 1), ('TA00652', 1), ('TA00655', 1), ('TA00677', 1), ('TA00702', 1)] --- Active job table status: ---------------------------------------------------------- Total 'success' count: 114 (flag=2 count: 7) (flag 2->1 downgrades: 2) | 'success' count for 2023-12-22: 107 (flag=2 count: 7) | 'success' count for 2023-12-21: 7 (flag=2 count: 0) | 'success' count for 2023-12-20: 0 (flag=2 count: 0) | 'success' count for 2023-12-19: 0 (flag=2 count: 0) | 'success' count for 2023-12-18: 0 (flag=2 count: 0) | 'success' count for 2023-12-17: 0 (flag=2 count: 0) | 'success' count for 2023-12-16: 0 (flag=2 count: 0) Anomalies (flag=2): TA00028 2023-12-22 | score: 13.836 (thresh: 4.314) -- 'pr' t: 0.034 mm n: (0.0 mm, 27 km), (5.39 mm, 54 km) TA00068 2023-12-22 | score: 271.972 (thresh: 163.651) -- 'pr' t: 44.083 mm n: (34.757 mm, 4 km), (0.765 mm, 11 km), (0.765 mm, 57 km) TA00069 2023-12-22 | score: 840.433 (thresh: 125.323) -- 'pr' t: 34.757 mm n: (44.083 mm, 4 km), (0.765 mm, 8 km), (0.765 mm, 60 km) TA00073 2023-12-22 | score: 73.969 (thresh: 62.936) -- 'pr' t: 0.000 mm n: (5.39 mm, 37 km), (0.0 mm, 57 km), (0.051 mm, 58 km), (0.068 mm, 61 km) TA00172 2023-12-22 | score: 167.928 (thresh: 130.022) -- 'pr' t: 0.000 mm n: (0.051 mm, 48 km), (11.082 mm, 75 km), (0.0 mm, 87 km) TA00327 2023-12-22 | score: 281.580 (thresh: 60.333) -- 'pr' t: 0.000 mm n: (21.398 mm, 76 km), (11.444 mm, 91 km), (6.109 mm, 102 km), (5.546 mm, 104 km) TA00361 2023-12-22 | score: 193.123 (thresh: 188.050) -- 'pr' t: 11.069 mm n: (37.765 mm, 37 km), (2.686 mm, 73 km) -------- Total 'failure' count: 317 Total record count: 1255 Job history table record count: 123590 Scoring job record table record count: 471

1 0

Re: [Rainqc-jobman] database problem processing jobs today
by Slater, Michael 23 Dec '23

23 Dec '23

Similar to about 10 days ago, there was a database issue processing the jobs today. Over the past year and a half, give or take, we've had only four unusual exceptions processing the jobs, including the one today: ---------- JOBMAN.LOG.20221212 2022-12-12 22:03:24.640 | ERROR : Exception in process_jobs_for_day: InterfaceError: connection already closed ---------- JOBMAN.LOG.20230903 2023-09-02 22:46:39.415 | ERROR : Exception in process_jobs_for_day: could not translate host name "31b393d2-9a3f-4c15-b86f-33579993dbb5.bc28ac43cf10402584b5f01db462d330.databases.appdomain.cloud" to address: Unknown host ---------- JOBMAN.LOG.20231213 2023-12-12 22:46:41.263 | ERROR : Exception in process_jobs_for_day: current transaction is aborted, commands ignored until end of transaction block ---------- JOBMAN.LOG 2023-12-22 23:34:29.115 | ERROR : Exception in process_jobs_for_day: current transaction is aborted, commands ignored until end of transaction block These last two appear as if some other process is accessing the database when we are processing the jobs. The first one above on 2022-12-12 might also be a database related error. The second one, from 2023-09-02, seems to be related to DNS/network routing, an area where I still see occasional problems, mostly related to Kubernetes. Michael

1 0