FYI, the jobs didn't run as what might have been a DNS update took longer than our retry timeout window and we couldn't resolve sensordx.tahmo.org. Note that the name resolution worked when the job manager was determining the data completeness level, but then failed when we went to add the jobs for today. I'll have to manually add today's scoring jobs to the queue tomorrow and then we should see the results when tomorrow's scoring jobs run. --ms

ERROR : get_all_models_impl: exception | HTTPSConnectionPool(host='sensordx.tahmo.org', port=443): Max retries exceeded with url: /api/models (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x000001D874C3F8E0>: Failed to resolve 'sensordx.tahmo.org' ([Errno 11001] getaddrinfo failed)"))
2024-06-13 23:16:52.535 | CRITICAL : add_daily_job_record() | date: 2024-06-13 -- unable to retrieve scoring models from RainQC!
2024-06-13 23:16:52.541 |    ERROR : process_jobs(): error adding daily job record, jobs will not be processed.

On Thu, Jun 13, 2024 at 11:17 PM RainQC Job Manager <noreply@tahmo.org> wrote:
Current UTC date: 2024-06-14 -> scoring models for previous day: 2024-06-13
---------------------------------------------------------------------------------
Daily Model Data Completeness Check:
data completeness    50% | complete models: 117 of 273 (42.86%)
data completeness    60% | complete models: 117 of 273 (42.86%)
data completeness    70% | complete models: 106 of 273 (38.83%)
data completeness    75% | complete models: 106 of 273 (38.83%)
data completeness    80% | complete models: 106 of 273 (38.83%)
data completeness    85% | complete models: 106 of 273 (38.83%)
data completeness    90% | complete models: 103 of 273 (37.73%)
data completeness    95% | complete models: 101 of 273 (37.00%)
data completeness   100% | complete models:  91 of 273 (33.33%)
----------------------------
station status | total: 313, delayed: 154, offline 24h: 84, offline week: 82
 | battery, min: 0, max: 100, mean: 60.35, std dev: 29.67
 | battery, common values: [(100, 151), (0, 88), (78, 4), (97, 3), (76, 3)]
 | battery <= mean, common countries: [('KE', 35), ('GH', 23), ('UG', 12), ('ZM', 7), ('ML', 6)]
----------------------------
105 LOW DATA (< 0.9) and 92 NO DATA weather stations impacted 170 RainQC models
LOW/NO data station impact on models: [('TA00057', 11), ('TA00651', 9), ('TA00715', 8), ('TA00182', 8), ('TA00061', 7), ('TA00185', 7), ('TA00199', 7), ('TA00016', 6), ('TA00413', 6), ('TA00129', 6), ('TA00569', 6), ('TA00327', 6), ('TA00621', 5), ('TA00320', 5), ('TA00045', 5), ('TA00587', 5), ('TA00317', 5), ('TA00243', 5), ('TA00067', 4), ('TA00231', 4), ('TA00301', 4), ('TA00530', 4), ('TA00565', 4), ('TA00543', 4), ('TA00700', 4), ('TA00636', 4), ('TA00080', 3), ('TA00035', 3), ('TA00222', 3), ('TA00041', 3), ('TA00267', 3), ('TA00115', 3), ('TA00116', 3), ('TA00126', 3), ('TA00484', 3), ('TA00217', 3), ('TA00482', 3), ('TA00385', 3), ('TA00289', 3), ('TA00436', 3), ('TA00430', 3), ('TA00542', 3), ('TA00020', 2), ('TA00024', 2), ('TA00308', 2), ('TA00050', 2), ('TA00072', 2), ('TA00256', 2), ('TA00118', 2), ('TA00133', 2), ('TA00136', 2), ('TA00165', 2), ('TA00148', 2), ('TA00487', 2), ('TA00210', 2), ('TA00223', 2), ('TA00232', 2), ('TA00224', 2), ('TA00271', 2), ('TA00399', 2), ('TA00691', 2), ('TA00374', 2), ('TA00339', 2), ('TA00373', 2), ('TA00451', 2), ('TA00462', 2), ('TA00471', 2), ('TA00014', 1), ('TA00031', 1), ('TA00044', 1), ('TA00062', 1), ('TA00070', 1), ('TA00078', 1), ('TA00091', 1), ('TA00095', 1), ('TA00123', 1), ('TA00157', 1), ('TA00212', 1), ('TA00219', 1), ('TA00650', 1), ('TA00237', 1), ('TA00251', 1), ('TA00268', 1), ('TA00269', 1), ('TA00290', 1), ('TA00336', 1), ('TA00343', 1), ('TA00362', 1), ('TA00369', 1), ('TA00382', 1), ('TA00389', 1), ('TA00396', 1), ('TA00422', 1), ('TA00432', 1), ('TA00433', 1), ('TA00493', 1), ('TA00528', 1), ('TA00529', 1), ('TA00533', 1), ('TA00535', 1), ('TA00652', 1), ('TA00655', 1), ('TA00677', 1), ('TA00685', 1), ('TA00702', 1)]
-----------------------------------------------------------
Processed daily jobs for UTC date: 2024-06-13
Start time: 2024-06-14T06:15:06+00:00
End time  : 2024-06-14T06:16:56+00:00
Elapsed time HH:MM:SS: 0:01:50
---------------------
Before job processing job table stats:
Total 'success' count: 73
Total 'failure' count: 200
Total record count: 1513
Job history table record count: 170561
Scoring job record table record count: 644
---------------------
After job processing job table stats:
Total 'success' count:                0 (flag=2 count:   0) (flag 2->1 downgrades:   0)
 | 'success' count for 2024-06-13:    0 (flag=2 count:   0)
 | 'success' count for 2024-06-12:    0 (flag=2 count:   0)
 | 'success' count for 2024-06-11:    0 (flag=2 count:   0)
 | 'success' count for 2024-06-10:    0 (flag=2 count:   0)
 | 'success' count for 2024-06-09:    0 (flag=2 count:   0)
 | 'success' count for 2024-06-08:    0 (flag=2 count:   0)
 | 'success' count for 2024-06-07:    0 (flag=2 count:   0)
Anomalies (flag=2):
--------
Total 'failure' count: 0
Total record count: 1240
Job history table record count: 170834
Scoring job record table record count: 644
-----------------------------------------------------------
RainQC JobMan LR | version 1.2