#6009 - Add better handling of notification and mail crashes
-
By PDStig
- Added
- 7 views
| Identifier | #6009 |
|---|---|
| Issue type | Minor issue (breaks specific functionality) |
| Title | Add better handling of notification and mail crashes |
| Status | Completed |
| Tags |
Roadmap: v11 (custom) |
| Handling member | PDStig |
| Version | 11 beta4 |
| Addon | core |
| Description | Sometimes a notification digest or an e-mail will cause an out-of-memory error. When this happens, the whole process is halted:
* For digests, it halts / locks the entire hook. * For the mail queue, the queue halts and no e-mails get sent out (except ones that bypass the queue). Add a sub-locking mechanism within each using the elective value system. "Lock" a specific notification tin when we start processing, then delete the value when we are done. Do the same for the mail queue. Skip any records which are locked. This will also help with troubleshooting because one can look into the values elective to see the record number that got locked. Also add a health check that searches for the presence of any of these value prefixes (and also for background tasks), and fails if any are present and older than 5 minutes. |
| Steps to reproduce | |
| Funded? | No |
The system will post a comment when this issue is modified (e.g., status changes). To be notified of this, click "Enable comment notifications".
Comments
* For mail, it will try up to 5 times (no more than once per hour) on failed mail. Permanently failed mail will be removed from the queue and dumped to a log file in data_custom/failed_mail. A health check will fail if that directory contains any log files.
* For notification digests, we're not using locks. Instead, two cut-offs are now used: item count and e-mail body size. If either is exceeded, the e-mail will be wrapped up. Remaining items will be sent in another digest one hour later [because the hook only runs once per hour] (truncation of items was a bad idea IMO because notification digests cannot easily be opened in full).