View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
3958Composrcorepublic2019-11-19 01:122024-07-23 16:57
ReporterChris Graham Assigned ToGuest  
PrioritynormalSeverityfeature 
Status newResolutionopen 
Summary3958: Opportunistic scheduler
DescriptionSome background Cron hooks are not very time critical, but may have a performance impact. Ideally we would run these hooks only when the server is under low load.

Code in a detector function to find if the server is under 'low load', based on I/O load, CPU usage (uptime command on Linux), and memory usage (configurable thresholds).
This check would be run before every Cron hook that is algorithmically determined as 'optional'.

Make the scheduler (Composr CRON system) use this to determine whether some low priority hooks should get done. For example, updating the sitemap. CRON hooks would specify:
 - a "frequency" (currently this logic is hard-coded in v10, and already implemented as the 'minutes_between_runs' setting in v11)
 - a "fluff" factor (to determine how much things should be allowed to be late, e.g. 1.6x -- tasks being deferred if not in a low load state)
 - a "priority" (i.e. tasks would run in priority order [well actually anything already late past it's fluff factor would be elevated to high priority])
...and the system would automatically launch CRON hooks accordingly.

Hooks should be able to keep calling back and checking to see if they should keep going. Not all hooks would implement that, but particularly long running low priority re-entrant ones (background caching/indexing) would.
TagsType: Performance
Attach Tags
Time estimation (hours)4
Sponsorship open

Sponsor

Date Added Member Amount Sponsored

Relationships

Search Commits
related to 3501 ResolvedPDStig Poor-mans-Cron 
related to 2236 ClosedChris Graham Opportunistic scheduler and background caching 
related to 3959 Not AssignedGuest Cache warm up via Cron 
related to 3185 Not AssignedGuest Pagination indexing table specifically for forum topics 
related to 3288 ResolvedChris Graham Major overhaul of fulltext search support, the "Composr fast custom index" 
related to 5656 ResolvedPDStig Optimise task queue more intelligently 

Activities

PDStig

2024-07-23 16:57

administrator   ~8897

The UI should probably indicate when a hook has been "deferred" and its reasoning. If a hook is deferred beyond the fluff factor, then Health Check should warn of this along with the UI.

Add Note

View Status
Note
Upload Files
Maximum size: 32,768 KiB

Attach files by dragging & dropping, selecting or pasting them.
You are not logged in You are not logged in. This means you will not get any e-mail notifications. And if you reply, we will not know for sure you are the original poster of the issue.

Issue History

Date Modified Username Field Change
2019-11-19 01:12 Chris Graham New Issue
2019-11-19 01:12 Chris Graham Tag Attached: ocProducts client-work (likely)
2019-11-19 01:12 Chris Graham Tag Attached: Type: Performance
2019-11-19 01:13 Chris Graham Relationship added related to 2236
2019-11-19 01:13 Chris Graham Relationship added related to 3501
2019-11-19 01:14 Chris Graham Relationship added related to 3959
2019-11-19 01:16 Chris Graham Relationship added related to 3185
2019-11-19 01:17 Chris Graham Relationship added related to 3288
2019-11-19 01:19 Chris Graham Description Updated
2021-04-06 01:29 Chris Graham Tag Detached: ocProducts client-work (likely)
2024-07-23 14:57 Chris Graham Relationship added related to 5656
2024-07-23 16:57 PDStig Note Added: 0008897