#2236 - Opportunistic scheduler and background caching

Identifier #2236
Issue type Feature request or suggestion
Title Opportunistic scheduler and background caching
Status Closed (duplicate)
Tags

ocProducts client-work (likely) (custom)

Type: Performance (custom)

Handling member Chris Graham
Addon core
Description Code in a detector function to find if the server is under 'low load', based on I/O usage, CPU usage, and memory usage (configurable).

Make the scheduler (Composr CRON system) use this to determine whether some low priority hooks should get done. For example, updating the sitemap. CRON hooks would specify:
- a "frequency" (currently this logic is hard-coded)
- a "fluff" factor (to determine how much things should be allowed to be late, e.g. 1.6x -- tasks being deferred if not in a low load state)
- a "priority" (i.e. low priority tasks happen last and only if CRON has only been running less than say 20 seconds already)
- a "ui_responsiveness" flag (set to false normally, see note about AJAX later)
...and the system would automatically launch CRON hooks accordingly. It would also handle locking (which we currently do with custom per-hook code).

Code a new CRON hook that simply goes through and compiles any non-compiled templates, Comcode pages, blocks (by evaluating the Comcode pages after compiling them), blocks (just using their default parameters, or perhaps all permutations of parameters a new block method says should be pre-cached), and language files. It should focus on things from 'core' addons first. This hook has the 'ui_responsiveness' flag.

Create a new AJAX system that *only* executes the ui_responsiveness flagged hooks. When an admin user is going around the site, this is being called in the background, and if the system is low-load, it is going through and compiling things. A good example of why this is nice, is on a fresh install, the Admin Zone is initially slow due to the sheer amount still to be compiled. As the user is reading through stuff, the stuff is getting compiled for them.

Automatically determine which of the admin zone screens are slow. Put a threshold on it, e.g. no screen should be more than 1 second upon second load (i.e. cache-ready load). Create an automated test for that, ensuring base performance on anything on the sitemap. If anything isn't hitting it, this is an indicator we should be implementing a block (with caching) for that screen. For example, the "choose a theme image to edit" screen is slow due to massive numbers of theme images needing categorising and laying out, but a block could handle this.
(Certain things may not be on the sitemap currently but should be, I think my example above isn't currently on the sitemap because it is per-theme, but it could be done anyway)
Steps to reproduce

Additional information I considered shipping pre-compiled stuff, and smarter (less harsh) decaching, but due to the number of different variables, neither of those are feasible.
Related to

#3501 - Poor-mans-Cron

#3958 - Opportunistic scheduler

#3959 - Cache warm up via Cron
Funded? No
The system will post a comment when this issue is modified (e.g., status changes). To be notified of this, click "Enable comment notifications".

Rating

Unrated