#6194 - Stats processing uses too much memory

Identifier	#6194
Issue type	Minor issue (breaks specific functionality)
Title	Stats processing uses too much memory
Status	Completed
Handling member	PDStig
Addon	stats
Description	Stats pre-processing can easily eat up too much PHP memory (e.g. the events hook) since buckets are stored by month, and the entire months of every bucket are loaded in. Instead, I think buckets should be stored by day. It will be much easier to process without running into memory issues. * Make stats pre-processing loop by each day when a range of start / end time extends over multiple days (it shouldn't though because of the updated Cron!). * Only load bucket data for the specific day we are processing on that iteration; empty the array of bucket data after each iteration (and updating the database) to conserve memory. * Ensure nothing is modifying stats outside of the designated day; if they are, modify so they don't.
Steps to reproduce
Additional information	High priority; this affects composr.app and thus needs fixed ASAP.
Funded?	No

Edited 19th Nov 2025

The system will post a comment when this issue is modified (e.g., status changes). To be notified of this, click "Enable comment notifications".

Unrated

Comments

By PDStig, By PDStig, posted 19th Nov 2025, 8:37 PM

The details or status of this tracker issue were updated by PDStig

By PDStig posted 19th Nov 2025, 8:41 PM

I am still monitoring. However, this is what I did:

Buckets are stored in a time interval according to the pivot
Cron runs every 15 minutes instead of every day (or every minute if we need to catch up)
Cron terminates after 15 seconds of processing or if memory use is getting high
Cron tracks the processing of each stats hook individually
Added a "delta"; when the stats hooks run, deltas are stored in a separate table instead of loading in current data (this saves on memory). Then, the Cron will merge the deltas batch by batch.

I'm still worried about the flat data because the flat data is still stored as one large set (and does not use deltas).

Last edit: 19th Nov 2025, 8:41 PM by PDStig