I am still monitoring. However, this is what I did:
Buckets are stored in a time interval according to the pivot
Cron runs every 15 minutes instead of every day (or every minute if we need to catch up)
Cron terminates after 15 seconds of processing or if memory use is getting high
Cron tracks the processing of each stats hook individually
Added a "delta"; when the stats hooks run, deltas are stored in a separate table instead of loading in current data (this saves on memory). Then, the Cron will merge the deltas batch by batch.
I'm still worried about the flat data because the flat data is still stored as one large set (and does not use deltas).
System message - Issue updated
I'm still worried about the flat data because the flat data is still stored as one large set (and does not use deltas).