Some details on referral tracking...

We currently use 'keep_referrer' as a standard for tracking referrals by members. This should be documented in codebook3.txt. These detected referrers would be logged to the stats_member_referrals table.

The current 'ad_success' non-bundled addon uses a '_t' parameter for tracking. These detected referrers would be logged to the stats_tracking_codes table. This should be documented into codebook3.txt.

HTTP referers would be logged to the stats_referers table.

We may also consider an addon for showing email delivery reports via e.g. Elastic Email. Also simple bounce reports, e.g. via Amazon SES.

Likely when this issue is implemented some parts of it will be split out into separate issues.

Also a note that we could do graphs for affiliate systems such as avantgate, revenue, clicks, conversion rate, etc.

I've been thinking about how the Admin Zone dashboard displays. It's overwhelming to people, and some of the stuff on there is dated in that it is over-simple compared to the massive breadth of analytics open to people now.

I think probably the staff links should be taken off and moved into links at the bottom of tutorials. There's too much to link to from a single block nowadays.

The monitoring block probably should just be removed. I don't think anyone is really using it as currently-is, because people serious are going to go into more detail and lay-users will not even use it. People can use something good like Moz to compare against competitors.
I'm not sure if we should keep the block as an option for people. Probably not, probably any integrations would happen under the new analytics system anyway, not the dashboard.

See https://docs.google.com/spreadsheets/d/1_yaJeGzDIsxq33I7Wg9I-lTBDk3YS22WPBwJ971v5tI/ sheet for more ideas about integrations.

This is now broadly implemented. There is stuff implemented not discussed here, as I did a clean room reimplementation before checking back over this topic. Some of the implementation details have changed a little.

It really is awesome, we have 69 different graphs, most of them supporting some filters. We have split testing. We have a really nice integrated tracking/referrals system. We have a custom-event tracking system. We have a KPI system. We have Google Analytics integration (22 extra graphs from GA can be displayed). We have Google Search Console integration.

The following is not implemented and won't be:
- Member map (we show a country bar chart instead, member map is a separate non-bundled addon and depends on Google Maps which we don't want in core)
- We're focusing more on what happens for sessions (e.g. page views per session), than what an average member does (e.g. page views per member); more useful
- Rather than tracking all attachment/download downloads, we have implemented a script that can be used to track any URL access, and also a JavaScript function to track any event -- this allows the webmaster to pinpoint what downloads they actually want to track on their site rather than having a huge amount of noise eating up database space
- We won't bother with "Newsletters sent", as it's too specific and we can filter to track any admin action anyway
- We'll show banner clicks not banner views, as banner views are just randomised anyway
- The banner graph will be filterable by banner name not banner type, as it's the specific banners you want to track performance for
- Google ranking will work via Google Search Console API, and show much more than just ranking (but it will be a non-bundled addon and need configuring)
- Errors in log, Server CPU load, Server memory usage, Server I/O, Free disk space -- we will not graph these and just leave them to the Health Check system, which can set up alerts on high usage; the timestamp of the alerts will provide the data the sysadmin needs
- We'll only show points gifted as a points graph, I don't think we need to see earnings over time
- "Rating actions by members per member" is bloaty
- "Custom Profile Field list value selection" is not going to fit into the graphing model well (we are really going to show graphs for every CPF by default?) - and already exists elsewhere in Composr with a more appropriate UI
- Not showing graph for closed support tickets, bloaty
- KPIs will be simplified:
- No configurable regular differentials; higher KPIs can be progressively configured via links in the KPI emails
- No setting responsibility, it's just a notification that staff will get; responsibility can be handled outside of the scope of Composr
- No main_staff_stats_kpi block; staff can add whatever graphs they want to the dashboard manually, and there'll be a link in the staff checklist to view all the KPI graphs
- Not removing main_staff_checklist hooks -- as these are a lot easier to use than setting up KPIs, and represent common needs
- We're not using block caching when rendering graphs, as it should be fast without it (and makes things a pain to debug)

Some differences:
- KPI notifications will be sent out based on the timbre of the graph they are for - e.g. KPIs on a monthly graph would have notifications sent monthly
- We already have made it so you can easily configure what is on the Admin Zone dashboard in the configuration

The third-party integrations in this issue would be implemented separately, if we so desire. They are already on our "Composr external integration" spreadsheet. Right now it only makes sense on a client project where those stats are really important.
For reference this issue mentioned:
- YouTube
- Twitter
- Facebook
- Google adsense
- Google Play store / Apple appstore
- affiliate systems such as avantgate, revenue, clicks, conversion rate
- email delivery reports via e.g. Elastic Email
- possible stuff on https://docs.google.com/spreadsheets/d/1_yaJeGzDIsxq33I7Wg9I-lTBDk3YS22WPBwJ971v5tI/

Side note, the original implementation quickly gets inefficient when you have more content / views; even on a small/medium site like composr.app, it threw memory errors.

In 11 beta9, I've put in place some optimisations to hopefully remedy this. Namely, buckets are not saved strictly by month, but according to the pivot type, to help spread the data out more. This means more rows in the database, but less likely to trigger OOM errors.

Furthermore, stats pre-processing now runs much more frequently, but dumps deltas into a separate temporary deltas table which is then gradually merged into the actual stats over time. This allows us to dump the $data_buckets variable to the database and start clean with new data when it starts to get large or when we get close to the PHP memory limit. We also don't have to load in any data into $data_buckets anymore prior to calculating stats; we just need to know what *changed*, and dump that into deltas, which will then merge later.

I'm still worried we might hit OOMs on large yearly-based stats or on the preprocessed_flat table. I'll continue to monitor and implement more optimisations if this happens.

I did run a stress test by populating stats with 80,000 unique garbage records and then had the stats system process those. Often a site will have millions, but many of them will share the same properties (user agent / browser / etc). This had 80,000 unique devices / agents / etc, which would have made the data in the buckets very long. And I haven't encountered any OOMs. So we might be good.