3650: Deep tracking - Composr CMS feature tracker

ID	Project	Category	View Status	Date Submitted	Last Update

3650	Composr	core	public	2018-07-30 20:21	2021-03-14 21:20

Reporter	Chris Graham	Assigned To
Priority	normal	Severity	feature
Status	new	Resolution	open

Summary	3650: Deep tracking
Description	Currently Composr can track activity by member, or by IP address. Back when I designed this, I thought we'd all be on ipv6 soon without us having NAT (Network Address Translation). Perhaps more than ever we have organisations were all users share a public facing IP. This means that we cannot effectively track a member when they're not logged in / prior to joining. Add more sophisticated tracking, using the session cookie, so we can are not relying on the IP address. I don't believe we need to implement any new special tracking cookie. If enabled the session cookie would not be literally implemented as a session cookie (i.e. it would survive browser closing). This tracking functionality would not be enabled by default. It also has heavy GDPR implications. Obviously it is controversial to track people using cookies, but for certain enterprise websites it can form a basic pillar of marketing - so a sales person knows the potential customer better. We'd have a new table... sessions_historic_tracking - ID_TEXT session_id - IP ip_address - MEMBER member_id - SHORT_TEXT user_agent - SHORT_TEXT first_referer - TIME time_first_active - TIME time_last_active It works to bind session/IP/member/user-agent identifier combinations. Hence any can be a key to find any other associated identifiers. On each hit (where we currently save to the stats table), we'd insert a new row into this table. Any existing conflicting row would be deleted first, with the time_first_active and first_referer values taken from it for reinsertion into the new record. Also prior to establishing a new session for a new member login, so we can make sure the guest session is associated with the member logging in (unless we are upgrading the guest session to a member session, I can't remember if we do that). The following tables would be modified to start storing session IDs (currently they do store member IDs and IP addresses)... - actionlogs - hackattack - link_tracker - booking - searches_logged - download_logging - banner_clicks - rating - f_posts - chat_messages - poll_votes - f_poll_votes The following tables would be modified to start storing session IDs (currently they do store IP addresses)... - failedlogins The following tables would be modified to start storing session IDs and IP addresses (currently they do store member IDs)... - quiz_entries - f_moderator_logs The following tables would be modified to start storing IP addresses (currently they do store member IDs and session IDs)... - ecom_transactions The following tables would be modified to store session IDs instead of member IDs, and use that for security... - incoming_uploads The following tables already store session IDs and member IDs and IPs, and are written here just for reference... - sessions - ecom_trans_expecting - shopping_logging - stats - post_tokens The admin_lookup module would be modified with enhanced functionality: 1) be able to search by session ID (as well as username etc) 2) when searching by any identifier it would show all possible matches, rather than just one. All data from sessions_historic_tracking would in fact be shown. For each of the matches all associated identifiers under those matches would show and be clickable. Therefore you could keep recursing through lookups. For example, you could lookup a member and find their IP addresses, and then from clicking one of those IP addresses you could find all members who have used that IP address. 3) ability to filter into other modules, by IP address, member ID, or session ID... i) admin_actionlog (actionlogs table, f_moderator_logs table) ii) admin_security (hackattack table, failedlogins table) iii) admin_ecommerce (ecom_transactions table) iv) admin_shopping (shopping_logging table) 4) display the full contents of what the find_user_metadata function can look up Enhance the find_user_metadata function to look up from these tables (clearly showing underneath each identifier, with the full metadetails of that identifier): 1) link_tracker 2) searches_logged 3) download_logging 4) poll_votes and f_poll_votes 5) quiz_entries (link to view individual entry) 6) banner_clicks 7) rating (stats is already used) And link to: 6) Action logs (links filtered by any identifier) 7) Chat moderation (links filtered by any identifier - filtering would need to be a new filter added to the chat moderation UI) 8) Forum post search (links filtered by any identifier - filtering would need to be a new filter added to post search) [making of forum posts isn't in the action log] Enhance the find_user_metadata function to (clearly showing underneath each identifier, with the full metadetails of that identifier): 1) Use the referers from the sessions_historic_tracking table, rather than the one from the current environment 2) Do geolocation (etc) on all IPs, not just the current IP 3) Show user-agent, even for guest users We would make sure session IDs can be sent to Google Analytics as a user identifier key. I'm not sure off-hand if we currently do this, or just do it with member IDs. I'm not sure if GA supports multiple user identifier keys at once - maybe not, and we should use session ID for non-logged in users, and member ID for logged in users. We would review anything using the f_members.m_ip_address property. It's never been good to just assume a member's IP address is whatever IP they last used. Where appropriate we should find all their IP addresses and use them all - or use session IDs. Generally review usage to make sure it is optimal. For example, the find_user_metadata function would definitely be modified to be smarter. This function is used for the SugarCRM addon, and contact functionality, to automatically track a user (for marketing purposes) - it will be one of the prime beneficiaries of our new tracking (along with improved admin_lookup module).
Tags	ocProducts client-work (likely), Roadmap: Over the horizon, Type: Legal compliance / Privacy, Type: Security
Attach Tags	(Separate by ",")

Time estimation (hours)	24
Sponsorship open	0

Date Added	Member	Amount Sponsored

View Status	Private (for security issues or disclosing of private information; if you submit as a guest, you will not be able to see your submission)
Note
Upload Files Maximum size: 32,768 KiB	Attach files by dragging & dropping, selecting or pasting them.
You are not logged in	You are not logged in. This means you will not get any e-mail notifications. And if you reply, we will not know for sure you are the original poster of the issue.

Date Modified	Username	Field	Change
2018-07-30 20:21	Chris Graham	New Issue
2018-07-30 20:22	Chris Graham	Tag Attached: ocProducts client-work (likely)
2018-07-30 20:28	Chris Graham	Tag Attached: Type: Legal compliance
2018-07-30 20:28	Chris Graham	Tag Attached: Type: Security
2018-07-30 23:02	Chris Graham	Note Added: 0005781
2021-03-14 21:17	Chris Graham	Relationship added	related to 1837
2021-03-14 21:20	Chris Graham	Tag Attached: Roadmap: v12
2022-09-01 02:23	Chris Graham	Tag Renamed	Type: Legal compliance => Type: Legal compliance / Privacy
2024-03-26 00:58	PDStig	Tag Renamed	Roadmap: v12 => Roadmap: Over the horizon