View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 3049 | Composr | core_upgrader | public | 2017-01-28 03:00 | 2025-02-27 16:36 |
| Reporter | Chris Graham | Assigned To | Guest | ||
| Priority | normal | Severity | feature | ||
| Status | new | Resolution | open | ||
| Summary | 3049: Major upgrade reimagining (including hypervisor, and default theme improvements) | ||||
| Description | I want us to do a completely overhaul of how we think about upgrading. This is to put us in line with the abilities and requirements of the average user. Related, is making themeing easier, but this is covered in other issues. These are the big 2 remaining problem points for Composr users, the stress of these things. This touches everything so this will be a long convoluted issue. Server API cleanup ================== Re-work all the composr_homesite addon scripts into a much cleaner API which has the following responsibilities: - Release management (including various kinds of version querying, and finding download IDs and news post IDs for particular versions) - Upgrade management - Changes listing (diffing) - Non-bundled addon management - Translation management - Demonstratr management - Error service (reporting [instead of direct emailing] and error message assistance) - Tracker & Hot-fix management - Newsletter joining - Patreons management - Site registration (including registration of up-time monitoring URLs and logging of what sites are live and at what URLs [if the user has enabled the option for this to be reported to us]) - Promo logos There would be a pure PHP API in sources_custom/composr_homesite/*.php - each responsibility would be in a separate file. There would be JSON endpoints that mirrored many functions in data_custom/composr_homesite_api/ Modules and blocks using the API would have minimal code, working just as front-ends to the clean API All the scripts in uploads/website_specific/compo.sr/scripts and uploads/website_specific/compo.sr/logos and uploads/website_specific/compo.sr/upgrades would be explicitly marked as 'LEGACY'. As far as possible these would be re-worked to draw off the new API too. Take off the upgrader block from news posts, make those news posts strictly only for news about the releases. Upgrades would usually only be through the JSON API which the Composr upgrader would directly communicate with. But add a manual upgrade generation page which can build any upgrader. Feature from/to drop-downs (from can be blank), and an ability to select which addons to include (by default all are selected). Remove the descriptions ("this is the latest version" etc) from release downloads. The API should be able to find out what the latest release is just by looking at version numbers. Composr would use the new JSON APIs to community with compo.sr, so we can do things more cleanly. Release scripts cleanup (build API) =================================== Currently we have make_release and build_addons scripts, and some others, for managing releases. Improve these to be more automated. For example, rather than manually FTPing up files, or separately building addons, there would be a single release wizard that would do this for you - so the server API would need methods for handling. The wizard would have a decent API using lots of checkboxes for confirming you've done things or deciding what exactly is required of it. Put forced checkbox to confirm releaser will be around for the next 48 hours in IRC during their working hours. Have a better API behind this so the wizard code can be pretty clean. There would be a number of sources_custom/composr_build/*.php files. The server API would be able to draw on these same build APIs for things like upgrade generation. We want the server API to essentially be able to do it's own headless building of any kind of Composr package. Allow posting a release without any associated news post. This is so we can have a faster timbre of releases without being spammy. Upgrader cleanup ================ Pull out most of the code from sources/upgrade.php into various individual files. Each function would be clearly delineated as either requiring Composr to be running, or working as standalone. Generally have a much cleaner API which the upgrade would then draw on. The upgrader would be able to upgrade non-bundled addons. To do that the server API needs improving to support it. The server API would generate a customised upgrader file for all the particular addons installed, and also to refresh any outdated uninstalled addons that are in the imports/addons directory. The upgrader would also have the ability to uninstall addons that are no longer supported (one's with no availability from compo.sr for the current installed version of Composr). When doing so it would add a note to the addons description (addon.inf inside the addon TAR) that it had been 'quarantined' due to no longer being supported. When the upgrader closes the site it should place the closed.html file there, as people with closed-site permission should not be able to access the site (and besides, accessing the closed-site screen when an upgrade is happening could cause some unforeseen issues in and of itself.) For each custom theme, provide a link to the "theme upgrading" screen within admin_themes, opened up for that particular theme. Remove automatic theme upgrading. Remove any references to it from tut_theme_lifecycle and tut_upgrade. The concept of automatic theme upgrading is now dead. Provide a link to the Health Check tool. Review the tut_upgrade tutorial and sup_professional_upgrading tutorial. Summary ======= To summarise there would be 4 sets of related APIs: - Server API (runs on server) - Build API (runs on dev machines and also server) - Addon API (a part of standard Composr, runs everywhere) - Upgrader API (a part of standard Composr that also partly runs standalone) (actually some of the things listing as composr_homesite responsibilities may get moved into the build API) New upgrade management model ============================ When installing you choose 1 of 2 possible methods for having your installation managed: - Automatic updates via hypervisor - Manual updates The default would be automatic updates. You could change it via config_editor.php. There is a major philosophical difference which we'd very clearly document through our platform. People using the hypervisor are not expected to do any custom programming or themeing (except for settings and theme images). Additionally, the user is granting the compo.sr server direct access to their hosting. That sacrifice is enough for us to then guarantee a good standard of automatic upgrades. Obviously this system implies a user having a high degree of trust in the compo.sr server, but that's inevitable. Of course this is why we have to make the system optional. We need to make sure a disclaimer is very prominent. If the hypervisor is enabled then we add strong discouraging warnings to any CSS/template editing screens (except for __extension files), and to code_editor.php, and to upgrader.php. The upgrade block in the Admin Zone would no longer have upgrade links, but instead link to the hypervisor. We would also add warnings at the top of themeing tutorials that if the hypervisor is in use you shouldn't do CSS/template editing (except for __extension files). The hypervisor requires CRON to be set up. The hypervisor requires SuExec. The hypervisor would be in English only. The hypervisor would need working HTTPS download support (for important security reasons). Developers, including those using git repositories for their code, would not want to use the hypervisor. Hypervisor ========== The new hypervisor would use the same API as the upgrader - but just the functions that work standalone. This allows it to inherit the functionality for pulling down the correct upgraders tuned for the site, including handling of non-bundled addons and uninstalled bundled addons. The hypervisor's responsibilities: - Doing fully headless upgrades using the upgrade API (*1) - Checking disk space, need some big arbitrary number like 1GB free to do any upgrade - Holding back major upgrades until the user logs into the hypervisor and provides permission (it'll have to link them to the news post that describes the new version's breakage); patch upgrades of the older release line would continue to be installed until the permission was granted - Emailing the user automatically when some exceptional situation occurs, such as running out of disk space or needing a major upgrade permission (so the hypervisor would need to know the staff address somehow; probably best to have it in _config.php) - Emailing the user whenever an automated upgrade happens, including a link for them to confirm all their system checks still pass (I'd say admin_phpinfo -- except that may not be installed, so maybe we need a new page for this) - Maintaining backups of each installed version in subdirectories, with the ability to hot-swap them in and out efficiently - Providing a master-password protected UI for: - Show error if an operation currently is in progress (i.e. hypervisor is locked), say what it is, don't allow any further manual actions - Pausing/resuming automatic updates - Rolling back to prior versions - Showing previous backups, including how much disk space they take - Deleting backups of prior versions - Warning the user if they have alien overrides that aren't a part of any properly defined addon OR a properly defined addon that is not recognised by the server API - Sending uptime details to compo.sr so that a developer can get involved if something about a site is breaking upon automated upgrade - Sending of known admin IPs to compo.sr - Temporary remote backdoor_ip setting via the server API, to an engineer's IP (an engineer would configure a site's profile on compo.sr to be IP-backdoored, and then compo.sr would tell the site to re-check the status, and the site would open up to the engineer). This would also need to trigger a refresh of the .htaccess file IP-whitelist if that feature is enabled. - Hidden feature for setting either fast-track/safe-track status (test sites are on fast-track so we can make sure sites are not gonna break when all do the upgrade) - The ability to force an early update - Locking, when an upgrade or rollback is happening - The ability to directly query compo.sr for code to run - Maintain JSON-index file of old versions in backup directory - Simple JSON API for calling externally, says if the hypervisor is enabled - Rollback a specific version if compo.sr requests it The hypervisor would be tied into data/cron_bridge.php, but run ahead of Composr (i.e. the hypervisor gets precedence). Each time an upgrade happens the old hypervisor would be available for running directly from within the backup directory. It must be possible for it's CRON-style running mode to be called manually via URL, so a developer (or script) can force some kind of automatic repair of an accidentally broken newer hypervisor that has been broken via harnessing an old working copy of a hypervisor. We'd want to add many unit tests for the hypervisor to minimise the chance of us breaking it on an update. We should be able to get it to do a full mocked upgrade on a test install and confirm it worked as expected. For the hypervisor upgrading to work, Composr must be able to essentially upgrade itself (as the hypervisor has no access to the database). Composr must essentially detect when an upgrade is pending, lock itself, do the upgrade, then unlock itself. *1 The upgrade backs up the whole site Composr-owned files to a new subdir, except the 'uploads' directory which would be symlinked from the new subdir to the original location of the uploads directory. The database would also be copied to a different table prefix IF it is more than a patch release. The _config.php file in the subdir would be altered so that the full backed up site could run from this subdir and table prefix. compo.sr site management ======================== The table of Composr websites would be enhanced with what sites are running the hypervisor, and the feature to set the backdoor IP setting for those that do temporarily. The hypervisor would be communicating with compo.sr if sites appeared down. These would be turned into staff notifications, but status would also be listed against the individual sites. Also Composr sites would be reporting errors via the server API now. These would be able to come through via emails, as a staff notification too, and you'd also see errors listed against the individual sites. The emails would highlight if the site is running the hypervisor. We'd have the feature to see what compo.sr users are associated with a hypervisor site, or what sites are associated with compo.sr users. We'd need a PHP console to each site running the hypervisor, it would essentially send PHP commands direct to the hypervisor. We could either send to a specific site, or every site. We could tell it to only target sites that have had at least one upgrade, and tell it to run through the last backed up copy of the hypervisor (as a fail-safe to repair a broken updated hypervisor). The default site would be our own test site. Rollbacking a release ===================== Have a new admin module that can totally roll-back a new release, unvalidating the download and news for it, and telling the hypervisor to do a rollback if it was installed. Maintenance scripts =================== Enhance the scripts to not require a master password if the backdoor IP setting is set against the current visitor (including the hypervisor). (This is so that developers can log in and fix things if they break) Language packs ============== Let's just have language pack addons re-generate nightly automatically from Transifex. This would be done on compo.sr via a CRON hook. Language packs would be deleted from git. The language pack generation would need to look at the existing addon and copy any additional files that addon has back into the new pack (e.g. png files). This should be a feature of the manual pack downloader links as well as the nightly generation. Security documentation ====================== Add "Automatic updates if hypervisor enabled"-like-text to our feature page security section and the Security tutorial. Installer ========= Add a bank of privacy options to the installer... - Send telemetry to developers - Keep your site registered with the compo.sr server (call home) - Allow listing your site as an example Composr CMS site - Allow automatic targeted upgrades via the hypervisor / allow developers to temporarily log in as an administrator on your site to run tests | ||||
| Tags | Roadmap: Over the horizon, Roadmap: v11 partial implementation | ||||
| Attach Tags | |||||
| Time estimation (hours) | 100 | ||||
| Sponsorship open | |||||
| has duplicate | 2680 | Closed | Chris Graham | Automatic (but optional) pushing of hot fixes and upgrades |
| related to | 2971 | Closed | Chris Graham | Theme tooling improvements |
| related to | 3314 | Resolved | Chris Graham | Automatic website tests ("Health Check") |
| related to | 3856 | Not Assigned | Guest | Addon isolation via virtual subtrees |
|
|
Article on how we should not repeat Wordpress's mistake: https://medium.com/@CiPHPerCoder/stopmullware-on-the-security-of-27-of-the-websites-on-the-internet-298a7e5b6871#.595l0mq0y [now deleted] |
|
|
Generally I'd like to see lots of unit tests, covering all the compo.sr APIs, to make sure things don't accidentally get broken around making releases. |
|
|
See comment(s) here for possible additional considerations: https://compo.sr/forum/topicview/browse/deploying/rolling-releases-in-v11.htm?post_id=5997&redirected=1#post_5997 |
|
|
"People using the hypervisor are not expected to do any custom programming or themeing (except for settings and theme images)." Seems like a lot of work for what may be the exception to the rule. It didn't take me long to start messing with forces I didn't fully understand. Perhaps a survey on this would be useful. |
|
|
Right. The only way we can really do automatic upgrades is if we know users are not overriding things. For now this had already been pushed back, it's no longer on the v11 roadmap. |
|
|
tut_software_feedback now references the desire for automatic upgrades in the future. It'll need to be updated once they are implemented. |
|
|
One thing I've learned recently is the power of simple off-by-default feature flags. New functionality: Build new functionality, only enabled if the feature flag is on. Change existing functionality: copy and paste into a new copy of the code, and only use the new if the feature flag is on. This would be a realy effective way for us to move towards stable rolling releases. |
|
|
2025 update: Server API cleanup ================== I've been working on this and took a different approach. I standardised hooks/endpoints (with authorization support) so that they can be used outside of the Composr Mobile SDK. And I migrated almost all homesite API calls as endpoint calls (v11 starts using these on composr.app). Telemetry is now also done through this API instead of e-mails. Every (unique) relayed error also gets a GUID so webmasters can check the status of their errors on composr.app. --- "Take off the upgrader block from news posts, make those news posts strictly only for news about the releases. Upgrades would usually only be through the JSON API which the Composr upgrader would directly communicate with." For now, I will not do this and will leave the block. It is very useful especially if someone needs to upgrade to a specific version first before upgrading to a later version. "Remove the descriptions ("this is the latest version" etc) from release downloads. The API should be able to find out what the latest release is just by looking at version numbers." Actually, I plan to use the advanced feature in downloads where you can specify a file of a newer version. Release scripts cleanup (build API) =================================== Will probably not implement the FTP idea at this time; too much work for not enough benefit. Upgrader cleanup ================ "The upgrader would be able to upgrade non-bundled addons." This is dangerous as non-bundled addons can break the site between major versions. Furthermore, we decided the upgrader should not upgrade that which is not officially supported. However, I did modify v11 so the upgrader runs in safe mode, and furthermore you can go to the site in safe mode and update non-bundled addons in safe mode. Hypervisor ========== Not going to go in-depth right now but I did want to say originally we decided this concept would be too risky to implement (if composr.app got compromised, then automatic updates could send malware to other sites). However, the new Telemetry service involves public and private key pairs both on the software level and the site level. It would now be theoretically possible to use these to stream encrypted automatic upgraders to people's sites using their site public key and the software private key (which is then decrypted by their site using the site private key and software public key). Even if a hacker compromises composr.app, they would not be able to stream malware through the upgrader without knowing the private keys of every individual site running Composr. And Composr does a quick site validation challenge to ensure when sites register themselves with telemetry, they are who they say they are. This, of course, requires telemetry to be enabled. But if it's disabled, we assume the webmaster wouldn't want automatic updates anyway as that violates the core principles of autonomy. |
|
|
Maintenance scripts =================== "Enhance the scripts to not require a master password if the backdoor IP setting is set against the current visitor (including the hypervisor)." Bad idea (e.g. multiple devices on the same IP). Instead, the maintenance password should be commented out / changed to a different one which we know what it is. Rollbacking a release ===================== We should take care with this. Our new developer policy should be NEVER to re-use the same version number again as soon as it went public, even if we later pull it, just in case someone upgraded to it. As such, if we roll back a release, the download should remain in the system (not validated) or some flag made so that the upgrade system skips it. And we use the next patch number when we fix the upgrade instead of re-using the same version. Language packs ============== "Let's just have language pack addons re-generate nightly automatically from Transifex." There aren't updates that often. Let's do weekly instead. Nightly may be annoying for some users. |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2017-01-28 03:00 | Chris Graham | New Issue | |
| 2017-01-28 03:01 | Chris Graham | Relationship added | has duplicate 2680 |
| 2017-01-29 17:12 | Chris Graham | Description Updated | |
| 2017-01-31 11:49 | Chris Graham | Relationship added | related to 2971 |
| 2017-02-14 16:45 | Chris Graham | Note Added: 0004788 | |
| 2017-04-13 21:49 | Chris Graham | Category | core => core_upgrader |
| 2017-07-05 13:35 | Chris Graham | Note Edited: 0004788 | |
| 2017-07-05 14:04 | Chris Graham | Relationship added | related to 3314 |
| 2017-07-09 16:45 | Chris Graham | Description Updated | |
| 2017-07-09 16:46 | Chris Graham | Summary | Major upgrade reimagining (including hypervisor) => Major upgrade reimagining (including hypervisor, and default theme improvements) |
| 2017-07-09 16:46 | Chris Graham | Description Updated | |
| 2017-11-20 00:08 | Chris Graham | Time estimation (hours) | 130 => 100 |
| 2017-11-20 00:08 | Chris Graham | Description Updated | |
| 2017-11-20 00:16 | Chris Graham | Relationship added | child of 3362 |
| 2018-01-24 17:09 | Chris Graham | Note Added: 0005398 | |
| 2019-06-27 18:07 | Chris Graham | Tag Attached: Roadmap: v11 | |
| 2019-09-10 00:44 | Chris Graham | Note Added: 0006084 | |
| 2020-02-20 02:11 | Chris Graham | Tag Detached: Roadmap: v11 | |
| 2020-02-20 02:11 | Chris Graham | Tag Attached: Roadmap: v12 | |
| 2020-02-22 23:13 | Adam Edington | Note Added: 0006437 | |
| 2020-02-22 23:45 | Chris Graham | Note Added: 0006438 | |
| 2020-02-22 23:45 | Chris Graham | Note Edited: 0006438 | |
| 2021-02-24 17:15 | Chris Graham | Relationship deleted | child of 3362 |
| 2022-08-16 20:38 | Chris Graham | Description Updated | |
| 2022-11-28 20:52 | Chris Graham | Note Added: 0007755 | |
| 2024-01-27 19:12 | Chris Graham | Note Added: 0008255 | |
| 2024-01-27 20:24 | PDStig | Relationship added | related to 3856 |
| 2024-03-26 00:58 | PDStig | Tag Renamed | Roadmap: v12 => Roadmap: Over the horizon |
| 2025-02-27 04:53 | PDStig | Note Added: 0009824 | |
| 2025-02-27 04:53 | PDStig | Tag Attached: Roadmap: v11 partial implementation | |
| 2025-02-27 04:56 | PDStig | Note Edited: 0009824 | |
| 2025-02-27 16:36 | PDStig | Note Added: 0009825 |
