#2680 - Automatic (but optional) pushing of hot fixes and upgrades

By Chris Graham
Added 29th Jun 2016, 1:47 PM
5 views

Identifier	#2680
Issue type	Feature request or suggestion
Title	Automatic (but optional) pushing of hot fixes and upgrades
Status	Closed (duplicate)
Handling member	Chris Graham
Addon	core_upgrader
Description	Have a CRON-linked script regularly checking our server for search-and-replace codified hot-fixes. It would not be for all hot fixes or git changes, just pointed fixes for major issues. The script will have strong support for doing rollbacks, either triggered by us (if we screw up) or by the user. As it uses search-and-replace and file-path wildcards we can target changes even if people are using overrides or their own hacked-up versions of files - and even target files to custom themes. We could also push full upgrades to users if they enable 'level 2' of the system. The script running on our server would be able to make decisions for individual sites and versions. For example, we would not upgrade v10 sites to v11 patch releases, but we might upgrade a v10 release to a later v10 release. For example, we could inject code into user's sites to be able to diagnose problems without having to request special access. Obviously this system implies a user having a high degree of trust in ocProducts, but that's inevitable. Of course this is why we have to make the system optional. I have attached my initial attempt at marking out the API/structure for the CRON-linked script and the interface. Additional Information below contains side tasks we'd need to do
Steps to reproduce
Additional information	Put forced checkbox in make_release script to confirm releaser will be around for the next 48 hours in IRC during their working hours. Add option to installer for code_hot_fixer_level, but only if SuEXEC is there Add code_hot_fixer_level config_editor.php option. Make description clear that SuEXEC is needed Add "Automatic updates" to our feature page security section Add new script as data/hot_fixer.php, with finished implementation and tested thoroughly Tie in data/hot_fixer.php to data/cron_bridge.php automatically Create initial uploads/website_specific/compo.sr/dynamic_hot_fixer.php script in composr_homesite git branch that doesn't do anything Configure live Demonstrar system to do hot fix updates Update lead developer guide to show basically how uploads/website_specific/compo.sr/dynamic_hot_fixer.php works. Say to test shared demo to make sure it has updated. Explain how to use wildcards so that even overrides can be updated. Document in upgrader tutorial
Related to	#3049 - Major upgrade reimagining (including hypervisor, and default theme improvements)
Funded?	No

The system will post a comment when this issue is modified (e.g., status changes). To be notified of this, click "Enable comment notifications".

Rating

Unrated

Comments

By Guest posted 29th Jun 2016, 1:50 PM

We can make the server preferentially upgrade certain sites, e.g. the Demonstratr system, so that advance-testing can be done by people on this, before everyone else is automatically upgraded.

By Guest posted 29th Jun 2016, 2:48 PM

Also add a way of querying a site to see if automatic updates are enabled.

Also add ability for system to add a backdoor_ip setting to _config.php easily, so manual testing can be done.

We'll have to be very clear that enabling automated updates grants ocProducts full admin access to the site to make fixes and changes as seen fit. Maybe a confirm in the installer if checking the option.

By Guest posted 29th Jun 2016, 4:11 PM

Actually we should probably add a bank of privacy options to the installer...

- Send error mails to ocProducts
- Keep your site registered with ocProducts (call home)
- Allow listing your site as an example Composr CMS site
- Automatically apply critical hot fixes
- Allow automatic targeted upgrades
- Allow ocProducts to temporarily log in as an administrator on your site to run tests

By Guest posted 29th Jun 2016, 4:16 PM

Also we should put something in the text of ocProducts error mails to say whether the site supports automatic updates. That will help us respond to issues more quickly, and identify issues with users running the latest code.

By Guest posted 29th Jun 2016, 5:05 PM

Ooh I really like this idea. My only concern is I think there should be a choice between automatic hotfixing updates and choosing which hot fixes the site manager wants to install.

By Guest posted 29th Jun 2016, 5:13 PM

I don't want to make a UI for manually installing individual fixes, it'd get pretty complicated to build a good UI and architecture for that and we'd have to debug sites being in intermediate states, all the different permutations.

However, what we could do is we could add a 'dryrun' mode, and we could add in a reason field for all the updates. When you manually call up a dry-run it would show everything that would change, and why. You'd then get a link to run it. We could put a link to the dry-run under the upgrade block in the Admin Zone dashboard if the automatic updates were not enabled.

By Guest posted 29th Jun 2016, 5:17 PM

Also the wording in the integrity checker would need updating to make it clear some changes may be due to automatic hot fixing.

By Guest posted 9th Jan 2017, 6:43 PM

Thinking about this more, it would be nice to have a really well-featured system for rolling forward/backward between versions.
Each time a new version is available, it schedules the upgrade and notifies the staff about that. Allow for jumping in and specifying an upgrade date/time manually too.

When an upgrade happens, it first checks it has enough disk space to clone the whole site. If not, it won't proceed.

The upgrade then backs up the whole site Composr-owned files to a new subdir, except the 'uploads' directory which would be symlinked from the new subdir to the original location of the uploads directory. The database would also be copied to a different table prefix. The _config.php file in the subdir would be altered so that the full backed up site could run from this subdir and table prefix.

Then the original site is upgraded automatically.

There'd be an upgrade panel which can activate temporary redirects into the subdirs of previous versions, in case an upgrade failed and you need the old version back while the situation is debugged. This would use the correct HTTP status code, so that Google knows to pause indexing. The redirects would include an exception if a keep_root_version=1 flag was in the URL.

The upgrade panel would also allow deleting of old versions, showing the space each takes plus the remaining disk space.

We can probably think about this some more. E.g. maybe we can make the redirect maintain old URLs by transparently redirecting everything into the subdir.
Maybe we can do automatic rollback if errors are detected.
Maybe we can send more emails out during the process.
Maybe we can auto-close the forum of a site to new posts in the backup, to stop community content getting lost when there's a temporary rollback.
Maybe we should have an option to only roll forward manually, so the webmaster could test it.
Maybe we can have keep_version=<whatever> flags that allow jumping into any prior backed up version.
Maybe we can put a tonne of buttons on the upgrader panel to control all of this.

By Guest posted 27th Jan 2017, 9:59 PM

I'm adding a new issue that is far more extensive, but takes a different approach in a number of ways.

Main difference:
- Instead of doing hot-fixes as well as upgrades, just do upgrades
- Do release manager improvements so we can do a much higher rate of upgrades without it being too time consuming or spammy
- Changes in how themeing and overrides are advised/managed to reduce the chance of failed upgrades
- Therefore we can drop the idea of having to worry about having temporary redirects to old versions
- A better integrated upgrade system even if automatic updates aren't on