Composr Supplementary: Professional upgrading

Written by Chris Graham

Potentially Outdated Tutorial

This supplementary tutorial might be outdated as it was written with ocProducts, Ltd (a now defunct company) in mind. As of version 11, Composr CMS is not commercially-backed (but companies and individuals can hire each other to develop Composr features).


This tutorial provides a professional process for updating customer website(s). Steps here may also be useful for professional website "tune ups".

It is only intended for programmers who are experienced in the use of Composr.

Table of contents

  1. Composr Supplementary: Professional upgrading
    1. Getting organised
    2. Process
    3. See also

Getting organised

If you are updating multiple websites then create a spreadsheet with the following columns:
  • Website URL
  • Prospective upgrade date
  • Cost quote
  • Website owner name
  • Website owner e-mail
  • Support ticket / CRM URL
  • Contacted owner?
  • Have access to website?
  • Cost agreed with owner?
  • Any concerns

This will keep you organised and prevent you from losing your sanity trying to remember all these details.

Process

This is the process repeated for each website being upgraded. The steps are not set in stone, but rather designed to be interpreted as guidelines by an experienced developer.
We assume you are running your own local AMP-style stack, have Git installed, and have a Composr Git checkout under <webroot>/Composr
<webroot> refers to your webroot, wherever it is. On my machine it is ~/www.

Step Description Extra code / notes
1 Transfer these instructions to a separate TODO list so you can work through them Copy to clipboard · Personalise paths
Downloading, assessing, reconfiguring
2 Check for any special circumstances If a multi-site-network or a third-party forum is involved, or other third-party integrations, extra consideration for compatibility will be needed
3 Make sure the website owner is aware that an upgrade cost doesn't include training for functionality changes Customers need to be aware of costs, if you don't protect yourself you could have a customer expecting you to include days of free training for what you're only invoicing as a few hours of work. Customers also need to be aware that things may change, including features being removed (link them to any announcement about this). Consider also asking for a list of all the design or code changes made on the site so that you can cost it up / plain it; anything non-listed might be charged extra for.
4 Download all the website files to your local machine, under <webroot>/<codename> It's much more efficient to work locally. Leave this running in the background while you work on other steps / tasks.
5 Check the remote server meets the Composr minimum requirements If the requirements are not met you'll need to circle back to the website owner; they may have intentionally held back software on their hosting and have the capability to update it, or they may need to do a server upgrade or host switch.
Run the first two steps of Composr's installer and see if it complains about anything (almost all requirements are checked by it).
6 Confirm the remote site is running a version of Composr that you can upgrade from If not you'll need to circle back and inform them of the cost of jumping through other intermediate Composr version(s).
† 7 Create <webroot>/<codename>/_temp and put a .htaccess file in it

Code

Require all denied
† 8 Download the website database to your local machine, put it in <webroot>/<codename>/_temp/initial.sql If you can't access phpMyAdmin, then use a script:

Code (PHP)

<?php

$filename = 'initial.sql';

$h = 'Content-Disposition: attachment;
filename="' . $filename . '"';
header($h);

$db_site = 'TODO';
$db_site_host = 'localhost';
$db_site_user = 'TODO';
$db_site_password = 'TODO';

if ($db_site == 'TODO') {
require('_config.php');
$db_site = $SITE_INFO['db_site'];
$db_site_host = $SITE_INFO['db_site_host'];
$db_site_user = $SITE_INFO['db_site_user'];
$db_site_password = $SITE_INFO['db_site_password'];
}

$sql_cmd = 'mysqldump';
$sql_cmd .= ' -u' . escapeshellarg($db_site_user);
$sql_cmd .= ' -p' . escapeshellarg($db_site_password);
$sql_cmd .= ' -h' . escapeshellarg($db_site_host);
$sql_cmd .= ' ' . escapeshellarg($db_site);
$sql_cmd .= ' 2>&1 > ' . $filename;
echo '/*';
echo shell_exec($sql_cmd);
echo '*/';

@ob_end_clean();
readfile($filename);

unlink($filename);

if (!empty($_GET['self_destruct'])) {
unlink('sqldump.php');
}
 
9 If your server is not suEXEC-style, set recursive world write permissions

Code (Bash)

cd <webroot>/<codename>
sudo chmod -R 777 .
 
† 10 (Following directly from the above) Change utf8mb4 to utf8 for just the following tables: cms_addons, cms_f_saved_warnings, cms_gsp, cms_wordfilter, cms_import_parts_done, cms_msp, cms_newsletter_subscribe, cms_theme_images, and cms_tickets If you are having to fudge a server running an old version of MySQL, use utf8 instead and be ready to tell the website owner they cannot use emojis.
† 11 Import the SQL dump into your local MySQL installation, in a database named client_<codename>, using the command line mysql client

Code (Bash)

cd <webroot>/<codename>
mysql client_<codename> < _temp/stage1.sql
 
12 Reconfigure _config.php to have local details by putting local ones on your development hostname

Code (PHP)

$host = gethostname();
switch ($host) {
case 'TODO': // dev machine
unset($SITE_INFO['db_forums']);
unset($SITE_INFO['db_forums_host']);
unset($SITE_INFO['db_forums_user']);
unset($SITE_INFO['db_forums_password']);
unset($SITE_INFO['cns_table_prefix']);
unset($SITE_INFO['cookie_domain']);
unset($SITE_INFO['cookie_path']);
$SITE_INFO['base_url'] = 'http://localhost/<codename>';
$SITE_INFO['db_site'] = 'client_<codename>';
$SITE_INFO['db_site_host'] = 'localhost';
$SITE_INFO['db_site_user'] = 'root';
$SITE_INFO['db_site_password'] = '';
$SITE_INFO['debug_mode'] = '0';
$SITE_INFO['dev_mode'] = '0';
$SITE_INFO['no_email_output'] = '1';

// These must not exist on any live site,
// they allow temporary unauthenticated
// local admin access to make upgrading
// easier a third-party developer
$SITE_INFO['admin_password'] = '';
$SITE_INFO['backdoor_ip'] = '127.0.0.1';
break;
}
 
13 Set up a local Git repository to reduce the chance of losing work

Code (Bash)

cd <webroot>/<codename>
git init
cp <webroot>/composr/.gitignore .
 
edit .gitignore to remove the second half of the file

Code (Bash)

git add .
git commit -a -m "Initial commit"
 
Main upgrade
† 14 Log into the website's upgrader on your local machine
† 15 Empty the caches
16 Do the file transfer via a custom upgrader build
17 Do the integrity checker scan and delete files from the prior version
† 18 Do the database upgrade
Cleanup and quarantine
19 Save a new SQL dump into the _temp folder

Code (Bash)

cd <webroot>/<codename>
mysqldump client_<codename> > _temp/stage2.sql
 
20 Delete old page revisions, old config files, old backups, old .latest_in_cms_edit files, etc
21 Temporarily move any custom themes into a parallel directory structure under _old; use the integrity check in the upgrader to quarantine alien files, which will also move them under _old
† 22 If jumping across versions run through the "Correct MySQL schema issues (for developers only)" tool in the upgrader to find and resolve any unexpected issues that might have developed Don't just trust the SQL given, check it really is correct to repair the problems found.
23 Run some cleanup tools Go to Admin Zone > Tools > Cleanup tools, run "Sync for lost disk content" and "Theme Wizard files"; if you're really enthusiastic also "Find orphaned uploads" and "Broken URLs"
24 Consider replacing all the TAR files in addons/imports so you're sure they're fresh If you download a custom upgrade package you can find all the TAR files in it (the from version number in the URL is irrelevant)
25 Do a Git commit

Code (Bash)

cd <webroot>/<codename>
git add .
git commit -a -m "Finished cleanup and quarantine"
 
Rebuild custom code and design
26 Build a new theme with the correct seed.
27 Use a diff tool to find all the changes made in the old theme (you put them in _old). You can usually diff-compare against the .editfrom files, otherwise you'll need to go back and compare with the originals from the version you are upgrading from.
There is a themechanges.sh script which may be useful, but realistically you probably just want to do it yourself on a file-by-file basis.
28 Fix for specific known theme changes.
29 Use a diff tool to find all the changes made in code files you put in _old. You'll need to compare overrides against the originals from the version you are upgrading from. Custom code and overridden code will need to be assessed to make sure it still is valid. By the time you're done you should be able to erase the _old directory.
30 Run CQC. Run the Code Quality Checker on any custom code. Do this by copying data_custom/functions.bin from a Composr GitLab pull, and then run the CQC out of that GitLab pull while reconfiguring the scan folder to the <webroot>/<codename> directory.
31 Do final local testing. Go through testing page by page, and also at least get a feel for content (such as news articles) working well.
32 Review errorlog.php then clear it out
33 Do a Git commit

Code (Bash)

cd <webroot>/<codename>
git add .
git commit -a -m "Finished code rebuild"
 
Get back in sync
 Defer this section if you are going to provide a client demo before the final upgrade
34 Close the remote site. If the site being upgraded has database content changing regularly (perhaps member activity), you need to update your local database so you have those changes. Skip this and the next step if you do not.
35 Get your database back in sync Do one of 3 things:
  1. Identify that nothing needs synching. If you are using MyISAM then Commandr can tell when tables were recently changed with ls /database.
  2. Take a new database backup and use a diff against your very first database backup, then manually copy those changes through. This is only a good idea if you expect a tiny number of simple changes. Of course in most cases you can ignore changes to logging tables.
  3. Install MySQL utilities and Connector/Python, put your _temp/stage1.sql file into client_<codename>_old then run something like:

    Code (Bash)

    mysqldbcompare \
    --server1=remote_user@remote_host \
    --server2=root@localhost \
    remote_database:client_<codename>_old \
    --run-all-tests \
    --difftype sql \
    --disable-binary-logging
     
    …to get SQL to do an update. Then carefully filter it to what you really want, then run it.
  4. Take a new database backup, repeat steps from this tutorial marked † to re-upgrade that database. It should not take so long given the file upgrading you've already done will still apply. You'll need to re-open the site of course, given you closed the live site just before backing it up.
Options 3 or 4 are the most likely you will want to use.
36 If the site being upgraded has files changing regularly (perhaps edited Comcode pages), download those changed files and extract locally This is a script to identify changed remote files and ZIP them:

Code (PHP)

<?php /*

Composr
Copyright (c) Christopher Graham, 2004-2024

See docs/LICENSE.md for full licensing information.

Meta Script:
Find files modified within the last 'days' days
*/

header('Content-Type: text/plain');

if (!isset($_GET['days'])) {
exit('Must give a days parameter');
}

$days = intval($_GET['days']);
$cutoff = time() - 60 * 60 * 24 * $days;

if (isset($_GET['path'])) {
chdir($_GET['path']);
}

$files = array();

$out = do_dir('.');
sort($out);
foreach ($out as $file) {
if (filemtime($file) > $cutoff) {
$files[] = $file;
}
}

if (!empty($_GET['zip'])) {
if (empty($files)) {
exit('No files');
}

header('Content-Type: application/octet-stream');
$h = 'Content-Disposition: attachment;
filename="new_files.zip"';
header($h);

$cmd = 'zip new_files.zip';
foreach ($files as $file) {
$cmd .= ' ' . escapeshellarg($file);
}
shell_exec($cmd);

@ob_end_clean();
readfile('new_files.zip');

unlink('new_files.zip');

exit();
}

foreach ($files as $file) {
echo $file . chr(10);
}

echo 'Done; set &zip=1 to download a ZIP file';

if (!empty($_GET['self_destruct']) {
unlink('find_new_files.php');
}

function do_dir($dir)
{
$out = array();
$_dir = ($dir == '') ? '.' : $dir;
$dh = opendir($_dir);
if ($dh) {
while (($file = readdir($dh)) !== false) {
if ($file[0] == '.') {
continue;
}

if (is_file($_dir . '/' . $file)) {
$out[] = $_dir . '/' . $file;
} elseif (is_dir($_dir . '/' . $file)) {
$_under = $dir .
(($dir != '') ? '/' : '') .
$file;
$_out = do_dir($_under);
$out = array_merge($out, $_out);
}
}
}
return $out;
}
 
Uploading
37 Put a copy of bigdump.php into the _temp folder and configure it for the live database http://www.ozerov.de/bigdump/
38 Upload a ZIP of the files into a _new folder, being careful to not include the .git folder in the ZIP If some upload directories are too big, omit them and resolve this by moving back the old ones from the live server.
For a further level of professionalism you may choose to deploy using your Git repository rather than a ZIP file, if so push Git up to a shared repository (e.g. GitLab) and do a pull from the server, rather than uploading and extracting a ZIP.
39 Extract the ZIP live on the server (control panel file managers can do this, or a simple custom PHP script)

Code (PHP)

<?php /*

Composr
Copyright (c) Christopher Graham, 2004-2024

See docs/LICENSE.md for full licensing information.

*/

if (!isset($_GET['file'])) {
exit('Need \'file\' parameter (filename to unzip)');
}

$filename = $_GET['file'];

error_reporting(E_ALL);
ini_set('display_errors', '1');

$zip = new ZipArchive();
$res = $zip->open(dirname(__FILE__) . '/' . $filename);
if ($res === true) {
$zip->extractTo('.');
$zip->close();
echo 'done';
} else {
echo 'failed ';
switch ($res) {
case ZIPARCHIVE::ER_EXISTS:
echo 'ER_EXISTS';
break;

case ZIPARCHIVE::ER_INCONS:
echo 'ER_INCONS';
break;

case ZIPARCHIVE::ER_INVAL:
echo 'ER_INVAL';
break;

case ZIPARCHIVE::ER_MEMORY:
echo 'ER_MEMORY';
break;

case ZIPARCHIVE::ER_NOENT:
echo 'ER_NOENT';
break;

case ZIPARCHIVE::ER_NOZIP:
echo 'ER_NOZIP';
break;

case ZIPARCHIVE::ER_OPEN:
echo 'ER_OPEN';
break;

case ZIPARCHIVE::ER_READ:
echo 'ER_READ';
break;

case ZIPARCHIVE::ER_SEEK:
echo 'ER_SEEK';
break;
}
}
 
40 Run bigdump
41 Delete the _temp folder and the ZIP file
42 Amend _config.php live to remove your temporary details, and possibly also add a backdoor_ip line

Code (PHP)

$SITE_INFO['backdoor_ip'] = 'TODO';
 

Smarter configuration management

If you want to have the exact same configuration file running across multiple machines, you can code in varying details using a switch statement…

Code (PHP)

switch (gethostname()) {
...
}
 
43 Set file permissions on live if necessary
44 Get the client to test out of the _new folder, if appropriate (in which case you'll need to temporarily set the base_url setting in _config.php)
45 Move the old files on server into an _old folder
46 Move files down from _new folder, changing base_url setting in _config.php back to normal if we changed it in step 44
47 Run Health Check (Admin Zone > Tools > Health Checks)
48 Check for broken URLs (Admin Zone > Tools > Broken URL scanning)
49 Final live testing
Customer interfacing
50 Contact the website owner with whatever you need to say, including details about old files and database tables
Your site has now been updated, the changes are live.

I have left a copy of your old site files under the ""_old"" folder.
Additionally, your old database tables are still in the database using the ""ocp_"" table prefix. The new database tables use the ""cms_"" table prefix.

I have granted my IP address administrative access so I can quickly run administrative tasks should I need to. Please let me know if you would like this to be removed.
51 Charge the website owner for the work

See also


Feedback

Please rate this tutorial:

Have a suggestion? Report an issue on the tracker.