Composr Tutorial: Website Health

Written by Chris Graham

This tutorial will cover some aspects of keeping your website running smoothly.

Table of contents

Composr Tutorial: Website Health

Health Check

On the modern advanced web there is too much to keep in mind to check. All kinds of things could go wrong without you noticing, which could be embarrassing.
For example, what if:

Your outgoing e-mail goes down, breaking sign-ups
You forget to renew SSL
You forget to renew your domain name
A hacker takes control of the domain name and puts up a fake site
You accidentally block Google from accessing the website

All the above scenarios are auto-detectable.
In fact, you can detect problems stemming from many kinds of situation, including:

Quality issues after building a new site or theme
Software compatibility issues
Problems after a Composr upgrade
Hardware failure
Configuration issues
Lack of server capacity
Hack-attacks
Lack of routine maintenance or website up-keep

Operation

The Health Check can be run manually, or regularly run in the background (requires the system scheduler to be set up and for you to enable the "Health Check results" notification).

Check sections

At the time of writing the list of check sections is:

Backups \ Backups
Bloated data \ Database size
Bloated data \ Directory size
Bloated data \ Log size
Bloated data \ Table size (row count)
Build mistakes \ Broken links (slow)
Build mistakes \ Broken web POST forms
Build mistakes \ Comcode page form fields
Build mistakes \ Comcode page headings
Build mistakes \ Common mistake patterns (page_errors.xml)
Build mistakes \ Default icons
Build mistakes \ Guest access
Build mistakes \ Incomplete content
Build mistakes \ Local linking
Build mistakes \ Manual checks for web standards
Build mistakes \ Spellchecking of all Comcode pages (slow)
Build mistakes \ Spellchecking of miscellaneous content
Build mistakes \ Spellchecking of pages
Build mistakes \ Web standards
Deployment mistakes \ Site open-status
Domains \ DNS resolution
Domains \ Domain expiry
E-mail \ DKIM configuration
E-mail \ DMARC configuration
E-mail \ E-mail configuration
E-mail \ E-mail configuration in PHP
E-mail \ E-mail operation (slow)
E-mail \ E-mail queue
E-mail \ E-mail templates
E-mail \ IMAP login
E-mail \ List-Unsubscribe header
E-mail \ SMTP blocklisting
E-mail \ SMTP login
E-mail \ Spam status
E-mail \ SPF
Hack-attacks \ Attack frequency
Hack-attacks \ Failed logins
Hack-attacks \ Overseas access
Hack-attacks \ Rate-limit spiking
Installation environment \ Base URL
Installation environment \ Database Unicode settings
Installation environment \ Directory naming
Installation environment \ Disk Space (Installation)
Installation environment \ Injected Ad Scripts
Installation environment \ ModSecurity
Installation environment \ MySQL version
Installation environment \ PCRE settings
Installation environment \ PHP platform
Installation environment \ PHP version
Installation environment \ SELinux settings
Installation environment \ Server software
Installation environment \ suEXEC
Installation environment \ Umask settings
Installation environment (PHP extensions) \ cURL
Installation environment (PHP extensions) \ GD
Installation environment (PHP extensions) \ Unicode
Installation environment (PHP extensions) \ Unzip
Installation environment (PHP extensions) \ XML
Installation environment (PHP) \ Deprecated options in php.ini
Installation environment (PHP) \ File uploads
Installation environment (PHP) \ max_execution_time
Installation environment (PHP) \ max_input_vars
Installation environment (PHP) \ mbstring overload
Installation environment (PHP) \ Memory limit
Installation environment (PHP) \ Needed functions
Installation environment (PHP) \ open_basedir
Installation environment (PHP) \ Suhosin
Marketing \ Analytics
Marketing \ Google Analytics
Marketing \ Social media
Network \ External access
Network \ Packet loss (slow)
Network \ Transfer latency
Network \ Transfer speed
Newsletter \ Newsletter queue
Performance \ Cookies
Performance \ Heavy bot activity
Performance \ HTTP optimisation
Performance \ Manual performance checks
Performance \ Page speed (slow)
Performance \ Setup
robots.txt \ robots.txt completeness
robots.txt \ robots.txt correctness
robots.txt \ robots.txt validity
robots.txt \ Sitemap linkage
Security \ Admin Script Access
Security \ Directory securing
Security \ Exposed backups
Security \ Exposed BigDump tool
Security \ Exposed execute_temp.php
Security \ Exposed PHP-Info script
Security \ Exposed PhpMyAdmin utility
Security \ IP backdoor left active
Security \ Malware
Security \ Site orphaning
Security \ WebShells in likely directories (backdoor scripts)
SEO \ Crawlers incorrectly banned
SEO \ H1 tags
SEO \ Manual SEO checks
SEO \ Meta description
SEO \ Meta keywords
SEO \ Page titles
SEO \ XML Sitemap
Server performance \ CPU load
Server performance \ CPU speed (slow)
Server performance \ CPU type
Server performance \ Disk space
Server performance \ Hanging processes
Server performance \ I/O load
Server performance \ I/O speed (slow)
Server performance \ RAM
Server performance \ Server uptime
Software integrity \ Addon upgrade completion
Software integrity \ Database corruption (slow)
Software integrity \ Database
Software integrity \ Files (slow)
Software integrity \ File permissions
Software integrity \ Upgrade completion
SSL \ Insecure embedding
SSL \ Insecure linking
SSL \ SSL correctness
SSL \ SSL grading
SSL \ SSL on
Stability \ Block integrity (slow)
Stability \ Error log
Stability \ Manual stability checks
Stability \ Page integrity
System scheduler \ Slow system scheduler
System scheduler \ System scheduler set up
Upkeep \ Admin account staleness
Upkeep \ Composr version
Upkeep \ Copyright date
Upkeep \ PHP version
Upkeep \ PHP version (if not distro default)
Upkeep \ Staff checklist
User-experience for mistakes \ 404 pages
User-experience for mistakes \ HTTPS redirection
User-experience for mistakes \ www redirection

Configuration

There are a large number of configuration options available, under Admin Zone > Setup > Configuration > Health Check options.
This includes:

setting which pages to do deep scans of
calibrating the checks (setting thresholds)
configuring system scheduler checks, including the frequency, and which check sections to run

Tips

Make sure that your system scheduler (Cron call) is running the same PHP options as normal web requests, otherwise you may get spurious errors caused by differences in configuration.
System scheduler configuration is documented in the Basic configuration and getting started tutorial, and this includes sample commands that will share your .user.ini PHP configuration.

What if things break too badly for the Health Check to run?

There are a couple of approaches you can take to make sure you know if the health checker is itself down:

Feed an uptime checker into data/health_check.php – if it gives an HTTP error or the server does not respond, then you know Health Checks do not run (and a none-blank result shows there is a failing health check)
Enable the "Send full reports" option and check you receive the report each day (either manually or using some kind of third party e-mail scanning tool)

Philosophy

There is a vast amount to check when it comes to web development. Even though we check over 100 things, we can not realistically check everything under the scope of the Health Check.
The Health Check focuses on issues likely faced by end-users, and assumes Composr itself has been well programmed and tested.

The Composr ecosystem also has:

Automated testing. This is used for testing a huge array of low-level coding issues (too detailed for Health Check, possibly destructive, and requiring a lot of testing framework code). The tests are regularly run prior to Composr releases being made by the developers.
PHP-Info. The Health Check is not intended as a way to report on all your settings, only problematic ones. If you want to list some information about the PHP environment, use Admin Zone > Tools > PHP-Info to do this.
Cleanup tools. The Health Check does not attempt to "cleanup" caches and so on, only check for problems. The cleanup tools can be accessed from Admin Zone > Tools > Cleanup tools.
Staff checklist. The staff checklist (on the Admin Zone dashboard) has detailed information on staff actions that need performing. There is integration of this into the Health Check, but only at a high level.
External scanning tools. The Health Check doesn\'t itself directly ensure your website is perfectly optimised, it just focuses on the more major issues, and does sign-posting to other tools. The Health Check will link to various external tools that can perform detailed scans, such as gauging your SSL security.

Additionally you may want to check out Chris Graham's Houndr project which can be used to remind yourself of manually things to check, or tell you when social media is not being updated.

Specific checks

"Upkeep \ PHP version" vs "Upkeep \ PHP version (if not distro default)"

The latter check is an alternative to the former check, and takes into account that some Linux distributions back-port fixes to their older bundled PHP versions. It will not check the version of PHP so long that it is installed as /usr/bin/php.
The latter check is only available on systems that support the which command (essentially, on Linux).

Make sure you keep your server patched though! If you aren't on managed hosting and are not able to regularly schedule yourself to do manual updates, turn on automatic updates. There are real vulnerabilities coming out for software you'll be using all the time, so if you don't patch you'll be vulnerable.

PHP-Info

'PHP-Info' displays information about the PHP server environment. This is based on a feature PHP itself provides, of the same name – but we also add in some extra checks of our own, putting out warnings if software requirements are not met.

Information includes all the technical details of the PHP configuration, such as the installed PHP extensions and the defined options.

Some PHP options can be defined in the .user.ini file (see our FAQ). Full details of this are in the PHP manual.

The PHP-Info is also convenient for identifying wider factors of the system environment, such as server paths.

PHP-Info can be reached from:
Admin Zone > Tools > PHP-Info / Server Checks

Failover mode

You may configure a special failover mode via the config_editor script. This mode serves cached versions of page if your server seems to be failing. It is not a perfect system because:

it serves hits as guests
it can only work if the static cache is populated
it assumes that your server is only failing for performance/database reasons, not totally failed (however most failures in practice are like this)

However, it's still a great system because it lets your site content continue to be available even under massive load.

Failover can automatically fall into place based on a number of criteria.

Note that the failover status messages (failover_message_place_after and failover_message_place_before options) are put in place at the caching stage, not dynamically. So you would need to empty the static cache if you change these messages to something else.

Failover mode requires the PHP usleep function not be disabled.

Cleanup tools

The website cleanup tools page will allow you to empty any website cache, as well as automatically locate and repair common problems, and remove any orphaned data. There are a number of cleanup tools, which are run individually; ideally none ever need to be used, but if things happen outside the norm (such as corruption, or bugs), they can be very useful.

The tools are divided into "De-cachers" and "Optimisers" (which don't all strictly involve optimising, some are for general other kinds of maintenance task).

Caches / De-cachers

For performance reasons, Composr defines a number of caches. During normal operation of Composr, these caches should be unnoticeable; if things are being edited from outside Composr theoretically Composr will automatically pick up on it, but you may sometimes need to force a cache flush.

On a default install all recommended caches are enabled.

Some kinds of cache may be turned off in the configuration, but leaving them on gives vastly improved performance.

If you can identify a situation where you need to rebuild or empty a cache that should not exist, please report it as a bug.

Self-learning cache

The self-learning cache is quite a sophisticated cache mechanism to speed up Composr. It learns what resources are used by a page, and then bulk-loads them in the future.

The self-learning cache is most relevant when it comes to language strings. To avoid having to load up all the language files a page might use (language files are relatively monolithic, so inefficient to load), it will remember what strings are used by a page. Pages may use different strings when executed in different contexts, so the cache will always fall-back to loading the full language files if it has to, with those strings then also being added to the cache for the future. However, what if a page is referencing a string that does not exist? The self-learning cache would always be loading the full language files to try and hunt for it. This is why the cache also learns what strings are used by a page but do not actually exist even in the full language files. A ramification of this is that if you are developing and add the strings later, you will need to clear out the self-learning cache for Composr to be able to pick up on the new strings.

Optimisers (advanced)

There are a number of optimisers:

Find orphaned uploads – find on-disk uploads that seem to no longer be referenced.
Delete page access statistics – remove old page statistics to reduce database usage.

Frankly most of these optimisers are not useful. We provide them for very rare situations, or for helping programmers manage complicated operations.

There are some additional tools for repairing database problems in the upgrader, see the Performing an upgrade tutorial.

Broken URL scanning

There is a powerful broken link scanner at Admin Zone > Tools > Broken URL scanning.
This can check:

internal links
inbound (external) links to your site, if you configure Moz and/or Google Search Console from Admin Zone > Configuration > Composr API Options

We have the following kinds of broken URL checking across composr:

The broken URL scanning tool discussed here (works at the content level)
The Health Check, scanning specific URLs to see if the page there has broken URLs (works at the HTML level)
Comcode, URLs are checked when Comcode is parsed

Feedback

Please rate this tutorial:

Have a suggestion? Report an issue on the tracker.