Composr Tutorial: Website Health

This tutorial will cover some aspects of keeping your website running smoothly.

Health Check

The Health Check screen

(Click to enlarge)

On the modern advanced web there is too much to keep in mind to check. All kinds of things could go wrong without you noticing, which could be embarrassing.
For example, what if:

1. Your outgoing e-mail goes down, breaking sign-ups
2. You forget to renew SSL
3. You forget to renew your domain name
4. A hacker takes control of the domain name and puts up a fake site
5. You accidentally block Google from accessing the website

All the above scenarios are auto-detectable.
In fact, you can detect problems stemming from many kinds of situation, including:

• Quality issues after building a new site or theme
• Software compatibility issues
• Problems after a Composr upgrade
• Hardware failure
• Configuration issues
• Lack of server capacity
• Hack-attacks
• Lack of routine maintenance or website up-keep

Operation

The Health Check can be run manually, or regularly run in the background (requires the system scheduler to be set up and for you to enable the "Health Check results" notification).

Check sections

At the time of writing the list of check sections is:

• Backups \ Backups
• Bloated data \ Database size
• Bloated data \ Directory size
• Bloated data \ Log size
• Bloated data \ Table size (row count)
• Build mistakes \ Broken links (slow)
• Build mistakes \ Broken web POST forms
• Build mistakes \ Comcode page form fields
• Build mistakes \ Comcode page headings
• Build mistakes \ Common mistake patterns (page_errors.xml)
• Build mistakes \ Default icons
• Build mistakes \ Guest access
• Build mistakes \ Incomplete content
• Build mistakes \ Local linking
• Build mistakes \ Manual checks for web standards
• Build mistakes \ Spellchecking of all Comcode pages (slow)
• Build mistakes \ Spellchecking of miscellaneous content
• Build mistakes \ Spellchecking of pages
• Build mistakes \ Web standards
• Deployment mistakes \ Site open-status
• Domains \ DNS resolution
• Domains \ Domain expiry
• E-mail \ DKIM configuration
• E-mail \ DMARC configuration
• E-mail \ E-mail configuration
• E-mail \ E-mail configuration in PHP
• E-mail \ E-mail operation (slow)
• E-mail \ E-mail queue
• E-mail \ E-mail templates
• E-mail \ IMAP login
• E-mail \ List-Unsubscribe header
• E-mail \ SMTP blocklisting
• E-mail \ SMTP login
• E-mail \ Spam status
• E-mail \ SPF
• Hack-attacks \ Attack frequency
• Hack-attacks \ Failed logins
• Hack-attacks \ Overseas access
• Hack-attacks \ Rate-limit spiking
• Installation environment \ Base URL
• Installation environment \ Database Unicode settings
• Installation environment \ Directory naming
• Installation environment \ Disk Space (Installation)
• Installation environment \ Injected Ad Scripts
• Installation environment \ ModSecurity
• Installation environment \ MySQL version
• Installation environment \ PCRE settings
• Installation environment \ PHP platform
• Installation environment \ PHP version
• Installation environment \ SELinux settings
• Installation environment \ Server software
• Installation environment \ suEXEC
• Installation environment \ Umask settings
• Installation environment (PHP extensions) \ cURL
• Installation environment (PHP extensions) \ GD
• Installation environment (PHP extensions) \ Unicode
• Installation environment (PHP extensions) \ Unzip
• Installation environment (PHP extensions) \ XML
• Installation environment (PHP) \ Deprecated options in php.ini
• Installation environment (PHP) \ File uploads
• Installation environment (PHP) \ max_execution_time
• Installation environment (PHP) \ max_input_vars
• Installation environment (PHP) \ mbstring overload
• Installation environment (PHP) \ Memory limit
• Installation environment (PHP) \ Needed functions
• Installation environment (PHP) \ open_basedir
• Installation environment (PHP) \ Suhosin
• Marketing \ Analytics
• Marketing \ Google Analytics
• Marketing \ Social media
• Network \ External access
• Network \ Packet loss (slow)
• Network \ Transfer latency
• Network \ Transfer speed
• Newsletter \ Newsletter queue
• Performance \ Cookies
• Performance \ Heavy bot activity
• Performance \ HTTP optimisation
• Performance \ Manual performance checks
• Performance \ Page speed (slow)
• Performance \ Setup
• robots.txt \ robots.txt completeness
• robots.txt \ robots.txt correctness
• robots.txt \ robots.txt validity
• robots.txt \ Sitemap linkage
• Security \ Admin Script Access
• Security \ Directory securing
• Security \ Exposed backups
• Security \ Exposed BigDump tool
• Security \ Exposed execute_temp.php
• Security \ Exposed PHP-Info script
• Security \ Exposed PhpMyAdmin utility
• Security \ IP backdoor left active
• Security \ Malware
• Security \ Site orphaning
• Security \ WebShells in likely directories (backdoor scripts)
• SEO \ Crawlers incorrectly banned
• SEO \ H1 tags
• SEO \ Manual SEO checks
• SEO \ Meta description
• SEO \ Meta keywords
• SEO \ Page titles
• SEO \ XML Sitemap
• Server performance \ CPU load
• Server performance \ CPU speed (slow)
• Server performance \ CPU type
• Server performance \ Disk space
• Server performance \ Hanging processes
• Server performance \ I/O load
• Server performance \ I/O speed (slow)
• Server performance \ RAM
• Server performance \ Server uptime
• Software integrity \ Addon upgrade completion
• Software integrity \ Database corruption (slow)
• Software integrity \ Database
• Software integrity \ Files (slow)
• Software integrity \ File permissions
• Software integrity \ Upgrade completion
• SSL \ Insecure embedding
• SSL \ Insecure linking
• SSL \ SSL correctness
• SSL \ SSL grading
• SSL \ SSL on
• Stability \ Block integrity (slow)
• Stability \ Error log
• Stability \ Manual stability checks
• Stability \ Page integrity
• System scheduler \ Slow system scheduler
• System scheduler \ System scheduler set up
• Upkeep \ Admin account staleness
• Upkeep \ Composr version
• Upkeep \ Copyright date
• Upkeep \ PHP version
• Upkeep \ PHP version (if not distro default)
• Upkeep \ Staff checklist
• User-experience for mistakes \ 404 pages
• User-experience for mistakes \ HTTPS redirection
• User-experience for mistakes \ www redirection

Configuration

There are a large number of configuration options available, under Admin Zone > Setup > Configuration > Health Check options.
This includes:

• setting which pages to do deep scans of
• calibrating the checks (setting thresholds)
• configuring system scheduler checks, including the frequency, and which check sections to run

Tips

Make sure that your system scheduler (Cron call) is running the same PHP options as normal web requests, otherwise you may get spurious errors caused by differences in configuration.
System scheduler configuration is documented in the Basic configuration and getting started tutorial, and this includes sample commands that will share your .user.ini PHP configuration.

What if things break too badly for the Health Check to run?

There are a couple of approaches you can take to make sure you know if the health checker is itself down:

1. Feed an uptime checker into data/health_check.php – if it gives an HTTP error or the server does not respond, then you know Health Checks do not run (and a none-blank result shows there is a failing health check)
2. Enable the "Send full reports" option and check you receive the report each day (either manually or using some kind of third party e-mail scanning tool)

Philosophy

There is a vast amount to check when it comes to web development. Even though we check over 100 things, we can not realistically check everything under the scope of the Health Check.
The Health Check focuses on issues likely faced by end-users, and assumes Composr itself has been well programmed and tested.

The Composr ecosystem also has:

• Automated testing. This is used for testing a huge array of low-level coding issues (too detailed for Health Check, possibly destructive, and requiring a lot of testing framework code). The tests are regularly run prior to Composr releases being made by the developers.
• PHP-Info. The Health Check is not intended as a way to report on all your settings, only problematic ones. If you want to list some information about the PHP environment, use Admin Zone > Tools > PHP-Info to do this.
• Cleanup tools. The Health Check does not attempt to "cleanup" caches and so on, only check for problems. The cleanup tools can be accessed from Admin Zone > Tools > Cleanup tools.
• Staff checklist. The staff checklist (on the Admin Zone dashboard) has detailed information on staff actions that need performing. There is integration of this into the Health Check, but only at a high level.
• External scanning tools. The Health Check doesn\'t itself directly ensure your website is perfectly optimised, it just focuses on the more major issues, and does sign-posting to other tools. The Health Check will link to various external tools that can perform detailed scans, such as gauging your SSL security.

Additionally you may want to check out Chris Graham's Houndr project which can be used to remind yourself of manually things to check, or tell you when social media is not being updated.

Specific checks

"Upkeep \ PHP version" vs "Upkeep \ PHP version (if not distro default)"

The latter check is an alternative to the former check, and takes into account that some Linux distributions back-port fixes to their older bundled PHP versions. It will not check the version of PHP so long that it is installed as /usr/bin/php.
The latter check is only available on systems that support the which command (essentially, on Linux).

Make sure you keep your server patched though! If you aren't on managed hosting and are not able to regularly schedule yourself to do manual updates, turn on automatic updates. There are real vulnerabilities coming out for software you'll be using all the time, so if you don't patch you'll be vulnerable.

PHP-Info

'PHP-Info' displays information about the PHP server environment. This is based on a feature PHP itself provides, of the same name – but we also add in some extra checks of our own, putting out warnings if software requirements are not met.

Information includes all the technical details of the PHP configuration, such as the installed PHP extensions and the defined options.

Some PHP options can be defined in the .user.ini file (see our FAQ). Full details of this are in the PHP manual.

The PHP-Info is also convenient for identifying wider factors of the system environment, such as server paths.

PHP-Info can be reached from:
Admin Zone > Tools > PHP-Info / Server Checks

Failover mode

You may configure a special failover mode via the config_editor script. This mode serves cached versions of page if your server seems to be failing. It is not a perfect system because:

1. it serves hits as guests
2. it can only work if the static cache is populated
3. it assumes that your server is only failing for performance/database reasons, not totally failed (however most failures in practice are like this)

However, it's still a great system because it lets your site content continue to be available even under massive load.

Failover can automatically fall into place based on a number of criteria.

Note that the failover status messages (failover_message_place_after and failover_message_place_before options) are put in place at the caching stage, not dynamically. So you would need to empty the static cache if you change these messages to something else.

Failover mode requires the PHP usleep function not be disabled.

Cleanup tools

The website cleanup tools page will allow you to empty any website cache, as well as automatically locate and repair common problems, and remove any orphaned data. There are a number of cleanup tools, which are run individually; ideally none ever need to be used, but if things happen outside the norm (such as corruption, or bugs), they can be very useful.

The tools are divided into "De-cachers" and "Optimisers" (which don't all strictly involve optimising, some are for general other kinds of maintenance task).

Caches / De-cachers

Self-learning cache

The self-learning cache is quite a sophisticated cache mechanism to speed up Composr. It learns what resources are used by a page, and then bulk-loads them in the future.

The self-learning cache is most relevant when it comes to language strings. To avoid having to load up all the language files a page might use (language files are relatively monolithic, so inefficient to load), it will remember what strings are used by a page. Pages may use different strings when executed in different contexts, so the cache will always fall-back to loading the full language files if it has to, with those strings then also being added to the cache for the future. However, what if a page is referencing a string that does not exist? The self-learning cache would always be loading the full language files to try and hunt for it. This is why the cache also learns what strings are used by a page but do not actually exist even in the full language files. A ramification of this is that if you are developing and add the strings later, you will need to clear out the self-learning cache for Composr to be able to pick up on the new strings.

Optimisers (advanced)

Broken URL scanning

There is a powerful broken link scanner at Admin Zone > Tools > Broken URL scanning.
This can check:

• internal links
• inbound (external) links to your site, if you configure Moz and/or Google Search Console from Admin Zone > Configuration > Composr API Options

We have the following kinds of broken URL checking across composr:

• The broken URL scanning tool discussed here (works at the content level)
• The Health Check, scanning specific URLs to see if the page there has broken URLs (works at the HTML level)
• Comcode, URLs are checked when Comcode is parsed

See also

• Webhosting for Composr (including Composr minimum requirements)
• Optimising Performance
• Security
• Guide to web technologies (including HTML, CSS, and JavaScript)
• Disaster recovery
• Improving your search engine ranking
• Understanding and configuring e-mail
• Supplementary web technologies (media, social networking, etc)
• Metadata
• Introduction to Cookies and sessions (and JavaScript)
• Helping improve site accessibility for disabled users
• How domain names work
• Performing an upgrade
• Internet Archivee (Find archived web pages that are no longer online)

Feedback

Have a suggestion? Report an issue on the tracker.