Test release of v10, with higher I/O performance

Post

Posted
Rating:
Item has a rating of 5 Item has a rating of 5 Item has a rating of 5 Item has a rating of 5 Item has a rating of 5 (Liked by Adam Edington)
#4358 (In Topic #876)
I invite you to help beta test this next patch version of Composr v10 that has improved I/O performance. Please don't test on a real live website.

I am providing both kinds of installer to choose from:
ERROR: A resource, -1, requested within some Comcode (the attachment tag) does not exist.
ERROR: A resource, -1, requested within some Comcode (the attachment tag) does not exist.

Explanation

Over the years we've noticed that some Windows machines have significantly worse performance with Composr than if those same machines were running Linux. Specifically, the I/O performance sometimes is much worse on Windows – presumably as Linux either has smarter caching, or has less cruft filling up the cache, or a larger default cache.

Good I/O caching is crucial, as if Composr does 200 file reads to make a request, but each is having to do disk access with a latency of 5 milliseconds, that's 1 second just reading from the disk for that request. And 5 milliseconds is pretty conservative  – a stressed or fragmented system might be much higher. 200 reads seems like a lot, but actually with loading up the PHP code files, and checking cached files aren't stale, that's pretty normal (as it would be for other large systems). With a properly functioning I/O cache Composr should just touch the disk minimally, except when warming up – so file reads would be much less than a millisecond on average.

We've improved things over the years a lot even for systems with bad I/O – but I had a report on Friday about the Setup Wizard timing out on a Windows machine after 30 seconds. The Setup Wizard might touch 2000 files on the final step, so if I/O caching is not working well that's a problem. Normally the Setup Wizard is allowed 600 seconds to complete the step, but some PHP configurations do not allow overriding the timeout.

I decided to take the opportunity to approach optimising I/O performance a different way. I developed an intermediate layer (a PHP "stream wrapper") that would allow me to log exactly what I/O Composr was doing and why, so I could do deep optimisation on it – and so I could also artificially slow-down I/O for benchmarking purposes. Through this I identified quite a few optimisations to make, possibly doubling the speed of the Setup Wizard on systems that are bound by slow I/O.

I have the wizard finishing in 65 seconds for simulated 5ms I/O time on a git clone. That's with every bundled and non-bundled addon running, which roughly doubles the amount of I/O required. I expect this will allow it to finish well within 30 seconds on a Windows system with an I/O caching problem.

Let me know how it fairs. I don't think I've introduced any bugs, and all tests pass, but I might have overlooked something. Usually we wouldn't do deep optimisation for an upcoming patch release, but good performance on any real world system is very important.
I think you'll notice a speed up, even if you don't have any I/O problems.

Post

Posted
Rating:
#4359
"This server appears to have a relatively slow hard disk, or is overloaded – we had to skip checking the integrity of the uploaded files as it was taking too long." Didn't get that message in earlier versions of v10 and this still makes the initial load of the installer take longer than it should before deciding to show this alert. A refresh makes the alert disappear although I am not sure if the disk files have been checked via that refresh or it's just hiding the alert.

The site does seem snappier in general though clicking the proceed buttons manually doesn't seem as snappy as leaving them to auto-proceed when installing. Will also mention that the initial welcome page didn't seem to load correctly when I clicked Visit Website, I had to refresh there too, though this could be my relatively slow hard disk (on a relatively new PC). This is on localhost Bitnami Wampstack, not Windows based hosting.
 

Post

Posted
Rating:
#4364
Thanks,

The integrity check is only in the manual installer, as it's not realistically possibly to have corruption come out of the quick installer (if it was corrupt, that wouldn't run). I didn't change that, but I ran some calculations, and I can see that we're now really pushing it for this to finish. It would take around 35 seconds to finish for 5ms disk reads, as our file count has crept up over the years passed 6000. I've made some adjustments now so it doesn't check the .php files, as if those are corrupt you'll know about it anyway via some direct error message from that file. In the ideal world we'd still check them, but we have to be practical, and the upgrader's integrity check can still do it.

The automatically proceeding may be snappier simply because it gives the machine a bit more time to settle down.

The start page not loading right on first load, probably something random outside our control, I'm unlikely to be able to reproduce that one.

Post

Posted
Rating:
#4398
0 guests and 0 members have recently viewed this.