View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 5473 | Composr | core | public | 2023-11-27 00:23 | 2024-11-28 15:43 |
| Reporter | PDStig | Assigned To | PDStig | ||
| Priority | normal | Severity | feature | ||
| Status | resolved | Resolution | fixed | ||
| Summary | 5473: v12: Consider *_compiled directories | ||||
| Description | In v12, consider making *_compiled directories in addition to the * and *_custom directories. Composr CMS will create / update PHP files in these directories containing the final compiled code when considering *_custom and contentious overrides. When requiring code, ultimately use the *_compiled file. This will make development easier especially regarding stack traces. With our current set-up, stack traces often mask the origin of errors due to eval code. We want to minimize use of eval so we can get proper stack traces. In this case, the stack trace would come from the *_compiled file. | ||||
| Additional Information | Perhaps make other considerations as well... Whenever an override was made, a comment is automatically added in the *_compiled PHP file indicating the location/file and line number of where the override takes place. Then add an END comment where that particular override ends. The *_compiled files should be treated as a cache and therefore a website cleanup tool should have the option of cleaning this out. Document that *_compiled files are not meant for editing (just like * files without _custom). Edits should be made by making or editing an override. In production, *_compiled files should normally be generated on-the-fly as needed. But have a script available that can be run by developers to compile the entire code base at once. When requiring, Composr should check the last file modification date/time for the * and *_custom files and compare it with the *_compiled file. If either are newer than the *_compiled file, then regenerate the *_compiled file, else just load the current *_compiled file. Of course, if config option to minimize disk accesses is set, then this does not happen and *_compiled files must be deleted (or the cleanup tools run) to regenerate them. (Need to consider how we would achieve the same for contentious overrides). If using an editor in Composr, it should automatically delete the relevant *_compiled file when saving so it can be regenerated. Or perhaps also regenerate it then and there. Perhaps we should consider whether or not this should be a "development only" feature, e.g. production builds do not include *_compiled directories and do not use them (instead using eval like before). And such feature only activates in dev mode or when running in a git repository. Normally, this feature would have little to no benefit outside of development. Consider modifying git hooks to clear out *_compiled files on pull. Also consider clearing out *_compiled files on decache. Consider utilizing *_compiled system when accessing /root (I'm not sure if contentious overrides currently apply to /root files). | ||||
| Tags | Risk: Changes core website behaviour , Roadmap: Over the horizon | ||||
| Attach Tags | |||||
| Time estimation (hours) | |||||
| Sponsorship open | |||||
|
|
Keep in mind that mtime checks are very slow. So perhaps this should be combined with self learning cache and only check mtimes every x minutes. However... a better solution: If we utilize runtime compilation, we could run into dog-piling on parallel servers (when the cache of a PHP file expires, multiple threads may try re-compiling it at the same time). Therefore, the job of compiling the PHP files should be left up to a Scheduler task (although we might consider runtime compilation when in dev mode). Require code looks for a compiled version first, and if it does not exist (either the cache was cleared or the scheduler was disabled), falls back to its previous behavior of eval-ing together the main and custom versions. To avoid heavy load, the scheduler task should still utilize mtime and only re-compile if either the main or the custom file's mtime is newer than the compiled file. CONTENTIOUS OVERRIDES We need to give special consideration to contentious overrides as well. Perhaps contentious overrides need a new info function which defines all applicable files it overrides. That way, the scheduler can also check and compare the mtimes of these files and re-run compilation (also compiling in the contentious override) when applicable. Some overrides might not have a clear cut and dry definition of files they override, so maybe the info function receives the name of a file being checked by the scheduler and returns true of that override applies to it, false otherwise... and every info function is run for every file checked. |
|
|
Implemented out of necessity through 5594 as 5594 made stack traces completely useless in dev mode. However, I did not follow the original plan. Here is what I changed: * There is only a single "_compiled" directory which contains the compiled PHP code in the same directory structure as Composr itself (e.g. _compiled/sources, _compiled/site/pages/modules, and so on). * Composr will not generate a _compiled file unless it is necessary (we have a code override in place for that particular script). Otherwise it will just include the original file. In dev mode, however, this amounts to almost every file because of 5594 injecting declare(strict_types=1) at the top of every PHP file which is not listed as untouchable third-party code. * We do not check file modification times, nor do we use self-learning cache (because it might not be initialized yet given compilation occurs extremely early within global.php). Instead, we determine first if any overrides are in place, and if so, what the final code will be. Then we take the crc32 hash of the final code and the crc32 hash of the _compiled file (if it exists). If the crc32 hashes do not match, or there is no _compiled file, we write. Otherwise, we do not to save on precious IO time. * "If using an editor in Composr, it should automatically delete the relevant *_compiled file". We're not doing that because of the above crc32 hash check. If the hashes don't match, the _compiled file will be modified. Additionally, if we determine there are no code overrides anymore, but a _compiled file exists, it will be deleted and the original files used instead. * Not a development only feature; this could be helpful for debugging in production as well. I've run some testing and, although the initial generation of _compiled files can be slow (if you have a lot of overrides), Composr operates quickly as it should after the _compiled files are generated thanks to the crc32 hash checks. Plus, normally only a small number of files need compiled; as mentioned, Composr will not compile any files that do not have overrides. * "Consider modifying git hooks to clear out *_compiled files on pull. Also consider clearing out *_compiled files on decache." Also not doing either of these because again, Composr is being smart about it with crc32. But if we absolutely must, there is a cleanup tool to erase the _compiled directory. * "Consider utilizing *_compiled system when accessing /root (I'm not sure if contentious overrides currently apply to /root files)." - Deferred as this is not necessary for what we need right now. * Composr is not using eval anymore for require scripts... ever. Everything is PHP included. Composr will save separate custom PHP files into _compiled and modified originals (to have the non_overridden function names) to ensure everything works as intended. Two major benefits to this are that we have proper stack traces even when overrides are in place, and we can utilise PHP's cache even on overridden code. Regarding dog-piling: The compilation process involves file locking. This was a pain in the buttocks to set up, but I think I have it working for the most part now. It is possible for multiple PHP processes to request the same scripts at the same time due to AJAX requests. So the compilation process must be entirely sync. * When compiling: - We skip all untouchable third-party code and anything we marked as already compiled by this process / execution. - Composr checks for overrides (including contentious). If none exist, we don't compile anything (and we return true to indicate if a _compiled file does exist, it should be deleted once we call this code). - Composr makes missing _compiled directories and index.html files to prepare for compilation. - Composr checks if the file already exists. If so, we take the crc32 hash of it and compare it with the file. If they do not match, or the file does not exist, we prepare to write. - We use file_write_contents in a while loop with an exclusive lock and non-blocking lock checks. If it returns false, we keep trying every 250ms for up to 5 seconds in case another process had a lock on it. - Once the script has been compiled (or we determined it did not need to be compiled), we begin the calling. * Calling: - We check if the Composr script we want to call has a _compiled version available. If it does, we use it, unless a parameter passed in said we actually want that to be deleted (the compiling stage will return true or false in that regard for us to pass in here). - Any scripts we already called in this process / execution, we skip. - We attempt a shared file lock on the file in a while loop. We keep trying every 100ms for up to 5 seconds. - Once the lock has been established, we PHP require the script in (we never use eval anymore), and then release the lock. |
|
|
I should also mention this makes a breaking change to init__ functions. You cannot use them anymore for code overrides; init functions no-longer pass in any parameters. Instead, contentious_overrides must be used. I migrated all of the existing init overrides over to contentious hooks. |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2023-11-27 00:23 | PDStig | New Issue | |
| 2023-11-27 00:23 | PDStig | Tag Attached: Roadmap: v12 | |
| 2023-11-27 00:23 | PDStig | Tag Attached: Risk: Changes core website behaviour | |
| 2023-11-27 00:25 | PDStig | Additional Information Updated | |
| 2024-01-27 20:09 | PDStig | Relationship added | related to 3856 |
| 2024-01-27 20:32 | PDStig | Note Added: 0008257 | |
| 2024-03-26 00:58 | PDStig | Tag Renamed | Roadmap: v12 => Roadmap: Over the horizon |
| 2024-03-30 03:29 | PDStig | Project | Composr alpha bug reports => Composr |
| 2024-03-30 03:51 | PDStig | Category | General / Uncategorised => core |
| 2024-11-28 15:01 | PDStig | Relationship added | related to 5594 |
| 2024-11-28 15:36 | PDStig | Assigned To | => user4172 |
| 2024-11-28 15:36 | PDStig | Status | Not Assigned => Resolved |
| 2024-11-28 15:36 | PDStig | Resolution | open => fixed |
| 2024-11-28 15:36 | PDStig | Note Added: 0009689 | |
| 2024-11-28 15:39 | PDStig | Note Added: 0009690 | |
| 2024-11-28 15:43 | PDStig | Note Edited: 0009689 |
