A simple default implementation of a CDN transfer hook (with associated config options) would allow just mapping of files onto a particular directory path and base URL combination.

A peripheral thing I'd like to solve with this work is case-insensitive filenames. If you develop on Mac or Windows there's a chance you'll mess up with case-mismatches but not notice until it goes to a Linux server.

The filesystem wrapper would have an option to force case-sensitivity, even if just mapping to a case-insensitive filesystem.

I gave this a lot more thought.

Notes...


function init__dyn_file_manager()
{
define('DATA_CLASS_SYSTEM', 1);
define('DATA_CLASS_SYSTEM_CUSTOM', 2);
define('DATA_CLASS_USER', 4);
define('DATA_CLASS_VOLATILE', 8);

$GLOBALS['DYN_MANAGER'] = new DynFileManager();
}

class DynFileManager
{
protected $hook_obs;

function __construct()
{
$this->hook_obs = find_all_hook_obs('systems', 'dyn_file_manager');
}

function find_path($type, $data_class)
{
// May return a path that is a filesystem wrapper path; normal file operations can then be done
}

function find_file_path($type, $subpath, $data_class_filter = null)
{
}

function find_url($type, $subpath, $relative = false, $data_class_filter = null)
{
}

function file_path_to_url($url)
{
}

function url_to_file_path($file_path)
{
}

function find_unique_filename($type, $subpath)
{
}

function copy_to($tmp_path, $type, $subpath)
{
}
}

class DatabaseFilesystemWrapper
{
}


Examples...

$GLOBALS['DYN_MANAGER']->find_file_path('uploads/banners', 'example.png');




Notes....

Instead of just "non-custom" and "custom", we now have "system", "system custom" and "user data" - and this is an override chain. Some things are probably only user data, e.g. uploads.

We move everything that changes during run-time and is shared between installs under a '_user_data' directory.
data/data_custom currently conflates too much. Have data/data_custom, resources/resources_custom, scripts/scripts_custom, logs.
uploads/website_specific will have to change, as this is not uploaded. Merge to resources_custom
Move caches and logs under a '_volatile' directory. Actually whether logs should be volatile or not should perhaps be configurable.
Document what '_volatile' and '_user_data' (i.e. _volatile is not to be synced, _user_data is). Document the whole override chain system.

Our API will allow hooks to override the functionality

Options for specifying which directories are 'system data' vs 'user data' (so you can for example decide all theme files and Comcode pages are 'system data')
Other directories are hard-coded as one or the other
Warnings if editing anything that would edit 'system data' and therefore should be done on a development level - but only if an option is enabled for these warnings
Both kinds of data would be managed via the same API
Search both locations for data, but in priority order (even for ones hard-coded - as shared installs may be using stuff as system data).
Themes and translations should definitely be system data, as otherwise it would complicate distribution of them as addons.

What about new data (e.g. a new download) that is being added at the staging stage? Programmer will have to deal with this manually.

This the time to drop non-suexec support? Put check in installer that all files are owned by web user. Quick installer will now not extract using FTP, just FS - and complain if no write access. Remove all chmodding references. Remove abstract file manager. Change written minimum requirements. Remove fix_permissions.

Auto-create missing directories.

Ability to store all user data in DB. Controlled via hidden option, function to switch between that is documented.

Merge in cdn_transfer hook functionality (broadly these will become dyn_file_manager hooks)

Another kind of hook that just listens to changes (dyn_file_manager_sync). Needs to be called by filesystem wrappers and DynFileManager functions.

Re-write the tut_optimisation tutorial. Maybe rename to tut_performance.
Document that '_user_data' can be placed under shared storage. Or you can put in DB. Or you can have an addon that puts it elsewhere (or multiple places). Or you can implement a dyn_file_manager_sync hook. Document advantages - DB may be best because it is synced across machines automatically, so minimises bottleneck.
Document to NOT try and use rsync, as there is no 'master' and thus deletes would be messed up.

Kill sync_file.

get_custom_file_base and get_custom_base_url can go, as 'user data' is now same as custom data. Each install gets it's own '_user_data' directory.

In dev-mode host the entire Composr filesystem under a filesystem wrapper and give errors if file-ops are done on things that should not be.

A peripheral thing I'd like to solve with this work is case-insensitive filenames. If you develop on Mac or Windows there's a chance you'll mess up with case-mismatches but not notice until it goes to a Linux server. The filesystem wrapper would have an option to force case-sensitivity, even if just mapping to a case-insensitive filesystem.

Implement Allow backup to use cdn_transfer mechanism https://compo.sr/tracker/view.php?id=2962 (almost done automatically)

Kill upload_syndication hooks. Over complex and not really user friendly.

This is now covered on this spreadsheet: https://docs.google.com/spreadsheets/d/1_yaJeGzDIsxq33I7Wg9I-lTBDk3YS22WPBwJ971v5tI
Services referenced:
Amazon S3
Cloudinary
Dropbox
Google Drive

" The filesystem wrapper would have an option to force case-sensitivity, even if just mapping to a case-insensitive filesystem. " - this is now done in debug_fs.php, although it is only implemented as a specific debug option, and not related to the rest of the functionality discussed here.

A fresh look at all this, which I think is both simpler and more powerful...

Overall strategy
================

We have many different kinds of "content across servers" scenarios we need to support well:

1) Content Delivery Networks [CDN] (locate asset files geographically close to users to minimise site download time)
2) Server farms (spread load across multiple servers)
3) Staging servers (pushing content from a staging server to a live server)
4) Multi-site (having content on a Demonstratr-style master site available to satellite sites - e.g. site-builder scenario)
5) Git (implementing content inside a Git repository then pulling it live)

And here's how we approach them...

1) Content Delivery Networks -- promote smart CDNs that will automatically pull assets from the master site on-demand, with proper cache management (avoiding the need for the server to ever proactively push anything)
2) Server farms -- support mounting network storage onto the new smart filesystem feature or implement hooks on it
3) Staging servers -- new Sync UI feature
4) Multi-site -- new smart filesystem feature
5) Git -- no special approach needed

Sync UI
=======

Have a new UI for synching between a staging site and a live site.

The details of the live site would need configuration, some kind of API-key system.

It would be laid out something like...

<table>
<thead>
<tr>
<th colspan="3">Repository object [sort]</th>
<th colspan="2">Modification date [sort]</th>
<th colspan="2">CRC</th>
<th rowspan="2">Sync action</th>
</tr>
<tr>
<th>GUID</th>
<th>Title</th>
<th>Moniker</th>
<th>Staging</th>
<th>Live</th>
<th>Staging</th>
<th>Live</th>
</tr>
</thead>
<tbody>
<tr>
<td>xxx</td>
<td>xxx</td>
<td>xxx</td>
<td>3rd Dec 2018 2:03 pm</td>
<td>(only on staging)</td>
<td>xxx</td>
<td>(only on staging)</td>
<td>
<select>
<option>Leave staging-only</option>
<option disabled>Leave live-only</option>
<option>Copy from staging to live (includes revision history)</option>
<option disabled>Copy from live to staging (includes revision history)</option>
<option disabled>Delete from live</option>
<option>Delete from staging</option>
<option disabled>Delete from both staging and live</option>
</select>
</td>
</tr>
...
</tbody>
</table>

Also clear caches aggressively on live after sync <input type="checkbox" />

Take database backup on staging <input type="checkbox" />

Take database backup on live <input type="checkbox" />

<button>Sync</button>

For this synching system to work well, we'd ideally want to completely remove IDs from Composr and replace with GUIDs. We currently do have GUIDs as an optional feature, but they're not usually used.

Also we need to consider that a sync may result in a moniker conflict, and we have to somehow ask how to resolve that.

Smart filesystem
================

Instead of:
- get_custom_file_base
- sync_file
- (much of what is considered in this issue with having a custom class to solve it)

... have all file I/O go through a custom PHP stream-wrapper

Allow configuring into _config.php how other paths mount onto the default base directory (which may themselves by PHP stream-wrappers. FUSE-mounts, or whatever).

Allow mounting multiple paths in the same position, with precedence. This allows a multi-site scenario to work well.

All I/O operations would support hooks, so you can for example write custom sync code to sync onto different servers.

This issue isn't fully updated, but we have been, and are, generally moving in this direction - hopefully concluding for v12.

I hope we can do the service integration leveraging Hybridauth. This way we don't need to be fully responsible for implementing/maintaining all the integrations we might ever want to do, we pool the work with other users of Hybridauth. For #2020 Hybridauth is getting lots of new code for sharing social networks 'Atoms' back and forth, for Hybridauth providers that implement that. I've designed the API to be flexible enough to also use this same API for filesystem access.

Hybridauth work needed...

Add new providers:
Amazon S3 (no auth)
Cloudinary (no auth)
Google Drive

Add Atom support to existing providers:
Dropbox

The capabilities code in sources_custom/hybridauth_admin.php will be important, and need to be extended for these new providers. We rely on this metadata to know what providers can actually be integrated for filesystem support, and how.
I was considering trying to merge this into Hybridauth too, but I think it would bloat up that project too much.

I am closing this issue.
Many superior new ideas have happened since it was created, and a lot has been completed - and really we're talking about a lot at once here.

The Hybridauth stuff was implemented last year and allows so many third party logins, pull of third party content via Atom, push of content via an Atom-like API (replacing our old syndication functionality).

Soon I will merge support for a new "cloud filesystem" within Composr, implemented on top of PHP stream wrappers. This achieves a lot of what is discussed in this issue, but in a simpler way.
A new logging API has also been completed, allowing syndication of individual log lines via the database or via syslog.

What remains will be moved to new issues.