View Issue Details

IDProjectCategoryView StatusLast Update
1389Composrcorepublic2015-07-14 23:45
ReporterChris Graham Assigned ToChris Graham  
PrioritynormalSeverityfeature 
Status resolvedResolutionfixed 
Summary1389: XML Sitemap index
DescriptionCurrently Composr builds a single sitemap nightly. For a large website, this is resource-intensive and risks going over the 50,000 sitemap URL/10MB limit.

Ideally we would support the sitemap API putting out everything since the last "archival date", and then iteratively updating the sitemap index file to include the refreshed reference to the latest sitemap. It would have to calculate the archival date in a smart way, noting when it approaches a certain limit (e.g. 5000 URLs per sitemap).
Additional InformationOld sitemaps would contain URLs duplicated in newer sitemaps, if the article had been refreshed. This is not a serious problem though.
TagsNo tags attached.
Attach Tags
Time estimation (hours)10
Sponsorship open

Sponsor

Date Added Member Amount Sponsored

Relationships

related to 142 ResolvedChris Graham Merged sitemap API 

Activities

Chris Graham

2015-07-14 23:45

administrator   ~2954

Resolved in https://github.com/ocproducts/composr/commit/29c9de41415bd5c07bd75377da7dc2ffa0d2eebc

Implementation differs to description. Actually works much better, this is very efficient (can scale to millions of URLs), and can handle deletions of URLs from the sitemap.

Issue History

Date Modified Username Field Change