#3032 - Bulletproof spreadsheet importing

Identifier #3032
Issue type Feature request or suggestion
Title Bulletproof spreadsheet importing
Status Completed
Tags

Roadmap: v11 (custom)

Type: Cross-cutting feature (custom)

Type: Usability (major) (custom)

Handling member Chris Graham
Addon core
Description You can currently import CSV files, and it also supports text files with ";" as a delimiter (we don't advertise this).

This is pretty techie for some users.

We should ideally be able to handle any of these scenarios, using a clean API:
1) User accidentally uploads CSV with tab delimiter (TSV) not proper CSV
2) User accidentally uploads CSV with semicolon delimiter not proper CSV
3) User uploads a spreadsheet with no header row in a situation where we don't strictly need one (newsletter raw email list import comes to mind)
4) User uploads a CSV with old-Mac-style line endings (this should not happen in 2016 but some ageing codebases export bizarre stuff; the PHP enable auto_detect_line_endings setting may help)
5) User uploads a spreadsheet with a different character set, esp UTF-16 or ANSI (to solve this we'll probably need to have a character-set selector box and also support BOM markers)
7) Use uploads a very large files that won't fit in memory in one go
Steps to reproduce

Additional information Code files that load CSV files at the moment...

adminzone/pages/modules/admin_newsletter.php
sources/hooks/systems/tasks/import_member_csv.php
sources/hooks/systems/tasks/import_newsletter_subscribers.php
sources/hooks/systems/tasks/import_catalogue.php
sources/newsletter.php
pages/modules/recommend.php

We'd want all this to be carefully unit tested.
Related to

#1908 - Support importing of spreadsheets other than CSV
Funded? No
The system will post a comment when this issue is modified (e.g., status changes). To be notified of this, click "Enable comment notifications".

Rating

Unrated