Make submitting for import fail if column headers aren't as expected

ConversazioniRecommend Site Improvements

Iscriviti a LibraryThing per pubblicare un messaggio.

Make submitting for import fail if column headers aren't as expected

1r.orrison
Apr 19, 2018, 6:39 am

I'm sure I've suggested this before, but probably as a comment in some other thread.

The importer often goes wrong, because people put the wrong things in the wrong columns. For example, most of the works on https://www.librarything.com/author/false look like someone put tags in the Title column and "False" in the author column.

CSV (and TSV) files include a header row. When a file is submitted for import, the site should immediately check all the field names in the header row before submitting it to the queue. If field names in the header aren't all as the importer will expect, then the site should notify the user right then and there, and offer them the download of the template file with the correct column layout.

The importer itself should also check all the field names in the header row before starting to process the file, just in case something slipped through.

2lorax
Apr 19, 2018, 9:41 am

This is a good idea, but I'd rather see the importer completely reworked from the ground up to be more robust to things like mis-ordered columns (obviously it cannot account for cases where the users' data is inconsistent with their stated headers) and - as everyone knows I want - to be able to import all user-editable fields in the LT export file.

3r.orrison
Modificato: Ott 23, 2018, 4:31 pm

Oh yes indeed! I'd rather see it fixed up right too, but this has the advantage of being much simpler to implement - it doesn't require any changes to the code that imports and creates the books, just a check before handing off to the existing importer.

6r.orrison
Modificato: Set 5, 2019, 6:10 pm

Bump. Also, obviously, check that it is a TSV or CSV file and not something else.

7r.orrison
Modificato: Apr 9, 2020, 8:38 am

Bump: https://www.librarything.com/topic/219866#7121523

Copying one existing book record isn't allowed, because that can cause proliferation of bad data. But importing thousands of records of bad data because the columns aren't in the right order isn't considered a problem.

8r.orrison
Mar 23, 2021, 4:11 am

Bump. This seems like a no-brainer to me, but if you're not going to do it can you come out and say "we're never going to verify that import fields are in the right order"?

9r.orrison
Giu 1, 2022, 1:23 pm

Bump.

10melannen
Giu 1, 2022, 7:03 pm

+1

11kristilabrie
Giu 2, 2022, 9:03 am

I've passed along the sentiments about import improvements (better verification/messaging upon upload, better messaging throughout the process e.g. which records failed to import and why, etc.) to the team. I can't promise any immediate changes but will post any updates.

12InfoChallenges
Modificato: Giu 10, 2022, 9:52 am

+1, import-column validation

13Nicole_VanK
Giu 8, 2022, 10:00 am

This is a good idea!

14r.orrison
Giu 14, 2022, 3:40 pm

From the other thread, prompted by a comment by AnnieMod:

How about after quickly checking just the first row of the file, if the headers don't match ask the user if they want to search the file for ISBNs?

E.g. "The submitted file is not a correctly formatted CSV file for full data import. Do you want to search the file for ISBNs and do the import solely on the basis of any ISBNs found in the file?" (If that's how it works.)