Menu

#51 Upload CSV - Migration helper

open
nobody
None
5
2008-06-23
2008-06-23
H.-J. Tappe
No

This patch adds a biblio CSV import which can be used for library migration to openbiblio.

The CSV import file must be a tab separated file (use any parser-converter to convert your CSV file to the usable format). It can include barcode numbers to
ease migration but it does not yet handle duplicates cleanly (currently, it will add a complete entry, not just a copy).

The import page describes the column names understood by the import tool. Most are represented by the MARC identifier, other (openbiblio-special enries) carry
own column names.

Before import, a dry-run feature is available (like in the MARC import) to check for a lot of error conditions (missing mandatory data).

Translation-ready (en part of this patch).

Discussion

1 2 > >> (Page 1 of 2)
  • Nobody/Anonymous

    Logged In: NO

    While imports are receiving error: "Fatal error: Maximum execution time of 30 seconds exceeded in D:\wamp\www\openbiblio\catalog\upload_csv.php on line 205".
    Why get this error?
    Please give us and a CSV file format for tests.

     
  • H.-J. Tappe

    H.-J. Tappe - 2008-07-26

    Logged In: YES
    user_id=2125563
    Originator: YES

    The error message "Maximum execution time of 30 seconds exceeded" can be circumvented by:
    a) increasing the maximum execution time in php.ini and restarting the web server or
    b) using a smaller CSV file (try the first 10, 100, 1000 entries for a test).
    For the reference library, an import of 16000 entries used 10-15 minutes with a 1GHz CPU, 1GB RAM.

     
  • H.-J. Tappe

    H.-J. Tappe - 2008-07-26

    Logged In: YES
    user_id=2125563
    Originator: YES

    I uploaded a patch update,including:
    - A one-line sample CSV file
    - A update of the table on the welcome page (Call number was missing)

    Thanks for testing.
    H.-J. Tappe
    File Added: upload_csv.patch-0.6.X-1.zip

     
  • Nobody/Anonymous

    Logged In: NO

    I used this file(http://openbiblioromania.xhost.ro/exampleUpload.zip) but without positive results.Get the same error:"Fatal error: Maximum execution time of 200 seconds exceeded in D:\wamp\www\openbiblio\catalog\upload_csv.php on line 254".
    I do not know what I'll try to solve the problem.
    It would be nice to add in "upload_csv.patch-0.6.X-1" and a test file.

     
  • Nobody/Anonymous

    Logged In: NO

    Fatal error: Maximum execution time of 60 seconds exceeded in C:\xampp\htdocs\openbiblio\classes\BiblioCopy.php on line 120

    This is the error transmitted by XAMPP.

     
  • H.-J. Tappe

    H.-J. Tappe - 2008-07-26

    Logged In: YES
    user_id=2125563
    Originator: YES

    Update 2 of the patch includes an updated dependency handling and the example CSV file (one biblio item). Especially, the situation of bad entries in the collection or material column now are a warning instead of an error and a fallback as given on the main page is applied.

    With this file, the import cannot use more than one or two seconds, even on slow systems. On the top left of the import page, there is a counter, indicating how many items have been imported. Most other error cases will show a distinct error message. Please try another browser if this does not help. Another try could be the insertion of (echo "HOOK";) in the php code to find out where it loops or stalls on your system (not reproducible here; Linux environment, Firefox).

    About the file mentioned in the comment: You should remove the leading "+", which is a patch "inserted line" indication.
    File Added: upload_csv.patch-0.6.X-2.zip

     
  • H.-J. Tappe

    H.-J. Tappe - 2008-07-27

    Upload CSV 0.6.X Update #3

     
  • H.-J. Tappe

    H.-J. Tappe - 2008-07-27

    Logged In: YES
    user_id=2125563
    Originator: YES

    Update #3 adds a hint text that all MARC fields as known to openbiblio can also be imported by use of a CSV column with the according heading.
    File Added: upload_csv.patch-0.6.X-3.zip

     
  • Nobody/Anonymous

    Logged In: NO

    My default browser is Firefox
    Fatal error: Maximum execution time of 60 seconds exceeded in C:\xampp\htdocs\openbiblio\classes\BiblioCopy.php on line 120
    Opera
    Fatal error: Maximum execution time of 60 seconds exceeded in C:\xampp\htdocs\openbiblio\catalog\upload_csv.php on line 211
    Internet Explorer
    Fatal error: Maximum execution time of 60 seconds exceeded in C:\xampp\htdocs\openbiblio\catalog\upload_csv.php on line 204

    Please send me the two files upload_csv_form.php and upload_csv.php to daviodan at yahoo dot com .

    Maybe importation can not be done in WIndows?

     
  • Nobody/Anonymous

    Logged In: NO

    I found where the problem was.I used the function Replace "+" with "" (space) and delete + from lines:
    $recCount++;
    for($colCount = 0; $colCount < count($targets); $colCount ++) {

    Thank you very much for this patch.

     
  • Nobody/Anonymous

    Logged In: NO

    It is possible that the CSV file to have the form:
    barCo;100$a;245$a;Coll.;mType;showO;Call1;Call2
    1;Mihai,Eminescu;Luceafarul;;;;LUC;1
    for import direct in Excel
    or
    "barCo","100$a","245$a","Coll.","mType","showO","Call1","Call2"
    1,"Mihai,Eminescu","Luceafarul",,,,"LUC",1
    for import direct in Calc ?
    Currently after the file is saved in CSV Calc must be removed quotes in an editor.

     
  • Nobody/Anonymous

    Logged In: NO

    1)It is possible that the CSV file to have the form:
    barCo;100$a;245$a;Coll.;mType;showO;Call1;Call2
    1;Mihai,Eminescu;Luceafarul;;;;LUC;1
    for import direct in Excel
    or
    "barCo","100$a","245$a","Coll.","mType","showO","Call1","Call2"
    1,"Mihai,Eminescu","Luceafarul",,,,"LUC",1
    for import direct in Calc ?
    Currently after the file is saved in CSV Calc must be removed quotes in an editor.

     
  • H.-J. Tappe

    H.-J. Tappe - 2008-07-28

    Logged In: YES
    user_id=2125563
    Originator: YES

    There are several reasons why the "tab with no text indication" format is preferred:
    - There is no field which contains a line break. Even a description of an item would need the HTML line break tag instead of a line break character to separate paragraphs.
    - the ";" and "," character may well be part of e.g. a title, so they don't always separate fields in the file, while the tab character will not occur in any data.
    - Only when using the ";" "," or line break character, the text indication ("") is necessary. To extract data, there is no standard PHP function and a more sophisticated parser is necessary (not impossible but significantly more work) to strip and respect the field boundaries as indicated.
    - For most migration purposes, a self-made parser will be necessary to convert another library database into the CSV values (i.e. split the call number).

    Last but not least, it is easy to convert from one CSV format into another.
    Excel has become much more sophisticated (irony) during the years and does not ask for the format any more (at least, I can't find it), try:
    - Save as "Text (tab separated, *.txt)"
    OpenOffice:
    - Save as "Text CSV (.csv)"; check "Edit Filter"
    - select {tab} and remove " from text indication

     
  • Nobody/Anonymous

    Logged In: NO

    I have made a test.
    I have imported data from CSV file then I have a backup database with phpMyAdmin.
    Get the same error with two programs: WAMP and XAMPP.
    Fatal error: Maximum execution time of 300 seconds exceeded in D:\wamp\apps\phpmyadmin2.11.5\libraries\import\sql.php on line 118
    Fatal error: Maximum execution time of 300 seconds exceeded in C:\xampp\phpMyAdmin\libraries\import\sql.php on line 121
    I made changes in php.ini but I did not initiate managed to solve the problem.
    http://openbiblioromania.xhost.ro/openbiblio.zip
    Zip archive contains CSV file and backup.
    Why do I get the same error?Smaller backup is imported without problems.

     
  • Nobody/Anonymous

    Logged In: NO

    Accuracy: error occurs when I want to restore the database

     
  • H.-J. Tappe

    H.-J. Tappe - 2008-07-29

    Logged In: YES
    user_id=2125563
    Originator: YES

    For problems involving MySQL and phpmyadmin, please try Google for available solutions. The solutions again are various:
    - increase the time limit in php.ini and phpmyadmin config.php (http://wiki.cihar.com/pma/Config#ExecTimeLimit)
    - Split the data and use partial import as available in phpmyadmin
    - Use any other tool which does not restrict the execution time to import back the SQL.
    For openbiblio backup purposes, search the web of a simple, frequent, automated mysql database backup.

     
  • Nobody/Anonymous

    Logged In: NO

    Thanks for the advice.

     
  • Orso

    Orso - 2008-08-28

    Logged In: YES
    user_id=1605054
    Originator: NO

    Hi!

    What's this patch for? Is it to import mysql data, or is it to import media saved in *.csv files? I'm a newbie and not sure about this.

    Is there a function in OpenBiblio to import media saved in *.csv? At the moment our library is managed by Excel and I'd like to import these data to OpenBiblio.

    TIA!

    Clemens

     
  • Nobody/Anonymous

    it must be the grammar focus so i thought it was an unforgettable story wich means awful and heartwarming americas frontier...it would be great for everyone of us

     
  • Nobody/Anonymous

    Could someone kindly provide instructions for non-technically minded person. I don't know how to install this patch. Also if the cvs files does not have the same entries will the import work anyway? Apologies to coders but I need to get this working to keep my job.

     
  • H.-J. Tappe

    H.-J. Tappe - 2009-02-11

    This patch adds a biblio CSV *import*, not any kind of export. Exports *to* CSV can be done via reports.

    This patch modifies several files. There is no less-technical way of including the patch than understanding the patch file format or "simply" using a patch program (ask wikipedia or hire an IT consultant).

     
  • Arash

    Arash - 2011-10-21

    Hi,

    Can you please show me how to install this patch, i couldn't find any HowTo file inside the zip file? This is how i did but it didn't work.

    krator@laptop:/var/www/openbiblio$ patch -p1 < upload_csv.patch-0.6.X-3/upload_csv.patch
    patching file openbiblio/catalog/upload_csv_form.php
    patching file openbiblio/catalog/upload_csv.php
    can't find file to patch at input line 638
    Perhaps you used the wrong -p or --strip option?
    The text leading up to this was:
    --------------------------
    |Index: quilt/openbiblio/navbars/cataloging.php
    |===================================================================
    |--- quilt.orig/openbiblio/navbars/cataloging.php 2008-06-16 21:06:52.000000000 +0200
    |+++ quilt/openbiblio/navbars/cataloging.php 2008-06-16 21:07:02.000000000 +0200
    --------------------------
    File to patch:

    Thank you

     
  • Hans van der Weij

    Patch Update #3 applied to 0.7.X

     
  • Hans van der Weij

    For easier installing on version 0.7.X: applied the patch to 0.7.1. The changed and new files are available in upload_csv.files-0.7.X-3.zip

     
  • Stephanie

    Stephanie - 2012-08-24

    Hi,
    I've been getting this warning and I was wondering if anyone had some advice.
    Warning: Missing argument 1 for BiblioCopy::validateData(), called in C:\xampp\htdocs\openbiblio\catalog\upload_csv.php on line 275 and defined in C:\xampp\htdocs\openbiblio\classes\BiblioCopy.php on line 43

    Thanks!

     
1 2 > >> (Page 1 of 2)

Log in to post a comment.