Work at SourceForge, help us to make it a better place! We have an immediate need for a Support Technician in our San Francisco or Denver office.

Close

problems with encoding

Help
2007-07-18
2013-05-28
  • Hi Forum,
    I have problems with displaying the right characters for my refbase installation.
    If I set entries by the client everything goes fine with displaying special characters like accents.
    If I do the direct import to the database from Endnote with endnote2mysql.php the results in the database will also be displayed in the right way. When I open the browser to have a look for the new entries the special characters disapear. Any suggestions whats going on here?

    I am using refbase 0.9 on Ubuntu Dapper with Mysql 5.0 from Repositories. Default character set is UTF-8 either for the database or in the source of the client webpages.

    Thanks in advance,
    Christian

     
    • Hi Christian,

      > I have problems with displaying the right characters for my refbase installation.

      I have tried to address issues regarding displaying, entering and searching of higher ASCII characters (such as accented characters or umlauts) in our refbase wiki at:

      http://wiki.refbase.net/index.php/Installation-Troubleshooting#Problems_with_special_characters

      Please read the given notes and tips carefully and make sure that your server & refbase installation is setup accordingly. Let us know if you still can't get it working correctly.

      Matthias

       
    • Hallo Matthias,

      I crosschecked all my settings to your refbase wiki and I didn't found the mistake. Everything on the data base side is set to UTF-8...
      Any further ideas? Set up the complet environment again?

      Christian

       
    • > Everything on the data base side is set to UTF-8...

      Ok, thanks for cross-checking. From my experience, the server's character set and collation settings are crucial to get things right:

      http://wiki.refbase.net/index.php/Troubleshooting#MySQL_migration_and_character_set_problems

      But I guess that you've also verified these to be 'utf8'?

      Just to make sure, you could post all your encoding-related server and refbase settings & variables here. Or, alternatively, send them to me via private mail (my Sourceforge email address will do).

      > Any further ideas? Set up the complet environment again?

      If you've really checked everything on the wiki page given in my previous post, and it all looks fine, then I don't think that setting up everything again will help.

      In your original post you wrote:

      > If I set entries by the client everything goes fine with displaying special characters like accents.

      So, am I understanding you correctly that you can successfully enter higher ascii chars via the refbase web interface?

      I.e., can you perform *all* of these steps successfully:

      1) enter special characters via the refbase web interface
      2) have the *newly entered* characters displayed correctly after saving your edits
      3) search for these *newly entered* characters

      It's important to differentiate between characters that were newly entered via the web interface, and incorrectly displayed characters that did already exist in the database (or that were added via anything other than the web GUI). If you can enter/display *and* search for newly entered special chars, then refbase has been setup correctly.

      > If I do the direct import to the database from Endnote with endnote2mysql.php
      > the results in the database will also be displayed in the right way. When I open
      > the browser to have a look for the new entries the special characters disapear.

      So is it just special characters from the Endote import via 'endnote2mysql.php' that don't display correctly?

      How did you import the file 'import.txt' into mysql? The file 'en-importer.sh' uses the 'mysqlimport' shell program. You may want to try to add " --default-character-set=utf8" in that file (on line 38) and see if that helps.

      Also, could it be that the encoding of file 'import.txt' is not UTF-8? If so, you may first need to convert the encoding to UTF-8 (using iconv or your text editor of choice).

      Also, have you tried this alternative route:

      1. export your Endnote records as Endnote tagged text (or RIS/Refman/Reference Manager)
      2. open the exported data in a text editor and make sure that special characters display correctly there
      3. copy & paste the data into the refbase import form (don't use the "upload file" button)
      4. click the "Import" button

      If you try these steps with a few test records that contain special chars, does this work better?

      Matthias