#80 how can I handle special characters?

[Apolo]

How can I handle special characters such umlaute?

I am using a Jython code to load a CSV into a table and getting an error if the CSV contains special character which I don't want to replace it.

import glob
import time
import os
import sys
import java.util.ArrayList as ArrayList
import java.lang.String as Ss
import java.sql.Connection as con
import java.sql.PreparedStatement as pps
...

*sqlstmt="INSERT INTO table VALUES ( ?, ?, ?, ?, ?, ?, ?, ?, ? )"
ps=sourceConnection.prepareStatement(sqlstmt)
txt=ArrayList()
txt=fileparse.ReadUsingOpenCsv.getArrayList(filename, logfilename) *

[Andrew Rucker Jones]

No special handling is necessary. As far as opencsv is concerned, all input is Unicode and is treated accordingly. You must make certain, of course, that the input actually is Unicode, and if it's not, convert it before passing it to opencsv.

I would not rule out the possibility from glancing over the meager information you sent that the error occurs not on reading, but on passing the data to the database.

[Scott Conway]

I concur. Try simplfying your problem into sections to narrow the cause. First verify that the file created is unicode. I have seen issues crop up where the database is set to one character set, their java is set with another (file.encoding property I believe), and opencsv is using a third.

If the file looks good copy it off and write a test that recreates your issue with just simple file read, if that works then the issue is somewhere between the database and file creation. If it fails then take the offending line and create a unit test in opencsv - just copy one of the test in CSVReaderTest and put your data in. If the test fails, and the data does not have any personal information that you cannot scrub/randomize, send that to us please.

Scott :)