#6 Add to select character encoding to reading / writing text file

[Anonymous]

It would very nice if Record Editor supports to select character encoding to reading / writing text files. it would be great if user can select different encoding for reading and writing.

Yuki

[Bruce Martin]

Can you explain what you want. Currently you can define a character-set with a layout; do you want
to overide this when loading a file ???. I could look at adding a character-set in the export function.

I am guess you are dealing with text files

[Anonymous]

I am guess you are dealing with text files

Yes, this is about reading / writing text file.

Currently you can define a character-set with a layout

Is this the font setting in extras tab of layout editor?
Should I select "Text I/O (byte based)" as File structure when reading text file of multibyte character encoding?

I could look at adding a character-set in the export function.

This feature is very helpful, because it's very hard to convert character encoding by manually with keeping the file format of fixed field length (as you know, byte length may be different after converting character encoding).

Yuki

[Yuki]

I am guess you are dealing with text files

Yes, this is about reading / writing text file.

Currently you can define a character-set with a layout

Is this the font setting in extras tab of layout editor?
Should I select "Text I/O (byte based)" as File structure when reading text file of multibyte character encoding?

I could look at adding a character-set in the export function.

This feature is very helpful, because it's very hard to convert character encoding by manually with keeping the file format of fixed field length (as you know, byte length may be different after converting character encoding).

[Anonymous]

I am guess you are dealing with text files

Yes, this is about reading / writing text file.

Currently you can define a character-set with a layout

Is this the font setting in extras tab of layout editor?
Should I select "Text I/O (byte based)" as File structure when reading text file of multibyte character encoding?

I could look at adding a character-set in the export function.

This feature is very helpful, because it's very hard to convert character encoding by manually with keeping the file format of fixed field length (as you know, byte length may be different after converting character encoding).

[Bruce Martin]

Is this the font setting in extras tab of layout editor? Yes this is the character set field.

Text I/O (byte based) - is byte based (c-style) single byte IO. When dealing with C/Cobol files the file will most likely be a single byte character-set. For example I have seen C program use x'00', x'01' etc as field delimiters in text files. Cobol programs can use low-values (x'00') or high-values (x'FF') in text files. You may run into trouble if used for some asian languages in UTF-8/16.

Text I/O (Unicode) - is a pure Character based IO.

---

For western languages

[Yuki]

Text I/O (Unicode) - is a pure Character based IO.

What is the Unicode character encoding of text files which this option assumes of?
(utf-8 / utf-16 big endian / utf-16 little endian)
Does this option detect character encoding automatically?

Text I/O (byte based) - is byte based (c-style) single byte IO. You may run into trouble if used for some Asian languages in UTF-8/16.

Which option should I select as File structure when I read text files of traditional Asian character encoding such as Shift_JIS or GB2312?
I would like to know how Record Editor scans one character in multibyte character encoding of flexible byte length per character.
Could you please take a look examples of character stream which I attached?

[Anonymous]

Text I/O (Unicode) - is a pure Character based IO.

What is the Unicode character encoding of text files which this option assumes of?
(utf-8 / utf-16 big endian / utf-16 little endian)
Does this option detect character encoding automatically?

Text I/O (byte based) - is byte based (c-style) single byte IO. You may run into trouble if used for some Asian languages in UTF-8/16.

Which option should I select as File structure when I read text files of traditional Asian character encoding such as Shift_JIS or GB2312?
I would like to know how Record Editor scans one character in multibyte character encoding of flexible byte length per character.
Could you please take a look examples of character stream which I attached?

[Bruce Martin]

You should be able to define your files using the Character-set and Text (Unicode)

---

I will look at add new File Structure where:

• RecordEditor will try and workout the Character-set from the file.
• Allow the User to set/correct the Character-Set

---

In the mean time, one trick useful is to:

• Define Record-Layout
• Use the Groups to create characterset versions of the layout

What I did is create a layout z_sjis using the sjis character-set.
I then defined a second layout z_UTF-16LE:

Record Type: Group of Record
Child Record: z_sjis
Font: z_UTF-16LE

This gives 2 layout with exactly the same field definitions but they use 2 different fonts

[Yuki]

Does Record Editor use InputStreamReader with giving the argument of character encoding selected as font setting when reading text files?

[Anonymous]

Does Record Editor use InputStreamReader with giving the argument of character encoding selected as font setting when reading text files?

[Bruce Martin]

Text I/O (Unicode) ** does use InputStreamReader with a character-set. When used lines are stored in character format.

Text I/O (byte based) does not use InputStreamReader , it converts \n back it to bytes and processes it as bytes. For fixed width files, the lines will be stored internally as bytes and fields are converted to Strings as needed.

[Yuki]

Text I/O (Unicode) does use InputStreamReader with a character-set. When used lines are stored in character format.

Does "Text I/O (Unicode)" option handle length defined in fixed length file layout as character length? or byte length?

[Bruce Martin]

It use character length.

Position = 3   ~ 3rd character in the record
Length = 3     ~ 3 characters long

[Yuki]

It use character length.

In this case, it may cause problem if the file layout is defined based on byte length and the file definition allow to put multibyte characters of flexible length per one character in a field.

In my understanding, There are 2 cases to define fixed length file layout of multibyte text file.

Precondition:
East Asian character code has single byte part and double byte part.
Single byte part is usually assigned to English characters and compatible with ASCII code.
Double byte part is usually assigned to Chinese characters, this part also may contain double byte English character and double byte space character.

Case1:
Forbid to put characters of different byte length into one field.
All characters in one field is must be same byte length, double byte field only keep double byte character, and double byte space character is used for padding.

Case2:
Allow mixed byte length characters into one field.
In this case, file layout is defined based on byte length, and a space character of shortest byte length is used for padding.

I think new option is necessary in File Structure of Record Editor to read text file of Case2(read text file according to file layout based on byte length, then convert character encoding per field one by one).

Could you please take a look attached document about file layout?

[Bruce Martin]

For Case 1, what As you would expect :

File Structure: Text (Unicode)
Font: Sift_JIS

Definition

Editing the file

[Bruce Martin]

For Case 2, what I did was :

File Structure: Text (Byte Based)
Font: Sift_JIS

Possible problem: Looking at Shit_JIS & UTF-8 it should work. The Text (Byte Based) reader just looks for \n and \n\r characters (it does not know about double byte characters) so if there was a posibility of the second byte being x'0A' or x'0D' there will be a problem.

Definition:

File:

RecordEditor Description

The RecordEditor has

• byte based reader's which store records in arrays of bytes and Position/length are byte positions and lengths

• Character based reader which store records in Arrays of characters and Position/Length

• Text IO (Byte based) - read as line as byte array (and convert to text at field level)

• Text IO (Unicode) - Read and store as charcter
• Text :
  • If SingleByte Charcterset or binary record - use Text IO (Byte based)
  • If Multi-Byte - use Text IO (Unicode)

[Yuki]

Thank you very much, I think I almost understood about how to use Record Editor.
I got some more requests and questions.

1.Could you please add knowledge in this topic to Record Editor document? I think these information are very helpful for users who are using systems based on multibyte encoding.

-> Guide to define file layout for Case1 and Case2 above
-> Detail about how options in File structure will works
-> Useful trick mentioned in this topic

2.It might be helpful if you change the caption of font setting from "Font Name" to "Encoding", because it affects not only for appearance but also for reading / writing file.

3.About Case2, Is there any option to scan one record only using byte length (not check \n and \r) with converting character encoding according to setting of file layout?
Can I use "Fixed Length Char" in File Structure option?

[Bruce Martin]

I have done some more testing for case_2; this will let you view existing files but updates to the variable sized fields might not work.

I will be making changes for this

[Bruce Martin]

1. I will look at; documentation is not my strength (or an interest).
2. Will look at it.
3. Yes a File Structure
   • Fixed Length Binary - Each record is a Fixed number Bytes long (based on the Layout). \n \r have no relavance - they are just bytes
   • Fixed Length Char - Character based equivalent; Each record is a Fixed number Characters long. \n \r have no relavance - they are just characters

[Bruce Martin]

There is a New Test version available as updated Jars:

• https://sourceforge.net/projects/record-editor/files/Test/Version_0.98.1_TestA

On Export screen (Text IO (Unicode)) you get the option entering a character set

---

Another way to Export with a character-set is to use a macro (also attached macro Write_utf16.js)

    var outName = RecordEditorData.view.getFileName() + ".utf16";
    var outStream = new java.io.FileOutputStream(outName);
    var writer =  new java.io.OutputStreamWriter(outStream, "UTF-16LE");
    var w = new java.io.BufferedWriter(writer);
    var rowCount = RecordEditorData.view.getRowCount();

    for (i = 0; i < rowCount; i++) {
        w.write(RecordEditorData.view.getTempLine(i).getFullLine());
        w.newLine();
    }
    w.close();
    RecordEditorData.outputFile = outName; // Tell the RecordEditor what the  output file was

• For Windows (Recordeditor_HSQL) the macro Write_Add@@@.js should go in

  C:\UsersYourUserName\RecordEditor_HSQL\User\Scripts

• For Linux

  Home/.RecordEditor/HSQLDB/User/Scripts

Once installed restart the RecordEditor and the macro should be on the Utilities >>> Runscript menu:

---

Other point with macro's:

• If you create a Directory in the User/Scripts directory and add macro scripts to it; The RecordEditor will display it as menu. So you could create a SaveAs directory and add various scripts into the directory.
• You can do basic editting using the Utilities >>> Script Test Panel test option.
• You can use can use scripting languages other than JavaScript (e.g. Groovy, Jython, JRuby). Just add the appropriate jar file to the RecordEditor/lib/jars directory.

[Yuki]

Thank you very much, these are really great, I will try.

On Export screen (Text IO (Unicode)) you get the option entering a character set

Do you have any plan to add this option also to "Text I/O(Byte based) and "Fixed Length Binay"?

[Yuki]

Thank you very much.

[Yuki]

By the way, is it possible to change comment which I posted as anonymous to read only to avoid that comments are deleted or modified unexpectedly? It seems that anyone can delete or modify comments posted from anonymous user.

[Bruce Martin]

I can not see any way to do it; It does not seem have any option. The only options are at the problem level

• Disable discussions might stop all updates
• Make the problem problem - since it was raised as Anonymous, it might hide it from you

[Yuki]

I can not see any way to do it; It does not seem have any option. The only options are at the problem level

I got it, I posted my comments using my account again.