Menu

#240 hfst-proc -z seems to treat NUL as EOF?

3.4
open
1
2015-09-25
2014-05-04
No

Here's an example of using lt-proc -z with NUL in input:

$ echo -e 'ja.[][\n]\0ja.[][\n]\0'| lt-proc -z nno.automorf.bin                                                                                
^ja/ja<ij>/ja<n><nt><sg><ind>/ja<n><nt><pl><ind>$^./.<sent><clb>$[][
]^ja/ja<ij>/ja<n><nt><sg><ind>/ja<n><nt><pl><ind>$^./.<sent><clb>$[][
]

Not shown in output, \0 is also printed in the same place it appears
in input (after the [\n]). This is also what we expect as output from
hfst-proc, but instead we get this:

$ echo -e 'ja.[][\n]\0ja.[][\n]\0'| hfst-proc -z sme-nob.automorf.hfst.ol 
^ja/ja<CC>$^./.<CLB>$[][
]

Ie. anything after the \0 is lost (there is no \0 in this output
either). It seems like \0 is treated as EOF here, instead of a flush
character.

Discussion

  • Kevin Brubeck Unhammer

    Tested on both -r3829 and -r3880

     
  • Erik Axelson

    Erik Axelson - 2014-05-08
    • assigned_to: Flammie Pirinen
     
  • Flammie Pirinen

    Flammie Pirinen - 2014-05-08
    • labels: --> proc, eol, later, misfeature
     
  • Kevin Brubeck Unhammer

    I think this can be closed as fixed in -r4452

     
MongoDB Logo MongoDB