Be the first to post a text review of Czech language tokenizer and segmenter. Rate and review a project by clicking thumbs up or thumbs down in the right column.

• czechtok czechtok-1_0 file released: czechtok-1_0.tgz

  The main change is the external definition of rules that allows the binary executable to work with multiple sets of rules (e.g. for multiple languages) without re-compilation.

  posted 626 days ago

• czechtok-1_0 released

  finished support for external definitions of rules, achieved PDT compatibility again.

  posted by kveton 626 days ago

• File released: /czechtok/czechtok-1_0/czechtok-1_0.tgz

  posted 626 days ago

• Work temporarily frozen

  I'm studying Unicode issues and possibilities to migrate from iswalpha etc. to Unicode Properties

  posted by kveton 647 days ago

• czechtok czechtok-0_4 file released: czechtok-0_4.tgz

  Bitwise interpreation of token type. Roman numerals flag is set up directly when the token is created. Upper/Lower/Titlecase is a token property. Segmenter processes all rules on every position (unlike tokenizer). LSSEQ introduced. lc() check supported.

  posted 670 days ago

• czechtok-0_4 releases

  latest release before migrating to external definition of non-trivial tokenizer and segmenter rules.

  posted by kveton 670 days ago

• File released: /czechtok/czechtok-0_4/czechtok-0_4.tgz

  posted 670 days ago

• pdt_comparison pdtcmp_czechtok-0_3 file released: pdtcmp_czechtok-0_3.tgz

  See czechtok-0_3

  posted 682 days ago

• czechtok czechtok-0_3 file released: czechtok-0_3.tgz

  <ul> <li>ct_numerals changed to ct_pdtcompat.</li> <li>removed PDT compat rules for decimal comma and a space-separated numbers</li> <li>fixed end-of-doc bugs</li> <li>in PDT-like output, fixed difference in <f> vs. <d> markup </li> <li>-n changed to --no_pdt</li> </ul>

  posted 682 days ago

• czechtok-0_3 released

  Achieved final compatibility with PDT tokenization. Fixed an end-of-doc bug.

  posted by kveton 682 days ago