#283 qex runs forever

[Giuseppe Attardi]

When trying to build an English tokenizer, quex keeps running forever.
I enclose the rules, which are a transliteration of the official lex rules for the Penn Treebank.

The command is run as follows:

quex/quex-exe.py -i tokenizer_en.qx -o tokenizer_en --iconv -b 4 \
--token-policy single \
--token-memory-management-by-user \
--template-compression

I am using quex version 0.64.8.

Thank you.

[Giuseppe Attardi]

Any hope for a solution?

[Frank-Rene Schäfer]

Does this issue remain in 0.65.* ?

[Giuseppe Attardi]

On 29/lug/2015, at 10:25, Frank-Rene Schäfer fschaef@users.sf.net wrote:

Does this issue remain in 0.65.* ?

Yes.

[bugs:#283] qex runs forever

Status: open
Group: v1.0_(example)
Labels: english tokenizer
Created: Thu Sep 04, 2014 10:19 AM UTC by Giuseppe Attardi
Last Updated: Sat Jul 11, 2015 09:55 PM UTC
Owner: nobody
Attachments:

tokenizer_en.qx (7.4 kB; text/plain)
When trying to build an English tokenizer, quex keeps running forever.
I enclose the rules, which are a transliteration of the official lex rules for the Penn Treebank.

The command is run as follows:

quex/quex-exe.py -i tokenizer_en.qx -o tokenizer_en --iconv -b 4 \ --token-policy single \ --token-memory-management-by-user \ --template-compression

I am using quex version 0.64.8.

Thank you.

Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/quex/bugs/283/

To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/

[DatabasesRules12345]

Encountered the same bug on Windows, fixed it by converting the files from CRLF to LF. Seems Quex can't handle CRLF well and sometimes hangs because of it. Can confirm this bug is still present in 0.71.2.

[Frank-Rene Schäfer]

Hi Guiseppe, thanks for coming back. I hope I can find time for that anytime soon.