Re: [Flex-devel] UTF8

[Peter M. <pet...@gm...>]

I'd be thrilled, actually.  I started trying to integrate the changes that
flex had gone through and  the patches you were sending into my branch
directly, and ended up deferring the work each time when I got intimidated
by the scale of the diffs :-)

Hopefully the commits in my branch are discrete enough to be useful.

On Thu, Jul 19, 2012 at 5:21 AM, Paul <pa...@pr...> wrote:

>  Would you mind if I took your changes, made them conditional on the utf8
> flag and integrated them into the unicode16 version.
> I would also like to add more tests.
>
>
> Paul
>
> On 12-07-18 11:29 AM, Peter Martini wrote:
>
> Not nearly as thoroughly as you did.  I confirmed that with or without
> UTF-8 parsing enabled, if its only ASCII text, the same exact tables are
> generated, and I added tests to make sure that the unicode escapes and the
> '.' work as designed.  As designed is a very broad definition though.
>
> Take a look at https://github.com/PeterMartini/flex/commits/master - I
> made a lot of small commits, so you can see the changes I was making.  The
> biggest change structurally was I changed the CCL bitmap to be a linked
> list of ranges, to speed up processing the 0x10 FFFF allowed values, and
> then added a secondary step to convert those unicode characters to an
> equivalent 8-bit pattern.
>
> But my commits also track some housekeeping I did to get rid of warnings
> and get those C++ tests to pass, and I think some of that has been merged
> into the main branch now.
>
> On Mon, Jul 16, 2012 at 5:03 PM, Paul <pa...@pr...> wrote:
>
>> How completely have you tested your utf8 version of flex?
>>
>> Paul
>>
>
>
>
>