Mason

On Thu, 16 Sep 2004, David Wheeler wrote:

> I've identified what I would call a bug with Compiler::ToObject's
> "preamble" parameter. Details (and patch) here:
>
>    http://www.masonhq.com/resources/todo/view.html?id=551

I think the patch is probably.  I wonder how many people are actually
using preamble anyway.


-dave

/*===========================
VegGuide.Org
Your guide to all that's veg.
===========================*/

On Sep 16, 2004, at 9:25 AM, Dave Rolsky wrote:

>>    http://www.masonhq.com/resources/todo/view.html?id=551
>
> I think the patch is probably.

Uh, was there more to that sentence? ;-)

> I wonder how many people are actually
> using preamble anyway.

I'm using it in Bricolage now that I need everything to have utf8 
turned on. So I do

   preamble => "use utf8;\n",

Cheers,

David

On Thu, 16 Sep 2004, Dave Rolsky wrote:
> On Thu, 16 Sep 2004, David Wheeler wrote:
> > I've identified what I would call a bug with Compiler::ToObject's
> > "preamble" parameter. Details (and patch) here:
> >
> >    http://www.masonhq.com/resources/todo/view.html?id=551
>
> I think the patch is probably.  I wonder how many people are actually
> using preamble anyway.

I didn't even notice "preamble" (I looked for something in Compiler
instead of Compiler::ToObject) and did the same thing using "preprocess"
recently.

If anyone wants to exchange utf8 horror stories, I have a few...

~ John Williams

On Thu, 16 Sep 2004, David Wheeler wrote:

> On Sep 16, 2004, at 9:25 AM, Dave Rolsky wrote:
>
> >>    http://www.masonhq.com/resources/todo/view.html?id=551
> >
> > I think the patch is probably.
>
> Uh, was there more to that sentence? ;-)

Oops, "probably ok".


-dave

/*===========================
VegGuide.Org
Your guide to all that's veg.
===========================*/

On Sep 16, 2004, at 10:04 AM, John Williams wrote:

> If anyone wants to exchange utf8 horror stories, I have a few...

Here you go:

   http://www.justatheory.com/computers/programming/perl/utf8_trials.html

Regards,

David

On Thu, 16 Sep 2004, David Wheeler wrote:
> > If anyone wants to exchange utf8 horror stories, I have a few...
>
> Here you go:
>
>    http://www.justatheory.com/computers/programming/perl/utf8_trials.html

I hit the string concatenation thing too.  If one string has the utf8 flag
on and the other has the flag off, the "off" string is "decoded" to utf8.
If it actually had utf8 data in it, you get doubly-utf8-encoded data,
where each byte>=128 is converted to 2 utf8 bytes.

URI::Escape also doesn't work with utf8 strings unless you have URI
version 1.31 (released in June) and change your code to call
uri_escape_utf8 instead of uri_escape.

Perl desperately needs a "default-to-utf8" mode.  My site has data coming
from so many different sources, that I found it easier to turn all the
utf8 flags off and treat everything as binary.  Luckily I'm not doing
character counting operations very much.

~ John Williams

On Thu, Sep 16, 2004 at 12:52:19PM -0600, John Williams wrote:
> Perl desperately needs a "default-to-utf8" mode.

It's called "use encoding 'utf8';".  It's also subtly enabling
lots of other knobs.  See encoding::warnings on CPAN for details.

Thanks,
/Autrijus/

On Fri, 17 Sep 2004, Autrijus Tang wrote:
> On Thu, Sep 16, 2004 at 12:52:19PM -0600, John Williams wrote:
> > Perl desperately needs a "default-to-utf8" mode.
>
> It's called "use encoding 'utf8';".  It's also subtly enabling
> lots of other knobs.  See encoding::warnings on CPAN for details.

Have you actually tried it?
That only sets your source file and STDIN/STDOUT to utf8.

"use open ':utf8'" will fix your other files, but only lexically, so other
modules you use will still give you non-utf8 strings.

LWP::UserAgent doesn't pay attention to the charset header.

libapreq2 doesn't mark the strings it gives you as utf-8.

And let's not even get started on DBI...

~ John Williams

P.S.  I should give kudos to Storable for doing the right thing, though.

On Thu, Sep 16, 2004 at 02:43:53PM -0600, John Williams wrote:
> On Fri, 17 Sep 2004, Autrijus Tang wrote:
> > On Thu, Sep 16, 2004 at 12:52:19PM -0600, John Williams wrote:
> > > Perl desperately needs a "default-to-utf8" mode.
> >
> > It's called "use encoding 'utf8';".  It's also subtly enabling
> > lots of other knobs.  See encoding::warnings on CPAN for details.
>=20
> Have you actually tried it?

Why yes, I'm one of its authors. :)

> That only sets your source file and STDIN/STDOUT to utf8.

Yes, but it enables utf8 semantics globally instead of the default
latin1 promotion, so high-bit characters you read from other handles
will be treated correctly when joined with utf8 data.

> "use open ':utf8'" will fix your other files, but only lexically, so other
> modules you use will still give you non-utf8 strings.

That is correct.  There are no good solution for it save for overriding
CORE::open.

Thanks,
/Autrijus/

On Fri, 17 Sep 2004, Autrijus Tang wrote:
> > > On Thu, Sep 16, 2004 at 12:52:19PM -0600, John Williams wrote:
> > > > Perl desperately needs a "default-to-utf8" mode.
> > >
> > > It's called "use encoding 'utf8';".  It's also subtly enabling
> > > lots of other knobs.  See encoding::warnings on CPAN for details.
> >
> > Have you actually tried it?
>
> Why yes, I'm one of its authors. :)

Oh good, then I can complain to you a bit.  :)

> > That only sets your source file and STDIN/STDOUT to utf8.
>
> Yes, but it enables utf8 semantics globally instead of the default
> latin1 promotion, so high-bit characters you read from other handles
> will be treated correctly when joined with utf8 data.

OK, I see where it says that in the docs now.  But it doesn't seem to be
working for me.

Right now I'm doing this (in my local copy of Mason::Buffer::receive),
because it doesn't matter whether utf8 is on or off once it gets to
Apache:

        use Encode;
        foreach ( grep { defined } @_) {
                Encode::_utf8_off($_);
                ${ $self->{buffer} } .= $_;
        }

I tried changing it to this:

        use encoding 'utf8';
        ${ $self->{buffer} } .= $_ foreach grep { defined } @_;

But all the doubly-encoded utf8 strings came back, indicating that it is
still doing the latin1->utf8 conversion on the strings which are "off".

I'm using perl-5.8.3, encoding-1.48, mod_perl-1.99_12.  Do I need to
upgrade something?

~ John Williams

On Thu, Sep 16, 2004 at 03:13:08PM -0600, John Williams wrote:
> I tried changing it to this:
>=20
>         use encoding 'utf8';
>         ${ $self->{buffer} } .=3D $_ foreach grep { defined } @_;
>=20
> But all the doubly-encoded utf8 strings came back, indicating that it is
> still doing the latin1->utf8 conversion on the strings which are "off".
>=20
> I'm using perl-5.8.3, encoding-1.48, mod_perl-1.99_12.  Do I need to
> upgrade something?

Hmm.  Try specifying "use encoding 'utf8'" before loading the rest of Mason
Modules, perhaps?  Also consider using encoding::warnings first (without
"use encoding 'utf8'") to find out where did the conversion really occur.

Thanks,
/Autrijus/

On Sep 16, 2004, at 11:52 AM, John Williams wrote:

> I hit the string concatenation thing too.  If one string has the utf8 
> flag
> on and the other has the flag off, the "off" string is "decoded" to 
> utf8.
> If it actually had utf8 data in it, you get doubly-utf8-encoded data,
> where each byte>=128 is converted to 2 utf8 bytes.

Oh, is that what that means? Ick.

> URI::Escape also doesn't work with utf8 strings unless you have URI
> version 1.31 (released in June) and change your code to call
> uri_escape_utf8 instead of uri_escape.

I don't use it. Phew!

> Perl desperately needs a "default-to-utf8" mode.  My site has data 
> coming
> from so many different sources, that I found it easier to turn all the
> utf8 flags off and treat everything as binary.  Luckily I'm not doing
> character counting operations very much.

Sooner or later you probably will. In my next major project, I'll use 
Autrijus' module to turn on utf8 everywhere and be done with it.

Regards,

David

On Fri, 17 Sep 2004, Autrijus Tang wrote:
> On Thu, Sep 16, 2004 at 03:13:08PM -0600, John Williams wrote:
> > I tried changing it to this:
> >
> >         use encoding 'utf8';
> >         ${ $self->{buffer} } .= $_ foreach grep { defined } @_;
> >
> > But all the doubly-encoded utf8 strings came back, indicating that it is
> > still doing the latin1->utf8 conversion on the strings which are "off".
> >
> > I'm using perl-5.8.3, encoding-1.48, mod_perl-1.99_12.  Do I need to
> > upgrade something?
>
> Hmm.  Try specifying "use encoding 'utf8'" before loading the rest of Mason
> Modules, perhaps?  Also consider using encoding::warnings first (without
> "use encoding 'utf8'") to find out where did the conversion really occur.

Thanks for the suggestions.  encoding::warnings looks like it will be very
helpful.

For technical reasons I cannot try it on our web site for a few days, but
I'll let you when I get some results using encoding::warnings.

~ John Williams

On Sep 17, 2004, at 2:38 PM, John Williams wrote:

> Thanks for the suggestions.  encoding::warnings looks like it will be 
> very
> helpful.
>
> For technical reasons I cannot try it on our web site for a few days, 
> but
> I'll let you when I get some results using encoding::warnings.

Recognizing that this is a just a hair off-topic, I see now that there 
is a perl-unicode list. Should we move this conversation there? I'm 
subscribing...

   http://lists.perl.org/showlist.cgi?name=perl-unicode

Regards,

David

On Sep 16, 2004, at 10:32 AM, Dave Rolsky wrote:

>>>>    http://www.masonhq.com/resources/todo/view.html?id=551
>>>
>>> I think the patch is probably.
>>
>> Uh, was there more to that sentence? ;-)
>
> Oops, "probably ok".

Uh, but then you went and added specific support for the prsence of $m 
in the premable, Dave:

--- lib/HTML/Mason/Compiler/ToObject.pm.~1.71.2.3~      Mon Sep 27 
22:21:56 2004
+++ lib/HTML/Mason/Compiler/ToObject.pm Mon Sep 27 22:21:20 2004
@@ -354,9 +354,9 @@
  {
      my $self = shift;

-    return join '', ( $self->preamble,
-                      $self->_set_request,
-                     $self->_arg_declarations,
+    return join '', ( $self->_set_request,
+                     $self->preamble,
+                      $self->_arg_declarations,
                        $self->_filter,
                       "\$m->debug_hook( \$m->current_comp->path ) if ( 
\%DB:: );\n\n",
                       $self->_blocks('init'),
@@ -612,11 +612,15 @@

  =item preamble

-Text given for this parameter is placed at the beginning of each 
component. See also P<postamble>.
+Text given for this parameter is placed at the beginning of each
+component.  See also P<postamble>.  The request will be available as
+C<$m> in preamble code.

  =item postamble

-Text given for this parameter is placed at the end of each component. 
See also P<preamble>.
+Text given for this parameter is placed at the end of each
+component. See also P<preamble>.  The request will be available as
+C<$m> in postamble code.

  =item use_strict


But IMO, it should be executed before the %once block, e.g., so I can 
set "use utf8;\n" as the preamble and be assured that everything in the 
rest of every component will be utf8.

So maybe we actually need some other parameter for this, like pre-once 
or something?

Regards,

David

David Wheeler wrote:

> On Sep 16, 2004, at 10:32 AM, Dave Rolsky wrote:
> 
>>>>>    http://www.masonhq.com/resources/todo/view.html?id=551
>>>>
>>>>
>>>> I think the patch is probably.
>>>
>>>
>>> Uh, was there more to that sentence? ;-)
>>
>>
>> Oops, "probably ok".
> 
> 
> Uh, but then you went and added specific support for the prsence of $m 
> in the premable, Dave:
> 
> --- lib/HTML/Mason/Compiler/ToObject.pm.~1.71.2.3~      Mon Sep 27 
> 22:21:56 2004
> +++ lib/HTML/Mason/Compiler/ToObject.pm Mon Sep 27 22:21:20 2004
> @@ -354,9 +354,9 @@
>  {
>      my $self = shift;
> 
> -    return join '', ( $self->preamble,
> -                      $self->_set_request,
> -                     $self->_arg_declarations,
> +    return join '', ( $self->_set_request,
> +                     $self->preamble,
> +                      $self->_arg_declarations,
>                        $self->_filter,
>                       "\$m->debug_hook( \$m->current_comp->path ) if ( 
> \%DB:: );\n\n",
>                       $self->_blocks('init'),
> @@ -612,11 +612,15 @@
> 
>  =item preamble
> 
> -Text given for this parameter is placed at the beginning of each 
> component. See also P<postamble>.
> +Text given for this parameter is placed at the beginning of each
> +component.  See also P<postamble>.  The request will be available as
> +C<$m> in preamble code.
> 
>  =item postamble
> 
> -Text given for this parameter is placed at the end of each component. 
> See also P<preamble>.
> +Text given for this parameter is placed at the end of each
> +component. See also P<preamble>.  The request will be available as
> +C<$m> in postamble code.
> 
>  =item use_strict
> 
> 
> But IMO, it should be executed before the %once block, e.g., so I can 
> set "use utf8;\n" as the preamble and be assured that everything in the 
> rest of every component will be utf8.
> 
> So maybe we actually need some other parameter for this, like pre-once 
> or something?
> 

I haven't been following this closely, but as far as that last
suggestion - oh lord, please no. I'd almost rather implement a use_utf8
flag. :)

Seriously, is it something that more and more peple are going to need to do?

Jon

On Mon, 27 Sep 2004, David Wheeler wrote:

> Uh, but then you went and added specific support for the prsence of $m
> in the premable, Dave:

So what's the problem?  Why would 'use utf8' need to come before that?
The whole purpose of 'use utf8' is to make sure Perl handles UTF8 chars
in the source code properly.  Since we know wha the "set the request" code
looks like, we can know it doesn't need 'use utf8'.


-dave

/*===========================
VegGuide.Org
Your guide to all that's veg.
===========================*/

On Sep 28, 2004, at 6:12 AM, Jonathan Swartz wrote:

> I haven't been following this closely, but as far as that last
> suggestion - oh lord, please no. I'd almost rather implement a use_utf8
> flag. :)
>
> Seriously, is it something that more and more peple are going to need 
> to do?

Yes, it is, because you either have to use it throughout your 
application or not use it at all. And if you have UTF-8 encoded text 
and need to use length() or stubtr(), then you *must* decode all of 
your strings to Perl's internal utf8 format. C<use utf8;> does that 
automatically.

But I'm also fine to turn the "preamble" parameter into a true 
preamble--that is, something that executes even before %once. That 
means that I can add C<use utf8;> myself, but that $m won't be 
available in the preamble. But frankly, I'd be happy if $m was 
available in %once blocks, too (for functional processing or even 
scomp(), but not output).

Regards,

David

On Sep 28, 2004, at 8:26 AM, Dave Rolsky wrote:

> So what's the problem?  Why would 'use utf8' need to come before that?
> The whole purpose of 'use utf8' is to make sure Perl handles UTF8 chars
> in the source code properly.  Since we know wha the "set the request" 
> code
> looks like, we can know it doesn't need 'use utf8'.

The preamble is output to the object files *after* the %once block. 
What if I have UTF-8 encoded data in the %once block? This is exactly 
the problem I ran into on the Radio Free Asia project.

Read more about my fun with utf8 here:

   http://www.justatheory.com/computers/programming/perl/utf8_trials.html

Regards,

David

On Tue, 28 Sep 2004, David Wheeler wrote:

> On Sep 28, 2004, at 8:26 AM, Dave Rolsky wrote:
>
> > So what's the problem?  Why would 'use utf8' need to come before that?
> > The whole purpose of 'use utf8' is to make sure Perl handles UTF8 chars
> > in the source code properly.  Since we know wha the "set the request"
> > code
> > looks like, we can know it doesn't need 'use utf8'.
>
> The preamble is output to the object files *after* the %once block.
> What if I have UTF-8 encoded data in the %once block? This is exactly
> the problem I ran into on the Radio Free Asia project.

Doh, I get it.  And on reflection, it doesn't particular make sense to
provide access to $m in the preamble, since we don't provide it in the
<%once> block, and the preamble is kind of like a <%once> block that is
set for every component.


-dave

/*===========================
VegGuide.Org
Your guide to all that's veg.
===========================*/

On Tue, 28 Sep 2004, Dave Rolsky wrote:

> On Tue, 28 Sep 2004, David Wheeler wrote:
>
> > On Sep 28, 2004, at 8:26 AM, Dave Rolsky wrote:
> >
> > > So what's the problem?  Why would 'use utf8' need to come before that?
> > > The whole purpose of 'use utf8' is to make sure Perl handles UTF8 chars
> > > in the source code properly.  Since we know wha the "set the request"
> > > code
> > > looks like, we can know it doesn't need 'use utf8'.
> >
> > The preamble is output to the object files *after* the %once block.
> > What if I have UTF-8 encoded data in the %once block? This is exactly
> > the problem I ran into on the Radio Free Asia project.
>
> Doh, I get it.  And on reflection, it doesn't particular make sense to
> provide access to $m in the preamble, since we don't provide it in the
> <%once> block, and the preamble is kind of like a <%once> block that is
> set for every component.

But we have stuff like MasonX::Profiler on CPAN that relies on having
access to $m in the preamble.  Doh, this is annoying.

I'm inclined to add a 'use_utf8' parameter to the compiler for this, as
Jonathan mentioned.


-dave

/*===========================
VegGuide.Org
Your guide to all that's veg.
===========================*/

On Sep 28, 2004, at 9:21 AM, Dave Rolsky wrote:

> But we have stuff like MasonX::Profiler on CPAN that relies on having
> access to $m in the preamble.  Doh, this is annoying.
>
> I'm inclined to add a 'use_utf8' parameter to the compiler for this, as
> Jonathan mentioned.

That's fine, but then I think that we should document that preamble 
isn't really a preamble, since it comes after %once blocks.

Regards,

David

On Sep 28, 2004, at 9:33 AM, David Wheeler wrote:

>> But we have stuff like MasonX::Profiler on CPAN that relies on having
>> access to $m in the preamble.  Doh, this is annoying.
>>
>> I'm inclined to add a 'use_utf8' parameter to the compiler for this, 
>> as
>> Jonathan mentioned.
>
> That's fine, but then I think that we should document that preamble 
> isn't really a preamble, since it comes after %once blocks.

John, Dave, can we get a ruling on this? I'd love to see better utf8 
support in 1.27.

Thanks,

David

We're not going to add a use_utf8 for 1.27 - as far as I'm concerned 
that branch is closed for new features. We can consider it for 1.3x.

In the meantime, David, have you tried writing a small subclass of
HTML::Mason::Compiler::ToObject that overrides _make_main_header?

David Wheeler wrote:

> On Sep 28, 2004, at 9:33 AM, David Wheeler wrote:
> 
>>> But we have stuff like MasonX::Profiler on CPAN that relies on having
>>> access to $m in the preamble.  Doh, this is annoying.
>>>
>>> I'm inclined to add a 'use_utf8' parameter to the compiler for this, as
>>> Jonathan mentioned.
>>
>>
>> That's fine, but then I think that we should document that preamble 
>> isn't really a preamble, since it comes after %once blocks.
> 
> 
> John, Dave, can we get a ruling on this? I'd love to see better utf8 
> support in 1.27.
> 
> Thanks,
> 
> David
> 
> 
>

On Oct 4, 2004, at 4:15 PM, Jonathan Swartz wrote:

> We're not going to add a use_utf8 for 1.27 - as far as I'm concerned 
> that branch is closed for new features. We can consider it for 1.3x.

Okay. But the behavior of preamble does need to be settled (does it 
come before %commit or not? Does it include $m or not?) and documented.

> In the meantime, David, have you tried writing a small subclass of
> HTML::Mason::Compiler::ToObject that overrides _make_main_header?

No, but good idea (even if that does look suspiciously like a private 
method!).

Regards,

David

>> We're not going to add a use_utf8 for 1.27 - as far as I'm concerned 
>> that branch is closed for new features. We can consider it for 1.3x.
> 
> 
> Okay. But the behavior of preamble does need to be settled (does it come 
> before %commit or not? Does it include $m or not?) and documented.

Yes, this should be documented. You can expect that whatever the current 
behavior is, that'll be documented, since we don't want to break anything...

> 
>> In the meantime, David, have you tried writing a small subclass of
>> HTML::Mason::Compiler::ToObject that overrides _make_main_header?
> 
> 
> No, but good idea (even if that does look suspiciously like a private 
> method!).

Yes, and aren't you glad Perl doesn't enforce private/public 
distinctions? :)

Jon

On Oct 4, 2004, at 4:39 PM, Jonathan Swartz wrote:

> Yes, this should be documented. You can expect that whatever the 
> current behavior is, that'll be documented, since we don't want to 
> break anything...

Well, the current behavior is that it doesn't have $m, IIRC, but Dave 
just changed that, I believe.

> Yes, and aren't you glad Perl doesn't enforce private/public 
> distinctions? :)

Okay, just making sure, since I got busted for overriding a private 
method back in the 1.0x days.

Cheers,

David

On Mon, 4 Oct 2004, David Wheeler wrote:

> On Oct 4, 2004, at 4:39 PM, Jonathan Swartz wrote:
>
> > Yes, this should be documented. You can expect that whatever the
> > current behavior is, that'll be documented, since we don't want to
> > break anything...
>
> Well, the current behavior is that it doesn't have $m, IIRC, but Dave
> just changed that, I believe.

No, it does have $m, or MasonX::Profiler wouldn't work.

> > Yes, and aren't you glad Perl doesn't enforce private/public
> > distinctions? :)
>
> Okay, just making sure, since I got busted for overriding a private
> method back in the 1.0x days.

Well, I'd say it's protected, not private, not that we really attempt to
make that distinction.


-dave

/*===========================
VegGuide.Org
Your guide to all that's veg.
===========================*/

On Oct 4, 2004, at 5:20 PM, Dave Rolsky wrote:

> No, it does have $m, or MasonX::Profiler wouldn't work.

Fine, just as long as it's documented, and it's documented that the 
preamble runs *after* any %once blocks. Do you need a patch to this 
effect?

> Well, I'd say it's protected, not private, not that we really attempt 
> to
> make that distinction.

Cool, thanks.

David

On Mon, 4 Oct 2004, David Wheeler wrote:

> On Oct 4, 2004, at 5:20 PM, Dave Rolsky wrote:
>
> > No, it does have $m, or MasonX::Profiler wouldn't work.
>
> Fine, just as long as it's documented, and it's documented that the
> preamble runs *after* any %once blocks. Do you need a patch to this
> effect?

Feel free to check it in.


-dave

/*===========================
VegGuide.Org
Your guide to all that's veg.
===========================*/

David Wheeler wrote:

> On Oct 4, 2004, at 5:20 PM, Dave Rolsky wrote:
> 
>> No, it does have $m, or MasonX::Profiler wouldn't work.
> 
> 
> Fine, just as long as it's documented, and it's documented that the 
> preamble runs *after* any %once blocks. Do you need a patch to this effect?

Or even better could you check the change in? (you still have 
permissions right?)

> 
>> Well, I'd say it's protected, not private, not that we really attempt to
>> make that distinction.
> 
> 
> Cool, thanks.

I was sort of suggesting the change as a stop-gap until there is a more 
supported way to do what you want.

Jon

On Oct 4, 2004, at 8:55 PM, Jonathan Swartz wrote:

> Or even better could you check the change in? (you still have 
> permissions right?)

Yes, committed.

> I was sort of suggesting the change as a stop-gap until there is a 
> more supported way to do what you want.

Understood.

Regards,

David