parseperl-discuss Mailing List for Parse::Perl

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

Hi

I looked at the archives on sf.net and the last emails seemed from 2010.
Is that correct ?

I just joined the mailing list and am looking to use PPI for a project.

Thanks
Vikas.

On 3/15/10 11:50 AM, Todd Rinaldo wrote:
> I'm sure you've seen many of these corner cases. How did you deal with them?

I haven't had to deal with $VERSION, so I haven't run into this.

Everything I've written has been to process code that I can change, and not arbitrary stuff from the CPAN.

> It seems to me META.yml is desperately in need of a 'provides' section
> which could properly document the author's modules and versions as they
> intended. Has anyone heard any discussion of this idea ever?

There is one: e.g. http://cpansearch.perl.org/src/ELLIOTJS/PPIx-Utilities-1.000001/META.yml.  Whether it is present or not depends upon what the author had installed at the time that the distribution tarball was created.  If you use current Module::Build/ExtUtils::MakeMaker, you'll get a MYMETA.yml after running Build.PL/Makefile.PL which will reflect the actual things that will/would be installed.

If you can't tell from the above URL, I've released the code that I discussed earlier: http://search.cpan.org/dist/PPIx-Utilities/.

On Mar 10, 2010, at 9:07 PM, Elliot Shank wrote:

> On 3/10/10 6:13 PM, Todd Rinaldo wrote:
>> I want to use PPI to determine all packages and their $VERSION in a
>> given pm file. I've determined how to get the list of packages in the
>> file, but not the version. How can I do this?
> 
> The only authoritative way of doing this is to load the module and check it at runtime.  (Of course, determining which namespaces got twiddled in the process is difficult.
> 
> You might want to look at http://search.cpan.org/dist/Module-Extract-VERSION/.
> 
> If you want to do it yourself via PPI, I would probably set up a state machine that tracked what the current package is and when a $VERSION assignment is found, save it off in a stash of some sort.
> 
> Alternatively, there's http://perlcritic.tigris.org/source/browse/perlcritic/trunk/distributions/PPIx-Utilities/lib/PPIx/Utilities/Node.pm?revision=3777&view=markup.  You could use it to split your module up into namespaces and then look for $VERSION assignments in that.  (I haven't released this yet due to wanting to move some code from Perl::Critic into the distribution first, but I'm still debating how to do it.)

Elliot, thanks for your advice. It took me about 6 hours of struggling with PPI before I realized you were right and that PPI was a lost cause. 

The approach I've taken so far is to parse the output from $ver = `perl -e 'require $file; print $module::VERSION'`; What I've discovered is how many crazy things people try to do outside a sub in some perl modules. Also I often have to have the module and it's dependencies installed before my parsing will succeed. I played with removing use/require entries from the modules before I try requiring it but this often causes further breaks, especially when modules like Net::DNS (may it eternally burn in a dark place in hell for their evilness) try to alter their identity inside the BEGIN block.

I've been forced to do a fallback to M::E::VERSION when I get weirdness in the output from my perl -e approach. I dislike this approach because it misses corner cases where multiple packages live inside a single .pm file.

I'm sure you've seen many of these corner cases. How did you deal with them?

It seems to me META.yml is desperately in need of a 'provides' section which could properly document the author's modules and versions as they intended. Has anyone heard any discussion of this idea ever?

Thanks,
Todd

On 3/10/10 6:13 PM, Todd Rinaldo wrote:
> I want to use PPI to determine all packages and their $VERSION in a
> given pm file. I've determined how to get the list of packages in the
> file, but not the version. How can I do this?

The only authoritative way of doing this is to load the module and check it at runtime.  (Of course, determining which namespaces got twiddled in the process is difficult.

You might want to look at http://search.cpan.org/dist/Module-Extract-VERSION/.

If you want to do it yourself via PPI, I would probably set up a state machine that tracked what the current package is and when a $VERSION assignment is found, save it off in a stash of some sort.

Alternatively, there's http://perlcritic.tigris.org/source/browse/perlcritic/trunk/distributions/PPIx-Utilities/lib/PPIx/Utilities/Node.pm?revision=3777&view=markup.  You could use it to split your module up into namespaces and then look for $VERSION assignments in that.  (I haven't released this yet due to wanting to move some code from Perl::Critic into the distribution first, but I'm still debating how to do it.)

I want to use PPI to determine all packages and their $VERSION in a given pm file. I've determined how to get the list of packages in the file, but not the version. How can I do this? I've included what I have so far:

use PPI;

my $file = 'Uplevel.pm'; # Testing with Sub::Uplevel for POC
my $doc = PPI::Document->new($file) or die;
my $pkgs = $doc->find('PPI::Statement::Package');

foreach my $pkg (@$pkgs) {
    my $package_name = $pkg->namespace;
    # I don't know how to get the $VERSION from here.
}

Dear list,

Currently there is no satisfactory (to me, at least) tool to parse regular expressions. Regexp::Parse has been fallow for three or four years, and complains mightily under Perl 5.010 unless patched.

My proposal is a regular expression parser that supports Perl 5.010 and provides the PPI navigational interface. I do not anticipate mutators (other than perhaps something to get rid of insignificant tokens). I also do not anticipate actually subclassing PPI elements, since there does not appear to be a documented way to do this.

Thoughts? Questions?

Tom Wyant

[ccing the PPI list]

On Dec 16, 2008, at 7:10 AM, jef...@gm... wrote:

> There seems to be a parsing problem. P::C gives for the following  
> code:
>
> #!/usr/bin/perl
> use warnings;
> use strict;
>
> sub temp {
> print $i++/1;
> return;
> }
>
> Subroutine does not end with "return" at line 5, column 1. See page  
> 197 of PBP. (Severity: 4)
>
> Regards
>
> Jeff

Yes, that appears to be a PPI parse flaw.

       PPI::Statement
         PPI::Token::Word        'print'
         PPI::Token::Whitespace          ' '
         PPI::Token::Symbol      '$i'
         PPI::Token::Operator    '++'
         PPI::Token::Regexp::Match       '/1;\n   return;\n}\n'

Chris

I'd like to move highest_explicit_perl_version() from Perl::Critic::Document to PPI::Document.  This is implemented using version.pm because I didn't want to get into the business of figuring out how to compare versions myself.

Moving highest_explicit_perl_version() would be the first step in implementing say_enabled(), switch_enabled(), and state_enabled().

version.pm's current minimum perl version is 5.005_04.

Let's discuss the following code:

    given ($foo) {
        when (@blah) { }
        default { }
    }

I would have thought that given would be a compound statement (especially in the face of the ability to use when/default in foreach loops); however, perlsyn does not include it in its list of compound statements, so it sounds like we'd need a new subclass of Statement for it.  (Also, this would help differentiation in Perl::Critic, especially in the determination of whether a specific Policy applies to an Element.)  Or we could ignore perlsyn's omission and treat it as Compound.

Should the parentheses be a Structure::Condition?  Or something analogous to Structure::ForLoop? (Again, from the P::C perspective, having a new class would be better.)

Then, the when/default.  The actual statements, like the given, could be Statement::Compound instances, and the parentheses after when could just be Conditions again.  Even if there's a new class, the when and default could be instances of the same one, since perlsyn says that default is precisely equivalent to "when (1 == 1) { ... }".

So, ignoring whitespace tokens, as I see it, here's the most conservative way of this parsing:

PPI::Document
  PPI::Statement::Compound
    PPI::Token::Word  	'given'
    PPI::Structure::Condition  	( ... )
      PPI::Statement::Expression
        PPI::Token::Symbol  	'$foo'
    PPI::Structure::Block  	{ ... }
      PPI::Statement::Compound
        PPI::Token::Word  	'when'
        PPI::Structure::Condition  	( ... )
          PPI::Statement::Expression
            PPI::Token::Symbol  	'@blah'
        PPI::Structure::Block  	{ ... }
      PPI::Statement::Compound
        PPI::Token::Word  	'default'
        PPI::Structure::Block  	{ ... }

OTOH, here's the most favorable structure from P::C's perspective (modulo the class names changing):

PPI::Document
  PPI::Statement::Switch
    PPI::Token::Keyword  	'given'
    PPI::Structure::Given  	( ... )
      PPI::Statement::Expression
        PPI::Token::Symbol  	'$foo'
    PPI::Structure::Block  	{ ... }
      PPI::Statement::When
        PPI::Token::Keyword  	'when'
        PPI::Structure::Match  	( ... )
          PPI::Statement::Expression
            PPI::Token::Symbol  	'@blah'
        PPI::Structure::Block  	{ ... }
      PPI::Statement::When
        PPI::Token::Keyword  	'default'
        PPI::Structure::Block  	{ ... }

Yes, I'm thinking of trying to implement this.  First, I need to figure out the parser and see whether I can fix some existing parsing problems.

I just added a test to t/data/05_lexer/05_compound_loops.code that has it parse the same way as if the qw<> had parentheses around it, except the ForLoop instance doesn't show any parentheses in the dump.  Whether that's the way that Adam thinks that it should look, I leave up to him.

Chris Dolan wrote:
> Elliot,
> Are you sure that's valid syntax?  I don't think it's legal to omit the 
> parens.

Perfectly valid.

Guy at work was running into a RequireFinalReturn violation because he used that form, even though he had a perfectly good return statement.

Well, not strictly so.  The form in the subject results in a "Missing $ on loop variable".  However, if you declare the loop variable like so:

perl -e 'foreach my $blah qw<blah blah> { print "$blah\n" }'

it works.  No parens anywhere.

Try it.

Elliot,
Are you sure that's valid syntax?  I don't think it's legal to omit  
the parens.

% perl -le'for qw<a b c> { }'
Missing $ on loop variable at -e line 1.

Did you mean "for (qw<a b c>) { }" ?

% ppidump 'for (qw<blah>) { }'
PPI::Document
   PPI::Statement::Compound
     PPI::Token::Word    'for'
     PPI::Structure::ForLoop     ( ... )
       PPI::Statement
         PPI::Token::QuoteLike::Words    'qw<blah>'
     PPI::Structure::Block       { ... }

Chris

On Jul 9, 2008, at 6:55 PM, Elliot Shank wrote:

> Currently, it's pretty ugly:
>
> PPI::Document
>   PPI::Statement::Compound
>     PPI::Token::Word  	'for'
>   PPI::Statement
>     PPI::Token::QuoteLike::Words  	'qw<blah>'
>     PPI::Structure::Block  	{ ... }
>
> Should that be
>
> PPI::Document
>   PPI::Statement::Compound
>     PPI::Token::Word  	'for'
>     PPI::Token::QuoteLike::Words  	'qw<blah>'
>     PPI::Structure::Block  	{ ... }
>
> or
>
> PPI::Document
>   PPI::Statement::Compound
>     PPI::Token::Word  	'for'
>     PPI::Structure::ForLoop  	...
>       PPI::Statement
>         PPI::Token::QuoteLike::Words  	'qw<blah>'
>     PPI::Structure::Block  	{ ... }
>
> ---------------------------------------------------------------------- 
> ---
> Sponsored by: SourceForge.net Community Choice Awards: VOTE NOW!
> Studies have shown that voting for your favorite open source project,
> along with a healthy diet, reduces your potential for chronic lameness
> and boredom. Vote Now at http://www.sourceforge.net/community/cca08
> _______________________________________________
> Parseperl-discuss mailing list
> Par...@li...
> https://lists.sourceforge.net/lists/listinfo/parseperl-discuss

Currently, it's pretty ugly:

PPI::Document
  PPI::Statement::Compound
    PPI::Token::Word  	'for'
  PPI::Statement
    PPI::Token::QuoteLike::Words  	'qw<blah>'
    PPI::Structure::Block  	{ ... }

Should that be

PPI::Document
  PPI::Statement::Compound
    PPI::Token::Word  	'for'
    PPI::Token::QuoteLike::Words  	'qw<blah>'
    PPI::Structure::Block  	{ ... }

or

PPI::Document
  PPI::Statement::Compound
    PPI::Token::Word  	'for'
    PPI::Structure::ForLoop  	...
      PPI::Statement
        PPI::Token::QuoteLike::Words  	'qw<blah>'
    PPI::Structure::Block  	{ ... }

Adam (and others), given the discussion at YAPC about classification methods returning true and false only if there is a definite answer and nothing otherwise, what do you think of the test cases for this method?  Are they correct?  What cases have I missed?

http://conferences.mongueurs.net/yn2008/wiki?node=PPI%20BOF

Elliot Shank wrote:
> Whoops.  Despite what I said about it on use.perl.org, the TODO test
> that involves the labels is not passing with this.  I'll investigate
> this evening.

And said failure is my fault, not PPI's.

Whoops.  Despite what I said about it on use.perl.org, the TODO test that involves the labels is not passing with this.  I'll investigate this evening.

I added an Ohloh.net project summary for PPI today.
   http://www.ohloh.net/projects/9254?p=PPI

Chris

Message-Id: <200...@p1...>
Date: Tue, 16 Oct 2007 12:49:17 +0200 (CEST)

Our Ref: CBN/IRD/CGX/NNPC/021/06

FROM THE DESK OF:MR TUNDE LEMO THE

DEPUTY GOVERNOR CENTRAL BANK OF NIGERIA (CBN)

Central Bank of Nigeria (CBN)

Tel: 234-80-66240858

RE: IMMEDIATE RELEASE OF YOUR OVERDUE FUNDS

After a joint meeting of the Federal Executive Council (FEC), the Senate Committee on Foreign Debts Reconciliation and the Presidential Payment and Implementation Panel on Contract/ inheritance funds under Category ?? which was addressed and headed by the New Elected president of the Federal Republic of Nigeria, Alhaji Umaru Musa Yar'dua, it became imperative to contact you on the subject matter. This meeting was initiated as part of the recent image laundering scheme of the federal government of Nigeria. 

During the meeting, so many negative reports were tabled on behalf of our numerous contractors and foreign personels on how unfairly they have been treated and extorted by some corrupt officials who were vested with the authority to pay them their entitlements. The most annoying and irritating aspect of it all is that they could not effect payment to these foreign beneficiaries aftersubjecting them to so much stress and trauma.

To this regards, this office has been given the SOLE authority by the President and commander-in-chief of the armed forces of the federal republic of Nigeria to handle and release all contract/inheritance funds and you are therefore advised NOT to contact ANY OTHER OFFICE to avoid further complications in your payment file. This information was published in one of the national dailies in Nigeria on the 29th of September, 2007. I will send you a copy of the newspaper publication as soon as you confirm reciept of this email.

Every arrangement for the remittance of your overdue funds payment of US$22m has been conluded and the funds will be released within 48hrs banking hours of your response. All documentation to this effect is inplace and we have already prepared your key tested telex (KTT) for your transfer to your bankaccount. In your file,we noticed that your designated bank account cordinates to be as follows and we want you to reconfirm this information first before remittance. 

The bank account is as follows:

UNION BANK OF SWITZERLAND, UBS, SA

16, PALACE DUMANOIR

ACC. # 240-384-287-601

ABA#101200329.

Kindly re-confirm to me if this is inline with what you have in your record and also re-confirm the informations below to enable this office proceed and finalize your fund remittance without further delays.

1) Your full name.

2) Phone, fax and mobile #.

3) Company name, position and address.

4) Profession, age and marital status.

5) Are you a property owner

6) A scanned copy of Working I.d/Int'l passport.

You are to contact this OFFICE as soon as you receive this message on via forimmediate remittance of your funds.I apologise on behalf of the Federal Republic of Nigeria for the delay in your payment,and we promise you that it will never repeat itself again as we are into our new government.Your response should be channeled to my alternate email account at (tun...@ho...)

Best regards,

MR TUNDE LEMO 

THE DEPUTY GOVERNOR CENTRAL BANK OF NIGERIA (CBN)

DIRECT LINE: 234-80-66240858

PRIVATE EMAIL: tun...@ho...

It's now been exactly 1 year since the last stable release of PPI.   
In that time, many valuable fixes and features have made it into SVN,  
some of which I eagerly anticipate for Perl::Critic.

I would like to start a discussion about the PPI roadmap.  In the  
short term, can we expect a v1.200 soon?  In the longer term:
  * Can I get my PPI commit bit back?
  * Can we create a prioritized list of which features and bugs  
should be the focus for post-1.200?
    I'm willing to create a first draft of that list, starting with  
the 17 open bugs on RT.
  * Can we consider co-maintenance?
    (not that I actually *want* to be a PPI maintainer...)

Thanks,
Chris

Attached is a patch with adds four failing test cases to PPI and  
fixes a trivial test counting bug.  I've mentioned some of these to  
Adam in personal email, but I wanted to archive it on this list so it  
doesn't get lost.  I seem to have lost SVN access to the PPI  
repository sometime in the last 9 months so am unable to apply this  
patch myself.

Chris

Hi Adam,

PPI 1.118 has a bug parsing hash constructors (it thinks they are  
hash subscripts).  SVN has the fix.  Would you please push out a new  
release?  This will help Perl::Critic.

Thanks,
Chris

--
Chris Dolan, Equilibrious LLC, http://equilibrious.net/
Public key: http://chrisdolan.net/public.key
vCard: http://chrisdolan.net/ChrisDolan.vcf

I have a line of code like:

foo(bar("Hello World"));

And I want to re-factor it into:

bar("Hello World");

Of course the real statements are likely to be more complex, multiple 
lines, conditionals, etc.  I can remove the 'foo' bareword easily enough 
but I can't figure out how to get rid of the outer list without loosing 
the inner 'bar' expression.

Is PPI currently capable of such things?

I have been trying variations on the code below.

thanks,
Stuart Johnston

my $code = q{foo(bar("Hello World"));};
my $d = PPI::Document->new(\$code);
my $foo = $d->find_first('PPI::Token::Word');
my $list = $foo->next_sibling;
my $exp = $list->child(0);
$exp->remove;
$list->delete;
$foo->insert_after($exp);

Hi,

Is there a plan to include support for variable attributes in PPI?

At first sight it looks like small change in =
__TOKENIZER__is_an_attribute()=20
(PPI\Token\Unknown.pm) will suffice.

Regards
Przemek Czerkas

Aloha,

since I have quite a bit of XS experience, I thought I'd tackle
implementing a few methods in PPI::XS to speed things up.  To find
bottlenecks I used the module I talked about in the earlier message and
let it analyze a ~3500 line program under the supervision of a
profiler.[1]  The profiles unveiled a few hotspots:

* PPI::Node::find_first, 9 calls, ~9 seconds
* PPI::Tokenizer::_process_next_char, 44305 calls, ~3.5 seconds
* PPI::Lexer::_lex_statement, 2908 calls, ~3 seconds

PPI::Node::find_first calls many methods and does little computation on
its own, so implementing it in XS wouldn't provide much benefit.  The
same is true for PPI::Lexer::_lex_statement.
PPI::Tokenizer::_process_next_char looked more promising.  So I wrote an
XS port of it.  Patch attached.

Well, the bad news is that this didn't improve performance measurably.

I looked at a few other candidates in the profiles but the all seem to
have the same issue: not much computation, lots of method calls.  I
don't think you can improve code like this very much by simply porting
it to XS verbatim.  perl already does a pretty good job when it creates
the optree, it seems.

So, any suggestions on what might be a good candidate for porting?

-- 
Bye,
-Torsten

[1] I tried Devel::DProf, Devel::Profile, and Devel::AutoProfiler.  All
yielded similar results.

2005	Jan	Feb	Mar	Apr	May	Jun (1)	Jul (10)	Aug (8)	Sep (1)	Oct	Nov (3)	Dec
2006	Jan (2)	Feb (2)	Mar	Apr	May	Jun	Jul (5)	Aug (2)	Sep (12)	Oct (26)	Nov	Dec
2007	Jan	Feb	Mar (2)	Apr	May (1)	Jun	Jul	Aug (1)	Sep (1)	Oct (2)	Nov	Dec
2008	Jan	Feb	Mar	Apr	May (3)	Jun (1)	Jul (3)	Aug	Sep (3)	Oct	Nov	Dec (1)
2009	Jan	Feb	Mar	Apr	May	Jun	Jul	Aug	Sep	Oct (2)	Nov	Dec
2010	Jan	Feb	Mar (4)	Apr	May	Jun	Jul	Aug	Sep	Oct	Nov	Dec
2014	Jan (1)	Feb	Mar	Apr	May	Jun	Jul	Aug	Sep	Oct	Nov	Dec

parseperl-discuss Mailing List for Parse::Perl

parseperl-discuss — Parse::Perl Discussion List