From: Steve Y. <st...@go...> - 2010-01-13 00:43:16
|
I've got a patch to submit for everyone's consideration. It's a static analyzer and indexer for Python source, built atop the Jython antlr parser. It was primarily designed to produce an index that can support IDE functionality, but it could potentially also be used to provide type information to the interpreter for optimization purposes. Some notes about the indexer: * it was an intern project from last summer, written from scratch by our intern Yin Wang - I've been tinkering with it for 4 or 5 months, getting it to index Google's entire Python code base * the foundation is a traditional bottom-up/top-down type-inferencer. Yin did some really great work here. It could be better (what inferencer couldn't?), but it does a nice job in the absence of type annotations. - Jim Baker's suggestion for using decorators for annotations seems like a great way to feed it type hints - I'm avoiding going that route until the Python community gains consensus around standardized type hints * it builds an index of uniquely qualified global names, prefixing them with package paths as necessary. This makes it possible to build a merged index across many projects without fear of name collisions. * it statically models a large portion of the standard library -- some 50 modules and perhaps 1000 definitions. - this would be easier and more effective if Python had a way to annotate return types, but it works OK. * it provides some error/warning lint diagnostics, although we haven't done too much here yet. It probably has enough information today to replicate the diagnostics from every other Python lint tool. * it has a decent unit-test suite, with around 500 assertions. could be better, but it's a good start. * it comes with a quick-and-dirty demo app I wrote last week. You point the app at a file or directory, and it recursively analyzes/indexes all the source and creates static html pages with LXR-like links between identifiers, per-file structural outlines, and syntax and (some) semantic highlighting. - the demo app doesn't show off some of the fancier features (e.g. inferred types, xrefs, diagnostics) There are bugs, of course, and there are many ways it can be improved. But it works pretty well. If nothing else, it's one of the most ass-kicking intern projects I've seen in the last twenty years. The indexer is mostly self-contained, residing in a new package org.python.indexer. We had to make some tweaks to Python.g and the support classes, but we didn't modify the AST; instead Yin designed a new AST format for the indexer, and wrote a conversion pass before the analysis. (We could use the current AST, but we wanted to minimize the diffs on the existing code for now.) Yin is back in his Ph.D. program, but he has expressed interest in continuing work on the analyzer. My team at Google also plans to continue work on it on a part-time basis, as it's an important part of a larger project we're doing in this space. I'd like to submit a patch for review . I'm happy to make changes as needed, provided they're not earth-shatteringly large feature requests. The patch is in two parts: files we needed to change in Jython, and a zip of new files and directories. I can't easily put them all in one patch because I can't svn add them to my working copy of head. If someone creates a bug or feature request at bugs.jython.org, I should be able to attach the patch to it. Or I can email it directly to one of the maintainers and they can do it; either way. Let me know how you'd like to proceed. Thanks, -steve |
From: Stefan B. <ste...@be...> - 2010-01-13 07:54:27
|
Steve Yegge, 13.01.2010 01:42: > I've got a patch to submit for everyone's consideration. It's a static > analyzer and indexer for Python > source, built atop the Jython antlr parser. It was primarily designed to > produce an index that can > support IDE functionality, but it could potentially also be used to provide > type information to the > interpreter for optimization purposes. Interesting. This sounds quite a bit better than what we currently have in Cython (especially the stdlib bits). What's it written in? Could you send me a copy? Stefan |
From: Oti <oh...@gm...> - 2010-01-13 14:56:47
|
Steve, this is great news! Please feel free to attach your patch to http://bugs.jython.org/issue1541. Many thanks, Oti. On Wed, Jan 13, 2010 at 1:42 AM, Steve Yegge <st...@go...> wrote: > I've got a patch to submit for everyone's consideration. It's a static > analyzer and indexer for Python > source, built atop the Jython antlr parser. It was primarily designed to > produce an index that can > support IDE functionality, but it could potentially also be used to provide > type information to the > interpreter for optimization purposes. > Some notes about the indexer: > * it was an intern project from last summer, written from scratch by our > intern Yin Wang > - I've been tinkering with it for 4 or 5 months, getting it to index > Google's entire Python code base > * the foundation is a traditional bottom-up/top-down type-inferencer. Yin > did some really great work here. > It could be better (what inferencer couldn't?), but it does a nice job > in the absence of type annotations. > - Jim Baker's suggestion for using decorators for annotations seems like > a great way to feed it type hints > - I'm avoiding going that route until the Python community gains > consensus around standardized type hints > * it builds an index of uniquely qualified global names, prefixing them > with package paths as necessary. > This makes it possible to build a merged index across many projects > without fear of name collisions. > * it statically models a large portion of the standard library -- some 50 > modules and perhaps 1000 definitions. > - this would be easier and more effective if Python had a way to > annotate return types, but it works OK. > * it provides some error/warning lint diagnostics, although we haven't > done too much here yet. > It probably has enough information today to replicate the diagnostics > from every other Python lint tool. > * it has a decent unit-test suite, with around 500 assertions. could be > better, but it's a good start. > * it comes with a quick-and-dirty demo app I wrote last week. You point > the app at a file or directory, > and it recursively analyzes/indexes all the source and creates static > html pages with LXR-like links > between identifiers, per-file structural outlines, and syntax and (some) > semantic highlighting. > - the demo app doesn't show off some of the fancier features (e.g. > inferred types, xrefs, diagnostics) > There are bugs, of course, and there are many ways it can be improved. But > it works pretty well. > If nothing else, it's one of the most ass-kicking intern projects I've seen > in the last twenty years. > The indexer is mostly self-contained, residing in a new package > org.python.indexer. We had > to make some tweaks to Python.g and the support classes, but we didn't > modify the AST; instead > Yin designed a new AST format for the indexer, and wrote a conversion pass > before the analysis. > (We could use the current AST, but we wanted to minimize the diffs on the > existing code for now.) > Yin is back in his Ph.D. program, but he has expressed interest in > continuing work on the analyzer. > My team at Google also plans to continue work on it on a part-time basis, as > it's an important part > of a larger project we're doing in this space. > I'd like to submit a patch for review . I'm happy to make changes as > needed, provided they're > not earth-shatteringly large feature requests. > The patch is in two parts: files we needed to change in Jython, and a zip > of new files and directories. > I can't easily put them all in one patch because I can't svn add them to my > working copy of head. > If someone creates a bug or feature request at bugs.jython.org, I should be > able to attach the > patch to it. Or I can email it directly to one of the maintainers and they > can do it; either way. > Let me know how you'd like to proceed. > Thanks, > -steve > ------------------------------------------------------------------------------ > This SF.Net email is sponsored by the Verizon Developer Community > Take advantage of Verizon's best-in-class app development support > A streamlined, 14 day to market process makes app distribution fast and easy > Join now and get one step closer to millions of Verizon customers > http://p.sf.net/sfu/verizon-dev2dev > _______________________________________________ > Jython-dev mailing list > Jyt...@li... > https://lists.sourceforge.net/lists/listinfo/jython-dev > > |
From: Oti <oh...@gm...> - 2010-01-13 21:07:33
|
Hi Steve, sorry about that - no idea what was hindering you in the bug tracker. I added your comment, and the .zip file. best wishes, Oti. On Wed, Jan 13, 2010 at 9:59 PM, Steve Yegge <st...@go...> wrote: > Bummer - I tried it and it said: "You do not have permission to create > file". > Here's what I'd added to the comments field: > --------------------------------------------------------- > The attached patchfile is a zip of 2 files: modified files and new files. > To apply it: > - unzip google-indexer.zip into .../jython (parent of src/ directory) > - tar zxf google-indexer-newfiles.tar.gz > - patch -p0 < google-indexer-patch.diff > You can run the unit tests with "ant idxtest". Instructions for running the > demo app are in the doc header of > src/org/python/indexer/demos/HtmlDemo.java. > --------------------------------------------------------- > I've attached google-indexer.zip to this email. > I'm happy to make changes as needed until you feel it's ready to put in the > trunk. We'll probably also want > a lightweight process for getting periodic batches of fixes in from my team. > We just launched it internally > at Google today, so we're likely to see lots of bug reports trickle in. > Cheers, > -steve > On Wed, Jan 13, 2010 at 6:56 AM, Oti <oh...@gm...> wrote: >> >> Steve, >> >> this is great news! >> Please feel free to attach your patch to http://bugs.jython.org/issue1541. >> >> Many thanks, >> Oti. >> >> On Wed, Jan 13, 2010 at 1:42 AM, Steve Yegge <st...@go...> wrote: >> > I've got a patch to submit for everyone's consideration. It's a static >> > analyzer and indexer for Python >> > source, built atop the Jython antlr parser. It was primarily designed >> > to >> > produce an index that can >> > support IDE functionality, but it could potentially also be used to >> > provide >> > type information to the >> > interpreter for optimization purposes. >> > Some notes about the indexer: >> > * it was an intern project from last summer, written from scratch by >> > our >> > intern Yin Wang >> > - I've been tinkering with it for 4 or 5 months, getting it to >> > index >> > Google's entire Python code base >> > * the foundation is a traditional bottom-up/top-down type-inferencer. >> > Yin >> > did some really great work here. >> > It could be better (what inferencer couldn't?), but it does a nice >> > job >> > in the absence of type annotations. >> > - Jim Baker's suggestion for using decorators for annotations seems >> > like >> > a great way to feed it type hints >> > - I'm avoiding going that route until the Python community gains >> > consensus around standardized type hints >> > * it builds an index of uniquely qualified global names, prefixing >> > them >> > with package paths as necessary. >> > This makes it possible to build a merged index across many projects >> > without fear of name collisions. >> > * it statically models a large portion of the standard library -- some >> > 50 >> > modules and perhaps 1000 definitions. >> > - this would be easier and more effective if Python had a way to >> > annotate return types, but it works OK. >> > * it provides some error/warning lint diagnostics, although we haven't >> > done too much here yet. >> > It probably has enough information today to replicate the >> > diagnostics >> > from every other Python lint tool. >> > * it has a decent unit-test suite, with around 500 assertions. could >> > be >> > better, but it's a good start. >> > * it comes with a quick-and-dirty demo app I wrote last week. You >> > point >> > the app at a file or directory, >> > and it recursively analyzes/indexes all the source and creates >> > static >> > html pages with LXR-like links >> > between identifiers, per-file structural outlines, and syntax and >> > (some) >> > semantic highlighting. >> > - the demo app doesn't show off some of the fancier features (e.g. >> > inferred types, xrefs, diagnostics) >> > There are bugs, of course, and there are many ways it can be improved. >> > But >> > it works pretty well. >> > If nothing else, it's one of the most ass-kicking intern projects I've >> > seen >> > in the last twenty years. >> > The indexer is mostly self-contained, residing in a new package >> > org.python.indexer. We had >> > to make some tweaks to Python.g and the support classes, but we didn't >> > modify the AST; instead >> > Yin designed a new AST format for the indexer, and wrote a conversion >> > pass >> > before the analysis. >> > (We could use the current AST, but we wanted to minimize the diffs on >> > the >> > existing code for now.) >> > Yin is back in his Ph.D. program, but he has expressed interest in >> > continuing work on the analyzer. >> > My team at Google also plans to continue work on it on a part-time >> > basis, as >> > it's an important part >> > of a larger project we're doing in this space. >> > I'd like to submit a patch for review . I'm happy to make changes as >> > needed, provided they're >> > not earth-shatteringly large feature requests. >> > The patch is in two parts: files we needed to change in Jython, and a >> > zip >> > of new files and directories. >> > I can't easily put them all in one patch because I can't svn add them to >> > my >> > working copy of head. >> > If someone creates a bug or feature request at bugs.jython.org, I should >> > be >> > able to attach the >> > patch to it. Or I can email it directly to one of the maintainers and >> > they >> > can do it; either way. >> > Let me know how you'd like to proceed. >> > Thanks, >> > -steve >> > >> > ------------------------------------------------------------------------------ >> > This SF.Net email is sponsored by the Verizon Developer Community >> > Take advantage of Verizon's best-in-class app development support >> > A streamlined, 14 day to market process makes app distribution fast and >> > easy >> > Join now and get one step closer to millions of Verizon customers >> > http://p.sf.net/sfu/verizon-dev2dev >> > _______________________________________________ >> > Jython-dev mailing list >> > Jyt...@li... >> > https://lists.sourceforge.net/lists/listinfo/jython-dev >> > >> > > > |
From: Nicholas R. <nj...@il...> - 2010-01-13 23:25:16
|
In article <ba9...@ma...>, Oti <oh...@gm...> wrote: > sorry about that - no idea what was hindering you in the bug tracker. > I added your comment, and the .zip file. I saw another complaint about a user not being able to create attachments in the bug tracker. This is a bit of a problem for people who want to commit patches. Is this configurable by the people who are Roundup admins or do we need to file a bug on the meta-tracker? -- Nicholas Riley <nj...@il...> |
From: Raghuram D. <dra...@gm...> - 2010-01-14 14:16:13
|
On Wed, Jan 13, 2010 at 6:18 PM, Nicholas Riley <nj...@il...> wrote: > Is this configurable by the people who are Roundup admins or do we need > to file a bug on the meta-tracker? Filing a bug on the meta-tracker is the best way to reach the admins. Can you please open a bug there? Thanks, Raghu |
From: Anders O. <and...@gm...> - 2010-01-14 14:33:44
|
On Tue, Jan 12, 2010 at 04:42:54PM -0800, Steve Yegge wrote: > I've got a patch to submit for everyone's consideration. It's a static > analyzer and indexer for Python > source, built atop the Jython antlr parser. It was primarily designed to > produce an index that can > support IDE functionality, but it could potentially also be used to provide > type information to the > interpreter for optimization purposes. Sorry for a complete newbie question but what is an indexer and hwo does the output look like? Best Regards Anders Olme |
From: Steve Y. <st...@go...> - 2010-01-14 21:22:27
|
On Thu, Jan 14, 2010 at 6:33 AM, Anders Olme <and...@gm...> wrote: > On Tue, Jan 12, 2010 at 04:42:54PM -0800, Steve Yegge wrote: > > I've got a patch to submit for everyone's consideration. It's a static > > analyzer and indexer for Python > > source, built atop the Jython antlr parser. It was primarily designed to > > produce an index that can > > support IDE functionality, but it could potentially also be used to > provide > > type information to the > > interpreter for optimization purposes. > Sorry for a complete newbie question but what is an indexer and hwo does > the output look like? > No worries. It's actually a nontrivial concept that's difficult to explain without providing some context. In a nutshell, it's a compiler for Python code, but it emits metadata rather than generated code. While that answer may be good enough for you as-is, I'll go ahead and elaborate for posterity. As you no doubt know, compilers perform a set of standard transformations: * scanning/lexing to get tokens * parsing to turn the tokens into a syntax tree * creating a symbol table to represent identifier scopes and other info * building a type representation of the program (sometimes using inference) * performing semantic checks such as used-before-assigned or incorrect-type * code generation * optimizations that can happen at every step of the way This is all typically done statically, without actually running the code, which means that compilation inevitably creates a model that is merely an approximation of the runtime semantics. The accuracy and completeness of this approximation can be improved by changing the language (e.g. Haskell > Java/C++ > Python in terms of support for static analysis) or by putting more work into the compiler (e.g. gcc does a better job than a C compiler produced by a student as a term project). There are many tools in the compiler family which generate simple static models sufficient (only) for their own needs -- for instance, linters, syntax highlighters, pretty printers, documentation generators, even classic code-generating compilers. This implies that lots of tools developers are reinventing the wheel just well enough for their own tools. But IDEs like Eclipse or Visual Studio are also in the compiler family. While they began as aggregators of other compiler tools, these days they actually create the richest general-purpose static models of the code, because they need to manipulate and surface a superset of the information from all the standalone tools. Hence, IDE backends are special-purpose compilers that do their best never to throw anything away, which differentiates them from all other compiler-like tools, including compilers themselves. The upside of this is that you run the indexer and get a general-purpose programming model and API which together allow you you query and manipulate the code as a set of data structures. At that point you can do anything you like: build a global symbol autocompletion index, or write some refactoring functions, or display fancy views of the class tree or type graph -- anything the model permits. An IDE backend should in theory be able to replace the static model for any other compiler-like tool, which is nice: no more wheel-reinventing necessary, at least provided the IDE actually exposes its indexer in headless mode, which a few of them do today. The downside is that it's fat and slow. Code produces a -lot- of metadata, and rich IDEs have to do lots of multi-level caching tricks to avoid blowing out memory or consuming all of your CPU, or both. The indexer I've submitted has had a fair bit of tuning to try to minimize its memory use, but it still struggles with projects with, say, 5000 files on a 32-bit JVM given -Xmx3000 or so. So it could use more work in this space. For now, we simply partition our code base into many projects, analyze them all in parallel, and merge the results into a large serialized version of the data structures produced by the analyzer shards -- a.k.a. "the index". The version of the indexer I've submitted here does not have a serialized representation. Internally at Google we have a language-neutral representation shared by all our language analyzers, but it's work-in-progress and relies heavily on Google infrastructure, so it's not yet appropriate for open-sourcing. At some point we hope to get there, but for now, the open-source Python indexer is suitable for small- to medium-sized projects. Serializing the index will be the next big evolutionary step, since if done properly, it can help the indexer scale to larger projects while simultaneously increasing the expressive power of the underlying model. I suspect that a relational database is likely to be the best choice, but defining the schema will be a tricky and ongoing task. Indexers like this one need to keep growing to be maximally useful -- at the very least they need to support the evolution of the language itself, but they also need to refine their static models to support the ever-increasing demands of the tools and their users. (As one concrete example, improving the indexer's internal model of Python's type system will yield better results across the board, e.g. picking up identifier references that it currently misses, but this comes at the cost of increasing both the time spent during analysis and the amount of information stored in the index.) The demo app I provided builds an index and then "walks" it using a visitor interface to produce a simple static-HTML view of the indexed code, with hyperlinks between identifiers similar to those produced by the Linux Cross-Referencer. But the indexer produces a great deal more metadata than is used by the demo. I'm hoping that some tools-inclined devs will jump in and start using the indexer to power their own tools efforts, making improvements as needed. With luck I may find time to write more demo apps, since it's easier to appreciate the indexer's value when you can visualize the information it provides. In the meantime, Python is a pretty big language at Google, and indexing all the code should improve life for our Python developers, so we'll keep hacking on the indexer and sending periodic updates. (This will of course be easier if it's in the svn repository. :) Hope that helps! -steve > Best Regards Anders Olme > > > ------------------------------------------------------------------------------ > Throughout its 18-year history, RSA Conference consistently attracts the > world's best and brightest in the field, creating opportunities for > Conference > attendees to learn about information security's most important issues > through > interactions with peers, luminaries and emerging and established companies. > http://p.sf.net/sfu/rsaconf-dev2dev > _______________________________________________ > Jython-dev mailing list > Jyt...@li... > https://lists.sourceforge.net/lists/listinfo/jython-dev > |
From: Anders O. <and...@gm...> - 2010-01-15 06:58:28
|
Hello, Thanks for the explanation! I moslty lurk here since im interested in python jython and to try to learn how computer languages work ( im an electrical engineer by trade so i havent studied any compiler courses ). Any how thanks for the explanation. Best Regards Anders Olme |
From: Oti <oh...@gm...> - 2010-02-05 12:04:29
|
Hi Steve, I'd like to let you know I just started to apply your patch to my local workspace. After converting some java6 only features back to java5 I will run the tests. If I should have bigger trouble, I will email you directly (if you don't mind). http://bugs.jython.org/issue1541 might contain some lower level details. best wishes, Oti. On Wed, Jan 13, 2010 at 9:59 PM, Steve Yegge <st...@go...> wrote: > Bummer - I tried it and it said: "You do not have permission to create > file". > Here's what I'd added to the comments field: > --------------------------------------------------------- > The attached patchfile is a zip of 2 files: modified files and new files. > To apply it: > - unzip google-indexer.zip into .../jython (parent of src/ directory) > - tar zxf google-indexer-newfiles.tar.gz > - patch -p0 < google-indexer-patch.diff > You can run the unit tests with "ant idxtest". Instructions for running the > demo app are in the doc header of > src/org/python/indexer/demos/HtmlDemo.java. > --------------------------------------------------------- > I've attached google-indexer.zip to this email. > I'm happy to make changes as needed until you feel it's ready to put in the > trunk. We'll probably also want > a lightweight process for getting periodic batches of fixes in from my team. > We just launched it internally > at Google today, so we're likely to see lots of bug reports trickle in. > Cheers, > -steve > On Wed, Jan 13, 2010 at 6:56 AM, Oti <oh...@gm...> wrote: >> >> Steve, >> >> this is great news! >> Please feel free to attach your patch to http://bugs.jython.org/issue1541. >> >> Many thanks, >> Oti. >> >> On Wed, Jan 13, 2010 at 1:42 AM, Steve Yegge <st...@go...> wrote: >> > I've got a patch to submit for everyone's consideration. It's a static >> > analyzer and indexer for Python >> > source, built atop the Jython antlr parser. It was primarily designed >> > to >> > produce an index that can >> > support IDE functionality, but it could potentially also be used to >> > provide >> > type information to the >> > interpreter for optimization purposes. >> > Some notes about the indexer: >> > * it was an intern project from last summer, written from scratch by >> > our >> > intern Yin Wang >> > - I've been tinkering with it for 4 or 5 months, getting it to >> > index >> > Google's entire Python code base >> > * the foundation is a traditional bottom-up/top-down type-inferencer. >> > Yin >> > did some really great work here. >> > It could be better (what inferencer couldn't?), but it does a nice >> > job >> > in the absence of type annotations. >> > - Jim Baker's suggestion for using decorators for annotations seems >> > like >> > a great way to feed it type hints >> > - I'm avoiding going that route until the Python community gains >> > consensus around standardized type hints >> > * it builds an index of uniquely qualified global names, prefixing >> > them >> > with package paths as necessary. >> > This makes it possible to build a merged index across many projects >> > without fear of name collisions. >> > * it statically models a large portion of the standard library -- some >> > 50 >> > modules and perhaps 1000 definitions. >> > - this would be easier and more effective if Python had a way to >> > annotate return types, but it works OK. >> > * it provides some error/warning lint diagnostics, although we haven't >> > done too much here yet. >> > It probably has enough information today to replicate the >> > diagnostics >> > from every other Python lint tool. >> > * it has a decent unit-test suite, with around 500 assertions. could >> > be >> > better, but it's a good start. >> > * it comes with a quick-and-dirty demo app I wrote last week. You >> > point >> > the app at a file or directory, >> > and it recursively analyzes/indexes all the source and creates >> > static >> > html pages with LXR-like links >> > between identifiers, per-file structural outlines, and syntax and >> > (some) >> > semantic highlighting. >> > - the demo app doesn't show off some of the fancier features (e.g. >> > inferred types, xrefs, diagnostics) >> > There are bugs, of course, and there are many ways it can be improved. >> > But >> > it works pretty well. >> > If nothing else, it's one of the most ass-kicking intern projects I've >> > seen >> > in the last twenty years. >> > The indexer is mostly self-contained, residing in a new package >> > org.python.indexer. We had >> > to make some tweaks to Python.g and the support classes, but we didn't >> > modify the AST; instead >> > Yin designed a new AST format for the indexer, and wrote a conversion >> > pass >> > before the analysis. >> > (We could use the current AST, but we wanted to minimize the diffs on >> > the >> > existing code for now.) >> > Yin is back in his Ph.D. program, but he has expressed interest in >> > continuing work on the analyzer. >> > My team at Google also plans to continue work on it on a part-time >> > basis, as >> > it's an important part >> > of a larger project we're doing in this space. >> > I'd like to submit a patch for review . I'm happy to make changes as >> > needed, provided they're >> > not earth-shatteringly large feature requests. >> > The patch is in two parts: files we needed to change in Jython, and a >> > zip >> > of new files and directories. >> > I can't easily put them all in one patch because I can't svn add them to >> > my >> > working copy of head. >> > If someone creates a bug or feature request at bugs.jython.org, I should >> > be >> > able to attach the >> > patch to it. Or I can email it directly to one of the maintainers and >> > they >> > can do it; either way. >> > Let me know how you'd like to proceed. >> > Thanks, >> > -steve >> > >> > ------------------------------------------------------------------------------ >> > This SF.Net email is sponsored by the Verizon Developer Community >> > Take advantage of Verizon's best-in-class app development support >> > A streamlined, 14 day to market process makes app distribution fast and >> > easy >> > Join now and get one step closer to millions of Verizon customers >> > http://p.sf.net/sfu/verizon-dev2dev >> > _______________________________________________ >> > Jython-dev mailing list >> > Jyt...@li... >> > https://lists.sourceforge.net/lists/listinfo/jython-dev >> > >> > > > |
From: Oti <oh...@gm...> - 2010-02-06 23:58:52
|
Hi Steve and Frank, I have all the tests passing locally, and I believe the build bots would pass as well. What I'd like to ask Steve to check is if the generated .html files are correct. I ran: org.python.indexer.demos.HtmlDemo ${workspace_loc}/jython-trunk/CPythonLib/email/ and then zipped the /html directory into http://bugs.jython.org/file764/html.zip This is the console output: http://bugs.jython.org/msg5507 (both links also can be found behind http://bugs.jython.org/issue1541) And I'd like to ask Frank to review the grammar changes. AFAICS they fit very well. Finally, is the copyright notice "Copyright 2009, Google Inc. All rights reserved." in the added files is OK for Jython? Please excuse my ignorance, but I keep forgetting where the potential legal problems are. I hope I get your thumbs up, and nobody else objects. Then this is ready for checkin. Thanks a lot! Oti |
From: Jim B. <jb...@zy...> - 2010-02-08 13:59:51
|
Oti, Google has an umbrella contributor agreement to the PSF for its employees (this was discussed on python-dev/jython-dev in April 2009). The contributor agreement (http://www.python.org/psf/contrib/contrib-form-python/) allows Google, like any other contributor, to retain copyright. We probably should also follow its letter by putting next to each copyright notice the following text: "Licensed to PSF under a Contributor Agreement." This practice has emerged into the Jython 2.5 source code and as of 2.6 core Python, it has become fairly standard usage; so you see entries like this: # Copyright 2007 Google, Inc. All Rights Reserved. # Licensed to PSF under a Contributor Agreement. - Jim On Sat, Feb 6, 2010 at 4:58 PM, Oti <oh...@gm...> wrote: > Hi Steve and Frank, > > I have all the tests passing locally, and I believe the build bots > would pass as well. > > What I'd like to ask Steve to check is if the generated .html files are > correct. > I ran: org.python.indexer.demos.HtmlDemo > ${workspace_loc}/jython-trunk/CPythonLib/email/ > and then zipped the /html directory into > http://bugs.jython.org/file764/html.zip > This is the console output: http://bugs.jython.org/msg5507 > (both links also can be found behind http://bugs.jython.org/issue1541) > > And I'd like to ask Frank to review the grammar changes. > AFAICS they fit very well. > > Finally, is the copyright notice > "Copyright 2009, Google Inc. All rights reserved." > in the added files is OK for Jython? > Please excuse my ignorance, but I keep forgetting where the potential > legal problems are. > > I hope I get your thumbs up, and nobody else objects. > Then this is ready for checkin. > > Thanks a lot! > Oti > > > ------------------------------------------------------------------------------ > The Planet: dedicated and managed hosting, cloud storage, colocation > Stay online with enterprise data centers and the best network in the > business > Choose flexible plans and management services without long-term contracts > Personal 24x7 support from experience hosting pros just a phone call away. > http://p.sf.net/sfu/theplanet-com > _______________________________________________ > Jython-dev mailing list > Jyt...@li... > https://lists.sourceforge.net/lists/listinfo/jython-dev > -- Jim Baker jb...@zy... |
From: Steve Y. <st...@go...> - 2010-02-08 20:29:41
|
Hi Oti, The HTML output needs some love -- the path to the css file is being hardcoded to your working directory, and the indexer is failing to pick up a lot of links, likely because I haven't provided the demo with a way to look in Jython for the stdlib. I'll fix these things and send you a replacement for the patch; I can also add in the "Licenced to PSF under a Contributor Agreement." to the files per Jim's suggestion. Thanks, -steve On Sat, Feb 6, 2010 at 3:58 PM, Oti <oh...@gm...> wrote: > Hi Steve and Frank, > > I have all the tests passing locally, and I believe the build bots > would pass as well. > > What I'd like to ask Steve to check is if the generated .html files are > correct. > I ran: org.python.indexer.demos.HtmlDemo > ${workspace_loc}/jython-trunk/CPythonLib/email/ > and then zipped the /html directory into > http://bugs.jython.org/file764/html.zip > This is the console output: http://bugs.jython.org/msg5507 > (both links also can be found behind http://bugs.jython.org/issue1541) > > And I'd like to ask Frank to review the grammar changes. > AFAICS they fit very well. > > Finally, is the copyright notice > "Copyright 2009, Google Inc. All rights reserved." > in the added files is OK for Jython? > Please excuse my ignorance, but I keep forgetting where the potential > legal problems are. > > I hope I get your thumbs up, and nobody else objects. > Then this is ready for checkin. > > Thanks a lot! > Oti > |
From: Oti <oh...@gm...> - 2010-03-02 19:31:06
|
Hi Steve, many thanks! I definitely will give it a try. When I tried the last version, I simply created a Run configuration in Eclipse, which automatically included all (or at least: most) of the .jar files in trunk/extlibs. Maybe it was because of that I did not have the posix problem. Next time I'll use the command line, to avoid confusion. best wishes, Oti. On Tue, Mar 2, 2010 at 5:05 AM, Steve Yegge <st...@go...> wrote: > Hi folks, sorry for the delay on this. I've attached a new file that > supercedes/replaces > the previous file I sent out. (Once again I tried submitting it on Jython > Tracker and got a > permissions error.) > The attachment fixes problems with my previous submission: > - added "Licensed to PSF under a Contributor Agreement" in all new files > - fixes to allow the demo app to use CPythonLib as its standard library > - inline the css so there's no trouble with paths > - misc bug fixes in the demo app (NPEs uncovered by indexing CPythonLib) > - merged in a new svn update from a few hours ago > The demo app should pick up an order of magnitude more links now. > (It would still be nice if it showed cross-references -- I'll think about > how to > do that statically for a future release.) > You can use the demo to index all of CPythonLib as follows: > ant jar-complete > java -classpath ./dist/jython.jar org.python.indexer.demos.HtmlDemo > ./CPythonLib ./CPythonLib > This tells it to use CPythonLib for the standard library, and to index > CPythonLib itself. > (This is more interesting than indexing, say, just CPythonLib/email, as it > shows more intra-file links.) > It writes about 1200 html files into ./html (103 mb uncompressed, or 55 mb > with bzip2 compression). > As before, the attachment is a zip of two files: a patch from svn diff, and > a tar.gz of new files. > Thanks for giving it a try! > -steve > On Mon, Feb 8, 2010 at 12:29 PM, Steve Yegge <st...@go...> wrote: >> >> Hi Oti, >> The HTML output needs some love -- the path to the css file is being >> hardcoded >> to your working directory, and the indexer is failing to pick up a lot of >> links, likely >> because I haven't provided the demo with a way to look in Jython for the >> stdlib. >> I'll fix these things and send you a replacement for the patch; I can also >> add in the >> "Licenced to PSF under a Contributor Agreement." to the files per Jim's >> suggestion. >> Thanks, >> -steve >> On Sat, Feb 6, 2010 at 3:58 PM, Oti <oh...@gm...> wrote: >>> >>> Hi Steve and Frank, >>> >>> I have all the tests passing locally, and I believe the build bots >>> would pass as well. >>> >>> What I'd like to ask Steve to check is if the generated .html files are >>> correct. >>> I ran: org.python.indexer.demos.HtmlDemo >>> ${workspace_loc}/jython-trunk/CPythonLib/email/ >>> and then zipped the /html directory into >>> http://bugs.jython.org/file764/html.zip >>> This is the console output: http://bugs.jython.org/msg5507 >>> (both links also can be found behind http://bugs.jython.org/issue1541) >>> >>> And I'd like to ask Frank to review the grammar changes. >>> AFAICS they fit very well. >>> >>> Finally, is the copyright notice >>> "Copyright 2009, Google Inc. All rights reserved." >>> in the added files is OK for Jython? >>> Please excuse my ignorance, but I keep forgetting where the potential >>> legal problems are. >>> >>> I hope I get your thumbs up, and nobody else objects. >>> Then this is ready for checkin. >>> >>> Thanks a lot! >>> Oti >> > > |
From: Steve Y. <st...@go...> - 2010-03-02 20:12:12
|
> Maybe it was because of that I did not have the posix problem. It turns out the little launcher script I wrote was referencing the wrong jar (jython-dev.jar rather than jython.jar). It was a bit tricky because there are only a few corner cases that trigger class-loads from 3 or 4 jars that aren't bundled in jython-dev.jar. But it boiled down to user error == mine. Anyway, let me know how it goes! Thanks, -steve On Tue, Mar 2, 2010 at 11:30 AM, Oti <oh...@gm...> wrote: > Hi Steve, > > many thanks! > I definitely will give it a try. > > When I tried the last version, I simply created a Run configuration in > Eclipse, which automatically included all (or at least: most) of the > .jar files in trunk/extlibs. Maybe it was because of that I did not > have the posix problem. Next time I'll use the command line, to avoid > confusion. > > best wishes, > Oti. > > > On Tue, Mar 2, 2010 at 5:05 AM, Steve Yegge <st...@go...> wrote: > > Hi folks, sorry for the delay on this. I've attached a new file that > > supercedes/replaces > > the previous file I sent out. (Once again I tried submitting it on > Jython > > Tracker and got a > > permissions error.) > > The attachment fixes problems with my previous submission: > > - added "Licensed to PSF under a Contributor Agreement" in all new > files > > - fixes to allow the demo app to use CPythonLib as its standard library > > - inline the css so there's no trouble with paths > > - misc bug fixes in the demo app (NPEs uncovered by indexing > CPythonLib) > > - merged in a new svn update from a few hours ago > > The demo app should pick up an order of magnitude more links now. > > (It would still be nice if it showed cross-references -- I'll think about > > how to > > do that statically for a future release.) > > You can use the demo to index all of CPythonLib as follows: > > ant jar-complete > > java -classpath ./dist/jython.jar org.python.indexer.demos.HtmlDemo > > ./CPythonLib ./CPythonLib > > This tells it to use CPythonLib for the standard library, and to index > > CPythonLib itself. > > (This is more interesting than indexing, say, just CPythonLib/email, as > it > > shows more intra-file links.) > > It writes about 1200 html files into ./html (103 mb uncompressed, or 55 > mb > > with bzip2 compression). > > As before, the attachment is a zip of two files: a patch from svn diff, > and > > a tar.gz of new files. > > Thanks for giving it a try! > > -steve > > On Mon, Feb 8, 2010 at 12:29 PM, Steve Yegge <st...@go...> wrote: > >> > >> Hi Oti, > >> The HTML output needs some love -- the path to the css file is being > >> hardcoded > >> to your working directory, and the indexer is failing to pick up a lot > of > >> links, likely > >> because I haven't provided the demo with a way to look in Jython for the > >> stdlib. > >> I'll fix these things and send you a replacement for the patch; I can > also > >> add in the > >> "Licenced to PSF under a Contributor Agreement." to the files per Jim's > >> suggestion. > >> Thanks, > >> -steve > >> On Sat, Feb 6, 2010 at 3:58 PM, Oti <oh...@gm...> wrote: > >>> > >>> Hi Steve and Frank, > >>> > >>> I have all the tests passing locally, and I believe the build bots > >>> would pass as well. > >>> > >>> What I'd like to ask Steve to check is if the generated .html files are > >>> correct. > >>> I ran: org.python.indexer.demos.HtmlDemo > >>> ${workspace_loc}/jython-trunk/CPythonLib/email/ > >>> and then zipped the /html directory into > >>> http://bugs.jython.org/file764/html.zip > >>> This is the console output: http://bugs.jython.org/msg5507 > >>> (both links also can be found behind http://bugs.jython.org/issue1541) > >>> > >>> And I'd like to ask Frank to review the grammar changes. > >>> AFAICS they fit very well. > >>> > >>> Finally, is the copyright notice > >>> "Copyright 2009, Google Inc. All rights reserved." > >>> in the added files is OK for Jython? > >>> Please excuse my ignorance, but I keep forgetting where the potential > >>> legal problems are. > >>> > >>> I hope I get your thumbs up, and nobody else objects. > >>> Then this is ready for checkin. > >>> > >>> Thanks a lot! > >>> Oti > >> > > > > > |