pyb-developer — Pyb Developer List

[Pyb-developer] FASD project: Online survey launched

[<ben...@id...>]

Dear Open Source developer

I am doing a research project on "Fun and Software Development" in which I kindly invite you to participate.
You will find the online survey under http://fasd.ethz.ch/qsf/. The questionnaire consists of 53 questions and you will need about 15 minutes to complete it.

With the FASD project (Fun and Software Development) we want to define the motivational significance of fun when software developers decide to engage in Open Source projects. What is special about our research project is that a similar survey is planned with software developers in commercial firms. This procedure allows the immediate comparison between the involved individuals and the conditions of production of these two development models. Thus we hope to obtain substantial new insights to the phenomenon of Open Source Development.


With many thanks for your participation,
Benno Luthiger


PS:
The results of the survey will be published under http://www.isu.unizh.ch/fuehrung/blprojects/FASD/.
We have set up the mailing list fa...@we... for this study. Please see http://fasd.ethz.ch/qsf/mailinglist_en.html for registration to this mailing list.

_______________________________________________________________________

Benno Luthiger
Swiss Federal Institute of Technology Zurich
8092 Zurich

Mail: benno.luthiger(at)id.ethz.ch
_______________________________________________________________________

Re: [Pyb-developer] Top level invoication questions

[Anthony E. <me...@an...>]

Andy Robinson wrote:
>>I was working on implementing parallel task execution and am still 
>>having problems with it.  That thread was an attempt to try to get 
>>things working properly by getting execution of the main thread, however 
>>it did not help.  It may not be necessary and if I (or someone else) can 
>>get parallel tasks working and it turns out the thread can be removed 
>>then it will be.
> 
> 
> I'm not into threads, I decided I wasn't clever enough to handle
> them years ago.  But having a top-level script to invoke parallel
> ones seems like a very good reason for an optional "pyb.bat" or "pyb.sh".
> I just think that with a VERY minor tweak, it would no longer
> be compulsory.

I agree that it is probably unnecessary but I need to get the parallel 
task execution working before I can prove that.  As it stands now there 
is a blocking problem that I just can't get around.

> There's a fundamental issue I have discussed with lots of people
> about whether an "ant for python" should use XML or Python. David
> Ascher felt that a build script should be in a separate language
> and process to what it builds; everyone else said "just use Python".

Well I think we already know where I stand on this. ;-)

>>Part of the reason pyb.bat exists is because I am still not convinced 
>>that the way Python distro works is the best way to do things.  This is 
>>probably because of my Java background - I don't like to have any code 
>>under the VM's home directory.  Maybe I'll just have to get over this 
>>and rework Pyb so it requires the more pythonic install process.  Is 
>>there a way, using the distutils, to set it up so I can just type pyb 
>>and have a the pyb.py script run?
> 
> 
> You mean "can the distutils setup process put pyb on your path"?
> 
> You now come to the reason most people hate distutils.  Yes, distutils
> can do anything, if you (a) write the python code you really want
> to have in your install process then (b) substitute in your own
> distutils command to be run on 'install' instead of its default
> implementation.  Doing anything nonstandard in distutils is always harder 
> than doing it in a Python script; and creating an icon, menu item or
> path addition is "nonstandard" :-(
> 
> On windows, I would recommend making either an "inno setup" installer
> or using Thomas Heller's windows distutils extension.  AFAIR this has a 
> procedural post-intall hook.  So we could add the pyb.bat to the system 
> path on install.  I'll have a look at how this works tonight or tomorrow.

I'd be happy to use Inno Setup on Windows.  I've used it in the past for 
building Java application installers and was very happy with it.  On a 
side note an inno task for Pyb would be a nice task to have around.

>>If at some point Pyb is changed to use a pythonic distutils install then 
>>it wouldn't be a problem supporting direct execution of your build 
>>scripts.  As it stands now that's not possible because it is the pyb.bat 
>>script which is executed and which ensures that the pyb modules are 
>>found by Python.
> 
> 
> So if I can make your package put the pyb executable on the path
> for both platforms, you won't mind me executing my build scripts
> directly?  

Correct, if you can provide a way for me to type pyb on the command line 
and have it execute the default build script (or a script specified via 
-f) then I have no problem switching to distutils thus allowing scripts 
to be executed directly.

>>How about a constructor flag or a global flag called autorun which can 
>>be set to False to disable that feature.  The reason I like having that 
>>there is because it keeps some stuff out of the build script which 
>>really isn't necessary.  I almost think a global flag would be better 
>>because it would allow you to keep the concept of autorun out of the 
>>build script itself and let the loading environment specify it instead.
> 
> 
> unittest offers a good parallel.  We have lots of test modules
> and we want the freedom to either (a) walk through and run them all
> at night or (b) run any one by hand.  See example:  we have a convention
> that every module exports a 'makeSuite' function to allow external
> invocation.  
> 
> More specifically, if it is named test_xxx.py and it has a makeSuite()
> function, I can call that and expect to get a TestSuite.  
> 
> I think this would work well for PYB projects too.  So would
> haveing a visible "project" object which doesn't run until told
> to.

The unit test you attached is really a functional test.  The one thing I 
really like about Python wrt unit tests is that it is easy to put the 
test right at the bottom of the file containing the unit which is being 
tested.  This really helps keep implementation and tests in sync and is 
a much better way of unittesting than separating out tests into their 
own files, IMHO.

Sincerely,
Anthony Eden

RE: [Pyb-developer] Top level invoication questions

[Andy R. <an...@re...>]

> I was working on implementing parallel task execution and am still 
> having problems with it.  That thread was an attempt to try to get 
> things working properly by getting execution of the main thread, however 
> it did not help.  It may not be necessary and if I (or someone else) can 
> get parallel tasks working and it turns out the thread can be removed 
> then it will be.

I'm not into threads, I decided I wasn't clever enough to handle
them years ago.  But having a top-level script to invoke parallel
ones seems like a very good reason for an optional "pyb.bat" or "pyb.sh".
I just think that with a VERY minor tweak, it would no longer
be compulsory.

There's a fundamental issue I have discussed with lots of people
about whether an "ant for python" should use XML or Python. David
Ascher felt that a build script should be in a separate language
and process to what it builds; everyone else said "just use Python".


> 
> Part of the reason pyb.bat exists is because I am still not convinced 
> that the way Python distro works is the best way to do things.  This is 
> probably because of my Java background - I don't like to have any code 
> under the VM's home directory.  Maybe I'll just have to get over this 
> and rework Pyb so it requires the more pythonic install process.  Is 
> there a way, using the distutils, to set it up so I can just type pyb 
> and have a the pyb.py script run?

You mean "can the distutils setup process put pyb on your path"?

You now come to the reason most people hate distutils.  Yes, distutils
can do anything, if you (a) write the python code you really want
to have in your install process then (b) substitute in your own
distutils command to be run on 'install' instead of its default
implementation.  Doing anything nonstandard in distutils is always harder 
than doing it in a Python script; and creating an icon, menu item or
path addition is "nonstandard" :-(

On windows, I would recommend making either an "inno setup" installer
or using Thomas Heller's windows distutils extension.  AFAIR this has a 
procedural post-intall hook.  So we could add the pyb.bat to the system 
path on install.  I'll have a look at how this works tonight or tomorrow.


> 
> If at some point Pyb is changed to use a pythonic distutils install then 
> it wouldn't be a problem supporting direct execution of your build 
> scripts.  As it stands now that's not possible because it is the pyb.bat 
> script which is executed and which ensures that the pyb modules are 
> found by Python.

So if I can make your package put the pyb executable on the path
for both platforms, you won't mind me executing my build scripts
directly?  

> How about a constructor flag or a global flag called autorun which can 
> be set to False to disable that feature.  The reason I like having that 
> there is because it keeps some stuff out of the build script which 
> really isn't necessary.  I almost think a global flag would be better 
> because it would allow you to keep the concept of autorun out of the 
> build script itself and let the loading environment specify it instead.

unittest offers a good parallel.  We have lots of test modules
and we want the freedom to either (a) walk through and run them all
at night or (b) run any one by hand.  See example:  we have a convention
that every module exports a 'makeSuite' function to allow external
invocation.  

More specifically, if it is named test_xxx.py and it has a makeSuite()
function, I can call that and expect to get a TestSuite.  

I think this would work well for PYB projects too.  So would
haveing a visible "project" object which doesn't run until told
to.

Thanks,

Andy

p.s. offline for 3 hours putting kids to bed, will be hacking
on pyb later...

Re: [Pyb-developer] Top level invoication questions

[Anthony E. <me...@an...>]

an...@re... wrote:
> Hi Anthony and friends,
> 
> I got distracted for a couple of weeks but am now trying hard to
> make Pyb work for packaging my code. Looks more and more promising.
> However I keep getting this "unpythonic" feeling at the very top
> level.

I'm not suprised.  That would be my reluctance to accept the way Python 
apps are distributed. :-)

> 
> (1) Thread questions:
> You run pyb.bat over a script, which has an
> execute-on-init object at the bottom. The only work pyb.bat
> does AFAICT is to start a thread and run the target module in it.
> Why is the thread important?

I was working on implementing parallel task execution and am still 
having problems with it.  That thread was an attempt to try to get 
things working properly by getting execution of the main thread, however 
it did not help.  It may not be necessary and if I (or someone else) can 
get parallel tasks working and it turns out the thread can be removed 
then it will be.

Part of the reason pyb.bat exists is because I am still not convinced 
that the way Python distro works is the best way to do things.  This is 
probably because of my Java background - I don't like to have any code 
under the VM's home directory.  Maybe I'll just have to get over this 
and rework Pyb so it requires the more pythonic install process.  Is 
there a way, using the distutils, to set it up so I can just type pyb 
and have a the pyb.py script run?

> (2) Executable scripts
> Assuming I can live without a thread, why not ditch
> it and just make an "executable  script" the norm?
> Python scripts are always executable on windows.
> If I saw a directory containing things like this,
> I'd naturally think to execute them (especially
> if they start with a verb)
>   download_dependents.py
>   build_minimal_distro.py
>   build_full_distro.py
> 
> 
> This feels like an Ant legacy.  XML files cannot be
> executed so the command line tool is needed there.
> 
> If I change the bottom of the script of my scripts to
> 
> |if __name__=='__main__':
> |   Project()
> then it all works.

In a way it is an ant legacy.  I want to be able to type pyb in any 
directory and have it locate the default build script and execute that 
script.

If at some point Pyb is changed to use a pythonic distutils install then 
it wouldn't be a problem supporting direct execution of your build 
scripts.  As it stands now that's not possible because it is the pyb.bat 
script which is executed and which ensures that the pyb modules are 
found by Python.

> (3) avoid auto-running on import
> The above change also stops scripts from executing when I
> import into Pythonwin, or run one of a number of documentation
> tools over my source directory.

OK, I can see how this is a problem.  I'm open to suggestions although I 
would prefer to avoid requiring if __name__ == '__main__' if possible.

Would it be possible to use exec(), execute the build script and then 
get ahold of the Project object which was created and execute that from 
within the code which called the original exec()?

> (3) explicit run method
> I would prefer to have an explicit run e.g.
> 
> |#project definition omitted
> |Project(name='Demo Project', default='build', targets=targets)
> |if __name__=='__main__':
> |    Project.run()
> 
> This opens the possibility to load up and examine PYB
> projects in analysis tools, make nice easy GUIs etc,
> simply by getting hold of the Project object and inspecting
> it.  Otherwise parsers are needed.

How about a constructor flag or a global flag called autorun which can 
be set to False to disable that feature.  The reason I like having that 
there is because it keeps some stuff out of the build script which 
really isn't necessary.  I almost think a global flag would be better 
because it would allow you to keep the concept of autorun out of the 
build script itself and let the loading environment specify it instead.

Sincerely,
Anthony Eden

Re: [Pyb-developer] Top level invoication questions

[<an...@re...>]

> |#project definition omitted
> |Project(name='Demo Project', default='build', targets=targets)
> |if __name__=='__main__':
> |    Project.run()
Doh.  I mean to say

proj = Project(...)
if __name__=='__main__':
    proj.run()

or some such..

[Pyb-developer] Top level invoication questions

[<an...@re...>]

Hi Anthony and friends,

I got distracted for a couple of weeks but am now trying hard to
make Pyb work for packaging my code. Looks more and more promising.
However I keep getting this "unpythonic" feeling at the very top
level.

(1) Thread questions:
You run pyb.bat over a script, which has an
execute-on-init object at the bottom. The only work pyb.bat
does AFAICT is to start a thread and run the target module in it.
Why is the thread important?

(2) Executable scripts
Assuming I can live without a thread, why not ditch
it and just make an "executable  script" the norm?
Python scripts are always executable on windows.
If I saw a directory containing things like this,
I'd naturally think to execute them (especially
if they start with a verb)
  download_dependents.py
  build_minimal_distro.py
  build_full_distro.py


This feels like an Ant legacy.  XML files cannot be
executed so the command line tool is needed there.

If I change the bottom of the script of my scripts to

|if __name__=='__main__':
|   Project()
then it all works.

(3) avoid auto-running on import
The above change also stops scripts from executing when I
import into Pythonwin, or run one of a number of documentation
tools over my source directory.

(3) explicit run method
I would prefer to have an explicit run e.g.

|#project definition omitted
|Project(name='Demo Project', default='build', targets=targets)
|if __name__=='__main__':
|    Project.run()

This opens the possibility to load up and examine PYB
projects in analysis tools, make nice easy GUIs etc,
simply by getting hold of the Project object and inspecting
it.  Otherwise parsers are needed.

Any thoughts?

Andy Robinson

RE: [Pyb-developer] RE: pyb collaboration

[Andy R. <an...@re...>]

Flipping the order for a moment....
> But for the moment my goals are:
>
>    Simplicity (in build scripts and implementation) and clarity in the
> implementation (minimize the magic and limit class explosion).

I strongly suggest to put these goals at the top of the web page and manual.
It will score a lot of points :-)


> Let's cross that bridge when we get to it. ;-)  Up until recently all of
> my open source projects have been released under an Apache-style license
> with basically no restrictions.  However recent reflection and the
> growing number of open source projects which are dual licensed and which
> are effectively making a profit, has be questioning whether such
> flexible licenses are a good thing.

I just chaired a session "Open Source Business Models That Work"
at Python-UK and it's clear GPL+commercial works for a lot of
people selling embedded software.

> Having said that if it was between
> keeping GPL and being able to incorporate something into the Python
> library I think the choice is fairly obvious. :-)

Good.  I see massive utility in having this or an equivalent in the
library one day. I agree on all the other points too.

I'm trying to build reportlab with this, will provide
feedback later...

- Andy

Re: [Pyb-developer] RE: pyb collaboration

[Anthony E. <me...@an...>]

Andy Robinson wrote:

>Hi folks.  I just joined.  I am starting a campaign for
>'best practice standards in enterprise deployment' in Python,
>something my company needs badly, and one big item on my wish
>list is something like Ant.  There are several contenders
>but this looks like the simplest and cleanest so far...
>
>I'd like to ask a few questions about where pyb might go,
>subject to sufficient resources.  
>
>0. GPL: fine for now, but if Guido agreed to get it into
>the standard Python library one day (which is BSD/LGPL-like),
>would you accept?  (we can dream....)
>
>  
>
Let's cross that bridge when we get to it. ;-)  Up until recently all of
my open source projects have been released under an Apache-style license
with basically no restrictions.  However recent reflection and the
growing number of open source projects which are dual licensed and which
are effectively making a profit, has be questioning whether such
flexible licenses are a good thing.  Having said that if it was between
keeping GPL and being able to incorporate something into the Python
library I think the choice is fairly obvious. :-)

>1. Ant has a standard way to keep config info external to the
>build script (ftp servers, hosts, passwords); you can thus safely 
>check in build scripts and have a config file in each environment.
>Is this desirable?
>  
>
Yes.   However I would like to be able to have external properties
appear in the build script as variables, since that is the way
properties are currently implemented.  The benefit as I see it is the
ability to define data structures such as lists and maps in a much more
concise fashion than can be done with properties (as implemented in Ant).

>2. filesets as objects: 
>The copy task and friends seem to be just walking the tree under where 
>you start with include and exclude filters.  Is this just a
>starting point or do you feel it is the optimum solution?
>
>  
>
It's just a starting point.  It was the simplest solution to the problem
at hand and seems to work well.  It would be nice to be able to reuse
filesets though in certain cases.

>I have seen half a dozen classes like filesets in Ant, including one 
>buried in distutils, and everyone doing deployment stuff implements one.
>I'd like to see one as a standard Python library module one day.  I 
>also  implemented the 'superglob' (e.g. include "spam/**/*.py") which
>I find very nice expressive syntax, and can contribute this if
>desired.  
>  
>
I'd be happy to consider other options, as long as they stay true to the
original design goals of pyb which is: do the simplest thing to get the
job done.

>3. Python versus XML: should build scripts be Python forever, or 
>would a mapping from XML be useful later?  (I have a 50/50 split 
>with people I have talked to, I also tend to feel GUIs to build/edit 
>build scripts are good, if further off..)
>
>  
>
OK, here's my thought on this: if you want XML build scripts then you
can use pyant.  I personally don't want to go that direction.  If I can
convert my XML build scripts into Python scripts which are half the size
and more concise then I have accomplished one of my goals with pyb.  If
pyb were to use XML syntax then that would effectively negate the need
for the library at all.

>4. tasks as objects vs. tasks as functions.
>The latter is the way you have gone and Java has no functions :-)
>I wonder if the former would be more useful if you ever want to 
>(a) instantiate from xml (i.e. have enough metadata to know
>the type of arguments)
>(b) give them all a channel for logging stuff as they do
>their work
>(c) create generic GUI dialogs for building scripts
>(d) have some global context for tasks (e.g. to externally
>store and make available
>
>  
>
a.) Sure, but like I said that has already been done with pyant and
doing the same thing in pyb would defeat the purpose.
b.) That already exists, they just call a function in util.
c.) This could be done without XML - the structure of the Python scripts
is remarkably similar to XML just without the verbosity and with
flexibility to take advantage of the Python language.  A GUI for
building scripts could simply be a code generator.
d.) This can be done today, and in fact is through global variables.

I'm not ready to create a class explosion if it isn't absolutely
necessary.  The way I see it a task is just a function and thus can be
most cleanly implemented as such since Python supports that type of
programming.

>5. Scons (scons.sourceforge.net) is really
>moving ahead in the C/C++ domain.  It deals with different issues.  
>But key people in the python world who don't have experience of 
>enterprise server stuff have said and will say again "why are 
>you duplicating this"?  Any ideas how to define the difference?  
>Has anyone here looked at scons to see if the models could be 
>merged or share tasks?
>
>  
>
Scons seems to be approaching the problem from a different direction
than Ant (and pyb).  Now, I've not used Scons, I've only read the
documentation, so I may be way off base here and I hope someone will
correct me if I am.  So far my impression of Scons is that it tries to
hide a lot from the build script.  If you want to compile something you
essentially do Program('file') and it will magically do what it needs to
do, detecting tools, determing what language the file is, so on and so
forth.  Personally I don't like this type of magic.  I want to be able
to control what I am doing.  Additionally a lot of developers are
familiar with Ant, especially Java developers, and thus it seems like a
good model to work with to provide a crossover into Python.  I count
myself as one of those more familiar with Ant than Scons or Cons, and
thus the tool was designed as such.

>I have no strong feelings on any of the above, just want to
>get a sense of where you are heading..
>
>  
>
It's good to get this out early.  Additionally my feelings for this
could change over time.  But for the moment my goals are:

   Simplicity (in build scripts and implementation) and clarity in the
implementation (minimize the magic and limit class explosion).

Sincerely,
Anthony Eden

[Pyb-developer] RE: pyb collaboration

[Andy R. <an...@re...>]

Hi folks.  I just joined.  I am starting a campaign for
'best practice standards in enterprise deployment' in Python,
something my company needs badly, and one big item on my wish
list is something like Ant.  There are several contenders
but this looks like the simplest and cleanest so far...

I'd like to ask a few questions about where pyb might go,
subject to sufficient resources.  

0. GPL: fine for now, but if Guido agreed to get it into
the standard Python library one day (which is BSD/LGPL-like),
would you accept?  (we can dream....)

1. Ant has a standard way to keep config info external to the
build script (ftp servers, hosts, passwords); you can thus safely 
check in build scripts and have a config file in each environment.
Is this desirable?

2. filesets as objects: 
The copy task and friends seem to be just walking the tree under where 
you start with include and exclude filters.  Is this just a
starting point or do you feel it is the optimum solution?

I have seen half a dozen classes like filesets in Ant, including one 
buried in distutils, and everyone doing deployment stuff implements one.
I'd like to see one as a standard Python library module one day.  I 
also  implemented the 'superglob' (e.g. include "spam/**/*.py") which
I find very nice expressive syntax, and can contribute this if
desired.  

3. Python versus XML: should build scripts be Python forever, or 
would a mapping from XML be useful later?  (I have a 50/50 split 
with people I have talked to, I also tend to feel GUIs to build/edit 
build scripts are good, if further off..)

4. tasks as objects vs. tasks as functions.
The latter is the way you have gone and Java has no functions :-)
I wonder if the former would be more useful if you ever want to 
(a) instantiate from xml (i.e. have enough metadata to know
the type of arguments)
(b) give them all a channel for logging stuff as they do
their work
(c) create generic GUI dialogs for building scripts
(d) have some global context for tasks (e.g. to externally
store and make available


5. Scons (scons.sourceforge.net) is really
moving ahead in the C/C++ domain.  It deals with different issues.  
But key people in the python world who don't have experience of 
enterprise server stuff have said and will say again "why are 
you duplicating this"?  Any ideas how to define the difference?  
Has anyone here looked at scons to see if the models could be 
merged or share tasks?


I have no strong feelings on any of the above, just want to
get a sense of where you are heading..

Thanks,

Andy Robinson
CEO/Chief Architect, 
ReportLab Europe Ltd.

[Pyb-developer] ANN: Pyb 0.4 Released

[Anthony E. <me...@an...>]

Pyb 0.4 is now available.

Recent changes include:

Started adding unit tests. Dependencies are now specified as a list of 
target names passed to the Target constructor. Added -d command line 
argument to enable debugging. Added jythonc task. Added 
platform-specific path building path to epydocs. Additional small 
presentational changes.

Sincerely,
Anthony Eden

Re: [Pyb-developer] Re: problems with cvs version

[Lawrence O. <l.o...@vi...>]

Anthony Eden ha scritto:
> Lawrence,
> 
> If you'd like to submit your changes I'll apply them to the CVS.
> 

I've made a temporary fix for the epydoc task to make it work on some 
Linux distro (epydoc is often in /usr/bin), and fixed a thing in core.py 
(a new line). In attachment you'll find the diffs of the files...

1 - pyb/core.py
2 - pyb/tasks/py.py

ps. may you keep the Changelog in sync? so the users and devs could know 
what has been changed and what are the news in deep.

bye

-- 
Lawrence

[Pyb-developer] Pyb 0.3 Released

[Anthony E. <me...@an...>]

Version 0.3 is now available.

This release includes the addition of command-line arguments for the
python task as well as fixes for the copy task and the javadocs task.

Sincerely,
Anthony Eden

[Pyb-developer] Re: problems with cvs version

[Anthony E. <ae...@si...>]

Lawrence,

If you'd like to submit your changes I'll apply them to the CVS.

-Anthony

Re: [Pyb-developer] problems with cvs version

[Lawrence O. <l.o...@vi...>]

problems solved, after reading the whole src i've gained some knowledge 
and I've modified the epydoc() function in py.py. So what next ;-) ?

-- 
Lawrence

[Pyb-developer] problems with cvs version

[<l.o...@vi...>]

I'm trying the cvs version (not tried the release to be honest) but it se=
ems
that it has a problem with finding epydoc. After building and installing
it (with setup.py) i added the dir to the PATH but when I execute pyb.py
(after setting on the exec bit) it searches epydoc.py in /usr/Scripts but=

my epydoc executable is in /usr/bin/epydoc (with no extension). I use Gen=
too
Linux, python 2.3 and epydoc 2.0. Anyway... I'm going to try to hack pyb/=
tasks/py.py
and fix the problem for my own... bye!

ps. the project seems a good idea... if I can I'll give you an hand.