From: Li S. <li...@gm...> - 2011-11-03 05:38:31
|
Hi, I'm pretty new to jython and I want to use jython to execute many existing python scripts. For performance reason, I'm wondering if I can pre-compile the python scripts, like below: private PyCode code = new PythonInterpreter().compile((new InputStreamReader(Thread.currentThread().getContextClassLoader().getResourceAsStream(SCRIPT_NAME)))); And then reuse the compiled code in a multi-thread environment, like below: public void someMethod() { PythonInterpreter interpreter = new PythonInterpreter(); interpreter.exec(code); // ... } Looks like the doc doesn't mention if PyCode is thead-safe. Could anyone please give out some clarifications, or suggestions if this approach is incorrect. Thanks! Li |
From: Jeff E. <jem...@fr...> - 2011-11-03 18:22:44
|
The interpreters ought to be constructed with independent namespaces and sys modules, otherwise they are shared among interpreters. See the 2-arg constuctor. http://www.jython.org/javadoc/org/python/util/PythonInterpreter.html#PythonInterpreter(org.python.core.PyObject, org.python.core.PySystemState) Pierre Thibault wrote: > 2011/11/3 Li Shen <li...@gm... <mailto:li...@gm...>> > > Hi, > > I'm pretty new to jython and I want to use jython to execute many > existing python scripts. > > For performance reason, I'm wondering if I can pre-compile the > python scripts, like below: > > private PyCode code = new PythonInterpreter().compile((new > InputStreamReader(Thread.currentThread().getContextClassLoader().getResourceAsStream(SCRIPT_NAME)))); > > And then reuse the compiled code in a multi-thread environment, > like below: > > public void someMethod() { > PythonInterpreter interpreter = new PythonInterpreter(); > interpreter.exec(code); > // ... > } > > Looks like the doc doesn't mention if PyCode is thead-safe. Could > anyone please give out some clarifications, or suggestions if this > approach is incorrect. > > Thanks! > > > > I think so. Each PythonInterpreter is an independent state machine. > > Pierre Thibault > > Python Developer/Développeur Python > Montréal, QC > /pie...@gm... <mailto:pie...@gm...>/ > > > > > > ------------------------------------------------------------------------ > > ------------------------------------------------------------------------------ > RSA(R) Conference 2012 > Save $700 by Nov 18 > Register now > http://p.sf.net/sfu/rsa-sfdev2dev1 > ------------------------------------------------------------------------ > > _______________________________________________ > Jython-users mailing list > Jyt...@li... > https://lists.sourceforge.net/lists/listinfo/jython-users > |
From: Li S. <li...@gm...> - 2011-11-04 07:58:10
|
Thanks so much Pierre & Jeff! Jeff, did you mean that I should do something like below to create a PythonInterpreter? PythonInterpreter interpreter = new PythonInterpreter(new PyStringMap(), new PySystemState()); Thanks! Li 2011/11/4 Jeff Emanuel <jem...@fr...> > > The interpreters ought to be constructed with independent > namespaces and sys modules, otherwise they are shared among > interpreters. See the 2-arg constuctor. > > > http://www.jython.org/javadoc/org/python/util/PythonInterpreter.html#PythonInterpreter(org.python.core.PyObject > , > org.python.core.PySystemState) > > Pierre Thibault wrote: > > 2011/11/3 Li Shen <li...@gm... <mailto:li...@gm...>> > > > > Hi, > > > > I'm pretty new to jython and I want to use jython to execute many > > existing python scripts. > > > > For performance reason, I'm wondering if I can pre-compile the > > python scripts, like below: > > > > private PyCode code = new PythonInterpreter().compile((new > > > InputStreamReader(Thread.currentThread().getContextClassLoader().getResourceAsStream(SCRIPT_NAME)))); > > > > And then reuse the compiled code in a multi-thread environment, > > like below: > > > > public void someMethod() { > > PythonInterpreter interpreter = new PythonInterpreter(); > > interpreter.exec(code); > > // ... > > } > > > > Looks like the doc doesn't mention if PyCode is thead-safe. Could > > anyone please give out some clarifications, or suggestions if this > > approach is incorrect. > > > > Thanks! > > > > > > > > I think so. Each PythonInterpreter is an independent state machine. > > > > Pierre Thibault > > > > Python Developer/Développeur Python > > Montréal, QC > > /pie...@gm... <mailto:pie...@gm...>/ > > > > > > > > > > > > ------------------------------------------------------------------------ > > > > > ------------------------------------------------------------------------------ > > RSA(R) Conference 2012 > > Save $700 by Nov 18 > > Register now > > http://p.sf.net/sfu/rsa-sfdev2dev1 > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > Jython-users mailing list > > Jyt...@li... > > https://lists.sourceforge.net/lists/listinfo/jython-users > > > > > ------------------------------------------------------------------------------ > RSA(R) Conference 2012 > Save $700 by Nov 18 > Register now > http://p.sf.net/sfu/rsa-sfdev2dev1 > _______________________________________________ > Jython-users mailing list > Jyt...@li... > https://lists.sourceforge.net/lists/listinfo/jython-users > |
From: Pierre T. <pie...@gm...> - 2011-11-04 11:24:49
|
2011/11/4 Li Shen <li...@gm...> > Thanks so much Pierre & Jeff! > > Jeff, did you mean that I should do something like below to create a > PythonInterpreter? > > PythonInterpreter interpreter = new PythonInterpreter(new PyStringMap(), > new PySystemState()); > > Thanks! > > Li > Hello Li, I don't think so. Your threads are probably sharing the same modules. The modules are shared across all the threads of the process by default in Python. Having separate modules for each thread will take a lot more memory. So use it only if needed. Pierre Thibault Python Developer/Développeur Python Montréal, QC *pie...@gm...* |
From: Li S. <li...@gm...> - 2011-11-05 02:05:41
|
Hi Pierre & Jeff, In my case, I need to execute many different python scripts with jython. I may assume those scripts could do anything they want and that's out of my control. So to ensure thread safety, the easiest approach is to create totally independent interpreter as Jeff mentioned? Memory consumption would be another concern to us, so we may have to use the non-arg and 2-args constructors of PythonInterpreter respectively depending on the behaviors of a python script to be interpreted? This approach would require python expertise to analyze each script. Not sure if my understandings are correct. Thanks so much for you help! Li 2011/11/4 Jeff Emanuel <jem...@fr...> > Hello Li, > > That's what I would do, but as Pierre points out, it depends on > how independent you want your threads to be. My preference would > be that when working with separate interpreter instances, they > ought to work completely independently. If you want your threads > to share an environment, then use the no-arg constructor. > > Jeff > > Li Shen wrote: > >> Thanks so much Pierre & Jeff! >> >> Jeff, did you mean that I should do something like below to create a >> PythonInterpreter? >> >> PythonInterpreter interpreter = new PythonInterpreter(new PyStringMap(), >> new PySystemState()); >> >> Thanks! >> >> Li >> >> 2011/11/4 Jeff Emanuel <jem...@fr... <mailto:jem...@fr...>> >> >> >> >> The interpreters ought to be constructed with independent >> namespaces and sys modules, otherwise they are shared among >> interpreters. See the 2-arg constuctor. >> >> http://www.jython.org/javadoc/**org/python/util/** >> PythonInterpreter.html#**PythonInterpreter(org.python.**core.PyObject<http://www.jython.org/javadoc/org/python/util/PythonInterpreter.html#PythonInterpreter(org.python.core.PyObject> >> <http://www.jython.org/**javadoc/org/python/util/** >> PythonInterpreter.html#**PythonInterpreter%28org.**python.core.PyObject<http://www.jython.org/javadoc/org/python/util/PythonInterpreter.html#PythonInterpreter%28org.python.core.PyObject> >> >, >> >> org.python.core.PySystemState) >> >> Pierre Thibault wrote: >> > 2011/11/3 Li Shen <li...@gm... <mailto:li...@gm...> >> <mailto:li...@gm... <mailto:li...@gm...>>> >> >> > >> > Hi, >> > >> > I'm pretty new to jython and I want to use jython to execute >> many >> > existing python scripts. >> > >> > For performance reason, I'm wondering if I can pre-compile the >> > python scripts, like below: >> > >> > private PyCode code = new PythonInterpreter().compile((**new >> > InputStreamReader(Thread.**currentThread().** >> getContextClassLoader().**getResourceAsStream(SCRIPT_**NAME)))); >> > >> > And then reuse the compiled code in a multi-thread environment, >> > like below: >> > >> > public void someMethod() { >> > PythonInterpreter interpreter = new PythonInterpreter(); >> > interpreter.exec(code); >> > // ... >> > } >> > >> > Looks like the doc doesn't mention if PyCode is thead-safe. >> Could >> > anyone please give out some clarifications, or suggestions >> if this >> > approach is incorrect. >> > >> > Thanks! >> > >> > >> > >> > I think so. Each PythonInterpreter is an independent state machine. >> > >> > Pierre Thibault >> > >> > Python Developer/Développeur Python >> > Montréal, QC >> > /pie...@gm... <mailto:pierre.thibault1@**gmail.com<pie...@gm...> >> > >> <mailto:pierre.thibault1@**gmail.com <pie...@gm...> >> >> <mailto:pierre.thibault1@**gmail.com <pie...@gm...>>>/ >> > >> > >> > >> > >> > >> > >> ------------------------------**------------------------------** >> ------------ >> > >> > >> ------------------------------**------------------------------** >> ------------------ >> > RSA(R) Conference 2012 >> > Save $700 by Nov 18 >> > Register now >> > http://p.sf.net/sfu/rsa-**sfdev2dev1<http://p.sf.net/sfu/rsa-sfdev2dev1> >> > >> ------------------------------**------------------------------** >> ------------ >> > >> > ______________________________**_________________ >> > Jython-users mailing list >> > Jython-users@lists.**sourceforge.net<Jyt...@li...> >> <mailto:Jython-users@lists.**sourceforge.net<Jyt...@li...> >> > >> >> > https://lists.sourceforge.net/**lists/listinfo/jython-users<https://lists.sourceforge.net/lists/listinfo/jython-users> >> > >> >> ------------------------------**------------------------------** >> ------------------ >> RSA(R) Conference 2012 >> Save $700 by Nov 18 >> Register now >> http://p.sf.net/sfu/rsa-**sfdev2dev1<http://p.sf.net/sfu/rsa-sfdev2dev1> >> ______________________________**_________________ >> Jython-users mailing list >> Jython-users@lists.**sourceforge.net<Jyt...@li...> >> <mailto:Jython-users@lists.**sourceforge.net<Jyt...@li...> >> > >> https://lists.sourceforge.net/**lists/listinfo/jython-users<https://lists.sourceforge.net/lists/listinfo/jython-users> >> >> >> >> ------------------------------**------------------------------** >> ------------ >> >> ------------------------------**------------------------------** >> ------------------ >> RSA(R) Conference 2012 >> Save $700 by Nov 18 >> Register now >> http://p.sf.net/sfu/rsa-**sfdev2dev1 <http://p.sf.net/sfu/rsa-sfdev2dev1> >> ------------------------------**------------------------------** >> ------------ >> >> ______________________________**_________________ >> Jython-users mailing list >> Jython-users@lists.**sourceforge.net <Jyt...@li...> >> https://lists.sourceforge.net/**lists/listinfo/jython-users<https://lists.sourceforge.net/lists/listinfo/jython-users> >> >> > |
From: Pierre T. <pie...@gm...> - 2011-11-05 14:48:59
|
2011/11/4 Li Shen <li...@gm...> > Hi Pierre & Jeff, > > In my case, I need to execute many different python scripts with jython. I > may assume those scripts could do anything they want and that's out of my > control. So to ensure thread safety, the easiest approach is to create > totally independent interpreter as Jeff mentioned? > > Memory consumption would be another concern to us, so we may have to use > the non-arg and 2-args constructors of PythonInterpreter respectively > depending on the behaviors of a python script to be interpreted? This > approach would require python expertise to analyze each script. > > Not sure if my understandings are correct. > > Thanks so much for you help! > > Li > > > Hello Li, I guess it would be great to experiment then. You may want to look at the Java console to know how much you are consuming resources. Maybe you can use external processes instead of threads. I believe you may reuse the interpreter, maybe having a pool, if you want to save resources. There is a risk of conflict but not that much and if you add a little cleanup after running a script it could be solution: (clear the global name space). But I don't know what it would be in practice. Pierre Thibault Python Developer/Développeur Python Montréal, QC *pie...@gm...* |
From: Li S. <li...@gm...> - 2011-11-07 03:43:54
|
Hi Pierre, Your suggestions sounds great and we can try them in practice. Thanks! Li 2011/11/5 Pierre Thibault <pie...@gm...> > 2011/11/4 Li Shen <li...@gm...> > >> Hi Pierre & Jeff, >> >> In my case, I need to execute many different python scripts with jython. >> I may assume those scripts could do anything they want and that's out of my >> control. So to ensure thread safety, the easiest approach is to create >> totally independent interpreter as Jeff mentioned? >> >> Memory consumption would be another concern to us, so we may have to use >> the non-arg and 2-args constructors of PythonInterpreter respectively >> depending on the behaviors of a python script to be interpreted? This >> approach would require python expertise to analyze each script. >> >> Not sure if my understandings are correct. >> >> Thanks so much for you help! >> >> Li >> >> >> > Hello Li, > > I guess it would be great to experiment then. You may want to look at the > Java console to know how much you are consuming resources. Maybe you can > use external processes instead of threads. I believe you may reuse the > interpreter, maybe having a pool, if you want to save resources. There is a > risk of conflict but not that much and if you add a little cleanup after > running a script it could be solution: (clear the global name space). But I > don't know what it would be in practice. > > > Pierre Thibault > > Python Developer/Développeur Python > Montréal, QC > *pie...@gm...* > > > > > > > > ------------------------------------------------------------------------------ > RSA(R) Conference 2012 > Save $700 by Nov 18 > Register now > http://p.sf.net/sfu/rsa-sfdev2dev1 > _______________________________________________ > Jython-users mailing list > Jyt...@li... > https://lists.sourceforge.net/lists/listinfo/jython-users > > |
From: Pierre T. <pie...@gm...> - 2011-11-03 18:15:50
|
2011/11/3 Li Shen <li...@gm...> > Hi, > > I'm pretty new to jython and I want to use jython to execute many existing > python scripts. > > For performance reason, I'm wondering if I can pre-compile the python > scripts, like below: > > private PyCode code = new PythonInterpreter().compile((new > InputStreamReader(Thread.currentThread().getContextClassLoader().getResourceAsStream(SCRIPT_NAME)))); > > And then reuse the compiled code in a multi-thread environment, like below: > > public void someMethod() { > PythonInterpreter interpreter = new PythonInterpreter(); > interpreter.exec(code); > // ... > } > > Looks like the doc doesn't mention if PyCode is thead-safe. Could anyone > please give out some clarifications, or suggestions if this approach is > incorrect. > > Thanks! > I think so. Each PythonInterpreter is an independent state machine. Pierre Thibault Python Developer/Développeur Python Montréal, QC *pie...@gm...* |