From: Jeff A. <ja...@fa...> - 2013-05-18 08:25:59
|
I have prototyped a solution to some difficulties with JLine (Issue 2046), but to make it work in the general case will require changes to how we go about initialisation that may impact users who have embedded the interpreter in applications. I'm not sure which aspects of the present classes constitute API we ought to keep stable. The root of these problems with JLine is the direct use by sys.stdin of a System.in, after its behaviour has been modified by JLine to suit itself. Use of jline.ConsoleReader.readLine through JLineConsole.raw_input works fairly well. Application writers probably also expect to use System.in without nasty surprises. By replacing System.in with a stream that wraps JLine (using jline.ConsoleReaderInputStream.setIn), I am able to get interactive behaviour very much like CPython's and a coherent System.in that benefits from JLine history and editing. dev>java -cp build\exposed;build\classes;dist\javalib\* org.python.util.jython Jython 2.7b1+ (default:bb71f88cbe80+, May 17 2013, 18:20:38) [Java HotSpot(TM) 64-Bit Server VM (Sun Microsystems Inc.)] on java1.6.0_35 >>> import sys >>> sys.stdin.readline() abc 'abc\n' >>> raw_input() Hello 'Hello' The purpose of JLine is to provide a more convenient way for a user to get (encoded) characters into a program than typing everything. I'll call this its "console" function. However, in our current design, JLineConsole is also a PythonInterpreter, working with lines of characters treated as commands. It is only by separating the two concerns, console and interpreter, that I've been able to make it work. Although one bug fix shouldn't drive architecture, I think this provides an architectural hint they should be separate. The second hint this provides is about initialisation. I've hard-wired use of JLine in my demonstration, but for the console/interpreter to be selected through configuration, I believe I have to break initialisation into phases that give the invoking code a chance to act between them. These phases are probably fill registry, init static state, create PySystemState instance. I get the feeling this is not the only case where it would be useful to respond to configuration, or to respond to static initialisation, before all the plates start spinning. What do others think? And where are the pitfalls, particularly API change breaking others' work? I guess we'd like the text-book examples still to work as written, but what else? Jeff (Please note change to IETF-compliant e-mail address) |
From: Jim B. <jb...@zy...> - 2013-05-20 16:44:03
|
Jeff, In Jython 2.5 development, up through 2.5.0 RCs/final, we gave ourselves the freedom to break existing Java APIs such as PythonInterpreter. For subsequent 2.5.x releases, we grew the Java API, but kept it backwards compatible. This required a fair amount of contortions. The general principle should continue to stand in 2.7. So you should feel free to do this refactoring. More below. On Sat, May 18, 2013 at 2:12 AM, Jeff Allen <ja...@fa...> wrote: > I have prototyped a solution to some difficulties with JLine (Issue > 2046), but to make it work in the general case will require changes to > how we go about initialisation that may impact users who have embedded > the interpreter in applications. I'm not sure which aspects of the > present classes constitute API we ought to keep stable. > > The root of these problems with JLine is the direct use by sys.stdin of > a System.in, after its behaviour has been modified by JLine to suit > itself. Use of jline.ConsoleReader.readLine through > JLineConsole.raw_input works fairly well. Application writers probably > also expect to use System.in without nasty surprises. By replacing > System.in with a stream that wraps JLine (using > jline.ConsoleReaderInputStream.setIn), I am able to get interactive > behaviour very much like CPython's and a coherent System.in that > benefits from JLine history and editing. > Sounds like a great wrapping. We probably should also look at the interaction with the readline module, which currently uses a workaround to get at JLine directly (through sys._jy_interpreter.reader). > > dev>java -cp build\exposed;build\classes;dist\javalib\* > org.python.util.jython > Jython 2.7b1+ (default:bb71f88cbe80+, May 17 2013, 18:20:38) > [Java HotSpot(TM) 64-Bit Server VM (Sun Microsystems Inc.)] on java1.6.0_35 > >>> import sys > >>> sys.stdin.readline() > abc > 'abc\n' > >>> raw_input() > Hello > 'Hello' > > The purpose of JLine is to provide a more convenient way for a user to > get (encoded) characters into a program than typing everything. I'll > call this its "console" function. However, in our current design, > JLineConsole is also a PythonInterpreter, working with lines of > characters treated as commands. It is only by separating the two > concerns, console and interpreter, that I've been able to make it work. > Although one bug fix shouldn't drive architecture, I think this provides > an architectural hint they should be separate. > Makes sense. > > The second hint this provides is about initialisation. I've hard-wired > use of JLine in my demonstration, but for the console/interpreter to be > selected through configuration, I believe I have to break initialisation > into phases that give the invoking code a chance to act between them. > These phases are probably fill registry, init static state, create > PySystemState instance. I get the feeling this is not the only case > where it would be useful to respond to configuration, or to respond to > static initialisation, before all the plates start spinning. > readline also provides functionality to get at this; I do *not* remember details, but set_startup_hook/set_pre_input_hook look like the entry points. > What do others think? And where are the pitfalls, particularly API > change breaking others' work? I guess we'd like the text-book examples > still to work as written, but what else? > This might also be the right time to consider revisiting other PythonIntepreter/PySystemState related changes. In particular, I would like to get rid of shadowed statics - that is statics that are not really statics but were left that way for backwards compatibility. There's a few that were made that way, but if they are not final, we really should revisit statics given their opportunity to create resource leaks, at the very least. Thanks for digging into this issue - it's awesome work! - Jim |
From: Jeff A. <ja...@fa...> - 2013-05-23 07:12:18
|
Thanks for those pointers Jim. It looks like readline.py only accesses what I called the "console" function of JLineConsole. By "shadowed statics" do you mean the use of ThreadLocal variables? It's not something I'd come across before. Naturally I'm going for minimal other change while reworking the JLine integration as cleanly as I can. But having opened and understood something (if I do), there may be an opportunity to improve it in other ways. I'm conscious however that PySystemState is something like the fragile heart of the whole animal. Jeff Allen On 20/05/2013 17:43, Jim Baker wrote: > Jeff, > > In Jython 2.5 development, up through 2.5.0 RCs/final, we gave > ourselves the freedom to break existing Java APIs such as > PythonInterpreter. For subsequent 2.5.x releases, we grew the Java > API, but kept it backwards compatible. This required a fair amount of > contortions. > > The general principle should continue to stand in 2.7. So you should > feel free to do this refactoring. More below. |
From: Jeff A. <ja...@fa...> - 2013-05-29 23:45:28
|
On 20/05/2013 17:43, Jim Baker wrote: > Jeff, > > In Jython 2.5 development, up through 2.5.0 RCs/final, we gave > ourselves the freedom to break existing Java APIs such as > PythonInterpreter. For subsequent 2.5.x releases, we grew the Java > API, but kept it backwards compatible. This required a fair amount of > contortions. > > The general principle should continue to stand in 2.7. So you should > feel free to do this refactoring. More below. > > On Sat, May 18, 2013 at 2:12 AM, Jeff Allen <ja...@fa... > <mailto:ja...@fa...>> wrote: > ... > > > The purpose of JLine is to provide a more convenient way for a user to > get (encoded) characters into a program than typing everything. I'll > call this its "console" function. However, in our current design, > JLineConsole is also a PythonInterpreter, working with lines of > characters treated as commands. It is only by separating the two > concerns, console and interpreter, that I've been able to make it > work. > Although one bug fix shouldn't drive architecture, I think this > provides > an architectural hint they should be separate. > > > Makes sense. > I spent a couple of days single-stepping through the initialisation and making notes. As a result understand this better. I found that initialisation is really orchestrated by PySystemState.doInitialize(). I found a spot in the middle of that where the registry is fully up, but the default PySystemState object has not yet been created. I can hook a custom console in at this point that provides a shim to System.in before sys.stdin gets created on it. I've only tried it with a do-nothing-much shim: >>> import sys >>> print (sys._jy_console, sys.stdin.isatty()) (org.python.util.JAInteractiveConsole@213a8eb1, True) >>> import java.lang.System >>> java.lang.System.in org.python.util.JAInteractiveConsole$Shim@1915470e >>> raw_input() stuff 'stuff' The names are temporary while I need old and new approaches concurrently. I've only sketched a JLine one, and how it would play with readline.py, but it looks possible. I believe we still need org.python.util.ReadlineConsole to work: is that right? Jeff |
From: Jeff A. <ja...@fa...> - 2013-09-07 18:34:28
|
It took a bit of getting right, but I have now implemented and pushed the console change indicated below. JLine and (on Linux) Readline consoles both work, and there is a plain console. The main structural change is that consoles and interpreters are separate ideas: the consoles implement an interface org.python.core.Console. Someone desiring to add a new console could implement that or extend PlainConsole. -Dpython.console=classname works as before. We still have org.python.util.JLineConsole and org.python.util.ReadlineConsole, but the fall-back console, and the one you get when Jython detects FileDescriptor.in is not interactive, is called org.python.core.PlainConsole. The implementation of these classes is much changed. The plain console is actually quite usable on Windows as it benefits from line recall and editing given it by the shell. (That surprised me.) sys.stdin.readline() echoes properly, recognises the end of line, and plays nicely with and raw_input(). I gave some attention to character encoding in the reworked consoles. We did not have bug reports, but support for anything beyond ASCII was poor (and confusing). It took me a while to figure out in just what way the faults in the libraries and the layered work-arounds were failing us. I think it is simpler now. There is a lot of transcoding one might avoid in special cases, but I don't think we care about performance here. Neither JLine nor Readline is entirely perfect for all purposes, and I can't get UTF-8 to work on Windows. But I can give myself a Greek keyboard and the characters I type are received in the encoding I set (ISO-8859-7 or UTF-8), and I have line editing and history recall with the cursor in the right place. I have kept the class called org.python.util.InteractiveConsole, because the name aligns with CPython, but it is an interpreter not a console. Specifying it will get you the plain console (and a warning), because that is what always happens when you specify a class that is not an org.python.core.Console. So it fails soft for people following old advice. sys._jy_console references the console you ended up with. readline.py uses that to access history as before, and test_readline passes. One thing I don't understand and would like to eliminate is the handling of ctrl-Z. I reproduced the logic from the original; JLineConsole, but I can't test it: on my Linux (Mint) system ctrl-Z sends Jython into the background, but when it comes back JLine will not recover (not before my changes either). I see what the code is supposed to do, but it is very ugly and on exactly what system does it do any good? Have I broken it? Jeff Jeff Allen On 30/05/2013 00:45, Jeff Allen wrote: > On 20/05/2013 17:43, Jim Baker wrote: >> Jeff, >> >> In Jython 2.5 development, up through 2.5.0 RCs/final, we gave >> ourselves the freedom to break existing Java APIs such as >> PythonInterpreter. For subsequent 2.5.x releases, we grew the Java >> API, but kept it backwards compatible. This required a fair amount of >> contortions. >> >> The general principle should continue to stand in 2.7. So you should >> feel free to do this refactoring. More below. >> >> On Sat, May 18, 2013 at 2:12 AM, Jeff Allen <ja...@fa... >> <mailto:ja...@fa...>> wrote: >> ... >> >> >> The purpose of JLine is to provide a more convenient way for a >> user to >> get (encoded) characters into a program than typing everything. I'll >> call this its "console" function. However, in our current design, >> JLineConsole is also a PythonInterpreter, working with lines of >> characters treated as commands. It is only by separating the two >> concerns, console and interpreter, that I've been able to make it >> work. >> Although one bug fix shouldn't drive architecture, I think this >> provides >> an architectural hint they should be separate. >> >> >> Makes sense. >> > I spent a couple of days single-stepping through the initialisation > and making notes. As a result understand this better. I found that > initialisation is really orchestrated by PySystemState.doInitialize(). > I found a spot in the middle of that where the registry is fully up, > but the default PySystemState object has not yet been created. > > I can hook a custom console in at this point that provides a shim to > System.in before sys.stdin gets created on it. I've only tried it with > a do-nothing-much shim: > >>> import sys > >>> print (sys._jy_console, sys.stdin.isatty()) > (org.python.util.JAInteractiveConsole@213a8eb1, True) > >>> import java.lang.System > >>> java.lang.System.in > org.python.util.JAInteractiveConsole$Shim@1915470e > >>> raw_input() > stuff > 'stuff' > > The names are temporary while I need old and new approaches > concurrently. I've only sketched a JLine one, and how it would play > with readline.py, but it looks possible. > > I believe we still need org.python.util.ReadlineConsole to work: is > that right? > > Jeff > > > ------------------------------------------------------------------------------ > Introducing AppDynamics Lite, a free troubleshooting tool for Java/.NET > Get 100% visibility into your production application - at no cost. > Code-level diagnostics for performance bottlenecks with <2% overhead > Download for free and get started troubleshooting in minutes. > http://p.sf.net/sfu/appdyn_d2d_ap1 > > > _______________________________________________ > Jython-dev mailing list > Jyt...@li... > https://lists.sourceforge.net/lists/listinfo/jython-dev |