gnuplot-beta — Developers and beta testers of new gnuplot code

Re: enhanced_recursion() and do_event(), can they be redesigned?

[Jérôme L. <lod...@us...>]

Hi,

	I got a bit confused about your motivations for introducing an event 
loop in the "inboard" qt terminal. In qt_term.cpp, we use, by design, a 
very limited set of the Qt library, and rather push as much 
functionality as possible the in the gnuplot_qt "outboard" program. And 
I think that this set of functions doesn't require an event loop to work 
properly (As a matter of fact, I can observe by commenting out the 
QCoreApplication declaration in qt_term.cpp that it doesn't even require 
an instantiation of QCoreApplication to work -- except from the Windows 
specific QCoreApplication::applicationDirPath() call). More 
specifically, we use
- Storage classes (QString, QImage, QColor...) that just gather 
information but use no event mechanism.
- QLocalSocket that, according to the Qt documentation, can work without 
an event loop:
"Although QLocalSocket is designed for use with an event loop, it's 
possible to use it without one. In that case, you must use 
waitForConnected(), waitForReadyRead(), waitForBytesWritten(), and 
waitForDisconnected() which blocks until the operation is complete or 
the timeout expires.". In fact, I can imagine that the gnuplot inboard 
driver is perhaps precisely the kind of programs that the Qt developers 
had in mind when they wrote this sentence: a program that cannot run a 
Qt event loop because it implements its own event loop, but that still 
wants to send messages to another auxiliary program that runs a Qt event 
loop to manage a GUI.

So could you point out a specific past or present problem in the Qt 
terminal that is due to the fact that no event loop is running in the 
inboard driver ?
To my knowledge, here are the issues that arose on non Linux platform:
- OSX: in the past, there was no gnuplot_qt independent process, and the 
GUI was rather managed in a thread. As you point out, this is not 
supported in Qt, and while it kind of happened to work with Linux, it 
failed on OSX.
- OSX: currently, the QtGnuplotWidget has a hard time to resize itself 
to the correct size, but it is specific to the gnuplot_qt outboard driver.
- Windows: Watching at the same time the standard input and a 
QLocalSocket file descriptor is not supported on Windows using select() 
or the Qt API (which might use select() under the hood). This issue is 
not related to the EventLoop or not EventLoop question and has been 
recently fixed by specific calls to the win32 API.
- All platforms: waitForConnected() immediatly returns if not socket is 
found (in particular, this happens when the gnuplot_qt program is still 
being initialized), even when a timeout is set. This was solved by 
introducing a custom timeout mechanism, but could more elegantly be 
solved by some communication mechanism between gnuplot ("inboard") and 
gnuplot_qt ("outboard"). I can see that part of your code actually 
implements this.

Concerning QApplication vs. QCoreApplication & font metrics:
Until recently, the inboard driver (qt_term.cpp) used to instantiate a 
QApplication to determine the font metrics (no event loop was required 
for this though, and more generally, no Qt event loop, nor Qt event 
based mechanism has ever been running in the main gnuplot thread). As 
Ethan said, this mechanism has been moved to the "outboard" gnuplot_qt 
program, and now the inboard driver only instantiate a QCoreApplication 
(which, as I said above, even turns out not to be necessary in practice, 
although the Qt documentation does not certify this point). The 
motivation for this was a significant performance increase: the 
initialization of a QApplication requires more than 0.5 s against a few 
10ms for a QCoreApplication. Moving back the font metric business in the 
main gnuplot program will require to instantiate again a QApplication 
and thus cancel this performance improvement.

Concerning the code you propose: apart from the synchronization problems 
between the Qt event loop and the main thread that you mention, I am 
worried about the performances. As far as I can see, the communication 
between the main thread and the thread running the Qt event loop relies 
on the signal/slot mechanism, which I think can be significantly slower 
that direct function calls. From rough testing, plotting a graph with a 
million points takes about 3x more time with the code you posted.

Jérôme



Le 24/02/2014 08:10, Daniel J Sebald a écrit :
> On 02/24/2014 12:16 AM, sfeam wrote:
>> On Sunday, 23 February 2014 11:22:42 PM Daniel J Sebald wrote:
>>> On 02/23/2014 10:35 PM, sfeam wrote:
>>>> On Sunday, 23 February 2014 10:04:42 PM Daniel J Sebald wrote:
>>>>>
>>>>> 	// Set plot size
>>>>> 	if (qt_setSize)
>>>>> 	{
>>>>> 		term->xmax = qt_oversampling*qt_setWidth;
>>>>> 		term->ymax = qt_oversampling*qt_setHeight;
>>>>> 		qt_setSize = false;
>>>>> 	}
>>>>>
>>>>> In a separate thread, the "term->xmax =" will be done asynchronously.
>>>>> Hence a mutex/wait is needed to make sure the code in the separate
>>>>> thread has updated term->xmax and term->max before the core thread can
>>>>> continue onward.
>>>>
>>>> Ah. Now I'm with you.
>>>> Yeah, this is the piece of code that has changed the most in qt
>>>> because nothing seems to work properly on both linux and OSX.
>>>> You are quite correct that term->foo should not be referenced in
>>>> this part of the terminal driver.  It is supposed to return the revised
>>>> font information via an event GP_fontprops.  And it _was_ doing that
>>>> at one point.  I've now lost track of all the work-arounds and what
>>>> exactly they fixed, but certainly it would be good if you can get
>>>> back to the original intent.  You can look at other terminal
>>>> drivers as a model if needed.
>>>
>>> OK, thanks.  I'll look that over.  I think that may be the one thing
>>> that isn't working yet for what I've done, i.e., some text isn't showing
>>> up probably owing to the core is told the font height is zero.
>>
>> Let me summarize a bit of the history.
>>
>> - The core code and the qt_term bits of Qt are in the same process.
>>     Call this "inboard"
>>     The screen display is being managed by a separate Qt process.
>>     Call this "outboard"
>>     The inboard and outboard processes operate asynchronously.
>>
>> - To reserve space for some text element on the next plot, the core
>>     code needs to know how big the current font is so it sends a query
>>     to the inboard terminal driver.  This request can come either via
>>     term->set_font()  or in enhanced text mode via term->put_text.
>>
>> - It would be simplest if the inboard terminal driver could just reply
>>     immediately with the requested font metrics. No communication back
>>     and forth with the outboard terminal driver is required.  This is what
>>     the Qt terminal used to do, and still does in version 4.6.
>>     The font size information is obtained by calling
>>     QFontMetrics metrics(QFont(qt_currentFontName, qt_currentFontSize));
>>     Since the inboard driver and the core code are in the same process,
>>     the inboard driver can just set term->h_char and term->v_char
>>     directly and that's the end of it.
>>
>> - Now here comes the problem.  Apparently calling QFontMetrics without
>>     there being a full QApplication and maybe [not sure] an event
>>     doesn't work properly.  In particular is was causing problems
>>     on OSX, and it was ugly even on linux.
>>     See the comments in version 4.6 qt_term.cpp
>>     // Create a QApplication without event loop for QObject's that need it,
>>     // namely font handling
>>     // A better strategy would be to transfer the font handling to the
>>     // QtGnuplotWidget, but it would require
>>     // some synchronization between the widget and the gnuplot process.
>>
>> - So in January Jérôme followed through on that comment and moved
>>     the QFontMetrics call over to the outboard driver for version 4.7.
>>     That simplified things in one way, but introduced a new complication.
>>     Now the inboard driver has to send a change font request to the
>>     outboard driver (a different process) and then wait for the resulting
>>     size information to be sent back via a QEvent.   That's where
>>     waitforinput() and do_event() suddenly become involved where
>>     they hadn't been previously.  It was also slower, so he later
>>     introduced a cache of font metrics on the inboard side but let's
>>     disregard that for now.
>>
>> - So now I gather you are working to have an event loop in the
>>     inboard driver again, although I've lost track of exactly why.
>
> Hopefully to make OSX and Windows behave the same as Unix.  I don't know
> for sure because Qt is a big creature, but I suspect that if there is no
> event loop that timers aren't guaranteed to work and who knows what
> else.  So by providing an event, I'm hoping Qt performs the same on all
> platforms.  It's just following the examples that Qt documentation gives.
>
>
>>     But in that case I think it makes sense to move the
>>     font metrics query back into the inboard driver as well.
>>     At which point we return to a setup in which waitforinput()
>>     and do_event() are not involved in font processing.
>>
>> So much for font handling and the involvement of enhanced text
>> processing.
>>
>> There is a separate issue, however, that has been at the heart of
>> the recent attempts to get OSX working.  Information about the
>> size of the display window necessarily comes from the outboard
>> driver, which is a separate process. Right now this window size
>> information is also passed using a GE_fontprops event, but there's no
>> good reason for that.  It really should have a separate event type
>> to reduce confusion.  But changing the name of the event wouldn't
>> change the information flow in any way.
>
> Oh, OK.  Thanks.  I've got the big picture now and I think we will
> probably converge on something here that is about right.
>
> I'll correct one thing in that list which is I believe the issue is not
> the presence of the QApplication in the main process, but that the
> QApplication cannot have an event loop.  That is, one can't issue
> application.exec(), because doing so will hang gnuplot while
> application.exec() blocks to handle all of its realtime traffic
> (signals, slots, system calls, etc).  Graphics in Qt must be done in a
> main thread; it can't be done in a separate thread.  That leaves the Qt
> graphics code to be run in a separate process.  (Note, I've fixed the
> process communications a bit so that there aren't these while loops that
> keep trying to establish a connection to some external service.)
>
> The QApplication issue is why I've set out to put the bulk of the
> terminal interface code in a separate thread because in the separate
> thread and event loop, i.e., thread.exec(), is perfectly fine.  The
> thread acts on its own.  My preliminary code indicates there is no
> problem sending signals back and forth across that thread.  So, with an
> event loop running in the second thread, all the timers and whatever
> else should be fine--the only restriction being that graphics cannot be
> done there.  If it turns out OSX doesn't behave the same in the thread
> that has an event loop, then something isn't right with Qt.  (We'll know
> soon.)
>
> So here is what I'm aiming for right now, sort of combining everything:
>
>      MAIN PROGRAM         |       QTHREAD        ||  GNUPLOT_QT
>                           |                      ||  (outboard)
>                           |                      ||
>    gnuplot core           |  QtTerminalInterface ||  Currently has a
>    QtTerminalEmitter      |  (active event loop) ||  few things we hope
>    (inactive event loop)  |                      ||  to move back into
>                           |                      ||  QtTerminalInterface
>
> where the single line means separate thread and the double line means a
> separate process.
>
> OK, for the most part I'll be offline for the week.
>
> Dan
>
> ------------------------------------------------------------------------------
> Flow-based real-time traffic analytics software. Cisco certified tool.
> Monitor traffic, SLAs, QoS, Medianet, WAAS etc. with NetFlow Analyzer
> Customize your own dashboards, set traffic alerts and generate reports.
> Network behavioral analysis & security monitoring. All-in-one tool.
> http://pubads.g.doubleclick.net/gampad/clk?id=126839071&iu=/4140/ostg.clktrk
> _______________________________________________
> gnuplot-beta mailing list
> gnu...@li...
> Membership management via: https://lists.sourceforge.net/lists/listinfo/gnuplot-beta
>

Re: Repository conversion

[Clark G. <cla...@fa...>]

I've been thinking for some time I might just do an hg conversion for the good of the order, at least to demonstrate. I agree with Eric - once a decision is made, you're better off to write up a little "how to" and have a flag day. Or two. ;-)

On Feb 24, 2014 10:22 AM, Daniel J Sebald <dan...@ie...> wrote:
>
> On 02/24/2014 08:30 AM, Eric S. Raymond wrote: 
>
> > (2) Where do you want to land? 
> > 
> > As bzr is now moribund, the realistic alternatives are Subversion, 
> > git, and hg.  But I mention Subversion only for completeness; the 
> > advantages of distributed VCS are huge and if you're going to incur 
> > the transition costs of moving away from CVS either git or hg would 
> > have a much higher functional payoff. 
>
> A few days ago I was thinking that perhaps a conversion to both git/hg 
> could be done and for a month the developers try to keep cvs/git/hg in 
> sync and at the end of the month decide which is preferred.  The summer 
> months are usually when there is most free time for that sort of thing. 
>
> Dan 

Re: Repository conversion

[Eric S. R. <es...@th...>]

Daniel J Sebald <dan...@ie...>:
> A few days ago I was thinking that perhaps a conversion to both
> git/hg could be done and for a month the developers try to keep
> cvs/git/hg in sync and at the end of the month decide which is
> preferred.  The summer months are usually when there is most free
> time for that sort of thing.

I advise against trying to maintain two repositories at once.  That way
lies only pain.

Fortunately the git suite recently added the capability to use hg as a
client to a remote git repository.  So the repository conversion would
only have to be done once.

-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Repository conversion

[Daniel J S. <dan...@ie...>]

On 02/24/2014 08:30 AM, Eric S. Raymond wrote:

> (2) Where do you want to land?
>
> As bzr is now moribund, the realistic alternatives are Subversion,
> git, and hg.  But I mention Subversion only for completeness; the
> advantages of distributed VCS are huge and if you're going to incur
> the transition costs of moving away from CVS either git or hg would
> have a much higher functional payoff.

A few days ago I was thinking that perhaps a conversion to both git/hg 
could be done and for a month the developers try to keep cvs/git/hg in 
sync and at the end of the month decide which is preferred.  The summer 
months are usually when there is most free time for that sort of thing.

Dan

Re: Repository conversion

[Eric S. R. <es...@th...>]

Clark Gaylord <cla...@fa...>:
> I'm not sure what is up with your subscription posting, but I very 
> strongly concur with the substance of your suggestion regarding the 
> repository and converting to a distributed source management system.

I've started getting list mail, so whatever glitch was involved has subsided.

> My personal preference is hg, but I agree that there are lots of git 
> users out there, especially due to Linux kernel development. In my 
> experience, it is fairly easy for cvs/svn users to convert to hg; I 
> suspect it is similar for git.

The learning barrier is a little higher for git because its UI is 
rather spiky, but - broadly - yes.  Neither is very difficult, and
anything not-CVS is a sufficiently large improvement over CVS that
the transition costs get paid back in decreased hassle very quickly.

>                                 I have a few dozen developers in my 
> group and elsewhere in my organization who have largely converted to hg 
> and git from CVS and SVN over the last few years, and reactions have 
> ranged from "doesn't negatively impact me" to enthusiastic adopters. I 
> think it is fair to say that on average the more sophisticated the 
> user, the more enthusiastic the reaction has been.

That matches my experience.

> Thanks for the suggestion. Please feel free to let me know if you have 
> any further data on your ability to post and I'll try to track it down.

As I said above, no longer a problem.

I'm trying to get the Emacs conversion squared away now. I'm not in a big
hurry to do gnuplot, but I do urge your project leadership to address three
questions:

(1) Are you ready to move away from CVS?

I hope so...it's pretty broken, and it's slowly dying; there hasn't
been a maintainance release since 2008.  Projects that stick with 
CVS face increasing difficulties in attracting new devs.

(2) Where do you want to land?

As bzr is now moribund, the realistic alternatives are Subversion,
git, and hg.  But I mention Subversion only for completeness; the
advantages of distributed VCS are huge and if you're going to incur
the transition costs of moving away from CVS either git or hg would
have a much higher functional payoff.

(3) When do you want to do it?

The real question here is whether you think you need to wait until after
your next point release.  My impression from the list traffic is that
your development tempo is relatively slow, so the minor distruption from
the switchover should be tolerable at any time.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Repository conversion

[Clark G. <cla...@fa...>]

Hello Eric --

I'm not sure what is up with your subscription posting, but I very 
strongly concur with the substance of your suggestion regarding the 
repository and converting to a distributed source management system.

My personal preference is hg, but I agree that there are lots of git 
users out there, especially due to Linux kernel development. In my 
experience, it is fairly easy for cvs/svn users to convert to hg; I 
suspect it is similar for git. I have a few dozen developers in my 
group and elsewhere in my organization who have largely converted to hg 
and git from CVS and SVN over the last few years, and reactions have 
ranged from "doesn't negatively impact me" to enthusiastic adopters. I 
think it is fair to say that on average the more sophisticated the 
user, the more enthusiastic the reaction has been.

Thanks for the suggestion. Please feel free to let me know if you have 
any further data on your ability to post and I'll try to track it down.

Regards
Clark

--
Clark Gaylord
gnuplot sysadmin
Blacksburg VA
cla...@gm...

On Fri, 21 Feb 2014 09:34:04 -0500 (EST), Eric S. Raymond wrote:
> Something unfortunate is going on with my list subscription.  Mailman
> tells me I'm subscribed, but I'm not seeing list mail; I'm having
> to read this thread through the archive interface.
> 
> Ethan A Merritt writes:
>> Can you summarize in what ways your conversion would be different
>> from the automated cvs->git conversion service offered by SourceForge
>> (which so far the project has declined to use)?
> 
> Before going into the details, I should make clear that I did not
> specify a target VCS in my offer because I can up-convert to either
> git or hg very easily - gnuplot can choose either.  I retain a
> lingering preference for hg myself, but I advise going with git 
> anyway - it presents the lowest entry to the largest number of
> potential contributors.
> 
> I don't know what SF uses for its conversions; the service is a recent
> invention and Google isn't turning up documentation on it for me.
> There aren't many possibilities - maintained and documented tools for
> this purpose are thin on the ground. I fear they are probably using
> cvsps called by git-cvsimport, because that is what the git suite
> ships.
> 
> Unfortunately, cvsps is so irreparably broken that I, acting as its last
> maintainer, end-of-lifed it a couple of months ago. Mojca Miklavec can
> confirm what a mess it is; she tried doing  a test conversion and got
> horrible results.  The git devs don't realize how bad the situation
> is.
> 
> Through an odd set of circumstances, last year I wound up as the
> maintainer of both cvsps and Keith Packard's parsecvs code, which I
> rewrote significantly and renamed cvs-fast-export.  After EOLing
> cvsps, I shifted my effort to cvs-fast-import, which is what Mojca
> Miklavec has been doing her most recent test conversions with.
> 
> cvsps and cvs-fast-export are two of only three possibilities SF
> might be using.  The third is cvs2git, which has some problems of
> its own that I am trying to help its maintainer fix.  
> 
> All three of these tools, by themselves, are too weak to produce 
> a really high-quality conversion.  The things they don't do include:
> 
> 1. Lifting CVS version references in comments into a form that
> will still be usable in git or hg.
> 
> 2. Complete coalescence of CVS change cliques into changesets.  They
> often only do this partially, one reason being that the default
> merge window is set too low.
> 
> 3. Mapping of .cvsignores to .gitignores.  My newer versions of
> cvs-fast-export do this but neither cvs2git nor the cvsps version in
> the git suite does.
> 
> 4. Cleaning up conversion artifacts and junk branches. I can't be
> more specific about this in advance because each CVS repo tends to
> have its own unique set of strange malformations.  Fortunately, 
> gnuplot's history seems exceptionally clean and I expect relatively
> few problems.
> 
> 5. (Optional) Canonicalizing change comments to git summary +
> continuation form.
> 
> Completing these tasks requires human judgment. The only way to get a
> really good conversion is to follow up application of a batch
> converter with skilled hand-editing, and reposurgeon is the only
> existing tool for that.  It's an interpreter for a domain-specific
> language built around repository-editing primitives. 
> 
> (As previously noted, I am currently in the process of lifting Emacs's
> history from bzr to git with reposurgeon.  That is an exceptionally
> large and messy conversion, far more complex than I expect gnuplot's
> to be.)
> 
> I have done about half a dozen large conversions before; the most
> recent completed one was GNU troff.  You can read about my procedures
> in the DVCS Migration HOWTO:
> 
>    http://www.catb.org/esr/dvcs-migration-guide.html
> 
> Ideally, I team with an inside developer who knows the project's
> history and idiosyncracies and is highly motivated to make the
> conversion succeed.  For this project, Mojca Miklavec has claimed that
> role - she has already supplied one of the key pieces of metadata, a
> map from developers' CVS usernames to full names and email addresses.
> 
> If the project elects to go ahead, our work product would be a lift
> script - that is, a file containing surgical instructions to
> reposurgeon - which would be available for review before the actual
> conversion day.
> -- 
> 		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>
> 
> Where rights secured by the Constitution are involved, there can be no
> rule making or legislation which would abrogate them.
>         -- Miranda vs. Arizona, 384 US 436 p. 491
> 
> 
------------------------------------------------------------------------------
> Managing the Performance of Cloud-Based Applications
> Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
> Read the Whitepaper.
> http://pubads.g.doubleclick.net/gampad/clk?id=121054471&iu=/4140/ostg.clktrk
> _______________________________________________
> gnuplot-beta mailing list
> gnu...@li...
> Membership management via: 
> https://lists.sourceforge.net/lists/listinfo/gnuplot-beta

Re: enhanced_recursion() and do_event(), can they be redesigned?

[Daniel J S. <dan...@ie...>]

On 02/24/2014 12:16 AM, sfeam wrote:
> On Sunday, 23 February 2014 11:22:42 PM Daniel J Sebald wrote:
>> On 02/23/2014 10:35 PM, sfeam wrote:
>>> On Sunday, 23 February 2014 10:04:42 PM Daniel J Sebald wrote:
>>>>
>>>> 	// Set plot size
>>>> 	if (qt_setSize)
>>>> 	{
>>>> 		term->xmax = qt_oversampling*qt_setWidth;
>>>> 		term->ymax = qt_oversampling*qt_setHeight;
>>>> 		qt_setSize = false;
>>>> 	}
>>>>
>>>> In a separate thread, the "term->xmax =" will be done asynchronously.
>>>> Hence a mutex/wait is needed to make sure the code in the separate
>>>> thread has updated term->xmax and term->max before the core thread can
>>>> continue onward.
>>>
>>> Ah. Now I'm with you.
>>> Yeah, this is the piece of code that has changed the most in qt
>>> because nothing seems to work properly on both linux and OSX.
>>> You are quite correct that term->foo should not be referenced in
>>> this part of the terminal driver.  It is supposed to return the revised
>>> font information via an event GP_fontprops.  And it _was_ doing that
>>> at one point.  I've now lost track of all the work-arounds and what
>>> exactly they fixed, but certainly it would be good if you can get
>>> back to the original intent.  You can look at other terminal
>>> drivers as a model if needed.
>>
>> OK, thanks.  I'll look that over.  I think that may be the one thing
>> that isn't working yet for what I've done, i.e., some text isn't showing
>> up probably owing to the core is told the font height is zero.
>
> Let me summarize a bit of the history.
>
> - The core code and the qt_term bits of Qt are in the same process.
>    Call this "inboard"
>    The screen display is being managed by a separate Qt process.
>    Call this "outboard"
>    The inboard and outboard processes operate asynchronously.
>
> - To reserve space for some text element on the next plot, the core
>    code needs to know how big the current font is so it sends a query
>    to the inboard terminal driver.  This request can come either via
>    term->set_font()  or in enhanced text mode via term->put_text.
>
> - It would be simplest if the inboard terminal driver could just reply
>    immediately with the requested font metrics. No communication back
>    and forth with the outboard terminal driver is required.  This is what
>    the Qt terminal used to do, and still does in version 4.6.
>    The font size information is obtained by calling
>    QFontMetrics metrics(QFont(qt_currentFontName, qt_currentFontSize));
>    Since the inboard driver and the core code are in the same process,
>    the inboard driver can just set term->h_char and term->v_char
>    directly and that's the end of it.
>
> - Now here comes the problem.  Apparently calling QFontMetrics without
>    there being a full QApplication and maybe [not sure] an event
>    doesn't work properly.  In particular is was causing problems
>    on OSX, and it was ugly even on linux.
>    See the comments in version 4.6 qt_term.cpp
>    // Create a QApplication without event loop for QObject's that need it,
>    // namely font handling
>    // A better strategy would be to transfer the font handling to the
>    // QtGnuplotWidget, but it would require
>    // some synchronization between the widget and the gnuplot process.
>
> - So in January Jérôme followed through on that comment and moved
>    the QFontMetrics call over to the outboard driver for version 4.7.
>    That simplified things in one way, but introduced a new complication.
>    Now the inboard driver has to send a change font request to the
>    outboard driver (a different process) and then wait for the resulting
>    size information to be sent back via a QEvent.   That's where
>    waitforinput() and do_event() suddenly become involved where
>    they hadn't been previously.  It was also slower, so he later
>    introduced a cache of font metrics on the inboard side but let's
>    disregard that for now.
>
> - So now I gather you are working to have an event loop in the
>    inboard driver again, although I've lost track of exactly why.

Hopefully to make OSX and Windows behave the same as Unix.  I don't know 
for sure because Qt is a big creature, but I suspect that if there is no 
event loop that timers aren't guaranteed to work and who knows what 
else.  So by providing an event, I'm hoping Qt performs the same on all 
platforms.  It's just following the examples that Qt documentation gives.


>    But in that case I think it makes sense to move the
>    font metrics query back into the inboard driver as well.
>    At which point we return to a setup in which waitforinput()
>    and do_event() are not involved in font processing.
>
> So much for font handling and the involvement of enhanced text
> processing.
>
> There is a separate issue, however, that has been at the heart of
> the recent attempts to get OSX working.  Information about the
> size of the display window necessarily comes from the outboard
> driver, which is a separate process. Right now this window size
> information is also passed using a GE_fontprops event, but there's no
> good reason for that.  It really should have a separate event type
> to reduce confusion.  But changing the name of the event wouldn't
> change the information flow in any way.

Oh, OK.  Thanks.  I've got the big picture now and I think we will 
probably converge on something here that is about right.

I'll correct one thing in that list which is I believe the issue is not 
the presence of the QApplication in the main process, but that the 
QApplication cannot have an event loop.  That is, one can't issue 
application.exec(), because doing so will hang gnuplot while 
application.exec() blocks to handle all of its realtime traffic 
(signals, slots, system calls, etc).  Graphics in Qt must be done in a 
main thread; it can't be done in a separate thread.  That leaves the Qt 
graphics code to be run in a separate process.  (Note, I've fixed the 
process communications a bit so that there aren't these while loops that 
keep trying to establish a connection to some external service.)

The QApplication issue is why I've set out to put the bulk of the 
terminal interface code in a separate thread because in the separate 
thread and event loop, i.e., thread.exec(), is perfectly fine.  The 
thread acts on its own.  My preliminary code indicates there is no 
problem sending signals back and forth across that thread.  So, with an 
event loop running in the second thread, all the timers and whatever 
else should be fine--the only restriction being that graphics cannot be 
done there.  If it turns out OSX doesn't behave the same in the thread 
that has an event loop, then something isn't right with Qt.  (We'll know 
soon.)

So here is what I'm aiming for right now, sort of combining everything:

    MAIN PROGRAM         |       QTHREAD        ||  GNUPLOT_QT
                         |                      ||  (outboard)
                         |                      ||
  gnuplot core           |  QtTerminalInterface ||  Currently has a
  QtTerminalEmitter      |  (active event loop) ||  few things we hope
  (inactive event loop)  |                      ||  to move back into
                         |                      ||  QtTerminalInterface

where the single line means separate thread and the double line means a 
separate process.

OK, for the most part I'll be offline for the week.

Dan

Re: enhanced_recursion() and do_event(), can they be redesigned?

[sfeam <sf...@us...>]

On Sunday, 23 February 2014 11:22:42 PM Daniel J Sebald wrote:
> On 02/23/2014 10:35 PM, sfeam wrote:
> > On Sunday, 23 February 2014 10:04:42 PM Daniel J Sebald wrote:
> >>
> >> 	// Set plot size
> >> 	if (qt_setSize)
> >> 	{
> >> 		term->xmax = qt_oversampling*qt_setWidth;
> >> 		term->ymax = qt_oversampling*qt_setHeight;
> >> 		qt_setSize = false;
> >> 	}
> >>
> >> In a separate thread, the "term->xmax =" will be done asynchronously.
> >> Hence a mutex/wait is needed to make sure the code in the separate
> >> thread has updated term->xmax and term->max before the core thread can
> >> continue onward.
> >
> > Ah. Now I'm with you.
> > Yeah, this is the piece of code that has changed the most in qt
> > because nothing seems to work properly on both linux and OSX.
> > You are quite correct that term->foo should not be referenced in
> > this part of the terminal driver.  It is supposed to return the revised
> > font information via an event GP_fontprops.  And it _was_ doing that
> > at one point.  I've now lost track of all the work-arounds and what
> > exactly they fixed, but certainly it would be good if you can get
> > back to the original intent.  You can look at other terminal
> > drivers as a model if needed.
> 
> OK, thanks.  I'll look that over.  I think that may be the one thing 
> that isn't working yet for what I've done, i.e., some text isn't showing 
> up probably owing to the core is told the font height is zero.

Let me summarize a bit of the history.

- The core code and the qt_term bits of Qt are in the same process.
  Call this "inboard"
  The screen display is being managed by a separate Qt process.
  Call this "outboard"
  The inboard and outboard processes operate asynchronously.

- To reserve space for some text element on the next plot, the core
  code needs to know how big the current font is so it sends a query
  to the inboard terminal driver.  This request can come either via
  term->set_font()  or in enhanced text mode via term->put_text.

- It would be simplest if the inboard terminal driver could just reply
  immediately with the requested font metrics. No communication back
  and forth with the outboard terminal driver is required.  This is what
  the Qt terminal used to do, and still does in version 4.6.
  The font size information is obtained by calling
  QFontMetrics metrics(QFont(qt_currentFontName, qt_currentFontSize));
  Since the inboard driver and the core code are in the same process,
  the inboard driver can just set term->h_char and term->v_char 
  directly and that's the end of it.

- Now here comes the problem.  Apparently calling QFontMetrics without
  there being a full QApplication and maybe [not sure] an event 
  doesn't work properly.  In particular is was causing problems
  on OSX, and it was ugly even on linux.
  See the comments in version 4.6 qt_term.cpp
  // Create a QApplication without event loop for QObject's that need it,
  // namely font handling
  // A better strategy would be to transfer the font handling to the
  // QtGnuplotWidget, but it would require
  // some synchronization between the widget and the gnuplot process.

- So in January Jérôme followed through on that comment and moved
  the QFontMetrics call over to the outboard driver for version 4.7.
  That simplified things in one way, but introduced a new complication.
  Now the inboard driver has to send a change font request to the
  outboard driver (a different process) and then wait for the resulting 
  size information to be sent back via a QEvent.   That's where
  waitforinput() and do_event() suddenly become involved where
  they hadn't been previously.  It was also slower, so he later 
  introduced a cache of font metrics on the inboard side but let's
  disregard that for now.

- So now I gather you are working to have an event loop in the 
  inboard driver again, although I've lost track of exactly why.
  But in that case I think it makes sense to move the
  font metrics query back into the inboard driver as well.
  At which point we return to a setup in which waitforinput()
  and do_event() are not involved in font processing.

So much for font handling and the involvement of enhanced text
processing.  

There is a separate issue, however, that has been at the heart of
the recent attempts to get OSX working.  Information about the
size of the display window necessarily comes from the outboard
driver, which is a separate process. Right now this window size
information is also passed using a GE_fontprops event, but there's no
good reason for that.  It really should have a separate event type
to reduce confusion.  But changing the name of the event wouldn't
change the information flow in any way.

	Ethan

Re: enhanced_recursion() and do_event(), can they be redesigned?

[Daniel J S. <dan...@ie...>]

On 02/23/2014 10:35 PM, sfeam wrote:
> On Sunday, 23 February 2014 10:04:42 PM Daniel J Sebald wrote:
>> On 02/23/2014 09:43 PM, sfeam wrote:
>>> On Sunday, 23 February 2014 09:26:26 PM Daniel J Sebald wrote:
>>>> In attempting to modify the qt_term.cpp Qt terminal, I ran into some
>>>> annoying problems with recursive calls, namely enhanced_recursion() and
>>>> do_event().
>>>>
>>>> First, let me summarize a few things:
>>>>
>>>> 1) There are a few uses of the global pointer term->.  My preference
>>>> would be to remove those, but I understand there needs to be a way to
>>>> get some information back to gnuplot core.  It seems to me that putting
>>>> non-const pointers in the API is the best way to do that.  But yes, it
>>>> is sort of the same difference.
>>>
>>> I'm afraid I'm not following you.
>>> A few uses for term->    in what piece of code, exactly?
>>
>> There aren't many cases, but here is one example in qt_term.cpp:
>>
>> // Called just before a plot is going to be displayed.
>> void QtTerminalInterface::qt_graphics(unsigned int v_char)
>> {
>> 	ensureOptionsCreated();
>> 	out<<  GEDesactivate;
>> 	qt_flushOutBuffer();
>> 	connectToServer();
>>
>> 	// Set text encoding
>> 	if (!(codec = qt_encodingToCodec(encoding)))
>> 		codec = QTextCodec::codecForLocale();
>>
>> 	// Set font
>> 	currentFontSize = qt_optionFontSize;
>> 	currentFontName = qt_option->FontName;
>>
>> 	// Set plot size
>> 	if (qt_setSize)
>> 	{
>> 		term->xmax = qt_oversampling*qt_setWidth;
>> 		term->ymax = qt_oversampling*qt_setHeight;
>> 		qt_setSize = false;
>> 	}
>>
>> In a separate thread, the "term->xmax =" will be done asynchronously.
>> Hence a mutex/wait is needed to make sure the code in the separate
>> thread has updated term->xmax and term->max before the core thread can
>> continue onward.
>
> Ah. Now I'm with you.
> Yeah, this is the piece of code that has changed the most in qt
> because nothing seems to work properly on both linux and OSX.
> You are quite correct that term->foo should not be referenced in
> this part of the terminal driver.  It is supposed to return the revised
> font information via an event GP_fontprops.  And it _was_ doing that
> at one point.  I've now lost track of all the work-arounds and what
> exactly they fixed, but certainly it would be good if you can get
> back to the original intent.  You can look at other terminal
> drivers as a model if needed.

OK, thanks.  I'll look that over.  I think that may be the one thing 
that isn't working yet for what I've done, i.e., some text isn't showing 
up probably owing to the core is told the font height is zero.


>>>> So, with that, I'll ask if there is some way of redesigning
>>>> enhanced_recursion() and do_event().
>>>
>>> These have nothing to do with each other, so I don't understand the
>>> question.  Both are part of the core, and are shared by all terminals.
>>
>> What they share is the fact that they go back to the core code and can
>> recursively issue further API calls before the active API returns.
>>
>>
>>>> What is the role of enhanced_recursion()?
>>>
>>> This is the routine that interprets enhanced text markup strings.
>>> It's part of the core text processing.  It is called whenever the
>>> core routines want to output a string in enhanced text mode.
>>
>> What you described sounds more like enhanced_writec(), whereas
>> enhanced_recursion() is issued by the terminal, at least it is for the
>> Qt terminal.
>
> All of those enhanced_foo() routines are part of the text output layer.
> enhanced_recursion() in particular is shared by all terminals and
> lives in term.c.   It is called whenever an enhanced text string is
> output, and then calls itself recursively as the name suggests to
> handle embedded fragments of the text markup.   E.g.
>     "{top_{sub1_{sub2_{sub3}}}}"
> Each left curly bracket triggers a new level of recursion.
> Please leave it alone.

Just wondering if that recursion couldn't be kept to the core.  It looks 
like the things needed (for Qt) to do that recursion are fontName, 
fontSize and enhanced_flush().  I think those are all currently 
accessible through the API.  In any case, I already have that worked out 
so no need to modify anything there.


> Anyhow, what's wrong with recursion?
> I posted an example before of a hot key definition that
> necessarily triggers recursion and works correctly on
> all the terminals I tried.  If there is a piece of the qt
> event code that is not re-entry safe, let's fix that first
> and then worry about whether something else is also problematic.

I can work this out, now that I've thought about it a bit.

Dan

Re: enhanced_recursion() and do_event(), can they be redesigned?

[sfeam <sf...@us...>]

On Sunday, 23 February 2014 10:04:42 PM Daniel J Sebald wrote:
> On 02/23/2014 09:43 PM, sfeam wrote:
> > On Sunday, 23 February 2014 09:26:26 PM Daniel J Sebald wrote:
> >> In attempting to modify the qt_term.cpp Qt terminal, I ran into some
> >> annoying problems with recursive calls, namely enhanced_recursion() and
> >> do_event().
> >>
> >> First, let me summarize a few things:
> >>
> >> 1) There are a few uses of the global pointer term->.  My preference
> >> would be to remove those, but I understand there needs to be a way to
> >> get some information back to gnuplot core.  It seems to me that putting
> >> non-const pointers in the API is the best way to do that.  But yes, it
> >> is sort of the same difference.
> >
> > I'm afraid I'm not following you.
> > A few uses for term->   in what piece of code, exactly?
> 
> There aren't many cases, but here is one example in qt_term.cpp:
> 
> // Called just before a plot is going to be displayed.
> void QtTerminalInterface::qt_graphics(unsigned int v_char)
> {
> 	ensureOptionsCreated();
> 	out << GEDesactivate;
> 	qt_flushOutBuffer();
> 	connectToServer();
> 
> 	// Set text encoding
> 	if (!(codec = qt_encodingToCodec(encoding)))
> 		codec = QTextCodec::codecForLocale();
> 
> 	// Set font
> 	currentFontSize = qt_optionFontSize;
> 	currentFontName = qt_option->FontName;
> 
> 	// Set plot size
> 	if (qt_setSize)
> 	{
> 		term->xmax = qt_oversampling*qt_setWidth;
> 		term->ymax = qt_oversampling*qt_setHeight;
> 		qt_setSize = false;
> 	}
> 
> In a separate thread, the "term->xmax =" will be done asynchronously. 
> Hence a mutex/wait is needed to make sure the code in the separate 
> thread has updated term->xmax and term->max before the core thread can 
> continue onward.

Ah. Now I'm with you.
Yeah, this is the piece of code that has changed the most in qt
because nothing seems to work properly on both linux and OSX.
You are quite correct that term->foo should not be referenced in 
this part of the terminal driver.  It is supposed to return the revised
font information via an event GP_fontprops.  And it _was_ doing that
at one point.  I've now lost track of all the work-arounds and what
exactly they fixed, but certainly it would be good if you can get
back to the original intent.  You can look at other terminal
drivers as a model if needed.

> >> So, with that, I'll ask if there is some way of redesigning
> >> enhanced_recursion() and do_event().
> >
> > These have nothing to do with each other, so I don't understand the
> > question.  Both are part of the core, and are shared by all terminals.
> 
> What they share is the fact that they go back to the core code and can 
> recursively issue further API calls before the active API returns.
> 
> 
> >> What is the role of enhanced_recursion()?
> >
> > This is the routine that interprets enhanced text markup strings.
> > It's part of the core text processing.  It is called whenever the
> > core routines want to output a string in enhanced text mode.
> 
> What you described sounds more like enhanced_writec(), whereas 
> enhanced_recursion() is issued by the terminal, at least it is for the 
> Qt terminal.

All of those enhanced_foo() routines are part of the text output layer.
enhanced_recursion() in particular is shared by all terminals and 
lives in term.c.   It is called whenever an enhanced text string is
output, and then calls itself recursively as the name suggests to
handle embedded fragments of the text markup.   E.g.
   "{top_{sub1_{sub2_{sub3}}}}"
Each left curly bracket triggers a new level of recursion.
Please leave it alone.

 
> >> What is the role of do_event()?
> >
> > do_event() is an asynchronous entry point in the core.
> > Interactive terminals use it to request some action, e.g.
> > replot, update mouse coords, zoom, respond to hot-key.
> >
> >> Is there some way a result can be sent back via the API that indicates
> >> to repeat the last event?
> >
> > Sent from whom to whom?  What sort of event?
> 
> Add a second variable to waitforinput, say:
> 
>      qt_waitforinput(int options, gp_event_t* event)
>
> and the gnuplot core does something like:
> 
>      gp_event_t *event_request = 0;
>      term->waitforinput(options, &event_request);
>      if (event_request)
> 	do_event(event_request);
> 
> That way there are no recursions...I'm assuming (hoping) that there 
> isn't recursions inside of recursions.

Sorry, I'm not following this at all.
You can't change waitforinput() for just qt;
it's an API shared by all the interactive terminals.

Anyhow, what's wrong with recursion?
I posted an example before of a hot key definition that
necessarily triggers recursion and works correctly on 
all the terminals I tried.  If there is a piece of the qt
event code that is not re-entry safe, let's fix that first
and then worry about whether something else is also problematic.

	Ethan

Re: enhanced_recursion() and do_event(), can they be redesigned?

[Daniel J S. <dan...@ie...>]

On 02/23/2014 10:04 PM, Daniel J Sebald wrote:

> Add a second variable to waitforinput, say:
>
>       qt_waitforinput(int options, gp_event_t* event)
>
> and the gnuplot core does something like:
>
>       gp_event_t *event_request = 0;
>       term->waitforinput(options,&event_request);
>       if (event_request)
> 	do_event(event_request);
>
> That way there are no recursions...I'm assuming (hoping) that there
> isn't recursions inside of recursions.

I suppose I could do something like the above inside the Qt terminal, so 
long as it is done on the gnuplot core side of things:


     MAIN PROGRAM         |       QTHREAD
                          |
   gnuplot core           |  QtTerminalInterface
   QtTerminalEmitter      |  (active event loop)
   (inactive event loop)  |
                          |
   Place do_event() here  |
   when other thread is   |
   complete               |

Dan

Re: enhanced_recursion() and do_event(), can they be redesigned?

[Daniel J S. <dan...@ie...>]

On 02/23/2014 09:43 PM, sfeam wrote:
> On Sunday, 23 February 2014 09:26:26 PM Daniel J Sebald wrote:
>> In attempting to modify the qt_term.cpp Qt terminal, I ran into some
>> annoying problems with recursive calls, namely enhanced_recursion() and
>> do_event().
>>
>> First, let me summarize a few things:
>>
>> 1) There are a few uses of the global pointer term->.  My preference
>> would be to remove those, but I understand there needs to be a way to
>> get some information back to gnuplot core.  It seems to me that putting
>> non-const pointers in the API is the best way to do that.  But yes, it
>> is sort of the same difference.
>
> I'm afraid I'm not following you.
> A few uses for term->   in what piece of code, exactly?

There aren't many cases, but here is one example in qt_term.cpp:

// Called just before a plot is going to be displayed.
void QtTerminalInterface::qt_graphics(unsigned int v_char)
{
	ensureOptionsCreated();
	out << GEDesactivate;
	qt_flushOutBuffer();
	connectToServer();

	// Set text encoding
	if (!(codec = qt_encodingToCodec(encoding)))
		codec = QTextCodec::codecForLocale();

	// Set font
	currentFontSize = qt_optionFontSize;
	currentFontName = qt_option->FontName;

	// Set plot size
	if (qt_setSize)
	{
		term->xmax = qt_oversampling*qt_setWidth;
		term->ymax = qt_oversampling*qt_setHeight;
		qt_setSize = false;
	}

In a separate thread, the "term->xmax =" will be done asynchronously. 
Hence a mutex/wait is needed to make sure the code in the separate 
thread has updated term->xmax and term->max before the core thread can 
continue onward.


>> I've managed to move enhanced_recursion() into the emitter code (i.e.,
>> same thread as gnuplot core) and that works.  Inelegant, but it works.
>
> Here I'm really lost. enhanced_recursion() is part of the gnuplot core.
> It is shared by all terminals.  What does this have to do with qt
> in particular?

It has to do with the fact I've placed the bulk of the qt terminal in a 
separate thread (which has an execution loop to ensure proper Qt 
behavior...that's the theory anyway).


>> So, with that, I'll ask if there is some way of redesigning
>> enhanced_recursion() and do_event().
>
> These have nothing to do with each other, so I don't understand the
> question.  Both are part of the core, and are shared by all terminals.

What they share is the fact that they go back to the core code and can 
recursively issue further API calls before the active API returns.


>> What is the role of enhanced_recursion()?
>
> This is the routine that interprets enhanced text markup strings.
> It's part of the core text processing.  It is called whenever the
> core routines want to output a string in enhanced text mode.

What you described sounds more like enhanced_writec(), whereas 
enhanced_recursion() is issued by the terminal, at least it is for the 
Qt terminal.


>> What is the role of do_event()?
>
> do_event() is an asynchronous entry point in the core.
> Interactive terminals use it to request some action, e.g.
> replot, update mouse coords, zoom, respond to hot-key.
>
>> Is there some way a result can be sent back via the API that indicates
>> to repeat the last event?
>
> Sent from whom to whom?  What sort of event?

Add a second variable to waitforinput, say:

     qt_waitforinput(int options, gp_event_t* event)

and the gnuplot core does something like:

     gp_event_t *event_request = 0;
     term->waitforinput(options, &event_request);
     if (event_request)
	do_event(event_request);

That way there are no recursions...I'm assuming (hoping) that there 
isn't recursions inside of recursions.

Dan

Re: enhanced_recursion() and do_event(), can they be redesigned?

[sfeam <sf...@us...>]

On Sunday, 23 February 2014 09:26:26 PM Daniel J Sebald wrote:
> In attempting to modify the qt_term.cpp Qt terminal, I ran into some 
> annoying problems with recursive calls, namely enhanced_recursion() and 
> do_event().
> 
> First, let me summarize a few things:
> 
> 1) There are a few uses of the global pointer term->.  My preference 
> would be to remove those, but I understand there needs to be a way to 
> get some information back to gnuplot core.  It seems to me that putting 
> non-const pointers in the API is the best way to do that.  But yes, it 
> is sort of the same difference.

I'm afraid I'm not following you.
A few uses for term->  in what piece of code, exactly?  

> I've managed to move enhanced_recursion() into the emitter code (i.e., 
> same thread as gnuplot core) and that works.  Inelegant, but it works.

Here I'm really lost. enhanced_recursion() is part of the gnuplot core.
It is shared by all terminals.  What does this have to do with qt
in particular?
 
> So, with that, I'll ask if there is some way of redesigning 
> enhanced_recursion() and do_event().  

These have nothing to do with each other, so I don't understand the
question.  Both are part of the core, and are shared by all terminals.

> What is the role of enhanced_recursion()?

This is the routine that interprets enhanced text markup strings.
It's part of the core text processing.  It is called whenever the
core routines want to output a string in enhanced text mode.

> What is the role of do_event()?

do_event() is an asynchronous entry point in the core.
Interactive terminals use it to request some action, e.g.
replot, update mouse coords, zoom, respond to hot-key.

> Is there some way a result can be sent back via the API that indicates 
> to repeat the last event?

Sent from whom to whom?  What sort of event?

Re: enhanced_recursion() and do_event(), can they be redesigned?

[Daniel J S. <dan...@ie...>]

On 02/23/2014 09:26 PM, Daniel J Sebald wrote:

> What is the role of do_event()?
>
> Is there some way a result can be sent back via the API that indicates
> to repeat the last event?

Or, return an event pointer to the API that if zero means the gnuplot 
core should do nothing, but if it is a valid pointer to an event type 
then gnuplot core should issue that event.  That would be much better 
than attempting to call core code from within the terminal and a 
separate thread.

Dan

Re: Reworked qt terminal is much faster – and fails to work on Mac

[Daniel J S. <dan...@ie...>]

On 02/23/2014 09:02 PM, Daniel J Sebald wrote:

> let's just start with OSX first.)  I'm saying that the thread concept is
> necessary, but it doesn't seem straightforward and conceptually it is
> nice having the terminal in a separate thread.

Oy, I'm NOT saying the thread concept is necessary but it DOES seem 
straightforward.

Dan

enhanced_recursion() and do_event(), can they be redesigned?

[Daniel J S. <dan...@ie...>]

In attempting to modify the qt_term.cpp Qt terminal, I ran into some 
annoying problems with recursive calls, namely enhanced_recursion() and 
do_event().

First, let me summarize a few things:

1) There are a few uses of the global pointer term->.  My preference 
would be to remove those, but I understand there needs to be a way to 
get some information back to gnuplot core.  It seems to me that putting 
non-const pointers in the API is the best way to do that.  But yes, it 
is sort of the same difference.

2) By placing the Qt terminal gnuplot_qt interface in a separate thread, 
a mutex/wait is necessary for any terminal function that a) modifies one 
of these global variables via term->, b) accesses anything globally via 
pointer such as a text string.  The reason being that gnuplot core 
cannot modify things that the separate thread is just about to access or 
use anything that the separate thread has yet to modify.  This mechanism 
works fine.  In the cases where it is only objects passed into the 
thread not using pointers, the API call can return immediately because 
all those signals are queued up in the second thread and don't need any 
global access.  So, if this works, I would probably go through and first 
copy the global strings into a QString and just send that via signal to 
the terminal slot.

3) So the global variables aren't so bad, but recursive calls back into 
gnuplot core (by the code in a separate thread) are trouble because that 
is not thread safe.  If the emitter code emits a signal and then sits to 
wait for the slot to signal it has finished and wake up the emitter, a 
do_event() back into the core is going to issue another signal and wait 
a second time.  The first slot call is going to either freeze or timeout 
because potentially the gnuplot core thread is waiting for two different 
things to finish.

I've managed to move enhanced_recursion() into the emitter code (i.e., 
same thread as gnuplot core) and that works.  Inelegant, but it works. 
But I raised the white flag with the do_event() callbacks that was 
coming from mouse code.  I just commented that line out to test things 
(and get some feedback from Mojca.)

So, with that, I'll ask if there is some way of redesigning 
enhanced_recursion() and do_event().  It really doesn't fit the terminal 
concept if term code is behaving that way, i.e., calling code that 
really isn't inside its domain.

What is the role of enhanced_recursion()?

What is the role of do_event()?

Is there some way a result can be sent back via the API that indicates 
to repeat the last event?

Dan

Re: Reworked qt terminal is much faster – and fails to work on Mac

[Daniel J S. <dan...@ie...>]

I've sent some sample code to Jérôme and Mojca to test out.  It isn't 
robust code just yet.  Rather I want to see how things behave on OSX. 
So there is no need for the general group to look at the code just yet.

If the behavior on OSX matches what I'm seeing on Linux, that is good. 
Why should it match?  Because I moved the bulk of the qt terminal inside 
a separate thread where timing should work consistently for all 
platforms.  (Now, whether Windows supports thread, I don't know... but 
let's just start with OSX first.)  I'm saying that the thread concept is 
necessary, but it doesn't seem straightforward and conceptually it is 
nice having the terminal in a separate thread.

Hopefully Jérôme can assess how the layout is and the thread idea.  I 
tried to change as little as possible, but there were some things I had 
to modify a bit with regard to recursive calls back into the gnuplot 
core.  I will follow up with a separate thread.

Dan

Version 4.6.5 released

[sfeam <sf...@us...>]

The source tarball, release notes, and user manual for gnuplot version 4.6.5
are now available on SourceForge.

    <https://sourceforge.net/projects/gnuplot/>

There is testing binary for windows available in the folder
in the files --> gnuplot --> "Testing (pre-release) binaries"

Release announcement appended below.

	Ethan Merritt (sfeam) - gnuplot development team



		   GNUPLOT VERSION 4.6.5
	    ===================================

This is an incremental release of gnuplot version 4.6.
A short list of changes since the previous patchlevel (version 4.6.4)
is given below and in the NEWS file.  Detailed information is in ChangeLog.

New features, changes and fixes since gnuplot version 4.6.4
===========================================================

* NEW monotonic cubic splines using "smooth mcsplines"
* NEW phase-jump removal filter "smooth unwrap"
* NEW allow '+' pseudofile to sample the T axis in 2D parametric plots
* NEW allow '++' pseudofile to sample the U/V axes in 3D parametric plots
* NEW "sixel" terminal driver
* NEW new object attribute clip/noclip
* CHANGE maximum number of using spec columns increased from 7 to 11
* CHANGE code in bitmap.c relicensed to remove restriction to noncommercial use
* FIX allow 'set pm3d' interpolate and top/bottom options to coexist
* FIX revised handling of defined palettes with explicit maxcolors
* FIX continue as normal after an interactive session error from "gnuplot -" 
* FIX empty first field in a tab-separated-values file was incorrectly ignored
* FIX several problems with color assignment to contour lines
* FIX qt terminal incorrectly changed linetype (dot/dash) to match line color
* FIX "pause mouse" worked only for right- or center- click, not left-click
* FIX emf terminal font initialization
* FIX wxt terminal vertical centering of enhanced text
* FIX win terminal filled polygon bugs
* FIX iteration over parametric function plots
* FIX autoscaling of polar mode plots
* FIX increase precision of xticlabel placement from (float) to (double)
* FIX allocation error affecting certain cvs files


		 NOTES TO PACKAGERS AND TESTERS 
		===============================

Configuration options for interactive use
-----------------------------------------

The 4.6 source code supports three primary cross-platform output modes
in addition to several platform-specific modes.

1) Cairo/pango/wxWidgets
   These terminals were introduced in version 4.4 and are now the most
   stable and full-featured option.  This set of terminals includes
   - pngcairo, pdfcairo, epscairo, and cairolatex for output to a file
   - wxt for interactive display
   This is the default configuration, but requires prior installation of
   libcairo, libpango, libcairo, libwxgtk, and related support libraries
   To disable these terminals:
       ./configure --disable-wxt --without-cairo

2) Qt
   The new qt terminal supports interactive display with menu-driven
   output to png, svg or pdf.
   Requires libqt version >= 4.5
       ./configure --enable-qt

3) X11 (the "classic" interactive interface)
   This used to be the preferred interactive interface, but the newer
   wxt and qt terminals offer nicer output and a wider range of features.


Options for output to files
---------------------------

Of course the terminals (output modes) present in previous gnuplot versions
are also still available. These include, among many more obscure options:
- png/jpeg/gif output via libgd
- PostScript
- Many flavors of TeX/LaTeX output, including TikZ and ConTeXt (new)
- Bitmapped output to support many older devices (e.g. HP deskjet, epson,
  seiko printers, pbm bitmapped graphics files) is available if needed
  but is no longer configured in by default.
      ./configure --with-bitmap-terminals

Options for generating interactive plots for web display
--------------------------------------------------------

- Mouseable output for display on the web can be created using either 
  the canvas terminal (HTML5 2D canvas element) or the svg terminal.
  Both allow zooming, toggling plot elements on/off, and user-scriptable
  hot keys.

Online demo plots
-----------------

Demo plots illustrating new and old features are online at

	http://gnuplot.sourceforge.net/demo/


		          OTHER NOTES
		===============================

Installation
------------

You can download a source tarball for gnuplot version 4.6.5 from the 
gnuplot development site on SourceForge.

	http://sourceforge.net/project/showfiles.php?group_id=2055

Installation instructions are available in the source itself; the short
version for linux/unix-like systems is to unpack the tarball and then
build it:
		cd gnuplot-4.6.5 ; ./configure ; make
test it:
		make check
install it:
		make install

Pay careful attention to the output of the ./configure script.
It may indicate that some output drivers have been omitted because the
necessary support libraries were not found.  In general you need to have
previously installed the "*-devel-*" versions of these libraries. 


Known issues
------------

- Mac OSX ships with a terminal input library that appears to be GNU
libreadline, but isn't really.  The program tries to cope with this, but
you may get better results by configuring gnuplot to use either its own
built-in readline routines or the real GNU libreadline.

- The gnuplot build system is not very good at figuring out where to find
or install LaTeX-related files.  This can affect use of the new lua/tikz
and ConTeXt terminals.

- You can configure support for both wxt and qt into the same gnuplot
executable, but only one of these two output modes can be used in any
given gnuplot session.


Support
-------
Please report all bugs and installation problems to the bug tracker
on SourceForge:

	http://sourceforge.net/tracker/?group_id=2055&atid=102055

There is also an gnuplot discussion forum on usenet group

	comp.graphics.apps.gnuplot


Development
-----------
Gnuplot development is quite active. The development branch on SourceForge
contains many new features.  The current development branch is labeled 
version 4.7, but will probably be released as version 5.
Feedback and contributions of code are very welcome.

Re: Reworked qt terminal is much faster – and fails to work on Mac

[Bastian M. <bma...@we...>]

Am 23.02.2014 04:46, schrieb Daniel J Sebald:
> On 02/22/2014 11:45 AM, Jérôme Lodewyck wrote:
>> I think it is fine to create the process as non-detached. We should just make
>> sure a new process is started (and without leaking memory) if the previous one
>> is killed.
>
> Oh yeah; gnuplot_qt could stop somehow.  I'll watch for that.
>
>
>>   Also, for the -persist option to work, the process should not be
>> deleted at exit.
>
> OK, good.  I'll try to figure out how to change a QProcess to persistent.

Hhhm. I am currently unaware of a Windows API to change a process to the 
"detached" state. If there is indeed such a thing, this could also help 
to finally implement -persist properly on Windows for all interactive 
terminals.

   Bastian


>
> Thanks,
>
> Dan
>
>
>>
>> Jérôme
>>
>> Le samedi 22 février 2014 10:55:39 Daniel J Sebald a écrit :
>>> It's fairly simple, really.  But first, I'd like to ask if it is alright
>>> to create the process not as detached, but just a normal process.  Is
>>> that OK?  If that is done, then after starting the QProcess that will
>>> create the local server, just wait for the process to write back the
>>> server's name and continue onward.  By writing the local server name to
>>> standard output (fprint/fflush) when everything is set up and ready to
>>> go, the first time that qt_term attempts to connect to the local server
>>> will be a success.
>>>
>>>
>>> IN QT_TERM.CPP:
>>>
>>>        if (!qtgnuplotProcess)
>>>            qtgnuplotProcess = new QProcess();
>>>        qtgnuplotProcess->start(filename, QStringList());
>>>        if (qtgnuplotProcess->waitForStarted())
>>>        {
>>>            qtgnuplotProcess->waitForReadyRead();
>>>            localServerName = qtgnuplotProcess->readAllStandardOutput();
>>>            qDebug()<<  localServerName;
>>>        }
>>>        else
>>>        {
>>>            qDebug()<<  "QProcess starting failed:"<<
>>> qtgnuplotProcess->errorString();
>>>        }
>>>
>>>
>>> IN QTGNUPLOTAPPLICATION.CPP:
>>>
>>> QtGnuplotApplication::QtGnuplotApplication(int&  argc, char** argv)
>>>
>>> 	: QApplication(argc, argv)
>>>
>>> {
>>> ...
>>>        connect(m_eventHandler, SIGNAL(disconnected()), this,
>>> SLOT(enterPersistMode()));
>>>
>>>        fprintf(stdout, sName.toLocal8Bit().data());
>>>        fflush(stdout);
>>> }
>>>
>>> Dan
>>
>>
>

Re: Reworked qt terminal is much faster – and fails to work on Mac

[Daniel J S. <dan...@ie...>]

On 02/22/2014 11:45 AM, Jérôme Lodewyck wrote:
> I think it is fine to create the process as non-detached. We should just make
> sure a new process is started (and without leaking memory) if the previous one
> is killed.

Oh yeah; gnuplot_qt could stop somehow.  I'll watch for that.


>  Also, for the -persist option to work, the process should not be
> deleted at exit.

OK, good.  I'll try to figure out how to change a QProcess to persistent.

Thanks,

Dan


>
> Jérôme
>
> Le samedi 22 février 2014 10:55:39 Daniel J Sebald a écrit :
>> It's fairly simple, really.  But first, I'd like to ask if it is alright
>> to create the process not as detached, but just a normal process.  Is
>> that OK?  If that is done, then after starting the QProcess that will
>> create the local server, just wait for the process to write back the
>> server's name and continue onward.  By writing the local server name to
>> standard output (fprint/fflush) when everything is set up and ready to
>> go, the first time that qt_term attempts to connect to the local server
>> will be a success.
>>
>>
>> IN QT_TERM.CPP:
>>
>>       if (!qtgnuplotProcess)
>>           qtgnuplotProcess = new QProcess();
>>       qtgnuplotProcess->start(filename, QStringList());
>>       if (qtgnuplotProcess->waitForStarted())
>>       {
>>           qtgnuplotProcess->waitForReadyRead();
>>           localServerName = qtgnuplotProcess->readAllStandardOutput();
>>           qDebug()<<  localServerName;
>>       }
>>       else
>>       {
>>           qDebug()<<  "QProcess starting failed:"<<
>> qtgnuplotProcess->errorString();
>>       }
>>
>>
>> IN QTGNUPLOTAPPLICATION.CPP:
>>
>> QtGnuplotApplication::QtGnuplotApplication(int&  argc, char** argv)
>>
>> 	: QApplication(argc, argv)
>>
>> {
>> ...
>>       connect(m_eventHandler, SIGNAL(disconnected()), this,
>> SLOT(enterPersistMode()));
>>
>>       fprintf(stdout, sName.toLocal8Bit().data());
>>       fflush(stdout);
>> }
>>
>> Dan
>
>

-- 

Dan Sebald
email: daniel(DOT)sebald(AT)ieee(DOT)org
URL: http://www(DOT)dansebald(DOT)com

Re: Reworked qt terminal is much faster – and fails to work on Mac

[Jérôme L. <lod...@us...>]

I think it is fine to create the process as non-detached. We should just make 
sure a new process is started (and without leaking memory) if the previous one 
is killed. Also, for the -persist option to work, the process should not be 
deleted at exit.

Jérôme

Le samedi 22 février 2014 10:55:39 Daniel J Sebald a écrit :
> It's fairly simple, really.  But first, I'd like to ask if it is alright
> to create the process not as detached, but just a normal process.  Is
> that OK?  If that is done, then after starting the QProcess that will
> create the local server, just wait for the process to write back the
> server's name and continue onward.  By writing the local server name to
> standard output (fprint/fflush) when everything is set up and ready to
> go, the first time that qt_term attempts to connect to the local server
> will be a success.
> 
> 
> IN QT_TERM.CPP:
> 
>      if (!qtgnuplotProcess)
>          qtgnuplotProcess = new QProcess();
>      qtgnuplotProcess->start(filename, QStringList());
>      if (qtgnuplotProcess->waitForStarted())
>      {
>          qtgnuplotProcess->waitForReadyRead();
>          localServerName = qtgnuplotProcess->readAllStandardOutput();
>          qDebug() << localServerName;
>      }
>      else
>      {
>          qDebug() << "QProcess starting failed:" <<
> qtgnuplotProcess->errorString();
>      }
> 
> 
> IN QTGNUPLOTAPPLICATION.CPP:
> 
> QtGnuplotApplication::QtGnuplotApplication(int& argc, char** argv)
> 
> 	: QApplication(argc, argv)
> 
> {
> ...
>      connect(m_eventHandler, SIGNAL(disconnected()), this,
> SLOT(enterPersistMode()));
> 
>      fprintf(stdout, sName.toLocal8Bit().data());
>      fflush(stdout);
> }
> 
> Dan

Re: Reworked qt terminal is much faster – and fails to work on Mac

[Daniel J S. <dan...@ie...>]

On 02/16/2014 05:47 PM, Daniel J Sebald wrote:

> I've a good start on the separate thread for Qt. I think it is going to
> work, but it will probably be next weekend until I finish. I just don't
> have the time right now. So Mojca, you'll have to wait a week I guess.
>
> It's been pretty straightforward up to this point. I just laid things
> out the way the Qt documentation shows. The separate thread starts and
> now timers definitely work as expected. I have signals going across
> without much problems. The external program gnuplot_qt shows up. There
> are two remaining issues:
>
> 1) Timing. Since this is signaling into a thread rather than calling,
> the signal returns right away and gnuplot core continues onward not
> having valid values for settings (i.e., x, y dimensions, etc.). While
> that goes on, the other thread has problems and 30 seconds later
> complaints show up in qDebug(), and 30 seconds later once again, and 30
> seconds later is the last try. I know how to fix this. It just takes a
> QMutex and putting it to sleep and have the object in the QThread wake
> it when it has finished or timed out. I've done it before.
>
> 2) Reorganization. I think the number of ways connectToServer tries to
> connect can be simplified, without recursion. The complaints from Qt are
> along the lines of some child objects being created in a thread
> different from its parent. Shouldn't be difficult to fix with some
> better reorganization. I've already broken the Qt class declarations
> into individual files QtTerminalEmitter.h (same main thread as gnuplot)
> and QtTerminalInterface.h (separate thread), which is similar to the
> rest of the Qt code in the qtterminal directory.
>
> More next weekend or later in the week.

Things are slowly coming along, but I've become rather busy with a 
number of projects at work.  I would like to point out that I have a new 
way of waiting for the gnuplot_qt server to be active that avoids that 
clumsy while loop.  Here's a brief summary if Ethan wants to try and 
tweak the CVS repository.  Otherwise, what I will have is a fairly big 
diff file in the coming week.

It's fairly simple, really.  But first, I'd like to ask if it is alright 
to create the process not as detached, but just a normal process.  Is 
that OK?  If that is done, then after starting the QProcess that will 
create the local server, just wait for the process to write back the 
server's name and continue onward.  By writing the local server name to 
standard output (fprint/fflush) when everything is set up and ready to 
go, the first time that qt_term attempts to connect to the local server 
will be a success.


IN QT_TERM.CPP:

     if (!qtgnuplotProcess)
         qtgnuplotProcess = new QProcess();
     qtgnuplotProcess->start(filename, QStringList());
     if (qtgnuplotProcess->waitForStarted())
     {
         qtgnuplotProcess->waitForReadyRead();
         localServerName = qtgnuplotProcess->readAllStandardOutput();
         qDebug() << localServerName;
     }
     else
     {
         qDebug() << "QProcess starting failed:" << 
qtgnuplotProcess->errorString();
     }


IN QTGNUPLOTAPPLICATION.CPP:

QtGnuplotApplication::QtGnuplotApplication(int& argc, char** argv)
	: QApplication(argc, argv)
{
...
     connect(m_eventHandler, SIGNAL(disconnected()), this, 
SLOT(enterPersistMode()));

     fprintf(stdout, sName.toLocal8Bit().data());
     fflush(stdout);
}

Dan

Re: Repository conversion

[Achim G. <Str...@ne...>]

Eric S. Raymond writes:
> Something unfortunate is going on with my list subscription.  Mailman
> tells me I'm subscribed, but I'm not seeing list mail; I'm having
> to read this thread through the archive interface.

You can switch off (and back on) delivery to your address seperately on
the subscription management page.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Repository conversion

[<es...@th...>]

Something unfortunate is going on with my list subscription.  Mailman
tells me I'm subscribed, but I'm not seeing list mail; I'm having
to read this thread through the archive interface.

Ethan A Merritt writes:
>Can you summarize in what ways your conversion would be different
>from the automated cvs->git conversion service offered by SourceForge
>(which so far the project has declined to use)?

Before going into the details, I should make clear that I did not
specify a target VCS in my offer because I can up-convert to either
git or hg very easily - gnuplot can choose either.  I retain a
lingering preference for hg myself, but I advise going with git 
anyway - it presents the lowest entry to the largest number of
potential contributors.

I don't know what SF uses for its conversions; the service is a recent
invention and Google isn't turning up documentation on it for me.
There aren't many possibilities - maintained and documented tools for
this purpose are thin on the ground. I fear they are probably using
cvsps called by git-cvsimport, because that is what the git suite
ships.

Unfortunately, cvsps is so irreparably broken that I, acting as its last
maintainer, end-of-lifed it a couple of months ago. Mojca Miklavec can
confirm what a mess it is; she tried doing  a test conversion and got
horrible results.  The git devs don't realize how bad the situation
is.

Through an odd set of circumstances, last year I wound up as the
maintainer of both cvsps and Keith Packard's parsecvs code, which I
rewrote significantly and renamed cvs-fast-export.  After EOLing
cvsps, I shifted my effort to cvs-fast-import, which is what Mojca
Miklavec has been doing her most recent test conversions with.

cvsps and cvs-fast-export are two of only three possibilities SF
might be using.  The third is cvs2git, which has some problems of
its own that I am trying to help its maintainer fix.  

All three of these tools, by themselves, are too weak to produce 
a really high-quality conversion.  The things they don't do include:

1. Lifting CVS version references in comments into a form that
will still be usable in git or hg.

2. Complete coalescence of CVS change cliques into changesets.  They
often only do this partially, one reason being that the default
merge window is set too low.

3. Mapping of .cvsignores to .gitignores.  My newer versions of
cvs-fast-export do this but neither cvs2git nor the cvsps version in
the git suite does.

4. Cleaning up conversion artifacts and junk branches. I can't be
more specific about this in advance because each CVS repo tends to
have its own unique set of strange malformations.  Fortunately, 
gnuplot's history seems exceptionally clean and I expect relatively
few problems.

5. (Optional) Canonicalizing change comments to git summary +
continuation form.

Completing these tasks requires human judgment. The only way to get a
really good conversion is to follow up application of a batch
converter with skilled hand-editing, and reposurgeon is the only
existing tool for that.  It's an interpreter for a domain-specific
language built around repository-editing primitives. 

(As previously noted, I am currently in the process of lifting Emacs's
history from bzr to git with reposurgeon.  That is an exceptionally
large and messy conversion, far more complex than I expect gnuplot's
to be.)

I have done about half a dozen large conversions before; the most
recent completed one was GNU troff.  You can read about my procedures
in the DVCS Migration HOWTO:

   http://www.catb.org/esr/dvcs-migration-guide.html

Ideally, I team with an inside developer who knows the project's
history and idiosyncracies and is highly motivated to make the
conversion succeed.  For this project, Mojca Miklavec has claimed that
role - she has already supplied one of the key pieces of metadata, a
map from developers' CVS usernames to full names and email addresses.

If the project elects to go ahead, our work product would be a lift
script - that is, a file containing surgical instructions to
reposurgeon - which would be available for review before the actual
conversion day.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Where rights secured by the Constitution are involved, there can be no
rule making or legislation which would abrogate them.
        -- Miranda vs. Arizona, 384 US 436 p. 491

Re: Offering help with repository conversion

[Ethan A M. <sf...@us...>]

On Wednesday, 19 February, 2014 10:43:49 Eric S. Raymond wrote:
> I maintain a tool called reposurgeon that edits and convers
> version-control repositories, and a front end for it called
> cvs-fast-export that does what its name says.
> 
> Recently a gnuplot contributor attempting to set up a local git mirror of
> the gnuplot repository for his own use approached me with a bug report
> about cvs-fast-export which I was able to resolve. In the process I
> ran a test conversion of the gnuplot history to git.
> 
> I am therefore in a position to state that converting your production
> repository out of CVS into a modern distributed VCS would not be
> very difficult. If I had an inside collaborator familiar with the
> project history I think it would be about a week's work.  Most of that 
> effort would be lifting CVS version references in comments into
> a useful form.

You are kind to offer.
Can you summarize in what ways your conversion would be different
from the automated cvs->git conversion service offered by SourceForge
(which so far the project has declined to use)?

	Ethan