Re: [Plplot-devel] Status report on remaining issues to be addressed for the forthcoming 5.11.1 release (wxwidgets issues)

[Phil R. <p.d...@gm...>]

Hi Alan - this is now slightly out of context as I found it in my
drafts folder unsent.
I can confirm similar odd timing results on my Centos machine. I did
have a couple of exit() calls in wxPLViewer, which to be honest was
just lazy of me. On was used to close the app when -np was used. I
have now replaced these both with clean exit methods. this ensures all
destructors are called and I can now say that at least as far as I can
tell on Windows the shared memory is _always_ released. It is
certainly released when the window is closed using -np. This is on the
viewer side. I presume the use of -np still gives a clean exit on the
console side and doesn't invoke any exit calls anywhere?

Unfortunately the odd timings still persist. Note that the application
only asks for 1 MB of shared memory so even if not freed I wouldn't
have thought it would slow anything down really after just one run.

I haven't got time right now to set up a decent debugging environment
on my Centos machine, but I will try to double check the shared memory
is released on that system as soon as I can.

Phil

On 3 July 2015 at 20:32, Alan W. Irwin <ir...@be...> wrote:
> To Phil and Andrew:
>
> @Andrew: I am addressing you directly (in addition to Phil) because I
> have questions for you below concerning whether you can replicate my
> weird timing results for wxwidgets and also concerning the issue of
> shared memory leaks on Linux.
>
>
> On 2015-07-03 13:53+0100 Phil Rosenberg wrote:
>
>> Hi Alan
>> So I think you should now find the wxPLViewer speed has significantly
>> improved. The item I have dealt with now was that there is a timer at
>> the OS level (initiated by wxWidgets) which sends a message to the
>> viewer which initiates checking for new commands from the console app.
>> At this point I used to check for a single message and return after
>> dealing with it. However it seems that for whatever reason (different
>> OS, different wxWidgets version?) on some systems the minimum time
>> between the timer calls was quite long. This meant that when we were
>> dealing with text which required a lot of small transmissions to the
>> viewer there was a huge overhead. I have now modified things so that
>> on every timer event I check for new transmissions at least 100 times
>> with a 1 millisecond sleep between each. This means that for many
>> small transmissions in a short time we remove most of the overhead.
>>
>> On my CentOS machine the execution time reported by time x08c -dev
>> wxWidgets -np has dropped from 40s to about 2 s. Note that this only
>> times the console part, not the viewer, but still we have clearly
>> moved to something I would call acceptable.
>>
>> Note that there will always be a significant overhead for the
>> transmission so this will never compete with Cairo, but this is
>> unavoidable because wxWidgets cannot be run in a library without
>> running it as a separate thread and PlPlot isn't thread safe. Note
>> also that when using the wxWidgets driver via the wxWidgets binding
>> (I.e. from within a wxWidgets app) this all becomes irrelevant as the
>> viewer is not used.
>>
>> Anyway I hope this fix closes the issue and you are happy Alan.
>
>
> @Phil:
>
> The timing results for current master tip (commit id 5e74b6a6,
> "Modification to previous wxPLViewer optimisation") are much improved
> on first execution of examples.  So big congratulations on that result!
>
> However, an important issue still remains as shown by the following
> repeated timing results for both xcairo (as a typical benchmark) and
> wxwidgets:
>
> software@raven> time examples/c/x00c -dev xcairo -np
>
> real    0m0.168s
> user    0m0.008s
> sys     0m0.016s
> software@raven> time examples/c/x00c -dev xcairo -np
>
> real    0m0.129s
> user    0m0.020s
> sys     0m0.004s
> software@raven> time examples/c/x00c -dev xcairo -np
>
> real    0m0.114s
> user    0m0.020s
> sys     0m0.008s
>
> software@raven> time examples/c/x00c -dev wxwidgets -np
>
> real    0m0.365s
> user    0m0.008s
> sys     0m0.012s
> software@raven> time examples/c/x00c -dev wxwidgets -np
>
> real    0m1.231s
> user    0m0.008s
> sys     0m0.012s
> software@raven> time examples/c/x00c -dev wxwidgets -np
>
> real    0m5.587s
> user    0m0.008s
> sys     0m0.012s
>
> So the xcairo case has subsequent time results that are up to 1.5
> times faster (in real time which is the most important component) then
> the first execution of the example while the wxwidgets case has
> subsequent time results that are up to 15 (!) times slower.  The
> initial fast wxwidgers result seems guaranteed if you wait a minute or
> so since your last attempt to use wxwidgets.  I mostly tested the
> above wxwidgets time results with the -np option (for convenience),
> but I also tried the test a few times without -np and got the fast,
> slow, slow,... pattern for that case as well.  Also, I usually get a
> consistent fast, slow, slow,.... pattern for wxwidgets, but just now I
> tried it again, and I got the fast result for something like 10 times
> in a row, then the slow result. So the wxwidgets inconsistency in time
> results is sometimes inconsistent. :-)
>
> The above results for the xcairo case are quite typical of all
> non-wxwidgets cases I have ever investigated before, where real time
> results are significantly shorter on second and subsequent
> execution because of the well-known effect on Linux of the system
> caching all reasonably small files in memory to improve small file
> access times for subsequent use.
>
> The very unusual much longer times that often but not always
> immediately appear on second and subsequent executions that occur for
> the wxwidgets case above strongly suggest to me a system resource is
> not being properly released by the first execution of the example so
> getting that resource back again often takes a lot of extra time on
> the second and subsequent runs.
>
> I brought up this issue with you off list much earlier this week, but
> you did not respond at that time, but I hope you do that now.  The
> first step in your response should be to try the above bash commands
> for at least the wxwidgets case on your Windows and Linux platforms.
> to (a) see if this issue also occurs on Windows, and (b) see if this
> issue occurs on all Linux systems accessible to you.
>
> @Andrew: could you please try the above time tests as well
> on your Linux platforms?
>
> @Both:
>
> If the issue is a Linux resource which I am chronically short of
> because of my prior wxwidgets testing and a shared memory leak (see
> discussion below) in wxwidgets, then you might not be able to
> replicate the issue on Linux until you do sufficient wxwidgets
> testing.  My Linux up time is currently 42 days, and if I reboot
> (which I won't do lightly because there are two users on the system
> and it is a bit of a pain for us to reset all desktops the way we like
> them), I might find the issue goes away for a while.
>
> @Phil:
>
> If the issue is a Linux-only one, then an obvious candidate for a
> system resource grabber is the shm_open calls in
> drivers/wxwidgets_comms.cpp that helps to allow shared memory access
> between the wxwidgets device driver and wxPLViewer.  I have just now
> read the shm_overview and shm_open/shm_unlink man pages, and it
> appears that shm_unlink must be called by _both_ the wxwidgets device
> driver and wxPLViewer to properly release the shared memory.  So a
> code review for those two cases to make sure that _always_ happens
> (i.e., there is no shared memory leak) is indicated.
>
> @Andrew: will you comment further here please about whether we should
> be concerned about possible shared memory leaks?  I now realize, for
> example, that both Phil's and my calling the IPC method that is used
> "shared memory" is likely not specific enough because it does not
> distinguish the old-fashioned shmget method (which we do not use) with
> the modern mmap-based method that we do use.  See
> <http://stackoverflow.com/questions/5656530/how-to-use-shared-memory-with-linux-in-c>
> for comments on this important "shared-memory" distinction. Furthermore,
> from the discussion at
> <http://stackoverflow.com/questions/22691621/how-to-avoid-shared-memory-leaks>
> it appears the mmap-based method cannot actually leak memory.  Is that
> your interpretation of those remarks as well?
>
> @Phil:
>
> One thing I noticed from reading those man pages is it is suggested
> the name used for shm_open should always start with a "/" for the most
> portable results.  It appears our current wxwidgets code does not
> follow that suggestion so this may affect Unix platforms that are not
> Linux.  So I suggest you will want to address that issue (even though
> it has nothing to do with timing.
>
>
> Alan
> __________________________
> Alan W. Irwin
>
> Astronomical research affiliation with Department of Physics and Astronomy,
> University of Victoria (astrowww.phys.uvic.ca).
>
> Programming affiliations with the FreeEOS equation-of-state
> implementation for stellar interiors (freeeos.sf.net); the Time
> Ephemerides project (timeephem.sf.net); PLplot scientific plotting
> software package (plplot.sf.net); the libLASi project
> (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
> and the Linux Brochure Project (lbproject.sf.net).
> __________________________
>
> Linux-powered Science
> __________________________