plplot-devel — Technical discussion about PLplot development

Re: [Plplot-devel] Fwd: [Bug 1295175] New: plplot contain problematic content

[Alan W. I. <ir...@be...>]

On 2016-01-04 09:08-0000 Phil Rosenberg wrote:

> To add to my last email, I just googled for compatibility between CC
> and LGPL - most of the resources I found were actually regarding GPL,
> not LGPL, but apparently they are incompatible - I think mostly
> because CC requires attribution
> http://programmers.stackexchange.com/questions/272335/how-and-when-had-the-cc-by-license-become-gnu-gpl-compatible
>
> Anyway, I think if we use a CC image, we must specify that the image
> is not LGPL and is in fact CC.

example 20 is an LPGLed combined work (consisting of LGPL application
and LGPL library), the image that example 20 reads is simply a data file, It seems
to me that any licensed software could read data that is licensed in
any way since software and some arbitrary datafile it reads are not a combined
work. For example, the GVPL'd gv application displays PostScript files
regardless of their license, free software browsers display web
content regardless of that content's license, etc., etc.

So my assumption is the discussion you referred to was considering the
possibility of combining software licensed under the GPL or LGPL with
_other software_ licensed under the CC license.  But that is a very
different situation than software reading a simple data file.

Assuming that legal theory is correct, then the only question for the
data file is do we have the right to distribute it in modified (PPM,
gray-scale) form?  And the CC license for the data file gives us that
right assuming we note the licensing terms for that image file in our
Copyright file.

Alan

__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________

[Plplot-devel] Fwd: [Bug 1295175] New: plplot contain problematic content

[Orion P. <or...@co...>]

This does look like a problem, unless there is specific permission for 
this file?


-------- Forwarded Message --------
Subject: [Bug 1295175] New: plplot contain problematic content
Date: Sun, 03 Jan 2016 08:08:59 +0000
From: bug...@re...
To: or...@co...

https://bugzilla.redhat.com/show_bug.cgi?id=1295175

             Bug ID: 1295175
            Summary: plplot contain problematic content
            Product: Fedora
            Version: rawhide
          Component: plplot
           Assignee: or...@co...
           Reporter: kam...@y2...
         QA Contact: ext...@fe...
                 CC: or...@co...



Hello.

plplot included non-free image.
This is "Lena" (Lenna) image. (PGM and IMG file)

This file license (Copyright) is non-free, and this content is violate 
Fedora
Packaging Guideline.

https://fedoraproject.org/wiki/Packaging:Guidelines#Code_Vs_Content

Thanks.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.

Re: [Plplot-devel] Linux wxwidgets idle issue not solved for text- or line-dominated plots

[Alan W. I. <ir...@be...>]

On 2016-01-02 23:51-0000 Phil Rosenberg wrote:

> The text dominated case idle issues doesn't surprise me. As I said
this requires 2 way comms, which is does involve both executables
sitting in while sleep loops. I think I can fix this and I will look
to do so asap. The line issue surprises me a bit. I will try to look
at the cause.

To help you narrow down what the Linux issues are I just tested a pure
ascii test case (the last 6 pages of example 23) and a pure line case
(example 00 with the pllab call commented out, and the last argument
to plenv set to -2 so absolutely no text is plotted at all by that
example).

and here are those results:

Number          cpu%  idle%  type of example

modified 23     25%   75%    pure ascii text with no graphics at all
modified 00     20%*  80%*   pure line graphics with no text

So whatever the idle problem is seems to be roughly the same for pure text
or pure line graphics.

Furthermore, I put an asterisk by that second result because the first
time I ran test_c_wxwidgets with 30 repetitions of the modified 00
example in plplot_test/test_c_interactive.sh.in, the cpu usage was
effectively zero (at the background level) for each of the first ~25
repetitions or so.  Essentially for each of these repetitions there
would be a blip of cpu activity at either the start or end of the
repetition but nothing else discernible on the graph for the
relatively long times it took to finish each repetition.  Then for the
last 5 repetitions the idle issue went from completely horrible to
merely bad (i.e., the idle% dropped from ~99 % to the 80% I state
above).  And then it stayed in that "bad" state for the next 30
repetitions I tried.

Anyhow, because that modified example 00 is so simple with no text at all,
that is the example I suggest you should investigate first for the cause of
this clear idle issue on Linux that varies between completely horrible
to just merely bad.  :-)

And if you cannot replicate that modified 00 idle issue on Windows,
then it is likely time to attempt to verify it on the Ubuntu Linux box
you have access to.

Alan
__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________

Re: [Plplot-devel] Linux wxwidgets idle issue not solved for text- or line-dominatedplots

[Phil R. <p.d...@gm...>]

Hi Alan. Just a quick reply, sorry for top posting, but I'm replying on my phone.

The text dominated case idle issues doesn't surprise me. As I said this requires 2 way comms, which is does involve both executables sitting in while sleep loops. I think I can fix this and I will look to do so asap. The line issue surprises me a bit. I will try to look at the cause.

Phil

-----Original Message-----
From: "Alan W. Irwin" <ir...@be...>
Sent: ‎02/‎01/‎2016 23:17
To: "Phil Rosenberg" <p.d...@gm...>
Cc: "PLplot development list" <Plp...@li...>
Subject: RE: Linux wxwidgets idle issue not solved for text- or line-dominatedplots

On 2016-01-02 16:28-0000 Phil Rosenberg wrote:

> I have reproduced your timings on windows with the command
> time ./x20.exe -dev wxwidgets -np. This gives me a similar ~23s real
time and a fraction of a second user and sys time. Note that this is
run from Cygwin bash shell, but x20.exe is built using visual studio.

Example 20 is not the one I measured since that is a special case in
many ways.  Instead, I ran the test_c_wxwidgets target which runs a
list of C examples for

-dev wxwidgets -np

That list of examples is given in
plplot_test/test_c_interactive.sh.in. For my previous tests I locally
modified that list here to be "01 04 08 14 16 24 30" for all devices,
but you can locally modify it to be any list of examples you want. For
example, you could modify it to "01 01 01 01 01 01 .... " to get good
timing numbers for a short example such as example 1, see below. Also,
your system will have very different speed than mine.  So what is
important is not comparisons with my numbers, but instead what
fraction of time is spent with _both_ example and wxPLViewer
application being idle (i.e., both in wait mode) compared to the
fraction of time when one of them is using the cpu.  This "idle" issue
is a substantial problem for the Linux case, and it is currently
unknown whether that is a substantial problem for the Windows case
until you do the equivalent of the Linux experiments I did below.

Note, I have now concluded you cannot reliably estimate the fraction
of time in the idle state using the Linux "time" command (or
equivalent Windows command-line app) because it only gathers time
statistics for the example it is given and obviously cannot account,
(say) for separate processes such as wxPLViewer. (So that makes its
user and system cpu estimates too small because they do not include
the wxPLViewer numbers for those, but the real time is reliable (as
also measured by a nearby clock) because wxPLViewer exits before the example
exits.)

So I redid the above test for the above list of examples, and found
the overall time spent on -dev wxwidgets + wxPLViewer is still an
order of magnitude larger than for the -dev qtwidget case (29 seconds
versus 3.6 seconds).  Some of that could be that the Linux wxwidgets
library might just be slower that the Linux Qt library. But I think
that ratio is unlikely to be such a large factor because ultimately
both libraries simply organize calls to the X library which does all
the real work on Linux.  Furthermore, while I was running those
comparisons I looked at a GUI of overall cpu usage, and for -dev
qtwidget, the cpu usage was 100 per cent during the time of the test.
However, for the combination of -dev wxwidgets + wxPLViewer the total
cpu usage varied markedly from example to example. So from this result
it appears that some examples have a mix of PLplot commands that don't
trigger idle issues on Linux, but many others do.

So I looked further at GUI cpu usage results (with the GUI expanded to
the whole page so I could look at results with good resolution) for
repeated (i.e., 01 01 01, etc.) examples for -dev wxwidgets with the
following results for average total cpu usage percentage (these
numbers are roughly good to +/- 05 per cent since I am using an
average value estimated by eye from a graph and subtracting a ~5 per
cent baseline for what the average graph level looks like when the
example is not running.  I presume this background level I subtract is
due to the GUI itself taking some CPU time to display the results as a
continously upgraded graph).  Also, note I have two cpus and for most
cases the plot showed the OS distributed the load 50-50 between them
so the numbers below refer to the average individual cpu usage for
each cpu.  i.e., 100 per cent minus the numbers tabulated below is
what I state in the 3rd column which is the fraction of the time the
entire system was completely idle (everything in wait state) while
running these PLplot examples.

example cpu %  idle %   dominant type of graphics

00      30 %   70 %     lines
01      30 %   70 %     lines
02      30 %   70 %     lines
03      25 %   75 %     lines
04      25 %   75 %     lines
05      20 %   80 %     lines
06      20 %   80 %     text
07      15 %   85 %     text
08      90 %   10 %     fills
15      40 %   60 %     lines
16      75 %   25 %     fills

For most of these examples I also looked at detailed cpu usage graphs
for -dev xwin and -dev qtwidget and they were all similar, i.e., above
90 per cent cpu usage with essentially no idle issue at all.

So it is pretty clear from the above wxwidgets Linux results there is
a severe idle issue for text, still fairly severe for lines, and
essentially no issue at all for fills (or else the issue is swamped by
some other wxwidgets fill inefficiency since in the case of repeated example 8,
-dev xwin, qtwidget, and qtwidgets took a total real time of 6, 5, and
35 seconds.  And the corresponding numbers for repeated example 16 were 9, 7,
and 26 seconds.  Note also that the example 20 you have been looking
at also may not have a perceptible "idle" issue because that example
is virtually all fills as well.

Assuming you can get access to a Windows GUI that plots total cpu usage as a
function of time, could you run similar tests to the above for a
selection of text, line, and fill examples for -dev wxwidgets and -dev
wingcc?  Typical ratios of total time taken for wxwidgets versus wingcc
for some/all of the examples would be useful as well.

> I tried quite hard to locate the source of the delay. There are a
few sleep calls while I have to wait for the viewer.

It only takes one sleep call in a loop to make both example and
wxwidgets idle a large fraction of the time....

> I tried the following tests to try to isolate the problem. With
every wxWidget driver callback returning immediately so the driver
does nothing, I still get a real time of ~5s. So even when the
wxWidgets driver does nothing there is still some unaccounted for
time.

Example 20, has some overhead due to reading in the Lena file so that
is an atypical example I would avoid from that regard.  Also, it
apparently is fill-dominated so if you are trying to deal with the
idle issue, that is not the example to use from my result about
that fill-dominated examples don't have perceptible idle issues.

> If I allow initialisation to occur the real time jumps to ~12
seconds. This presumably involves some time waiting for the viewer to
initialise and for the shared memory to be allocated.

> Allowing everything other than getting the size of text ups the time to around 15s.

> Enabling getting text size ups the total to 23s again.

> So on my system I seem to have the following approximate timing breakdown
> 5s plplot core
> 7s wxViewer initialisation
> 8s getting text size
> 3s everything else wx related
>

> The reason getting text size takes so long I that it requests the
size from the viewer and that two way comms takes a lot of time.
However, I have just realised how I can do that without the comma, so
I should be able to eliminate that. The rest, well I'm not sure.

Disabling parts of the wxwidgets device as you have done is a useful
idea, but you have to break down the real time numbers above into the
part that is cpu and the part that is idle time to make this idea
truly effective.  So something like the experiments I did above for
the Linux case is required in addition to your disabling parts of
wxwidgets. If it turns out such experiments show there are no
wxwidgets idle issues on Windows, then that narrows down what you have
to look at to just the code that is run only for the Linux case.

So to summarize this, my recent experiments show that text- and
line-dominated examples have a pretty severe idle issue on Linux while
fill-dominated examples show no perceptible idle issue.  So the
highest priority question at the moment is whether or not there is a
similar idle issue on Windows. Once we definitely know that, then it
makes it much easier for us to decide on the best strategy for
tracking down and fixing the idle issue.

Alan
__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________

Re: [Plplot-devel] Linux wxwidgets inefficiency not solved

[Alan W. I. <ir...@be...>]

On 2016-01-02 22:53-0000 Phil Rosenberg wrote:

> Also also I forgot to say, the gcc driver does not appear to get
built by default on my system. Do I have to manually enable it?

It should be enabled by default on Windows.  But it may
be disabled by some of the tests in cmake/modules/wingcc.cmake.

For example, what are the messages you are getting from your cmake
invocation concerning "Looking for gdi32 header and library"?

Are those found or not found?  If the latter, you need to address that
issue.

Alan

__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________

Re: [Plplot-devel] Linux wxwidgets idle issue not solved for text- or line-dominated plots

[Alan W. I. <ir...@be...>]

On 2016-01-02 16:28-0000 Phil Rosenberg wrote:

> I have reproduced your timings on windows with the command
> time ./x20.exe -dev wxwidgets -np. This gives me a similar ~23s real
time and a fraction of a second user and sys time. Note that this is
run from Cygwin bash shell, but x20.exe is built using visual studio.

Example 20 is not the one I measured since that is a special case in
many ways.  Instead, I ran the test_c_wxwidgets target which runs a
list of C examples for

-dev wxwidgets -np

That list of examples is given in
plplot_test/test_c_interactive.sh.in. For my previous tests I locally
modified that list here to be "01 04 08 14 16 24 30" for all devices,
but you can locally modify it to be any list of examples you want. For
example, you could modify it to "01 01 01 01 01 01 .... " to get good
timing numbers for a short example such as example 1, see below. Also,
your system will have very different speed than mine.  So what is
important is not comparisons with my numbers, but instead what
fraction of time is spent with _both_ example and wxPLViewer
application being idle (i.e., both in wait mode) compared to the
fraction of time when one of them is using the cpu.  This "idle" issue
is a substantial problem for the Linux case, and it is currently
unknown whether that is a substantial problem for the Windows case
until you do the equivalent of the Linux experiments I did below.

Note, I have now concluded you cannot reliably estimate the fraction
of time in the idle state using the Linux "time" command (or
equivalent Windows command-line app) because it only gathers time
statistics for the example it is given and obviously cannot account,
(say) for separate processes such as wxPLViewer. (So that makes its
user and system cpu estimates too small because they do not include
the wxPLViewer numbers for those, but the real time is reliable (as
also measured by a nearby clock) because wxPLViewer exits before the example
exits.)

So I redid the above test for the above list of examples, and found
the overall time spent on -dev wxwidgets + wxPLViewer is still an
order of magnitude larger than for the -dev qtwidget case (29 seconds
versus 3.6 seconds).  Some of that could be that the Linux wxwidgets
library might just be slower that the Linux Qt library. But I think
that ratio is unlikely to be such a large factor because ultimately
both libraries simply organize calls to the X library which does all
the real work on Linux.  Furthermore, while I was running those
comparisons I looked at a GUI of overall cpu usage, and for -dev
qtwidget, the cpu usage was 100 per cent during the time of the test.
However, for the combination of -dev wxwidgets + wxPLViewer the total
cpu usage varied markedly from example to example. So from this result
it appears that some examples have a mix of PLplot commands that don't
trigger idle issues on Linux, but many others do.

So I looked further at GUI cpu usage results (with the GUI expanded to
the whole page so I could look at results with good resolution) for
repeated (i.e., 01 01 01, etc.) examples for -dev wxwidgets with the
following results for average total cpu usage percentage (these
numbers are roughly good to +/- 05 per cent since I am using an
average value estimated by eye from a graph and subtracting a ~5 per
cent baseline for what the average graph level looks like when the
example is not running.  I presume this background level I subtract is
due to the GUI itself taking some CPU time to display the results as a
continously upgraded graph).  Also, note I have two cpus and for most
cases the plot showed the OS distributed the load 50-50 between them
so the numbers below refer to the average individual cpu usage for
each cpu.  i.e., 100 per cent minus the numbers tabulated below is
what I state in the 3rd column which is the fraction of the time the
entire system was completely idle (everything in wait state) while
running these PLplot examples.

example cpu %  idle %   dominant type of graphics

00      30 %   70 %     lines
01      30 %   70 %     lines
02      30 %   70 %     lines
03      25 %   75 %     lines
04      25 %   75 %     lines
05      20 %   80 %     lines
06      20 %   80 %     text
07      15 %   85 %     text
08      90 %   10 %     fills
15      40 %   60 %     lines
16      75 %   25 %     fills

For most of these examples I also looked at detailed cpu usage graphs
for -dev xwin and -dev qtwidget and they were all similar, i.e., above
90 per cent cpu usage with essentially no idle issue at all.

So it is pretty clear from the above wxwidgets Linux results there is
a severe idle issue for text, still fairly severe for lines, and
essentially no issue at all for fills (or else the issue is swamped by
some other wxwidgets fill inefficiency since in the case of repeated example 8,
-dev xwin, qtwidget, and qtwidgets took a total real time of 6, 5, and
35 seconds.  And the corresponding numbers for repeated example 16 were 9, 7,
and 26 seconds.  Note also that the example 20 you have been looking
at also may not have a perceptible "idle" issue because that example
is virtually all fills as well.

Assuming you can get access to a Windows GUI that plots total cpu usage as a
function of time, could you run similar tests to the above for a
selection of text, line, and fill examples for -dev wxwidgets and -dev
wingcc?  Typical ratios of total time taken for wxwidgets versus wingcc
for some/all of the examples would be useful as well.

> I tried quite hard to locate the source of the delay. There are a
few sleep calls while I have to wait for the viewer.

It only takes one sleep call in a loop to make both example and
wxwidgets idle a large fraction of the time....

> I tried the following tests to try to isolate the problem. With
every wxWidget driver callback returning immediately so the driver
does nothing, I still get a real time of ~5s. So even when the
wxWidgets driver does nothing there is still some unaccounted for
time.

Example 20, has some overhead due to reading in the Lena file so that
is an atypical example I would avoid from that regard.  Also, it
apparently is fill-dominated so if you are trying to deal with the
idle issue, that is not the example to use from my result about
that fill-dominated examples don't have perceptible idle issues.

> If I allow initialisation to occur the real time jumps to ~12
seconds. This presumably involves some time waiting for the viewer to
initialise and for the shared memory to be allocated.

> Allowing everything other than getting the size of text ups the time to around 15s.

> Enabling getting text size ups the total to 23s again.

> So on my system I seem to have the following approximate timing breakdown
> 5s plplot core
> 7s wxViewer initialisation
> 8s getting text size
> 3s everything else wx related
>

> The reason getting text size takes so long I that it requests the
size from the viewer and that two way comms takes a lot of time.
However, I have just realised how I can do that without the comma, so
I should be able to eliminate that. The rest, well I'm not sure.

Disabling parts of the wxwidgets device as you have done is a useful
idea, but you have to break down the real time numbers above into the
part that is cpu and the part that is idle time to make this idea
truly effective.  So something like the experiments I did above for
the Linux case is required in addition to your disabling parts of
wxwidgets. If it turns out such experiments show there are no
wxwidgets idle issues on Windows, then that narrows down what you have
to look at to just the code that is run only for the Linux case.

So to summarize this, my recent experiments show that text- and
line-dominated examples have a pretty severe idle issue on Linux while
fill-dominated examples show no perceptible idle issue.  So the
highest priority question at the moment is whether or not there is a
similar idle issue on Windows. Once we definitely know that, then it
makes it much easier for us to decide on the best strategy for
tracking down and fixing the idle issue.

Alan
__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________

Re: [Plplot-devel] Linux wxwidgets inefficiency not solved

[Phil R. <p.d...@gm...>]

Also also I forgot to say, the gcc driver does not appear to get built by default on my system. Do I have to manually enable it?

-----Original Message-----
From: "Phil Rosenberg" <p.d...@gm...>
Sent: ‎02/‎01/‎2016 16:34
To: "Phil Rosenberg" <p.d...@gm...>; "Alan W. Irwin" <ir...@be...>; "PLplot development list" <Plp...@li...>
Subject: RE: Linux wxwidgets inefficiency not solved

Also, i forgot to say, the following command
time ./x20c.exe -dev null -np
Gives a real time of ~12  s too.


From: Phil Rosenberg
Sent: ‎02/‎01/‎2016 16:28
To: Alan W. Irwin; PLplot development list
Subject: RE: Linux wxwidgets inefficiency not solved


Hi Alan
I have reproduced your timings on windows with the command 
time ./x20.exe -dev wxwidgets -np. This gives me a similar ~23s real time and a fraction of a second user and sys time. Note that this is run from Cygwin bash shell, but x20.exe is built using visual studio.

I tried quite hard to locate the source of the delay. There are a few sleep calls while I have to wait for the viewer.
I tried the following tests to try to isolate the problem. With every wxWidget driver callback returning immediately so the driver does nothing, I still get a real time of ~5s. So even when the wxWidgets driver does nothing there is still some unaccounted for time.
If I allow initialisation to occur the real time jumps to ~12 seconds. This presumably involves some time waiting for the viewer to initialise and for the shared memory to be allocated.
Allowing everything other than getting the size of text ups the time to around 15s.
Enabling getting text size ups the total to 23s again.

So on my system I seem to have the following approximate timing breakdown
5s plplot core
7s wxViewer initialisation
8s getting text size
3s everything else wx related

The reason getting text size takes so long I that it requests the size from the viewer and that two way comms takes a lot of time. However, I have just realised how I can do that without the comma, so I should be able to eliminate that. The rest, well I'm not sure.

Phil



From: Alan W. Irwin
Sent: ‎30/‎12/‎2015 21:20
To: PLplot development list
Cc: Phil Rosenberg
Subject: Linux wxwidgets inefficiency not solved


On 2015-12-30 17:16-0000 Phil Rosenberg wrote:

> Hi Alan
> Thanks for the test results. I will look to see if I can do a similar test on windows, perhaps using Cygwin bash. Then I will see where we stand.
> Just to be sure are you building an optimised or debug build on Linux?

Hi Phil:

I would like to keep this discussion on list in case someone else here
can comment on the IPC question.

Good question above.  By historical accident (I have been debugging C and
Fortran issues a lot for the new Fortran binding), my compile flags were

software@raven> printenv |grep FL
CXXFLAGS=-O3 -fvisibility=hidden -Wuninitialized
CFLAGS=-g
FFLAGS=-g

So assuming -O3 will give a factor of ~2 speed increase, the C++-based
qtwidget and wxwidgets devices should have a two-fold speed advantage
compared to the C-based xwin and xcairo devices.

But the important point is that if you look at the timings for all the
devices, then the sum of user + system timings is normally roughly
equal to the real time required to do the test.  But not so for
wxwidgets where the real time is an order of magnitude larger than
that sum, i.e., the problem is not to reduce cpu cycles.  Therefore,
the -O3 flag above won't make much difference for the wxwidgets case.
Instead the problem is each example and/or associated wxPLViewer are
simultaneously doing large amounts of waiting while the cpu is
completely idle.  My hypothesis to explain those long waits is they
are both waiting too long for communications between them because of
some issue with the way you have set up the IPC method (perhaps
only on Linux, but maybe on Windows also).

My IPC expertise is confined to light skimming of
<https://en.wikipedia.org/wiki/Inter-process_communication> and
<https://en.wikipedia.org/wiki/Shared_memory_(interprocess_communication)>.
I took a look at the latter since it appears you are using the POSIX
shared memory method of IPC on Linux.  So I am definitely no expert,
and the above articles only talk about using shared memory for a very
fast way to communicate data between processes.  So that part of
efficiency considerations should be fine.  But there must be more to
it than that since the processes must communicate between themselves
(presumably with some sort of signal) concerning what data has been
written to that shared memory and how one process wants the other to
service the request implied by the signal that has been sent.

That is, my mental model of the ideal IPC is the -dev wxwidgets code
launches a wxPLViewer application (WA) and the two processes set up
IPC between them. Then the IPC between the two processes is the one
with control goes about its business until it needs the other to
service a request.  Then it simply writes needed data for that request
to a shared memory area and sends a specific signal to the other
process to deal with that data, and then that other process receives
that specific signal and deals with that data and continues until it
needs the other process to service a request. Thus, with this ideal
IPC model, the cpu is busy 100 per cent of the time either running
real code (not a wait loop) in either the device or WA.  That is, when
either device or WA are waiting while the other is busy, they are
waiting for explicit signals from the other to proceed and are not in
some sort of inefficient sleep x amount of time and wakeup and check
for activity from IPC partner loop.

If you do have such an inefficient wait x amount of time loop in your
code for the Windows case as well, I think you will also find a
similar inefficiency for test_c_wxwidgets compared to test_c_wingcc.
So I will be most interested in your results for that comparison.

Alan
__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________

Re: [Plplot-devel] Linux wxwidgets inefficiency not solved

[Phil R. <p.d...@gm...>]

Also, i forgot to say, the following command
time ./x20c.exe -dev null -np
Gives a real time of ~12  s too.

-----Original Message-----
From: "Phil Rosenberg" <p.d...@gm...>
Sent: ‎02/‎01/‎2016 16:28
To: "Alan W. Irwin" <ir...@be...>; "PLplot development list" <Plp...@li...>
Subject: RE: Linux wxwidgets inefficiency not solved

Hi Alan
I have reproduced your timings on windows with the command 
time ./x20.exe -dev wxwidgets -np. This gives me a similar ~23s real time and a fraction of a second user and sys time. Note that this is run from Cygwin bash shell, but x20.exe is built using visual studio.

I tried quite hard to locate the source of the delay. There are a few sleep calls while I have to wait for the viewer.
I tried the following tests to try to isolate the problem. With every wxWidget driver callback returning immediately so the driver does nothing, I still get a real time of ~5s. So even when the wxWidgets driver does nothing there is still some unaccounted for time.
If I allow initialisation to occur the real time jumps to ~12 seconds. This presumably involves some time waiting for the viewer to initialise and for the shared memory to be allocated.
Allowing everything other than getting the size of text ups the time to around 15s.
Enabling getting text size ups the total to 23s again.

So on my system I seem to have the following approximate timing breakdown
5s plplot core
7s wxViewer initialisation
8s getting text size
3s everything else wx related

The reason getting text size takes so long I that it requests the size from the viewer and that two way comms takes a lot of time. However, I have just realised how I can do that without the comma, so I should be able to eliminate that. The rest, well I'm not sure.

Phil



From: Alan W. Irwin
Sent: ‎30/‎12/‎2015 21:20
To: PLplot development list
Cc: Phil Rosenberg
Subject: Linux wxwidgets inefficiency not solved


On 2015-12-30 17:16-0000 Phil Rosenberg wrote:

> Hi Alan
> Thanks for the test results. I will look to see if I can do a similar test on windows, perhaps using Cygwin bash. Then I will see where we stand.
> Just to be sure are you building an optimised or debug build on Linux?

Hi Phil:

I would like to keep this discussion on list in case someone else here
can comment on the IPC question.

Good question above.  By historical accident (I have been debugging C and
Fortran issues a lot for the new Fortran binding), my compile flags were

software@raven> printenv |grep FL
CXXFLAGS=-O3 -fvisibility=hidden -Wuninitialized
CFLAGS=-g
FFLAGS=-g

So assuming -O3 will give a factor of ~2 speed increase, the C++-based
qtwidget and wxwidgets devices should have a two-fold speed advantage
compared to the C-based xwin and xcairo devices.

But the important point is that if you look at the timings for all the
devices, then the sum of user + system timings is normally roughly
equal to the real time required to do the test.  But not so for
wxwidgets where the real time is an order of magnitude larger than
that sum, i.e., the problem is not to reduce cpu cycles.  Therefore,
the -O3 flag above won't make much difference for the wxwidgets case.
Instead the problem is each example and/or associated wxPLViewer are
simultaneously doing large amounts of waiting while the cpu is
completely idle.  My hypothesis to explain those long waits is they
are both waiting too long for communications between them because of
some issue with the way you have set up the IPC method (perhaps
only on Linux, but maybe on Windows also).

My IPC expertise is confined to light skimming of
<https://en.wikipedia.org/wiki/Inter-process_communication> and
<https://en.wikipedia.org/wiki/Shared_memory_(interprocess_communication)>.
I took a look at the latter since it appears you are using the POSIX
shared memory method of IPC on Linux.  So I am definitely no expert,
and the above articles only talk about using shared memory for a very
fast way to communicate data between processes.  So that part of
efficiency considerations should be fine.  But there must be more to
it than that since the processes must communicate between themselves
(presumably with some sort of signal) concerning what data has been
written to that shared memory and how one process wants the other to
service the request implied by the signal that has been sent.

That is, my mental model of the ideal IPC is the -dev wxwidgets code
launches a wxPLViewer application (WA) and the two processes set up
IPC between them. Then the IPC between the two processes is the one
with control goes about its business until it needs the other to
service a request.  Then it simply writes needed data for that request
to a shared memory area and sends a specific signal to the other
process to deal with that data, and then that other process receives
that specific signal and deals with that data and continues until it
needs the other process to service a request. Thus, with this ideal
IPC model, the cpu is busy 100 per cent of the time either running
real code (not a wait loop) in either the device or WA.  That is, when
either device or WA are waiting while the other is busy, they are
waiting for explicit signals from the other to proceed and are not in
some sort of inefficient sleep x amount of time and wakeup and check
for activity from IPC partner loop.

If you do have such an inefficient wait x amount of time loop in your
code for the Windows case as well, I think you will also find a
similar inefficiency for test_c_wxwidgets compared to test_c_wingcc.
So I will be most interested in your results for that comparison.

Alan
__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________

Re: [Plplot-devel] Linux wxwidgets inefficiency not solved

[Phil R. <p.d...@gm...>]

Hi Alan
I have reproduced your timings on windows with the command 
time ./x20.exe -dev wxwidgets -np. This gives me a similar ~23s real time and a fraction of a second user and sys time. Note that this is run from Cygwin bash shell, but x20.exe is built using visual studio.

I tried quite hard to locate the source of the delay. There are a few sleep calls while I have to wait for the viewer.
I tried the following tests to try to isolate the problem. With every wxWidget driver callback returning immediately so the driver does nothing, I still get a real time of ~5s. So even when the wxWidgets driver does nothing there is still some unaccounted for time.
If I allow initialisation to occur the real time jumps to ~12 seconds. This presumably involves some time waiting for the viewer to initialise and for the shared memory to be allocated.
Allowing everything other than getting the size of text ups the time to around 15s.
Enabling getting text size ups the total to 23s again.

So on my system I seem to have the following approximate timing breakdown
5s plplot core
7s wxViewer initialisation
8s getting text size
3s everything else wx related

The reason getting text size takes so long I that it requests the size from the viewer and that two way comms takes a lot of time. However, I have just realised how I can do that without the comma, so I should be able to eliminate that. The rest, well I'm not sure.

Phil


-----Original Message-----
From: "Alan W. Irwin" <ir...@be...>
Sent: ‎30/‎12/‎2015 21:20
To: "PLplot development list" <Plp...@li...>
Cc: "Phil Rosenberg" <p.d...@gm...>
Subject: Linux wxwidgets inefficiency not solved

On 2015-12-30 17:16-0000 Phil Rosenberg wrote:

> Hi Alan
> Thanks for the test results. I will look to see if I can do a similar test on windows, perhaps using Cygwin bash. Then I will see where we stand.
> Just to be sure are you building an optimised or debug build on Linux?

Hi Phil:

I would like to keep this discussion on list in case someone else here
can comment on the IPC question.

Good question above.  By historical accident (I have been debugging C and
Fortran issues a lot for the new Fortran binding), my compile flags were

software@raven> printenv |grep FL
CXXFLAGS=-O3 -fvisibility=hidden -Wuninitialized
CFLAGS=-g
FFLAGS=-g

So assuming -O3 will give a factor of ~2 speed increase, the C++-based
qtwidget and wxwidgets devices should have a two-fold speed advantage
compared to the C-based xwin and xcairo devices.

But the important point is that if you look at the timings for all the
devices, then the sum of user + system timings is normally roughly
equal to the real time required to do the test.  But not so for
wxwidgets where the real time is an order of magnitude larger than
that sum, i.e., the problem is not to reduce cpu cycles.  Therefore,
the -O3 flag above won't make much difference for the wxwidgets case.
Instead the problem is each example and/or associated wxPLViewer are
simultaneously doing large amounts of waiting while the cpu is
completely idle.  My hypothesis to explain those long waits is they
are both waiting too long for communications between them because of
some issue with the way you have set up the IPC method (perhaps
only on Linux, but maybe on Windows also).

My IPC expertise is confined to light skimming of
<https://en.wikipedia.org/wiki/Inter-process_communication> and
<https://en.wikipedia.org/wiki/Shared_memory_(interprocess_communication)>.
I took a look at the latter since it appears you are using the POSIX
shared memory method of IPC on Linux.  So I am definitely no expert,
and the above articles only talk about using shared memory for a very
fast way to communicate data between processes.  So that part of
efficiency considerations should be fine.  But there must be more to
it than that since the processes must communicate between themselves
(presumably with some sort of signal) concerning what data has been
written to that shared memory and how one process wants the other to
service the request implied by the signal that has been sent.

That is, my mental model of the ideal IPC is the -dev wxwidgets code
launches a wxPLViewer application (WA) and the two processes set up
IPC between them. Then the IPC between the two processes is the one
with control goes about its business until it needs the other to
service a request.  Then it simply writes needed data for that request
to a shared memory area and sends a specific signal to the other
process to deal with that data, and then that other process receives
that specific signal and deals with that data and continues until it
needs the other process to service a request. Thus, with this ideal
IPC model, the cpu is busy 100 per cent of the time either running
real code (not a wait loop) in either the device or WA.  That is, when
either device or WA are waiting while the other is busy, they are
waiting for explicit signals from the other to proceed and are not in
some sort of inefficient sleep x amount of time and wakeup and check
for activity from IPC partner loop.

If you do have such an inefficient wait x amount of time loop in your
code for the Windows case as well, I think you will also find a
similar inefficiency for test_c_wxwidgets compared to test_c_wingcc.
So I will be most interested in your results for that comparison.

Alan
__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________

Re: [Plplot-devel] New Fortran binding: current status

[Alan W. I. <ir...@be...>]

On 2015-12-25 03:31-0800 Alan W. Irwin wrote:

> Here is a summary of what still needs to be dealt with for the new
> Fortran binding.
>
> f95
>   Missing examples            :  09 14 14a 15 16 18 19 20 21 22 23 28 31
>   Differing graphical output  :
>   Missing stdout              :
>   Differing stdout            :

Happy New Year, everyone!

Just to give you an update, that report now reads like this for the
new Fortran binding project:

  f95
    Missing examples            :  16 19 20 21 22
    Differing graphical output  :
    Missing stdout              :
    Differing stdout            :

with perfect valgrind results for all non-missing examples as well. So
Arjen and I are making some excellent progress with this project, and
I hope to land it on master within a week or so.

Alan
__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________

Re: [Plplot-devel] Linux wxwidgets inefficiency not solved

[Alan W. I. <ir...@be...>]

On 2016-01-01 10:36-0000 Phil Rosenberg wrote:

> Hi Alan

> Just a quick email while I procrastinate, rather than start tidying
from last night's festivities - happy new year by the way :-)

Hi Phil:

And Happy New Year to you as well!

> You are almost right regarding the ipc. But actually for most things
it is a bit simpler than that. The shared memory has a small header
area and then a data area which is set up as a circular buffer. The
header includes a read pointer and a write pointer. Whenever the
driver receives a command it simply adds the new part of the plplot
buffer to the data area, then updates the write pointer. The viewer
repeatedly checks the read and write pointers and if they are not
equal then it reads the new data and updates the read pointer. In this
model, the only time the driver should be waiting is if the buffer
fills up and the write pointer catches up with the read pointer. Then
the driver must wait for the viewer to update.

That "repeatedly checks the read and write pointer" sounds like the
logic is in a loop that waits N nanosecs and checks the read and write
pointer, and that wait could be the source of the issue.

> Actually things then got a bit more complicated, because there are
other things to deal with such as the cursor position requests etc.
But that is the basic model.

> So in most cases the communication is only one way. Therefore
semphores are generally not needed and named semaphores can be very
slow - on Linux I think they involve creating a locked file - so I try
to avoid them when possible.

>> Just found a useful illustrative example via google. Take a look at
>> the "Unamed semaphore example" in
>> <http://man7.org/conf/lca2013/IPC_Overview-LCA-2013-printable.pdf>
>> which uses POSIX semaphores to signal between two processes that are
>> accessing the same POSIX shared memory.  Naively, this looks quite
>> efficient to me with no unnecessary simultaneous waits by the two
>> processes since sem_wait just blocks until the other process is
>> finished serving the request.

Perhaps the above illustrative example was avoiding named semaphores
for that very efficiency reason?  Some more googling found
<https://sendreceivereply.wordpress.com/2007/04/10/mutex-or-semaphore-for-performance/>
which implies efficiency does increase from named semaphore to unamed.

Anyhow, the current situation is the wait time (where _neither_ -dev
wxwidgets or wxPLViewer are using the cpu) is roughly equal to 10
times the actual cpu time on Linux used by either of those processes.
So that is a gross wait inefficiency and not a cpu inefficiency, and I
would not be too concerned about any extra cpu cyles that an unnamed
semaphore would take. Therefore, I suggest you replace any wait loops
with an unnamed semaphore to see if that does solve the wait
inefficiency issue.  And, of course, I would be happy to test any
possible solution to the wait inefficiency that you send me if you are
having trouble getting access to Linux yourself.

I would also recommend a detailed timing comparisons between -dev
wingcc and -dev wxwidgets on MSVC just to make sure you don't have a
gross wait inefficiency in that case as well.

By the way.  I emphasize MSVC for those timing tests because Cygwin 
has additional timing issues for the interactive PLplot devices which
could obfuscate the results. According to what Arjen has told me, the
timing results for interactive devices on Cygwin depend very much on
whether the underlying interactive library has been built against the
native Windows display API (whose name I have forgotten) or X (which
in turn is built against the native display API, but which is very
slow because of that).  So the wingcc device on Cygwin must be built
against the native display API and is therefore fast, but the
possibility exists that the Cygwin package for the wxwidgets library
may build the X version of wxwidgets rather than the native display
version which would greatly slow down -dev wxwidgets on Cygwin. There
also may be similar concerns on MinGW-w64/MSYS2 depending on how
wxwidgets is built on that platform.  So I suggest you do timing tests
on MSVC first before you start evaluating timing results on those
other Unix-like platforms.

Alan
__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________

Re: [Plplot-devel] Linux wxwidgets inefficiency not solved

[Phil R. <p.d...@gm...>]

Hi Alan
Just a quick email while I procrastinate, rather than start tidying from last night's festivities - happy new year by the way :-)

You are almost right regarding the ipc. But actually for most things it is a bit simpler than that. The shared memory has a small header area and then a data area which is set up as a circular buffer. The header includes a read pointer and a write pointer. Whenever the driver receives a command it simply adds the new part of the plplot buffer to the data area, then updates the write pointer. The viewer repeatedly checks the read and write pointers and if they are not equal then it reads the new data and updates the read pointer. In this model, the only time the driver should be waiting is if the buffer fills up and the write pointer catches up with the read pointer. Then the driver must wait for the viewer to update.

Actually things then got a bit more complicated, because there are other things to deal with such as the cursor position requests etc. But that is the basic model.

So in most cases the communication is only one way. Therefore semphores are generally not needed and named semaphores can be very slow - on Linux I think they involve creating a locked file - so I try to avoid them when possible.

Phil

-----Original Message-----
From: "Alan W. Irwin" <ir...@be...>
Sent: ‎30/‎12/‎2015 22:51
To: "PLplot development list" <Plp...@li...>
Subject: Re: [Plplot-devel] Linux wxwidgets inefficiency not solved

On 2015-12-30 13:20-0800 Alan W. Irwin wrote:

> That is, my mental model of the ideal IPC is the -dev wxwidgets code
> launches a wxPLViewer application (WA) and the two processes set up
> IPC between them. Then the IPC between the two processes is the one
> with control goes about its business until it needs the other to
> service a request.  Then it simply writes needed data for that request
> to a shared memory area and sends a specific signal to the other
> process to deal with that data, and then that other process receives
> that specific signal and deals with that data and continues until it
> needs the other process to service a request. Thus, with this ideal
> IPC model, the cpu is busy 100 per cent of the time either running
> real code (not a wait loop) in either the device or WA.  That is, when
> either device or WA are waiting while the other is busy, they are
> waiting for explicit signals from the other to proceed and are not in
> some sort of inefficient sleep x amount of time and wakeup and check
> for activity from IPC partner loop.

Just found a useful illustrative example via google. Take a look at
the "Unamed semaphore example" in
<http://man7.org/conf/lca2013/IPC_Overview-LCA-2013-printable.pdf>
which uses POSIX semaphores to signal between two processes that are
accessing the same POSIX shared memory.  Naively, this looks quite
efficient to me with no unnecessary simultaneous waits by the two
processes since sem_wait just blocks until the other process is
finished serving the request.

I don't understand drivers/wxwidgets_comms.cpp at all, but I do notice
you use sem_open, etc., there.  Thus, it appears you are using a
similar POSIX semaphore basis of method of control between the two
processes that are accessing POSIX shared memory.  If so, then I
frankly cannot see how both processes would spend so much time
simultanously waiting with completely idle cpu unless there is some
issue with how you have implemented the use of semaphores.  So maybe
this would be a good time to carefully review that.

Alan

__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________

------------------------------------------------------------------------------
_______________________________________________
Plplot-devel mailing list
Plp...@li...
https://lists.sourceforge.net/lists/listinfo/plplot-devel

Re: [Plplot-devel] Linux wxwidgets inefficiency not solved

[Alan W. I. <ir...@be...>]

On 2015-12-30 13:20-0800 Alan W. Irwin wrote:

> That is, my mental model of the ideal IPC is the -dev wxwidgets code
> launches a wxPLViewer application (WA) and the two processes set up
> IPC between them. Then the IPC between the two processes is the one
> with control goes about its business until it needs the other to
> service a request.  Then it simply writes needed data for that request
> to a shared memory area and sends a specific signal to the other
> process to deal with that data, and then that other process receives
> that specific signal and deals with that data and continues until it
> needs the other process to service a request. Thus, with this ideal
> IPC model, the cpu is busy 100 per cent of the time either running
> real code (not a wait loop) in either the device or WA.  That is, when
> either device or WA are waiting while the other is busy, they are
> waiting for explicit signals from the other to proceed and are not in
> some sort of inefficient sleep x amount of time and wakeup and check
> for activity from IPC partner loop.

Just found a useful illustrative example via google. Take a look at
the "Unamed semaphore example" in
<http://man7.org/conf/lca2013/IPC_Overview-LCA-2013-printable.pdf>
which uses POSIX semaphores to signal between two processes that are
accessing the same POSIX shared memory.  Naively, this looks quite
efficient to me with no unnecessary simultaneous waits by the two
processes since sem_wait just blocks until the other process is
finished serving the request.

I don't understand drivers/wxwidgets_comms.cpp at all, but I do notice
you use sem_open, etc., there.  Thus, it appears you are using a
similar POSIX semaphore basis of method of control between the two
processes that are accessing POSIX shared memory.  If so, then I
frankly cannot see how both processes would spend so much time
simultanously waiting with completely idle cpu unless there is some
issue with how you have implemented the use of semaphores.  So maybe
this would be a good time to carefully review that.

Alan

__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________

[Plplot-devel] Linux wxwidgets inefficiency not solved

[Alan W. I. <ir...@be...>]

On 2015-12-30 17:16-0000 Phil Rosenberg wrote:

> Hi Alan
> Thanks for the test results. I will look to see if I can do a similar test on windows, perhaps using Cygwin bash. Then I will see where we stand.
> Just to be sure are you building an optimised or debug build on Linux?

Hi Phil:

I would like to keep this discussion on list in case someone else here
can comment on the IPC question.

Good question above.  By historical accident (I have been debugging C and
Fortran issues a lot for the new Fortran binding), my compile flags were

software@raven> printenv |grep FL
CXXFLAGS=-O3 -fvisibility=hidden -Wuninitialized
CFLAGS=-g
FFLAGS=-g

So assuming -O3 will give a factor of ~2 speed increase, the C++-based
qtwidget and wxwidgets devices should have a two-fold speed advantage
compared to the C-based xwin and xcairo devices.

But the important point is that if you look at the timings for all the
devices, then the sum of user + system timings is normally roughly
equal to the real time required to do the test.  But not so for
wxwidgets where the real time is an order of magnitude larger than
that sum, i.e., the problem is not to reduce cpu cycles.  Therefore,
the -O3 flag above won't make much difference for the wxwidgets case.
Instead the problem is each example and/or associated wxPLViewer are
simultaneously doing large amounts of waiting while the cpu is
completely idle.  My hypothesis to explain those long waits is they
are both waiting too long for communications between them because of
some issue with the way you have set up the IPC method (perhaps
only on Linux, but maybe on Windows also).

My IPC expertise is confined to light skimming of
<https://en.wikipedia.org/wiki/Inter-process_communication> and
<https://en.wikipedia.org/wiki/Shared_memory_(interprocess_communication)>.
I took a look at the latter since it appears you are using the POSIX
shared memory method of IPC on Linux.  So I am definitely no expert,
and the above articles only talk about using shared memory for a very
fast way to communicate data between processes.  So that part of
efficiency considerations should be fine.  But there must be more to
it than that since the processes must communicate between themselves
(presumably with some sort of signal) concerning what data has been
written to that shared memory and how one process wants the other to
service the request implied by the signal that has been sent.

That is, my mental model of the ideal IPC is the -dev wxwidgets code
launches a wxPLViewer application (WA) and the two processes set up
IPC between them. Then the IPC between the two processes is the one
with control goes about its business until it needs the other to
service a request.  Then it simply writes needed data for that request
to a shared memory area and sends a specific signal to the other
process to deal with that data, and then that other process receives
that specific signal and deals with that data and continues until it
needs the other process to service a request. Thus, with this ideal
IPC model, the cpu is busy 100 per cent of the time either running
real code (not a wait loop) in either the device or WA.  That is, when
either device or WA are waiting while the other is busy, they are
waiting for explicit signals from the other to proceed and are not in
some sort of inefficient sleep x amount of time and wakeup and check
for activity from IPC partner loop.

If you do have such an inefficient wait x amount of time loop in your
code for the Windows case as well, I think you will also find a
similar inefficiency for test_c_wxwidgets compared to test_c_wingcc.
So I will be most interested in your results for that comparison.

Alan
__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________

[Plplot-devel] Master default build issue SOLVED (but Linux wxwidgets inefficiency not solved)

[Alan W. I. <ir...@be...>]

On 2015-12-29 17:48-0000 Phil Rosenberg wrote:

> Hi Alan
> I think this should be fixed. I don't have easy access to a Linux box
> right now, but I have tested and the new version builds with gcc on
> Cygwin.

Hi Phil:

I confirm the build of the master tip (as of commit 36bd058 Fix
compile error for wxPLplotwindow for gcc) now works again on Linux, and the
build of the test_c_wxwidgets target also shows no run-time issues.

Thank you for quickly addressing this default build issue for master
tip.

You asked me off list to perform some Linux timing tests of your
latest code version.

Here are the latest results for the xwin, xcairo, qtwidget, and
wxwidgets devices for the interactive test_c_<interactive device> targets
which generate interactive results for the 01 04 08 14 16 24 30 C
examples (note these targets were temporarily modified for these tests
to exclude example 17 since that is much slower than all the rest and
particularly slow for xcairo and qtwidget so in any case we always drop
it from the list for those devices, see
plplot_test/test_c_interactive.sh.in):

software@raven> time make -j4 test_c_xwin >& /dev/null

real    0m2.896s
user    0m1.152s
sys     0m0.348s
software@raven> time make -j4 test_c_xcairo >& /dev/null

real    0m4.514s
user    0m3.556s
sys     0m0.392s
software@raven> time make -j4 test_c_qtwidget >& /dev/null

real    0m3.144s
user    0m1.688s
sys     0m0.372s
software@raven> time make -j4 test_c_wxwidgets >& /dev/null

real    0m26.273s
user    0m1.108s
sys     0m0.344s

In all cases the tests were done two or more times to build all
prerequisites and get everything into memory for the early tests
while only the last test result is reported above.

So there is still roughly an order of magnitude of inefficiency on
Linux for wxwidgets compared to the other devices which to my mind is
just barely acceptable.  My cpu meter and also the comparison of real
and user times for the above results indicates most of that time is
spent waiting rather than executing cpu instructions which is why I
suspect some badly tuned aspect of the Linux IPC between application
and wxPLViewer that generates inordinate amounts of wait time is still
the principal cause of these bad efficiency results for wxwidgets on
Linux.

For the MSVC case can you similarly compare speed results (excluding
example 17, see plplot_test/test_c_interactive.sh.in to see where to
exclude that example from the list of examples) for the test_c_wingcc
target and the test_c_wxwidgets target?  If those two times are
comparable to each other, then my working hypothesis to explain these
results is the Windows IPC is well tuned, but that is not the case for
the Linux IPC method that you are currently using.  On the other hand,
if test_c_wingcc is an order of magnitude faster, perhaps the Windows
IPC method is equally badly tuned, and you can find some common
factor to speed that up as well as the Linux IPC method.

Alan
__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________

Re: [Plplot-devel] git blog: git am conflict resolution

[Alan W. I. <ir...@be...>]

On 2015-12-29 14:16-0800 Alan W. Irwin wrote:

Part II.

> The above URL
[<http://stackoverflow.com/questions/25846189/git-am-error-patch-does-not-apply>]
> implies you should use the --reject git am option when
> -3 does not work, and I will say more about the practical details of
> how you resolve conflicts in that case once I have evaluated the
> contents of the first two patches and made any further commits to
> allow comprehensive testing of those two.

After amending the last of Arjen's patches to make comprehensive
testing work for example 9 and 14 (and many others such that the missing
Fortran examples list is now reduced down to just "15 16 19 20 21 22" which is a
huge improvement from where we started), the -3 option continued not to
work (as expected).  Therefore I followed up by

software@raven> git am --reject < 0003-Implement-the-simplest-plshade-interface-used-by-x15.patch
Applying: Implement the simplest plshade interface - used by x15f This test case brought a deficiency in the interface for plpat to light. It has been corrected, but it might result in some incompatibilities
Checking patch bindings/f95/included_plplot_real_interfaces.f90...
error: while searching for:
     end interface plsdimap
     private :: plsdimap_impl

     interface plsmaj
         module procedure plsmaj_impl
     end interface plsmaj

error: patch failed: bindings/f95/included_plplot_real_interfaces.f90:344
error: while searching for:
                      real(dimxpmm,kind=private_plflt), real(diypmm,kind=private_plflt) )
end subroutine plsdimap_impl

subroutine plsmaj_impl( def, scale )
     real(kind=wp), intent(in) :: def, scale


error: patch failed: bindings/f95/included_plplot_real_interfaces.f90:2266
Checking patch bindings/f95/plplot_bindings.f90...
Hunk #1 succeeded at 758 (offset 45 lines).
Checking patch examples/f95/CMakeLists.txt...
Checking patch examples/f95/x15f.f90...
Applying patch bindings/f95/included_plplot_real_interfaces.f90 with 2 rejects...
Rejected hunk #1.
Rejected hunk #2.
Applied patch bindings/f95/plplot_bindings.f90 cleanly.
Applied patch examples/f95/CMakeLists.txt cleanly.
Applied patch examples/f95/x15f.f90 cleanly.
Patch failed at 0001 Implement the simplest plshade interface - used by x15f This test case brought a deficiency in the interface for plpat to light. It has been corrected, but it might result in some incompatibilities
The copy of the patch that failed is found in:
    /home/software/plplot/HEAD/plplot.git/.git/rebase-apply/patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

and

software@raven> git status
On branch new_f95_update
You are in the middle of an am session.
   (fix conflicts and then run "git am --continue")
   (use "git am --skip" to skip this patch)
   (use "git am --abort" to restore the original branch)

Changes not staged for commit:
   (use "git add <file>..." to update what will be committed)
   (use "git checkout -- <file>..." to discard changes in working directory)

         modified:   bindings/f95/plplot_bindings.f90
         modified:   examples/f95/CMakeLists.txt
         modified:   examples/f95/x15f.f90

Untracked files:
   (use "git add <file>..." to include in what will be committed)

         bindings/f95/included_plplot_real_interfaces.f90.rej

no changes added to commit (use "git add" and/or "git commit -a")

So this (much less convenient result than for the -3 option) is the
patch was applied where possible, but the remainder of it that
could not be applied because of conflicts is stored in a *.rej file.

Fortunately, that rejected part of the patch was because of broken
context lines where I had made changes that Arjen had not gotten yet
so it was an easy matter to edit
bindings/f95/included_plplot_real_interfaces.f90 to include the
relevant parts of bindings/f95/included_plplot_real_interfaces.f90.rej
without the leading "+" sign on all the lines that Arjen had added.

I followed up by using the command-line "rm" command to
remove the untracked file
bindings/f95/included_plplot_real_interfaces.f90.rej and also ran

software@raven> git add bindings examples
software@raven> git status
On branch new_f95_update
You are in the middle of an am session.
   (fix conflicts and then run "git am --continue")
   (use "git am --skip" to skip this patch)
   (use "git am --abort" to restore the original branch)

Changes to be committed:
   (use "git reset HEAD <file>..." to unstage)

         modified:   bindings/f95/included_plplot_real_interfaces.f90
         modified:   bindings/f95/plplot_bindings.f90
         modified:   examples/f95/CMakeLists.txt
         modified:   examples/f95/x15f.f90
software@raven> git log --oneline -1
f4f6e4b Implement the simplest plshade interface - used by x15f This
test case brought a deficiency in the interface for plpat to light. It
has been corrected, but it might result in some incompatibilities
software@raven> git status
On branch new_f95_update
nothing to commit, working directory clean

==> complete success with committing Arjen's 3rd patch.

I hope these details of what is needed to resolve conflicts will be of
help to the rest of you when you inevitably encounter your own
conflicts (typically when attempting to use "git rebase" or "git am").

Now, off-list again to rush the latest stack of commits with my work
that Arjen had not had access to before and with all conflicts
resolved to Arjen so he can make a clean start with that for
his future changes on this private topic branch concerning the
new Fortran binding.

Alan
__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________

[Plplot-devel] git blog: git am conflict resolution

[Alan W. I. <ir...@be...>]

Arjen and I are working together on the new Fortran binding and since
we are both doing lots of editing in the same files, the probability
of conflicts (where we both edit the same area of the file in our
individual commits so that the patch process fails) is higher than
normal.  Here is the first part of my notes on how I resolved those conflicts
with a second part still to come.

(By the way, Arjen.  You will see many long lines below starting with
"Implement the new Fortran interface for plcont...." The reason for
those long lines is you forgot to put a line break after the first
(summary) sentence in your commit message so we see the whole
paragraph rather than the first sentence below in a number of
different contexts.)

1.  First try

software@raven> git am < 0001-Implement-the-new-Fortran-interface-for-plcont.-This.patch
Applying: Implement the new Fortran interface for plcont. This involves defining callback functions - the interface to these functions depends on the actual form they take their data arguments in. This is not entirely ideal as there is more code involved but earlier attempts via type(c_funptr) caused coredumps.
error: patch failed: bindings/f95/plplot_bindings.f90:27
error: bindings/f95/plplot_bindings.f90: patch does not apply
Patch failed at 0001 Implement the new Fortran interface for plcont. This involves defining callback functions - the interface to these functions depends on the actual form they take their data arguments in. This is not entirely ideal as there is more code involved but earlier attempts via type(c_funptr) caused coredumps.
The copy of the patch that failed is found in:
    /home/software/plplot/HEAD/plplot.git/.git/rebase-apply/patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

The above makes no change in the working directory according to git
diff.  When I google searched for this issue I found
<http://stackoverflow.com/questions/25846189/git-am-error-patch-does-not-apply>
which explains the above result is because git am by default wants to
preserve atomicity.  The discussion there mentioned the -3 flag for
git am (although apparently that sometimes will not work), so I tried
that instead.

2. Second try

I backed out of the first try with

git am --abort

then ran

software@raven> git am -3 < 0001-Implement-the-new-Fortran-interface-for-plcont.-This.patch
Applying: Implement the new Fortran interface for plcont. This involves defining callback functions - the interface to these functions depends on the actual form they take their data arguments in. This is not entirely ideal as there is more code involved but earlier attempts via type(c_funptr) caused coredumps.
Using index info to reconstruct a base tree...
M       bindings/f95/included_plplot_real_interfaces.f90
M       bindings/f95/plplot_bindings.f90
Falling back to patching base and 3-way merge...
Auto-merging bindings/f95/plplot_bindings.f90
CONFLICT (content): Merge conflict in bindings/f95/plplot_bindings.f90
Auto-merging bindings/f95/included_plplot_real_interfaces.f90
Failed to merge in the changes.
Patch failed at 0001 Implement the new Fortran interface for plcont. This involves defining callback functions - the interface to these functions depends on the actual form they take their data arguments in. This is not entirely ideal as there is more code involved but earlier attempts via type(c_funptr) caused coredumps.
The copy of the patch that failed is found in:
    /home/software/plplot/HEAD/plplot.git/.git/rebase-apply/patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

This obviously got much further.  Also, if I run git diff, I get
conflict resolution markers in the affected files, e.g.,

++<<<<<<< HEAD
  +  private :: private_single, private_double
  + 
++=======
+ 
++>>>>>>> Implement the new Fortran interface for plcont. This involves defining callback functions - the interface to these functions depends on the actual form they take their data arguments in. This is not entirely ideal as there is more code involved but earlier attempts via type(c_funptr) caused coredumps.

in bindings/f95/plplot_bindings.f90, 
i.e., Arjen inserted a blank line (which destroyed patch context) in his
commit close to a line in that file where I had inserted

private :: private_single, private_double

followed by a blank line in my own commit.

Resolving such simple conflicts is easy (by editing the file, removing
the <<<<<<<*, =======, and >>>>>>>* lines which are the conflict
resolution markers and deciding which version to use (or merge both
versions sometimes). In this simple case I resolved the conflict by
choosing my version and deleting the blank line from Arjen's version.

There were a small number of other conflicts marked in that same file,
but in each case it was easy to resolve them.

3. Finishing up

3a.

After I had edited bindings/f95/plplot_bindings.f90 to resolve the
conflicts the status was as follows:

software@raven> git status
On branch new_f95_update
You are in the middle of an am session.
   (fix conflicts and then run "git am --continue")
   (use "git am --skip" to skip this patch)
   (use "git am --abort" to restore the original branch)

Changes to be committed:
   (use "git reset HEAD <file>..." to unstage)

         modified:   bindings/f95/included_plplot_real_interfaces.f90

Unmerged paths:
   (use "git reset HEAD <file>..." to unstage)
   (use "git add <file>..." to mark resolution)

         both modified:   bindings/f95/plplot_bindings.f90

I then ran

git add bindings/f95/plplot_bindings.f90

to get my conflict resolutions into the index.

3b.  The status after that was

software@raven> git status
On branch new_f95_update
You are in the middle of an am session.
   (fix conflicts and then run "git am --continue")
   (use "git am --skip" to skip this patch)
   (use "git am --abort" to restore the original branch)

Changes to be committed:
   (use "git reset HEAD <file>..." to unstage)

         modified:   bindings/f95/included_plplot_real_interfaces.f90
         modified:   bindings/f95/plplot_bindings.f90

3c.

I then ran

software@raven> git am --continue
Applying: Implement the new Fortran interface for plcont. This
involves defining callback functions - the interface to these
functions depends on the actual form they take their data arguments
in. This is not entirely ideal as there is more code involved but
earlier attempts via type(c_funptr) caused coredumps.
software@raven> git log --oneline -1
996b4a7 Implement the new Fortran interface for plcont. This involves defining callback functions - the interface to these functions depends on the actual form they take their data arguments in. This is not entirely ideal as there is more code involved but earlier attempts via type(c_funptr) caused coredumps.
software@raven> git status
On branch new_f95_update
nothing to commit, working directory clean

==> Complete success for the first patch.

In sum, for this case could use "git am -3" which lead to insertion of
conflict resolution markers directly into the affected file and easy
resolution of the issue by editing, "git add <edited file name>", "git
am --continue".

Arjen's second patch applied without issues, but his third patch again
had conflicts.  However, in this case the -3 option does not work (probably
because of the previous conflict resolution in this series of commits).

The error message was

software@raven> git am -3 < /home/irwin/Arjen.Markus/20151229/new_fortran/0003-Implement-the-simplest-plshade-interface-used-by-x15.patch 
Applying: Implement the simplest plshade interface - used by x15f This test case brought a deficiency in the interface for plpat to light. It has been corrected, but it might result in some incompatibilities
fatal: sha1 information is lacking or useless (bindings/f95/included_plplot_real_interfaces.f90).
Repository lacks necessary blobs to fall back on 3-way merge.
Cannot fall back to three-way merge.
Patch failed at 0001 Implement the simplest plshade interface - used by x15f This test case brought a deficiency in the interface for plpat to light. It has been corrected, but it might result in some incompatibilities
The copy of the patch that failed is found in:
    /home/software/plplot/HEAD/plplot.git/.git/rebase-apply/patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

The above URL implies you should use the --reject git am option when
-3 does not work, and I will say more about the practical details of
how you resolve conflicts in that case once I have evaluated the
contents of the first two patches and made any further commits to
allow comprehensive testing of those two.

But so far so good, and note the same conflict resolution would also
have had to be done if Arjen and I had both been attempting to push
our local commits to master rather than collaborating on this private
topic branch using the "git format-patch"/"git am" method.

Alan
__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________

Re: [Plplot-devel] wxPLViewer can no longer be built on Linux ==> master default build issue

[Phil R. <p.d...@gm...>]

Hi Alan
I think this should be fixed. I don't have easy access to a Linux box
right now, but I have tested and the new version builds with gcc on
Cygwin.

The issue is actually a C++ syntax error and nothing to do with
wxWidgets. The short explanation is that this was GCC being stricter
than Visual studio regarding syntax. In case you are interested in the
long explanation the function wxPLplotwindow<WXWINDOW>::OnMouse is
part of a template class which inherits from the template type
WXWINDOW. In the case of wxPLViewer this type is a wxFrame. The
function GetGlientSize() is a member of wxFrame. Visual Studio
realises that the template parameter is instantiated as a wxFrame,
sees the inheritance and allows the syntax in your error message to
call wxFrame::GetClientSize(), however GCC sees the call to
GetClientSize() as part of its pre instatiation checks and cannot find
a function definition so gives an error. I am not sure if this is GCC
being overly strict or VS being overly permissive or clever. As you
can see from the error, passing the permissive flag would allow this
to compile with GCC, so who knows.

Anyway, the new syntax WXWINDOW::GetClientSize() specifies the scope
fully so there should be no worries at all there. It compiles fine on
Cygwin GCC so it should be fine on Linux GCC. If you still have any
problems let me know.

Phil

On 28 December 2015 at 23:09, Alan W. Irwin <ir...@be...> wrote:
> Hi Phil:
>
> For the current tip of master (commit 08d4864) wxPLViewer can no
> longer be built on Linux, i.e., those following our git version must
> currently use -DENABLE_wxwidgets=OFF in order to build the master
> branch. Because this is a default build issue for the master branch
> that affects a considerable number of people who are attempting to use
> the master tip version, I hope we will be able to find a fix in short
> order.  The last time I attempted to build the master branch was some
> time ago, but I am pretty sure this build issue has been introduced by
> your most recent wxwidgets change, but if there is any uncertainty
> about which of your commits caused the issue, git bisect is your friend.
>
> To help you figure out this build issue, I have attached a compressed
> version of the output results from
>
> make VERBOSE=1 wxPLViewer >& wxPLViewer.out
>
> I ran that command after an initial attempt with VERBOSE=1 not set
> failed. In other words, all prerequisites have already been built, and
> wxPLViewer.out just gives the VERBOSE=1 part of the build that failed.
>
> In case this is a wxWidgets versioning issue, I currently have installed
> on my Debian Jessie platform the following wx-related packages:
>
> irwin@raven> dpkg --list |grep -i wx
> ii  libwxbase3.0-0:amd64                  3.0.2-1+b1
> amd64        wxBase library (runtime) - non-GUI support classes of wxWidgets
> toolkit
> ii  libwxbase3.0-dev                      3.0.2-1+b1
> amd64        wxBase library (development) - non-GUI support classes of
> wxWidgets toolkit
> ii  libwxgtk3.0-0:amd64                   3.0.2-1+b1
> amd64        wxWidgets Cross-platform C++ GUI toolkit (GTK+ runtime)
> ii  libwxgtk3.0-dev                       3.0.2-1+b1
> amd64        wxWidgets Cross-platform C++ GUI toolkit (GTK+ development)
> ii  wx-common                             3.0.2-1+b1
> amd64        wxWidgets Cross-platform C++ GUI toolkit (common support files)
> ii  wx3.0-headers                         3.0.2-1+b1
> amd64        wxWidgets Cross-platform C++ GUI toolkit (header files)
>
> Good luck figuring this out, and let me know if there is any
> additional information I need to supply.
>
> Alan
> __________________________
> Alan W. Irwin
>
> Astronomical research affiliation with Department of Physics and Astronomy,
> University of Victoria (astrowww.phys.uvic.ca).
>
> Programming affiliations with the FreeEOS equation-of-state
> implementation for stellar interiors (freeeos.sf.net); the Time
> Ephemerides project (timeephem.sf.net); PLplot scientific plotting
> software package (plplot.sf.net); the libLASi project
> (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
> and the Linux Brochure Project (lbproject.sf.net).
> __________________________
>
> Linux-powered Science
> __________________________

[Plplot-devel] wxPLViewer can no longer be built on Linux ==> master default build issue

[Alan W. I. <ir...@be...>]

Hi Phil:

For the current tip of master (commit 08d4864) wxPLViewer can no
longer be built on Linux, i.e., those following our git version must
currently use -DENABLE_wxwidgets=OFF in order to build the master
branch. Because this is a default build issue for the master branch
that affects a considerable number of people who are attempting to use
the master tip version, I hope we will be able to find a fix in short
order.  The last time I attempted to build the master branch was some
time ago, but I am pretty sure this build issue has been introduced by
your most recent wxwidgets change, but if there is any uncertainty
about which of your commits caused the issue, git bisect is your friend.

To help you figure out this build issue, I have attached a compressed
version of the output results from

make VERBOSE=1 wxPLViewer >& wxPLViewer.out

I ran that command after an initial attempt with VERBOSE=1 not set
failed. In other words, all prerequisites have already been built, and
wxPLViewer.out just gives the VERBOSE=1 part of the build that failed.

In case this is a wxWidgets versioning issue, I currently have installed
on my Debian Jessie platform the following wx-related packages:

irwin@raven> dpkg --list |grep -i wx
ii  libwxbase3.0-0:amd64                  3.0.2-1+b1                           amd64        wxBase library (runtime) - non-GUI support classes of wxWidgets toolkit
ii  libwxbase3.0-dev                      3.0.2-1+b1                           amd64        wxBase library (development) - non-GUI support classes of wxWidgets toolkit
ii  libwxgtk3.0-0:amd64                   3.0.2-1+b1                           amd64        wxWidgets Cross-platform C++ GUI toolkit (GTK+ runtime)
ii  libwxgtk3.0-dev                       3.0.2-1+b1                           amd64        wxWidgets Cross-platform C++ GUI toolkit (GTK+ development)
ii  wx-common                             3.0.2-1+b1                           amd64        wxWidgets Cross-platform C++ GUI toolkit (common support files)
ii  wx3.0-headers                         3.0.2-1+b1                           amd64        wxWidgets Cross-platform C++ GUI toolkit (header files)

Good luck figuring this out, and let me know if there is any
additional information I need to supply.

Alan
__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________

Re: [Plplot-devel] Documentation bug

[Alan W. I. <ir...@be...>]

On 2015-12-27 20:26-0000 Phil Rosenberg wrote:

> Hi Alan
> Thanks, the parallel build seems to have been the issue. I have added
> a note to the documentation readme to warn that the recommended -j4
> option does not work under Cygwin.

Hi Phil:

I am glad you discovered the source of the issue by following up on
that possibility I suggested.

The lack of reliable parallel build is an important issue on Cygwin
for those with access to multi-cpu PC's where substantial build
speedups will become available once parallel builds are reliable again
on Cygwin.

However, it is guaranteed that no Cygwin developer is going to do
anything about this important issue until someone brings a
reproducible problem to their attention. If the problem seems
reproducible in your case (i.e., -j4 always gives you a bad result
from a fresh start for 3 or more tries on your particular hardware),
then I suggest it is worth reporting the issue as a Cygwin bug report.

Assuming you are satisfied on the reproducibility question, and have
written such a bug report, it should be sent to the Cygwin mailing
list (the Cygwin developers prefer that way of reporting Cygwin bugs).
That report should contain a good description of your hardware
(especially the frequency and number of cpu(s) in case those hardware
attributes affect the reproducibility), and a cookbook (probably with
most wording lifted from doc/docbook/README.developers) of how our
documentation should be built on that platform.  Such details should
give the Cygwin developers a reasonable chance to reproduce the issue
consistently for themselves which is the first step in finding
and eventually fixing the bug.

Alan
__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________

Re: [Plplot-devel] Documentation bug

[Phil R. <p.d...@gm...>]

Hi Alan
Thanks, the parallel build seems to have been the issue. I have added
a note to the documentation readme to warn that the recommended -j4
option does not work under Cygwin.

Phil

On 24 December 2015 at 22:40, Alan W. Irwin <ir...@be...> wrote:
> On 2015-12-22 22:40-0000 Phil Rosenberg wrote:
>
>> Hi
>> I have added some documentation to plGraphicsIn and moved the
>> documentation I wrote recently for plTranslateCursor to the C API.
>> However now I am getting the following error on Cygwin when I try to
>> build the pdf
>>
>> Build plplotdoc-print.pdf
>>
>> Unexpected error occured
>>
>> Error: [Errno 16] Device or resource busy:
>>
>> '/cygdrive/d/usr/local/src/plplot-plplot/build/cygwin/doc/docbook/src/plplot-5.11.1.pdf'
>>
>> doc/docbook/src/CMakeFiles/pdf_target.dir/build.make:87: recipe for
>> target 'doc/docbook/src/plplot-5.11.1.pdf' failed
>>
>> make[3]: *** [doc/docbook/src/plplot-5.11.1.pdf] Error 1
>>
>> CMakeFiles/Makefile2:1802: recipe for target
>> 'doc/docbook/src/CMakeFiles/pdf_target.dir/all' failed
>>
>> make[2]: *** [doc/docbook/src/CMakeFiles/pdf_target.dir/all] Error 2
>>
>> CMakeFiles/Makefile2:1601: recipe for target
>> 'doc/docbook/src/CMakeFiles/build_docbook_documentation.dir/rule'
>> failed
>>
>> make[1]: ***
>> [doc/docbook/src/CMakeFiles/build_docbook_documentation.dir/rule]
>> Error 2
>>
>> Makefile:457: recipe for target 'build_docbook_documentation' failed
>>
>> make: *** [build_docbook_documentation] Error 2
>>
>> I don't suppose anyone has seen this before or could try to reproduce
>> it on a proper Linux system to check it's not a Cygwin related issue?
>> I have attached a patch
>
>
> Hi Phil:
>
> It turns out your patch was LF rather than CRLF (which
> is obviously fine with me).  Is that because you produced it with
> Cygwin git?
>
> Anyhow, your patch applied without issues using "git am <patch file
> name>".
>
>
> One small workflow issue with your patch is these two lines
>
> -          (<literal>PLINT</literal>, input)
> +          (<literal>PLINT </literal>, input)
>
> which restored that blank which I had just removed in one of my recent
> "tweak" commits.  I don't really understand how this happened, but my
> guess is you forgot to git rebase before you sent me the git
> format-patch result.
>
> Anyhow, I have used "git commit --amend" to restore my blank removal
> tweak.
>
> Finally, I tweaked the wording of your documentation change a bit
> when I amended your commit.  The final version is given
> in commit 9863b15.
>
>
> I forgot to say in that commit message that there were no validation
> errors and the results built (in parallel with "make -j4 all") without
> issues on Debian jessie.
>
> Please try building commit 9863b15
>
> If you continue to get the above "resource busy" issue, is it possible
> you are attempting a parallel build on Cygwin?  Previous experience by
> Arjen and Greg indicate parallel builds are problematic on modern Cygwin
> (although they used to work fine for older versions of Cygwin).
>
>
> Alan
> __________________________
> Alan W. Irwin
>
> Astronomical research affiliation with Department of Physics and Astronomy,
> University of Victoria (astrowww.phys.uvic.ca).
>
> Programming affiliations with the FreeEOS equation-of-state
> implementation for stellar interiors (freeeos.sf.net); the Time
> Ephemerides project (timeephem.sf.net); PLplot scientific plotting
> software package (plplot.sf.net); the libLASi project
> (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
> and the Linux Brochure Project (lbproject.sf.net).
> __________________________
>
> Linux-powered Science
> __________________________

[Plplot-devel] New Fortran binding: current status

[Alan W. I. <ir...@be...>]

Merry Christmas everyone.

Over the last two weeks I have been focussed on following up on a
private topic branch implementation by Arjen of a new Fortran binding
for PLplot. This implementation uses the modern Fortran's
iso_c_binding module to do virtually all the interfacing with our core
C library, and I thought it would be appropriate to let everyone on
this list know how this topic is maturing.

Just to give some background, Arjen's new approach is quite powerful,
and has the following desireable properties:

* It is implemented using much more compact code with virtually none
of the interfacing done in C (i.e., the plplotf95c library has
shrunk to a tiny fraction of its previous size).

* It is much easier to understand and maintain than the previous effort.

* Linking issues are simplified.  (nm --undefined-only shows that
the Fortran examples only need to be linked to libplplotf95
and not libplplotf95c or libplplot)

* There is no need anymore for the "c_" prefixes on our C API to avoid
nameclashes with the Fortran versions of the routines. For now, I am
keeping those prefixes, but once the new Fortran binding is completed
and well-tested, I intend to follow up by dropping all the "braindead"
"c_" prefixes in our C library.

For backwards compatibility (just in case some user has latched onto
the "c_" prefixed form of our C API)

I will replace, e.g.,

#define    plcol0                   c_plcol0

with

#define c_plcol0 plcol0

But the second form of these #defines will be the only place that the
"c_" prefix will occur in our C code, and the "c_" prefix will also be
gone from all our routine names in the C library.  I am looking
forward to these C simplifications/side benefits due to
the Fortran binding reimplementationn, and I assume everyone else
here will be looking forward to this side benefit as well.

* The Fortran user will no longer be forced to use one particular
Fortran real precision that agrees with however PLFLT was configured
(which can be double or single precision depending on how the PLplot
library build was configured). Instead, they can conveniently use
single precision or double precision (even mixed in the same code so
long as a given PLplot call has consistent real precision), and our new
Fortran binding does the appropriate conversions to PLFLT using modern
Fortran's overloading mechanism.

My work so far consists of 24 separate commits on top of Arjen's
(large) initial effort that he sent me as a tarball, and Arjen plans
to start sending additional commits to me as well on this joint
private topic.

The current status is that as a result of our combined effort,
libplplotf95, libplplotf95c, and many of the Fortran examples build,
and that same list of examples produce identical results to the
corresponding C examples and also perfect (0 errors, no leaks are
possible) valgrind results.

Here is a sumary of what still needs to be dealt with for the new
Fortran binding.

f95
   Missing examples            :  09 14 14a 15 16 18 19 20 21 22 23 28 31
   Differing graphical output  :
   Missing stdout              :
   Differing stdout            :

The missing examples 09 14 14a 15 16 (and 16a which is a variant of
16) are due just to plcont, plshade, and plshades missing from the new
Fortran API. Arjen has apparently thought up a good way to deal with
the callback function arguments to those functions so he is taking
responsibility for their implementation with a good chance that he
will finish before January 1st.  While he is working on those I hope
to add the rest of the missing routines to our Fortran API so that 18
19 20 21 22 23 28 31 all build and produce perfect results by that
same date as well.

In sum, I anticipate this private topic is going to mature and land on
the master branch quite soon (likely within a week) due to our
collaborative work on it. Furthermore, after a month or two of further
testing on as many Fortran platforms as we can get our hands on, we
should all feel confident of making this new Fortran binding part of
our next release.

Alan
__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________

Re: [Plplot-devel] Documentation bug

[Alan W. I. <ir...@be...>]

On 2015-12-22 22:40-0000 Phil Rosenberg wrote:

> Hi
> I have added some documentation to plGraphicsIn and moved the
> documentation I wrote recently for plTranslateCursor to the C API.
> However now I am getting the following error on Cygwin when I try to
> build the pdf
>
> Build plplotdoc-print.pdf
>
> Unexpected error occured
>
> Error: [Errno 16] Device or resource busy:
> '/cygdrive/d/usr/local/src/plplot-plplot/build/cygwin/doc/docbook/src/plplot-5.11.1.pdf'
>
> doc/docbook/src/CMakeFiles/pdf_target.dir/build.make:87: recipe for
> target 'doc/docbook/src/plplot-5.11.1.pdf' failed
>
> make[3]: *** [doc/docbook/src/plplot-5.11.1.pdf] Error 1
>
> CMakeFiles/Makefile2:1802: recipe for target
> 'doc/docbook/src/CMakeFiles/pdf_target.dir/all' failed
>
> make[2]: *** [doc/docbook/src/CMakeFiles/pdf_target.dir/all] Error 2
>
> CMakeFiles/Makefile2:1601: recipe for target
> 'doc/docbook/src/CMakeFiles/build_docbook_documentation.dir/rule'
> failed
>
> make[1]: *** [doc/docbook/src/CMakeFiles/build_docbook_documentation.dir/rule]
> Error 2
>
> Makefile:457: recipe for target 'build_docbook_documentation' failed
>
> make: *** [build_docbook_documentation] Error 2
>
> I don't suppose anyone has seen this before or could try to reproduce
> it on a proper Linux system to check it's not a Cygwin related issue?
> I have attached a patch

Hi Phil:

It turns out your patch was LF rather than CRLF (which
is obviously fine with me).  Is that because you produced it with
Cygwin git?

Anyhow, your patch applied without issues using "git am <patch file
name>".


One small workflow issue with your patch is these two lines

-          (<literal>PLINT</literal>, input)
+          (<literal>PLINT </literal>, input)

which restored that blank which I had just removed in one of my recent
"tweak" commits.  I don't really understand how this happened, but my
guess is you forgot to git rebase before you sent me the git
format-patch result.

Anyhow, I have used "git commit --amend" to restore my blank removal
tweak.

Finally, I tweaked the wording of your documentation change a bit
when I amended your commit.  The final version is given
in commit 9863b15.


I forgot to say in that commit message that there were no validation
errors and the results built (in parallel with "make -j4 all") without
issues on Debian jessie.

Please try building commit 9863b15

If you continue to get the above "resource busy" issue, is it possible
you are attempting a parallel build on Cygwin?  Previous experience by
Arjen and Greg indicate parallel builds are problematic on modern Cygwin
(although they used to work fine for older versions of Cygwin).

Alan
__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________

[Plplot-devel] Documentation bug

[Phil R. <p.d...@gm...>]

Hi
I have added some documentation to plGraphicsIn and moved the
documentation I wrote recently for plTranslateCursor to the C API.
However now I am getting the following error on Cygwin when I try to
build the pdf

Build plplotdoc-print.pdf

Unexpected error occured

Error: [Errno 16] Device or resource busy:
'/cygdrive/d/usr/local/src/plplot-plplot/build/cygwin/doc/docbook/src/plplot-5.11.1.pdf'

doc/docbook/src/CMakeFiles/pdf_target.dir/build.make:87: recipe for
target 'doc/docbook/src/plplot-5.11.1.pdf' failed

make[3]: *** [doc/docbook/src/plplot-5.11.1.pdf] Error 1

CMakeFiles/Makefile2:1802: recipe for target
'doc/docbook/src/CMakeFiles/pdf_target.dir/all' failed

make[2]: *** [doc/docbook/src/CMakeFiles/pdf_target.dir/all] Error 2

CMakeFiles/Makefile2:1601: recipe for target
'doc/docbook/src/CMakeFiles/build_docbook_documentation.dir/rule'
failed

make[1]: *** [doc/docbook/src/CMakeFiles/build_docbook_documentation.dir/rule]
Error 2

Makefile:457: recipe for target 'build_docbook_documentation' failed

make: *** [build_docbook_documentation] Error 2

I don't suppose anyone has seen this before or could try to reproduce
it on a proper Linux system to check it's not a Cygwin related issue?
I have attached a patch

Re: [Plplot-devel] Problems applying patches

[Alan W. I. <ir...@be...>]

On 2015-12-22 09:04-0000 Arjen Markus wrote:

> Hi Alan,
>
>
>
> See below.
>
>> -----Original Message-----
>> From: Alan W. Irwin [mailto:ir...@be...]
>>
>> Hi Arjen:
>>
>> I too am really glad the process is working for you now, but I am also just as startled
>> as you were by your discovery of an LF working directory.  Therefore, I would
>> appreciate you systematically investigating this question further so we can draw
>> some definite conclusions with regard to other attempts at Window/Unix
>> communications via "git format-patch"/"git am" results.
>>
>> 1. Just to double check your LF conclusion above, do you also get LF for the text
>> files of a pristine working directory for master created with git clone before you make
>> any local changes at all? If so, I guess that means Cygwin git is now interpreting
>> your git platform as Unix with LF endings on text files, and your bad results
>> yesterday were due to you creating that working directory in the past with an old
>> Cygwin git version (or else some non-Cygwin git version) that interpreted your git
>> platform to be CRLF.  Does that seem like a reasonable hypothesis to explain
>> yesterday's bad results and today's good results?
>>
> No, I have checked several working directories I have for PLplot, accumulated over the years :), but I find LF line endings in various files that I did not touch myself.

Hi Arjen:

>From these results my guess is it was the clean start itself (e.g.,
the actual file contents in bindings/f95 rather than line endings
there) that was essential. For example, my initial commit included all
your Fortran changes so git am for that first patch file would fail it
you attempted to apply it on top of your Fortran changes.

>
>> 2. Assuming LF what happens on a throwaway topic branch if you use a normal
>> Windows editor to make a file change?  In particular you should check that the file
>> created by that editor is CRLF, and if so, what what happens if you attempt to
>> commit that file?  From my own experience with attempting to commit CRLF files, my
>> prediction is the commit will tell you it is changing the repository result to LF (as it
>> should) and leaving the working directory file as CRLF (which is problematic in my
>> view, but that is what happened to me).  Anyhow, if your working directory starts out
>> as LF, you are going to have to work pretty hard to keep it LF when using ordinary
>> Windows tools to make changes.  So you might want to get clarification on the
>> Cygwin list about why Cygwin's git is creating LF working directories for a repository
>> like ours where the .gitattributes files says text=auto, i.e., all working directories
>> should have native line endings.
>
> If I edit a file which is then saved with CRLF and commit it, I get a warning about the CR, but the file is left as it is. Luckily my text editors don't mind.

OK.  Good to know.

>
>>
>> 2. Could you double-check that when you unpacked the tarball from me the patchfile
>> results were LF and not CRLF?  (It might make a difference whether you unpack with
>> Cygwin tar or some other tar.)
>>
> No, the individual patch files are simply LF, as I expected actually - the unpacking facility does not interpret files in any way.

OK.  Good to know.
>
>> 3. Assuming you confirm above that both your working directory and patch files are
>> LF could you try the experiments of using "git am"
>> (from a fresh start each time) with (1) "--keep-cr", (2) "--no-keep-cr", and (3) neither
>> option? My extremely tentative prediction is the --keep-cr option will fail (because it
>> will attempt to convert the patch file to CRLF before applying), and the other two will
>> succeed but it would be good for you to let us know exactly what happens in the
>> three different cases as a guide to what to expect on git CRLF platforms.
>>
> Actually when I succeeded with applying the patch yesterday, I used the -keep-cr option. So that is working correctly in my setup.
>
> Just now I tried the other possibilities: --no-keep-cr and no options at all. They both work in exactly the same way as -keep-cr. A few messages about trailing whitespace and that is it.
>
>> One sure conclusion is that Windows users attempting to communicate with "git
>> format-patch"/"git am" must look at their working directory (especially if they are
>> using Cygwin git) to determine if it is CRLF or LF, and similarly for any patch files. I
>> am pretty sure the --no-keep-cr and ---keep-cr "git am" options will help sort out
>> when there is a mismatch in line-endings, but the experiments mentioned in
>> 3 should help confirm/disprove that hypothesis.
>>
> Not sure what to conclude from the above experiments.

Thanks for reporting all these results.

There was web comment that could be interpreted (probably wishful
thinking on my part when I thought you had a CRLF platform) as ---keep-cr 
transforms  LF to CRLF.  Your experiments have confirmed that is not the
case, and it acts like it is documented which is that git-am
merely passes that flag to git mailsplit which in turn documents it as

--keep-cr
     Do not remove \r from lines ending with \r\n

So it appears by default that git am and git mailsplit transform from
CRLF to LF.  So to preserve CRLF you must use the --keep-cr option but it does not
assert it, i.e., if the input patch files are LF it does not transform
them to CRLF (as you have confirmed).

So I think on LF Cygwin you should never use this option since either
it is a no-op if communicating with another LF platform or else it
would mess you up if communicating with a CRLF platform.  However,
from web comment it still appears to be a must for CRLF platforms
communicating with other CRLF platforms.  And for cases where CRLF
platforms are trying to apply LF patches, I suspect the only recourse
is to transform the patches to CRLF before applying them with the
--keep-cr option of git am.

Alan
__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________