plplot-devel — Technical discussion about PLplot development

Re: [Plplot-devel] Planning for the release of 5.13.0

[Alan W. I. <ir...@be...>]

On 2017-08-15 08:03-0000 Arjen Markus wrote:

> I expand the path to get access to git in the comprehensive tests, but then CMake fails:

Hi Arjen:

How do you expand the PATH?  If you mean put /usr/bin on it,
doesn't that already happen automatically when you set
the appropriate shell for MinGW-w64/MSYS2?

The reason I mention /usr/bin is that is where make.exe, bash.exe,
sh.exe, git.exe and other Unix tool executables should be located.  If
those Unix tools are not all there, then there is likely something
wrong/incomplete with your automated install procedure compared to
your previous hand-installed MinGW-w64/MSYS2 snapshot where you had no
trouble accessing all of those Unix tools (other than git.exe which
was not installed for that snapshot) in /usr/bin.

In fact, in general, can't you debug your automated install snapshot
by comparing directory listings with your prevous snapshot you created
by hand?

Also, look carefully at the file
../comprehensive_test_disposeable/comprehensive_test_env.out generated
by the comprehensive test script.  For example, if there was something
wrong with the PATH in your invocation script, it will be recorded
there.

I hope these ideas and the ideas in my previous post are a significant
help to you but if not, a few days break might clear your mind
sufficiently so you quickly spot what is wrong with your automatic
install once you look at it again.

Best wishes,

Alan
__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________

Re: [Plplot-devel] Planning for the release of 5.13.0

[Alan W. I. <ir...@be...>]

On 2017-08-15 07:11-0000 Arjen Markus wrote:

> Hi Alan,
>
>
>
>> -----Original Message-----
>> From: Alan W. Irwin [mailto:ir...@be...]
>> Sent: Friday, August 11, 2017 6:33 PM
>>
>> Once you perform (1), (2'), and (3), I should have two report tarballs to look at as
>> well as one ~62MB tarball collection of plot file results, and assuming those reveal
>> no more issues (other than the "missing pyqtconfig.py" issue we will be dealing with
>> after the release), then these should be the last noninteractive tests you need to
>> make on this platform before the release.
>>
> This is turning out to be very frustrating - a sort of spring procession of Echternach.
>

> While my installation of MinGW-w64/MSYS2 is clean (I did not mess
with it other than via pacman), I thought it would be a good idea to
do a parallel installation anyway - as a test for my automated
installation procedure and as future help to others.

I agree this was a really good idea.  It's just too bad that
you ran into so many initial troubles with it (as you
discuss below), but I am sure you will eventually figure
out all of those issues that you encountered.

>
> So:
>

> - The automated installation went fairly smooth, but CMake did not
acknowledge the MSYS or Unix generators that my original installation
did have.

> - I had to resort to the MinGW generator, but then you have to move
"sh.exe" out of the way.

I can see why you attempted to try "MinGW Makefiles" when "MSYS2
Makefiles" was temporarily not working, but now the latter is working
again I suggest you stick with "MSYS2 Makefiles" until you complete
your tests to avoid the sh.exe complications implicit in using "MinGW
Makefiles".

However, after the release if you ever want to pursue "MinGW
Makefiles" again, I would suggest the following slightly different
approach that does not mess with any files installed by pacman and 
which worked very well for me for the MinGW/MSYS case.

* Copy everything in your pristine (i.e., without sh.exe removed) /usr/bin to some other location called say
/usr/bin_without_sh.

* Remove sh.exe from /usr/bin_without_sh.  (Since this
is a different location then your pristine /usr/bin, this change does
not clobber that pristine location and therefore does not affect pacman
results in any way.)

* Replace /usr/bin on your PATH with /usr/bin_without_sh

This PATH change in at least the MinGW/MSYS platform case lead to good
comprehensive test results for the "MinGW Makefiles" generator so I
think that will be generally true in the MinGW-w64/MSYS2 platform case
as well.  However, you may also run into find difficulties, e.g., for
cmake/modules/findwxWidgets.cmake.  Therefore, debugging such issues
will likely take some time to sort out which is one other reason why I
have suggested you stick with "MSYS Makefiles" for now.

> - I had some trouble getting "git" to be available via the
MinGW-w64/MSYS2 shell - I later realised that was my fault. So I
corrected that this morning by expanding the PATH correcrly. Only to
discover that the git installation then brings in its own "sh.exe".

That makes sense.  The git package is from the Unix tools (msys2)
repository of MinGW-w64/MSYS2.  So any installation such as that
likely updates everything from the Unix tools side and therefore likely
figures out there is a sh.exe file missing from that side and
fixes it.

> - Yesterday I got stuck because CMake was invoking the MicroSoft
version of make - despite the fact that "which make" in the
MinGW-w64/MSYS2 shell reports "/usr/bin/make".

The "which" command is sometimes unreliable.  For example, suppose
your PATH was "fullpath_one:fullpath_two", i.e., anything in
fullpath_one is higher on the PATH than anything in fullpath_two. Now,
suppose there is an executable (let's call it make.exe) located in
fullpath_two but not in fullpath_one, then "which make" will (rightly)
tell you the location is fullpath_two/make.exe.  Now suppose you
subsequently install a completely different make.exe in fullpath_one.
Then "which make" will still (in error) report fullpath_two/make.exe!
This occurs even though that new version of make is higher on your PATH and
will get executed when you execute "make" even though the fullpathname
reported by "which make" is different!

To avoid this "which" unreliability (especially when doing installs!) I
suggest you use the "hash" builtin command available from bash.exe
instead which is more complicated to use then "which" but completely
reliable.  The complication is hash has to be called twice (once to
update the hash table to fill it with the correct full pathname
corresponding to current PATH and specified <command> and once to
print out that full pathname corresponding to <command>).  So always
invoke it like this,

hash <command>; hash -t <command>

e.g.,

irwin@raven> hash make; hash -t make
/usr/bin/make

So here are my guesses to explain the above peculiar result you
describe for cmake.  At the time when you invoked cmake, you may not
have set the correct PATH to start with or your PATH might have been
wrong because you were not in the correct shell.  Alternatively your
"which" result could have been unreliable or else made with a
different PATH or shell setting compared to what you had when you invoked
cmake.

> - I realised, however, that I could instruct the comprehensive test
script to explicitly use "/usr/bin/make". So I added that option to
the invocation.

Yes, fullpaths are always reliable, but so is "hash make; hash -t make"

> - To my surprise CMake did acknowledge the "MSYS generator" this
morning (*), but because I had moved "sh.exe" out of the way,
"/usr/bin/make" refused to do its job, as it could not find the
"sh.exe" executable. So CMake failed. Sigh.

[out of order]
> (*) MinGW-w64 seems to require that you restart the shell after
installing things if you want to take full advantage of the freshly
installed package. That may be an explanation for yesterday's failure
with CMake. A small mistake in my install script required me to
install CMake manually.

> Sorry to sound so grumpy, but it was rather frustrating as I already
said.

My sympathies.  Tiny details that are slightly wrong can drive you crazy.

This is why I agreed with you above that an automated install
procedure is absolutely the best way to go.  Debugging such an
automated process forces most of the tiny details to eventually be
correct.  Furthermore, using automated procedures makes it much easier
to reproduce previous results.

> The whole thing is requiring more of my attention than I can
actually spare for the next few days (I have to revise an article and
that is mostly in my spare time).  I will try to get bits and pieces of
the tests done inbetween all other stuff, but I am not going to
promise anything as far as time of delivery is concerned.

Thanks for being honest about all the other critical calls on your
time right now. That and your current inability to reproduce what you
did before make it pretty likely it is going to be quite a long time
from now before you can finish your testing. And, of course, that
conflicts with my strong desire to get out the release in a timely
manner now that I have made my last planned change to the source code.

So given this situation and my additional strong desire for PLplot to
be a fun spare-time hobby activity for you (just as it is for me)
rather than some horrible slog, it appears there are two general
courses open to us.

1. Release soon.  It is unlikely you will have complete testing
results for your three platforms any time soon so this option
virtually guarantees that we will release with incomplete testing on
your part.

2. Wait indefinitely until you do have some relaxed time to
   complete your testing, then release.

Let's make this tough decision between these two general choices on
Friday when you will presumably have a much better idea about how your article
revision is going and therefore how much time you will have for
PLplot testing in the near future.  Meanwhile, a day or two off from
testing may clear your mind and make it possible to do it again in
a way that reproduces what you have done before.

Alan
__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________

Re: [Plplot-devel] Planning for the release of 5.13.0

[Arjen M. <Arj...@de...>]

Hi Alan,



Okay, a bit more grumpiness:



I expand the path to get access to git in the comprehensive tests, but then CMake fails:



-- Looking for gdi32 header and library

-- Looking for gdi32 header and library - found

CMake Error at cmake/modules/FindwxWidgets.cmake:933 (message):

  wxWidgets wx/version.h file not found in

  C:/progra~1/git/mingw64/lib/wx/include/msw-unicode-3.0;C:/progra~1/git/mingw64/include/wx-3.0.

Call Stack (most recent call first):

  cmake/modules/wxwidgets.cmake:49 (find_package)

  cmake/modules/drivers.cmake:99 (include)

  cmake/modules/plplot.cmake:556 (include)

  CMakeLists.txt:125 (include)



I will turn off the expansion of the PATH, losing "git", but hopefully getting a useable CMake output.



Regards,



Arjen

DISCLAIMER: This message is intended exclusively for the addressee(s) and may contain confidential and privileged information. If you are not the intended recipient please notify the sender immediately and destroy this message. Unauthorized use, disclosure or copying of this message is strictly prohibited. The foundation 'Stichting Deltares', which has its seat at Delft, The Netherlands, Commercial Registration Number 41146461, is not liable in any way whatsoever for consequences and/or damages resulting from the improper, incomplete and untimely dispatch, receipt and/or content of this e-mail.

Re: [Plplot-devel] Planning for the release of 5.13.0

[Arjen M. <Arj...@de...>]

Hi Alan,



> -----Original Message-----
> From: Alan W. Irwin [mailto:ir...@be...]
> Sent: Friday, August 11, 2017 6:33 PM
>
> Once you perform (1), (2'), and (3), I should have two report tarballs to look at as
> well as one ~62MB tarball collection of plot file results, and assuming those reveal
> no more issues (other than the "missing pyqtconfig.py" issue we will be dealing with
> after the release), then these should be the last noninteractive tests you need to
> make on this platform before the release.
>
This is turning out to be very frustrating - a sort of spring procession of Echternach.

While my installation of MinGW-w64/MSYS2 is clean (I did not mess with it other than via pacman), I thought it would be a good idea to do a parallel installation anyway - as a test for my automated installation procedure and as future help to others.

So:

-        The automated installation went fairly smooth, but CMake did not acknowledge the MSYS or Unix generators that my original installation did have.

-        I had to resort to the MinGW generator, but then you have to move "sh.exe" out of the way.

-        I had some trouble getting "git" to be available via the MinGW-w64/MSYS2 shell - I later realised that was my fault. So I corrected that this morning by expanding the PATH correcrly. Only to discover that the git installation then brings in its own "sh.exe".

-        Yesterday I got stuck because CMake was invoking the MicroSoft version of make - despite the fact that "which make" in the MinGW-w64/MSYS2 shell reports "/usr/bin/make". The MicroSoft make utility does not do anything useful with the makefiles as they are generated by CMake in this constellation :(. Except perhaps causing the process to hang.

-        I realised, however, that I could instruct the comprehensive test script to explicitly use "/usr/bin/make". So I added that option to the invocation.

-        To my surprise CMake did acknowledge the "MSYS generator" this morning (*), but because I had moved "sh.exe" out of the way, "/usr/bin/make" refused to do its job, as it could not find the "sh.exe" executable. So CMake failed. Sigh.

Sorry to sound so grumpy, but it was rather frustrating as I already said. The whole thing is requiring more of my attention than I can actually spare for the next few days (I have to revise an article and that is mostly in my spare time). I will try to get bits and pieces of the tests done inbetween all other stuff, but I am not going to promise anything as far as time of delivery is concerned.

Regards,

Arjen

(*) MinGW-w64 seems to require that you restart the shell after installing things if you want to take full advantage of the freshly installed package. That may be an explanation for yesterday's failure with CMake. A small mistake in my install script required me to install CMake manually.

DISCLAIMER: This message is intended exclusively for the addressee(s) and may contain confidential and privileged information. If you are not the intended recipient please notify the sender immediately and destroy this message. Unauthorized use, disclosure or copying of this message is strictly prohibited. The foundation 'Stichting Deltares', which has its seat at Delft, The Netherlands, Commercial Registration Number 41146461, is not liable in any way whatsoever for consequences and/or damages resulting from the improper, incomplete and untimely dispatch, receipt and/or content of this e-mail.

Re: [Plplot-devel] Planning for the release of 5.13.0

[Alan W. I. <ir...@be...>]

On 2017-08-14 02:11-0700 Alan W. Irwin wrote:

> The other uncertainty (other than how much time you have for testing
> this week) in the release schedule is I still have a set of wxwidgets
> issues remaining here which is the uninitialized variable in the
> wxwidgets superscript/subscript logic that you discovered with MSVC,
> and several other related issues I discovered for that
> superscript/subscript logic. These issues only affect the
> noninteractive test for MSVC (a build issue due to the uninitialized
> variable), and the wxwidgets-only interactive tests on all platforms.
> Which is why I encourage you to start your final noninteractive
> testing for MinGW-w64/MSYS2 and Cygwin immediately rather than
> waiting for my set of fixes.
>
> I am not done yet with this set of wxwidgets superscript/subscript
> fixes, but the uninitialized variable issue is solved, and I had
> several breakthroughs in understanding that wxwidgets code today so I
> am pretty confident I can finish this up by early Tuesday (UTC) which
> should get rid of the wxwidgets build error for the noninteractive
> MSVC case, and also allow you to interactively test wxwidgets on all
> three platforms.

As promised, DONE as of commit ac753c6.  This set of fixes did not
solve all issues with the superscript/subscript logic for wxwidgets
(see the commit message for the details).  However, none of the
remaining issues I identified are release critical, and they are all
difficult to understand/fix without a much better understanding of the
transformation between the wxwidgets text string origin and the
corresponding PLplot text string origin.  Therefore, I plan to deal
with these issues some time after the release as a low priority. As a
result, this commit is the last source code change for this release I
plan to make (unless our further testing turns up one or more
additional release critical issues).

Alan
__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________

Re: [Plplot-devel] Planning for the release of 5.13.0 - findings regarding Cygwin

[Alan W. I. <ir...@be...>]

On 2017-08-14 09:35-0000 Arjen Markus wrote:

> Hi Alan,
>
>
>
> I will ignore wxWidgets for the moment. Cmake seems to find something, but is quiet about the reasons for its final decision:

Hi Arjen:

I don't want to distract you now from the important goal of
immediately finishing off your noninteractive testing for
MinGW-w64/MSYS2 and Cygwin.  But once those tasks are completed (early
Tuesday?) I am virtually positive the revised ideas below will help
you to quickly solve your current find issues for wxwidgets on Cygwin.
The reason I revised my ideas, is I did do a bit more research this
morning.  What follows should be skimmed lightly for now, but studied
in more detail once your noninteractive tests of MinGW-w64/MSYS2 and
Cygwin are completed.

One possibility I dismissed before (since you should be a veteran at
installing wxwidgets by now on all your platforms) is you might not
have the right set of wxwidgets packages installed on Cygwin.

For Cygwin, I believe the result is you will automatically be using
the Unix part of cmake/modules/FindwxWidgets.cmake, and that uses some
name variant of the wx-config script.

So after your noninteractive tests are done, you need to be sure to
install the package that contains some name variant of wx-config, and
from
<https://cygwin.com/cgi-bin2/package-grep.cgi?grep=wx-config&arch=x86_64>
the 3.x choice seems to be

libwx_gtk3u3.0-devel

Furthermore, if you click on that package name in the GUI provided
above, the name variant provided by that package is wx-config-3.0
(which is one of the variant names of wx-config that are supported by
cmake/modules/FindwxWidgets.cmake).

Thus, if you have libwx_gtk3u3.0-devel (and all its dependencies) installed
already, then execution of wx-config-3.0 within
cmake/modules/FindwxWidgets.cmake should result in a consistent set of
compile flags and libraries.  So it should all "just work".

However, in the case that normally reliable executable fails for some
strange reason within cmake/modules/FindwxWidgets.cmake, i.e., you
still get wxwidgets "not found" messages, then as I have previously
mentioned you can use the cmake --debug-out and --trace options to
debug what is going on. But use of those cmake options are a really
blunt tool that generate a tonne of output.  So save that possibility
for a last resort and instead first use a much more selective choice
for debugging cmake/modules/FindwxWidgets.cmake which is to uncomment
the message commands in the DBG_MSG and DBG_MSG_V macros that occur
within that module.  If it turns out you cannot figure out what is
going wrong with that find module from that selective debug output,
please send that cmake debug output to me to see if I can use that
information to help you out.

But I emphasize again, all the above ideas are for after your completion
of the noninteractive tests.

Alan
__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________

Re: [Plplot-devel] Planning for the release of 5.13.0 - findings regarding Cygwin

[Alan W. I. <ir...@be...>]

On 2017-08-14 09:35-0000 Arjen Markus wrote:

> Hi Alan,
>
>
>
> I will ignore wxWidgets for the moment. Cmake seems to find something, but is quiet about the reasons for its final decision:
>
>
>
> -- wxWidgets_FOUND             : FALSE
>
> -- wxWidgets_INCLUDE_DIRS      : /usr/lib/wx/include/gtk3-unicode-3.0;/usr/include/wx-3.0
>
> -- wxWidgets_LIBRARIES         :
>
> -- wxWidgets_LIBRARY_DIRS      :
>
> -- wxWidgets_DEFINITIONS       : _FILE_OFFSET_BITS=64;WXUSINGDLL;__WXGTK__
>
> -- wxWidgets_DEFINITIONS_DEBUG :
>
> -- WARNING: wxWidgets or its libraries not found so setting all wxwidgets devices to OFF.
>
> -- WARNING: PLD_wxwidgets is OFF so setting ENABLE_wxwidgets to OFF.
>
> -- WARNING: ENABLE_wxwidgets is OFF so setting all wxwidgets devices to OFF.
>
>
>
> But analysing that will be for another time.

Yes.  You should complete noninteractive testing first since wxwidgets
issues (unless they are actual compile or link issues) don't affect
noninteractive tests.  But right after that (still Tuesday, I hope),
you should run cmake with the --debug-output and --trace options to
help you to locate and fix the source of this wxwidgets find issue very quickly.

Alan

__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________

Re: [Plplot-devel] Planning for the release of 5.13.0 - findings regarding Cygwin

[Arjen M. <Arj...@de...>]

Hi Alan,



I will ignore wxWidgets for the moment. Cmake seems to find something, but is quiet about the reasons for its final decision:



-- wxWidgets_FOUND             : FALSE

-- wxWidgets_INCLUDE_DIRS      : /usr/lib/wx/include/gtk3-unicode-3.0;/usr/include/wx-3.0

-- wxWidgets_LIBRARIES         :

-- wxWidgets_LIBRARY_DIRS      :

-- wxWidgets_DEFINITIONS       : _FILE_OFFSET_BITS=64;WXUSINGDLL;__WXGTK__

-- wxWidgets_DEFINITIONS_DEBUG :

-- WARNING: wxWidgets or its libraries not found so setting all wxwidgets devices to OFF.

-- WARNING: PLD_wxwidgets is OFF so setting ENABLE_wxwidgets to OFF.

-- WARNING: ENABLE_wxwidgets is OFF so setting all wxwidgets devices to OFF.



But analysing that will be for another time.



Regards,



Arjen



> -----Original Message-----
> From: Alan W. Irwin [mailto:ir...@be...]
> Sent: Monday, August 14, 2017 11:27 AM
> To: Arjen Markus
> Cc: PLplot development list
> Subject: RE: [Plplot-devel] Planning for the release of 5.13.0 - findings regarding
> Cygwin
>
> On 2017-08-14 08:19-0000 Arjen Markus wrote:
>
> > It looks as if some component is missing - as it still didn't work
> when I removed version 2.8. Which component is a riddle though. At least it does
> not seem to be the shear logic itself.
>
> Where is the find process failing?  If the cmake output does not get to the version
> check and instead says that wxwidgets is not found, then my educated guess is
> there is some issue with cmake/modules/Findwxwidgets.cmake for the Cygwin case.
> You have debugged that logic previously for the MinGW-w64/MSYS2 case so you
> know what to do which is to insert some message(STATUS...) and/or try the cmake
> --debug-output and --trace options to see what is going on with that find module for
> the Cygwin case.
>
> Anyhow, wxwidgets does not matter for the noninteractive comprehensive test on
> Cygwin so please finish that test first and also the corresponding noninteractive
> comprehensive test for MinGW-w64/MSYS2, and deal with this Cygwin wxwidgets
> issue later (or even post-release since interactive testing on Cygwin is so slow
> compared to your other platforms).
>
> Alan
>
> __________________________
> Alan W. Irwin
>
> Astronomical research affiliation with Department of Physics and Astronomy,
> University of Victoria (astrowww.phys.uvic.ca).
>
> Programming affiliations with the FreeEOS equation-of-state implementation for
> stellar interiors (freeeos.sf.net); the Time Ephemerides project (timeephem.sf.net);
> PLplot scientific plotting software package (plplot.sf.net); the libLASi project
> (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux
> Brochure Project (lbproject.sf.net).
> __________________________
>
> Linux-powered Science
> __________________________

DISCLAIMER: This message is intended exclusively for the addressee(s) and may contain confidential and privileged information. If you are not the intended recipient please notify the sender immediately and destroy this message. Unauthorized use, disclosure or copying of this message is strictly prohibited. The foundation 'Stichting Deltares', which has its seat at Delft, The Netherlands, Commercial Registration Number 41146461, is not liable in any way whatsoever for consequences and/or damages resulting from the improper, incomplete and untimely dispatch, receipt and/or content of this e-mail.

Re: [Plplot-devel] Planning for the release of 5.13.0 - findings regarding Cygwin

[Alan W. I. <ir...@be...>]

On 2017-08-14 08:19-0000 Arjen Markus wrote:

> It looks as if some component is missing - as it still didn't work
when I removed version 2.8. Which component is a riddle though. At
least it does not seem to be the shear logic itself.

Where is the find process failing?  If the cmake output does not get
to the version check and instead says that wxwidgets is not found,
then my educated guess is there is some issue with
cmake/modules/Findwxwidgets.cmake for the Cygwin case.  You have
debugged that logic previously for the MinGW-w64/MSYS2 case so you
know what to do which is to insert some message(STATUS...) and/or try
the cmake --debug-output and --trace options to see what is going on
with that find module for the Cygwin case.

Anyhow, wxwidgets does not matter for the noninteractive comprehensive
test on Cygwin so please finish that test first and also the
corresponding noninteractive comprehensive test for MinGW-w64/MSYS2,
and deal with this Cygwin wxwidgets issue later (or even post-release
since interactive testing on Cygwin is so slow compared to your other
platforms).

Alan

__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________

Re: [Plplot-devel] Planning for the release of 5.13.0

[Alan W. I. <ir...@be...>]

On 2017-08-14 06:50-0000 Arjen Markus wrote:

> Hi Alan,
>
>
>
>> -----Original Message-----
>> From: Alan W. Irwin [mailto:ir...@be...]
>> Sent: Friday, August 11, 2017 6:33 PM
>>
>> Thanks for the reminder (on the MSYS2 list) that our build system automatically
>> drops pyqt4 without any build error due to the "missing pyqtconfig.py " issue on this
>> platform.  As a result, this step should simply read
>>
>> (2') Run the noninteractive comprehensive test and send the generated report
>> tarball to me.
>>
>
> :)
>
> Unfortunately, it has been rather quiet wrt this issue. I have not seen any other reactions.

I don't want to puff myself up too much, but I think I did really good
research there and came up with the definitive answer for you. In any
case, the others there felt there was nothing more to be said.  :-)

>>>
>>> (3) After success (with or without pyqt4) on MinGW-w64/MSYS2, follow
>>> up with the constrained noninteractive comprehensive test I suggested
>>> where you collect the ~62MB of plot results in a tarball and post that
>>> tarball somewhere I can download it for my viewing pleasure (hah).
>>
>> Once you perform (1), (2'), and (3), I should have two report tarballs to look at as
>> well as one ~62MB tarball collection of plot file results, and assuming those reveal
>> no more issues (other than the "missing pyqtconfig.py" issue we will be dealing with
>> after the release), then these should be the last noninteractive tests you need to
>> make on this platform before the release.
>>
> Yes, I hope to get those done today or tomorrow.

Good.  That sounds quite promising for getting the release done this
week.  Reaching that release goal assumes you would you be willing to
finish all noninteractive testing for both MinGW-w64/MSYS2 and Cygwin
today or Tuesday, and noninteractive testing of MSVC and
wxwidgets-only interactive testing on all 3 platforms started on
Tuesday (see below) and finished by Wednesday at the latest.

The other uncertainty (other than how much time you have for testing
this week) in the release schedule is I still have a set of wxwidgets
issues remaining here which is the uninitialized variable in the
wxwidgets superscript/subscript logic that you discovered with MSVC,
and several other related issues I discovered for that
superscript/subscript logic. These issues only affect the
noninteractive test for MSVC (a build issue due to the uninitialized
variable), and the wxwidgets-only interactive tests on all platforms.
Which is why I encourage you to start your final noninteractive
testing for MinGW-w64/MSYS2 and Cygwin immediately rather than
waiting for my set of fixes.

I am not done yet with this set of wxwidgets superscript/subscript
fixes, but the uninitialized variable issue is solved, and I had
several breakthroughs in understanding that wxwidgets code today so I
am pretty confident I can finish this up by early Tuesday (UTC) which
should get rid of the wxwidgets build error for the noninteractive
MSVC case, and also allow you to interactively test wxwidgets on all
three platforms.

In sum, if I can deliver the wxwidgets fixes and my final
noninteractive and interactive comprehensive test results for this
release by early Tuesday (UTC), and you can deliver all those
comprehensive test reports for this release by Wednesday, and if our
joint comprehensive tests reveal no showstopping issues, then I plan
to start the official release process that is documented in
README.Release_Manager_Cookbook on Wednesday, and likely finish the
release a day later.  Which is a most satisifying prospect!

Alan
__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________

Re: [Plplot-devel] Planning for the release of 5.13.0 - findings regarding Cygwin

[Arjen M. <Arj...@de...>]

Hi Alan,



> -----Original Message-----
> From: Alan W. Irwin [mailto:ir...@be...]
> Sent: Monday, August 14, 2017 10:16 AM
> > This 2.8 versus 3.0 issue is something we need to look into. For the moment I will
> just continue with 3.0 only.
>
> I don't really understand the module, but from the comments in
> cmake/modules/FindwxWidgets.cmake, that module supports version selection,
> and does the right thing if multiple versions are available.  So I have put into my
> post-release ToDo list that we should drop our own brute-force version check
> (which doesn't work for multiple versions like you had installed) and use the
> FindwxWidgets.cmake version check instead.
>
> But for now, you can work around the problem (as you have stated
> above) by installing 3.0 only,
>
It looks as if some component is missing - as it still didn't work when I removed version 2.8. Which component is a riddle though. At least it does not seem to be the shear logic itself.

Regards,

Arjen

DISCLAIMER: This message is intended exclusively for the addressee(s) and may contain confidential and privileged information. If you are not the intended recipient please notify the sender immediately and destroy this message. Unauthorized use, disclosure or copying of this message is strictly prohibited. The foundation 'Stichting Deltares', which has its seat at Delft, The Netherlands, Commercial Registration Number 41146461, is not liable in any way whatsoever for consequences and/or damages resulting from the improper, incomplete and untimely dispatch, receipt and/or content of this e-mail.

Re: [Plplot-devel] Planning for the release of 5.13.0 - findings regarding Cygwin

[Alan W. I. <ir...@be...>]

On 2017-08-14 07:00-0000 Arjen Markus wrote:

> Hi Alan,
>
>
>
> As promised: I have found that currently wxWidgets is not acknowledged by PLplot on my laptop and I had a look at the report to find out why, because in previous tests I did have it. Here is the situation:
>
> -        In december last year, the build did include wxWidgets, as witnessed by a CMake output report from that date that I have on my computer. Currently, however, it is refused because PLplot finds a version 2.8 of wxWidgets and not 3.0.
>
> -        My Cygwin installation has both 2.8 and 3.0 - the latter was updated after said report (a matter of comparing the file dates).
>
> -        Why PLplot now does not find that 3.0 version is not clear to me. I guess unstalling version 2.8 should work.
>
> -        I updated the setup utility for Cygwin a weeks ago, but that new version works differently than the previous one. Searching for "wxWidgets" does not show which packages were installed, but I just found out that searching for "wx" does. So I can uninstall 2.8 and try again.
>
>
>
> This 2.8 versus 3.0 issue is something we need to look into. For the moment I will just continue with 3.0 only.

I don't really understand the module, but from the comments in
cmake/modules/FindwxWidgets.cmake, that module supports version
selection, and does the right thing if multiple versions are
available.  So I have put into my post-release ToDo list that we
should drop our own brute-force version check (which doesn't work for
multiple versions like you had installed) and use the
FindwxWidgets.cmake version check instead.

But for now, you can work around the problem (as you have stated
above) by installing 3.0 only,

Alan
__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________

Re: [Plplot-devel] Planning for the release of 5.13.0 - findings regarding Cygwin

[Arjen M. <Arj...@de...>]

Hi Alan,



As promised: I have found that currently wxWidgets is not acknowledged by PLplot on my laptop and I had a look at the report to find out why, because in previous tests I did have it. Here is the situation:

-        In december last year, the build did include wxWidgets, as witnessed by a CMake output report from that date that I have on my computer. Currently, however, it is refused because PLplot finds a version 2.8 of wxWidgets and not 3.0.

-        My Cygwin installation has both 2.8 and 3.0 - the latter was updated after said report (a matter of comparing the file dates).

-        Why PLplot now does not find that 3.0 version is not clear to me. I guess unstalling version 2.8 should work.

-        I updated the setup utility for Cygwin a weeks ago, but that new version works differently than the previous one. Searching for "wxWidgets" does not show which packages were installed, but I just found out that searching for "wx" does. So I can uninstall 2.8 and try again.



This 2.8 versus 3.0 issue is something we need to look into. For the moment I will just continue with 3.0 only.



Regards,



Arjen



DISCLAIMER: This message is intended exclusively for the addressee(s) and may contain confidential and privileged information. If you are not the intended recipient please notify the sender immediately and destroy this message. Unauthorized use, disclosure or copying of this message is strictly prohibited. The foundation 'Stichting Deltares', which has its seat at Delft, The Netherlands, Commercial Registration Number 41146461, is not liable in any way whatsoever for consequences and/or damages resulting from the improper, incomplete and untimely dispatch, receipt and/or content of this e-mail.

Re: [Plplot-devel] Planning for the release of 5.13.0

[Arjen M. <Arj...@de...>]

Hi Alan,



> -----Original Message-----
> From: Alan W. Irwin [mailto:ir...@be...]
> Sent: Friday, August 11, 2017 6:33 PM
>
> Thanks for the reminder (on the MSYS2 list) that our build system automatically
> drops pyqt4 without any build error due to the "missing pyqtconfig.py " issue on this
> platform.  As a result, this step should simply read
>
> (2') Run the noninteractive comprehensive test and send the generated report
> tarball to me.
>

:)

Unfortunately, it has been rather quiet wrt this issue. I have not seen any other reactions.
> >
> > (3) After success (with or without pyqt4) on MinGW-w64/MSYS2, follow
> > up with the constrained noninteractive comprehensive test I suggested
> > where you collect the ~62MB of plot results in a tarball and post that
> > tarball somewhere I can download it for my viewing pleasure (hah).
>
> Once you perform (1), (2'), and (3), I should have two report tarballs to look at as
> well as one ~62MB tarball collection of plot file results, and assuming those reveal
> no more issues (other than the "missing pyqtconfig.py" issue we will be dealing with
> after the release), then these should be the last noninteractive tests you need to
> make on this platform before the release.
>
Yes, I hope to get those done today or tomorrow.

Meanwhile I found something else which is puzzling, about the Cygwin environment this time. I will describe that in a separate post.

Regards,

Arjen

DISCLAIMER: This message is intended exclusively for the addressee(s) and may contain confidential and privileged information. If you are not the intended recipient please notify the sender immediately and destroy this message. Unauthorized use, disclosure or copying of this message is strictly prohibited. The foundation 'Stichting Deltares', which has its seat at Delft, The Netherlands, Commercial Registration Number 41146461, is not liable in any way whatsoever for consequences and/or damages resulting from the improper, incomplete and untimely dispatch, receipt and/or content of this e-mail.

Re: [Plplot-devel] Planning for the release of 5.13.0

[Alan W. I. <ir...@be...>]

On 2017-08-09 12:06-0700 Alan W. Irwin wrote:

> On 2017-08-09 10:46-0000 Arjen Markus wrote:
>
>> Hi Alan,
>> 
>> 
>> 
>> My apologises - I thought I had selected the right one. Here is the correct 
>> tarball. More comments below.
>
> Hi Arjen:
>
> I have now looked at what you sent me
> (comprehensive_test.tar_mingw_aug.gz), and it is an older result
> without qt4 or diff i.e., not what I asked for.
>
> But actually that missing report tarball does not really matter since
> that test with qt4 and diff that you described did not have git
> installed or all packages you needed for pyqt4. Therefore, here is
> what I would like you to do to finish off noninteractive testing
> on the MinGW-w64/MSYS2 platform.
>
> (1) Install a new snapshot (i.e., with its own unique install prefix)
> from scratch that includes git, diffutils, qt4, and all the
> pyqt4-related packages that we discussed (including
> mingw-w64-x86_64-python3-pyqt4). This install procedure (which I hope you 
> have
> almost entirely automated by now) should not only install all needed
> PLplot packages, but also remove any doubts about the effect of
> overwriting your current MinGW-w64/MSYS2 snapshot system files by
> attempting to use external packages rather than official
> MinGW-w64/MSYS2 installs to update your current system snapshot.
>
> (2) Run the noninteractive comprehensive test and send the generated
> report tarball to me.  If that script does not show
> a pyqt4 build error, that will be the
> report for MinGW-w64/MSYS2 that I summarize on our Wiki prior to the
> release of 5.13.0. However, in the event that the report tarball shows
> a pyqt4 build failure, then please send that report to me as well as a
> report for a subsequent script launch where you disabled the pyqt4
> componenent of PLplot. The former report would help me to advise you
> after the 5.13.0 release about what to do about pyqt4 and the latter
> will be the report for MinGW-w64/MSYS2 that I summarize on our Wiki
> prior to the release of 5.13.0 (including a note about why the pyqt4
> component of PLplot had to be dropped from the test).

Thanks for the reminder (on the MSYS2 list) that our build system
automatically drops pyqt4 without any build error due to the "missing
pyqtconfig.py " issue on this platform.  As a result, this step should
simply read

(2') Run the noninteractive comprehensive test and send the generated
report tarball to me.

>
> (3) After success (with or without pyqt4) on MinGW-w64/MSYS2, follow
> up with the constrained noninteractive comprehensive test I suggested
> where you collect the ~62MB of plot results in a tarball and post that
> tarball somewhere I can download it for my viewing pleasure (hah).

Once you perform (1), (2'), and (3), I should have two report tarballs
to look at as well as one ~62MB tarball collection of plot file
results, and assuming those reveal no more issues (other than the
"missing pyqtconfig.py" issue we will be dealing with after the
release), then these should be the last noninteractive tests
you need to make on this platform before the release.

Alan
__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________

Re: [Plplot-devel] Finding the sip system directory and sip command flags for PyQt4 and PyQt5

[Arjen M. <Arj...@de...>]

Hi Alan,



> -----Original Message-----
> From: Alan W. Irwin [mailto:ir...@be...]
> Sent: Thursday, August 10, 2017 11:19 PM
...

> * The PyQt4 case
>
>    In this case we currently depend on the pyqtconfig Python module to
>    find the sip system directory and sip command flags, and although
>    both the Debian and Cygwin packagers of PyQt4 provide this
>    deprecated (by the PyQt4 developers) module, the MinGW-w64/MSYS2
>    packager of PyQt4 does not.  For more details about this situation,
>    see the recent discussion on the MSYS2 mailing list that Arjen
>    started with the subject line "How to complete the installation of
>    PyQt4?" Arjen is going to submit a MinGW-w64/MSYS2 bug report
>    concerning this MinGW-w64/MSYS2 issue.  However, they may decide not to
>    supply this deprecated module, and by definition of what deprecated
>    means, it is virtually guaranteed that all software distributions
>    will eventually stop providing pyqtconfig so we have to find
>    an alternative to use for our build system.
>
>

FIY, I have reported this issue in https://github.com/msys2/msys2/issues/69.

Regards,

Arjen

DISCLAIMER: This message is intended exclusively for the addressee(s) and may contain confidential and privileged information. If you are not the intended recipient please notify the sender immediately and destroy this message. Unauthorized use, disclosure or copying of this message is strictly prohibited. The foundation 'Stichting Deltares', which has its seat at Delft, The Netherlands, Commercial Registration Number 41146461, is not liable in any way whatsoever for consequences and/or damages resulting from the improper, incomplete and untimely dispatch, receipt and/or content of this e-mail.

[Plplot-devel] Finding the sip system directory and sip command flags for PyQt4 and PyQt5

[Alan W. I. <ir...@be...>]

Our build system's abilities to find the sip system directory and sip
command flags for PyQt4 and PyQt5 are not implemented very well. So
after the 5.13.0 release has been completed, I want to improve this
situation.

The current issues that will be part of the 5.13.0 release are as follows:

* The PyQt4 case

   In this case we currently depend on the pyqtconfig Python module to
   find the sip system directory and sip command flags, and although
   both the Debian and Cygwin packagers of PyQt4 provide this
   deprecated (by the PyQt4 developers) module, the MinGW-w64/MSYS2
   packager of PyQt4 does not.  For more details about this situation,
   see the recent discussion on the MSYS2 mailing list that Arjen
   started with the subject line "How to complete the installation of
   PyQt4?" Arjen is going to submit a MinGW-w64/MSYS2 bug report
   concerning this MinGW-w64/MSYS2 issue.  However, they may decide not to
   supply this deprecated module, and by definition of what deprecated
   means, it is virtually guaranteed that all software distributions
   will eventually stop providing pyqtconfig so we have to find
   an alternative to use for our build system.

   The PyQt4 developers have decided to deprecate this module because
   they claim all it is really good for is to determine the sip system
   directory and sip command flags where other alternatives exist to
   determine those. That turns out to be our exact use case (see
   cmake/modules/qt.cmake) so using those alternatives does appear to be
   a promising approach.  Indeed, from
   <http://pyqt.sourceforge.net/Docs/PyQt4/build_system.html>) it does
   appear that you can determine the sip flags without the pyqtconfig
   module using PyQt4.QtCore.PYQT_CONFIGURATION which is

   "a dict that describes how
   PyQt was configured. At the moment it contains a single value called
   sip_flags which is a string containing the appropriate -t and -x
   flags."

   So that alternative looks OK.

   At this stage it is not clear what is the alternative to using
   pyqtconfig to find the sip system directory, but I believe the
   "fool-proof" approach here is to determine a standard sip file that
   is guaranteed to always be installed in that directory, and use the
   CMake find_path command to find the directory that contains that
   file.

* The PyQt5 case

   The PyQT developers (consistent with their deprecation warning for the PyQt4 case)
   do not provide the pyqtconfig Python module at all for this case.

   So our current approach is to use the above PYQT_CONFIGURATION method for determinng
   sip flags, and it appears to work (at least on Linux but probably on
   all other systems as well).  But our current approach to determining
   the sip system directory is problematic.  For example, it has
   the comment

# FIXME: This is almost surely wrong

   for the WIN32 AND NOT CYGWIN part of the logic that would
   have been used if Arjen had tried out Qt5 (and PyQt5) on
   MinGW-w64/MSYS2.

   So I think the solution here is to try the idea I had above
   for determining the sip system directory, but we will need
   to experiment on all platforms to find out what is the best approach.

While writing the above summary, I *finally* remembered to search our
mailing list archive for previous mentions of pyqtconfig, and it turns
out that Greg Jung thoroughly discussed this issue two years ago!  In
fact, at that time Greg sent me in an off-list e-mail a patch to deal
with the "missing pyqtconfig issue" for pyqt4 that attempted to find
the sip system directory in what looks like a fairly problematic way
and the sip command flags using the above PYQT_CONFIGURATION way. I
didn't apply that patch at the time because of style issues in it, and
because of the problematic nature of how the sip system directory was
found.  Also, Greg said at the time that as a result of his patch our
build system configured the pyqt4 component build without any obvious
configure-time issues, but the actual build itself failed.

So it appears that after the release of 5.13.0, Arjen and I have a
relatively small amount of work to do to figure out the remaining
problematic configuration issues for determining the sip system
directory for pyqt4 and pyqt5 on Cygwin and MinGW-w64/MSYS2.  But then
after that is all solved it is possible (unless MinGW-w64/MSYS2 has
been improved in the last two years in this regard) that additional
work might be required to get both pyqt4 and pyqt5 built and running
properly on MinGW-w64/MSYS2.  And Arjen will likely want to make
similar tests of both pyqt4 and pyqt5 on Cygwin (since interactive
testing on Cygwin has not been done very often in the past and never
for Qt5), and he might turn up some configure, build or run time
issues that we have to address in that case as well.

In sum, the goal is to have a good run-time experience for our pyqt4
(if we are using Qt4) or our pyqt5 (if we are using Qt5) components on
both MinGW-w64/MSYS2 and Cygwin similar to what is now possible for
Linux.  To achieve that goal is obviously going to take quite a bit of
testing and development effort, but we do have a good start and at
least the remaining configuration issues when we move completely away
from the deprecated pyqtconfig Python module are now clear.  So I hope
we do reach this goal in the early stages of the release cycle just
after 5.13.0 is released, and to remind me of that possibility I have
summarized what needs to be done in my post-5.13.0 ToDo list.

Alan
__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________

Re: [Plplot-devel] Planning for the release of 5.13.0

[Arjen M. <Arj...@de...>]

Hi Alan,



> -----Original Message-----
> From: Alan W. Irwin [mailto:ir...@be...]
> Sent: Wednesday, August 09, 2017 7:56 PM
>
> "MSYS Makefiles" historically worked for MinGW/MSYS so I would expect (as you
> have found) it would continue to work for MinGW-w64/MSYS2.
> From your results it also appears that "Unix Makefiles" (which I never tried for
> MinGW/MSYS) almost works now for MinGW-w64/MSYS2.  It is possible we could
> get "Unix Makefiles" to work completely for MinGW-w64/MSYS with some changes
> to cmake/modules/FindwxWidgets.cmake,
> but I think that is something we should only pursue post-release.
>
No problem - CMake for MinGW-w64/MSYS2 lists three possible generators and IIRC the MinGW one complains about bash. Which is why I have used the other two. And then stumbled on the difference.

> > No luck sofar with PyQt4 - it appears I need both SIP (got that),
> pyqt4 (got that) and pyqtconfig (presenting a puzzle). The latter component has me
> puzzled, as I do not know anything about Python package management. First I
> downloaded a tarball for pyqtconfig, from PyPI, but it has no instructions on the
> actual installation. It seems I need PIP, so I got that, but that does not show up as
> a runnable command (either in the command shell or in the Python shell). So I am
> stuck, unless it is a matter of unpacking the tarball in the right location.
>
> > Please advise me on how to proceed.
>
> If what you have done is unpacked an external tarball (i.e., the pyqtconfig tarball
> you mentioned) on top of your existing
> MinGW-w64/MSYS2 snapshot system files, then I cannot emphasize enough, that
> procedure is absolutely not recommended.  The issue is that overwriting your
> system that way can introduce system incompatibilities and even security issues for
> your platform.

No, I expressly did not.

>
> I suggested you install mingw-w64-x86_64-python3-sip AND all its dependencies,
> and I assume you have done that, but the issue is likely that you need to install one
> or more additional packages to get access to "pyqtconfig"
>
As you can see in the attached report from "pacman", the package manager under MinGW-w64/MSYS2, I have installed both SIP and PyQt4 for Python 3. I have searched the installation directory for any file "pyqt*.*", but it did not turn up "pyqtconfig.py" or the like. When I tried "pacman -Q -o pyqtconfig.py" to search for the package that "owns" that file, I got the reply that there was nothing there. Though I am not sure if it is the correct command - it complains about the file not being found via the PATH.

> For example, on Debian Jessie my search for packages which contain any file with
> "pyqtconfig" somewhere in the filename yielded the following:
>
> irwin@raven> apt-file search pyqtconfig
> python-qt4: /usr/lib/python2.7/dist-packages/PyQt4/pyqtconfig.py
> python-qt4-dbg: /usr/lib/python2.7/dist-packages/PyQt4/pyqtconfig_d.py
> python3-pyqt4: /usr/lib/python3/dist-packages/PyQt4/pyqtconfig.py
> python3-pyqt4: /usr/lib/python3/dist-packages/PyQt4/pyqtconfig_nd4.py
> python3-pyqt4-dbg: /usr/lib/python3/dist-packages/PyQt4/pyqtconfig_d4.py
>
> I only have the names of the MinGW-w64/MSYS2 packages available to me and
> not their file contents, but if you currently do not have
> mingw-w64-x86_64-python3-pyqt4 installed, I suggest you install that package to
> see if that solves the missing pyqtconfig issue.

That is not the solution.

I will ask on the MinGW-w64/MSYS2 mailing list for advice.

Regards,

Arjen



DISCLAIMER: This message is intended exclusively for the addressee(s) and may contain confidential and privileged information. If you are not the intended recipient please notify the sender immediately and destroy this message. Unauthorized use, disclosure or copying of this message is strictly prohibited. The foundation 'Stichting Deltares', which has its seat at Delft, The Netherlands, Commercial Registration Number 41146461, is not liable in any way whatsoever for consequences and/or damages resulting from the improper, incomplete and untimely dispatch, receipt and/or content of this e-mail.

Re: [Plplot-devel] Planning for the release of 5.13.0

[Alan W. I. <ir...@be...>]

On 2017-08-09 10:46-0000 Arjen Markus wrote:

> Hi Alan,
>
>
>
> My apologises - I thought I had selected the right one. Here is the correct tarball. More comments below.

Hi Arjen:

I have now looked at what you sent me
(comprehensive_test.tar_mingw_aug.gz), and it is an older result
without qt4 or diff i.e., not what I asked for.

But actually that missing report tarball does not really matter since
that test with qt4 and diff that you described did not have git
installed or all packages you needed for pyqt4. Therefore, here is
what I would like you to do to finish off noninteractive testing
on the MinGW-w64/MSYS2 platform.

(1) Install a new snapshot (i.e., with its own unique install prefix)
from scratch that includes git, diffutils, qt4, and all the
pyqt4-related packages that we discussed (including
mingw-w64-x86_64-python3-pyqt4). This install procedure (which I hope you have
almost entirely automated by now) should not only install all needed
PLplot packages, but also remove any doubts about the effect of
overwriting your current MinGW-w64/MSYS2 snapshot system files by
attempting to use external packages rather than official
MinGW-w64/MSYS2 installs to update your current system snapshot.

(2) Run the noninteractive comprehensive test and send the generated
report tarball to me.  If that script does not show
a pyqt4 build error, that will be the
report for MinGW-w64/MSYS2 that I summarize on our Wiki prior to the
release of 5.13.0. However, in the event that the report tarball shows
a pyqt4 build failure, then please send that report to me as well as a
report for a subsequent script launch where you disabled the pyqt4
componenent of PLplot. The former report would help me to advise you
after the 5.13.0 release about what to do about pyqt4 and the latter
will be the report for MinGW-w64/MSYS2 that I summarize on our Wiki
prior to the release of 5.13.0 (including a note about why the pyqt4
component of PLplot had to be dropped from the test).

(3) After success (with or without pyqt4) on MinGW-w64/MSYS2, follow
up with the constrained noninteractive comprehensive test I suggested
where you collect the ~62MB of plot results in a tarball and post that
tarball somewhere I can download it for my viewing pleasure (hah).

Alan
__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________

Re: [Plplot-devel] Planning for the release of 5.13.0

[Alan W. I. <ir...@be...>]

On 2017-08-09 10:46-0000 Arjen Markus wrote:

>> -----Original Message-----
>> From: Alan W. Irwin [mailto:ir...@be...]
>> Sent: Wednesday, August 09, 2017 10:51 AM
>>
>> That is a report for Cygwin which you apparently ran on 2017-08-07, but which you
>> have not described yet.  Is that the one where you addressed all the remaining
>> Cygwin platform regressions?
>>
> No, I thought I would run the tests again without the "-DNON_TRANSITIVE" option to make sure nothing ontowards would occur.

OK.  That sounds like that was an interim Cygwin result that I should ignore.

>
>> What I requested was the report for the MinGW-w64/MSYS2 test you did recently
>> with Qt4 and diffutils installed where there were unexpected PostScript differences.
>>
> Note:
>

> I noticed there is a difference between the results with the "MSYS
Makefiles" generator and the "Unix Makefiles" generator using
MinGW-w64/MSYS2. It is subtle (see the extra files with only the
output from CMake) but it causes wxWidgets not to be found in the
latter case. The recommendation should probably be (at this moment) to
use the former, though in principle there should not matter, I think.

"MSYS Makefiles" historically worked for MinGW/MSYS so I would expect
(as you have found) it would continue to work for MinGW-w64/MSYS2.
>From your results it also appears that "Unix Makefiles" (which I never
tried for MinGW/MSYS) almost works now for MinGW-w64/MSYS2.  It is
possible we could get "Unix Makefiles" to work completely for
MinGW-w64/MSYS with some changes to cmake/modules/FindwxWidgets.cmake,
but I think that is something we should only pursue post-release.

> No luck sofar with PyQt4 - it appears I need both SIP (got that),
pyqt4 (got that) and pyqtconfig (presenting a puzzle). The latter
component has me puzzled, as I do not know anything about Python
package management. First I downloaded a tarball for pyqtconfig, from
PyPI, but it has no instructions on the actual installation. It seems
I need PIP, so I got that, but that does not show up as a runnable
command (either in the command shell or in the Python shell). So I am
stuck, unless it is a matter of unpacking the tarball in the right
location.

> Please advise me on how to proceed.

If what you have done is unpacked an external tarball (i.e., the
pyqtconfig tarball you mentioned) on top of your existing
MinGW-w64/MSYS2 snapshot system files, then I cannot emphasize enough,
that procedure is absolutely not recommended.  The issue is that
overwriting your system that way can introduce system
incompatibilities and even security issues for your platform.

So if you did overwrite your system that way, then to retrieve the
  situation (i.e., go back to a pure form of MinGW-w64/MSYS2 without
  such overwriting you need to install a new snapshot from scratch. So
  I hope you have automated that procedure as I suggested to make that
  procedure convenient for you to do.

Once you are back to pure MinGW-w64/MSYS2 (or if you are already there
because you did not do what I described above), then the question
is why is pyqtconfig missing (which I presume is the issue you
were trying to solve by unpacking that external tarball)?

I suggested you install mingw-w64-x86_64-python3-sip AND all its
dependencies, and I assume you have done that, but the issue
is likely that you need to
install one or more additional packages to get access to "pyqtconfig"

For example, on Debian Jessie my search for packages which contain
any file with "pyqtconfig" somewhere in the filename yielded the following:

irwin@raven> apt-file search pyqtconfig
python-qt4: /usr/lib/python2.7/dist-packages/PyQt4/pyqtconfig.py
python-qt4-dbg: /usr/lib/python2.7/dist-packages/PyQt4/pyqtconfig_d.py
python3-pyqt4: /usr/lib/python3/dist-packages/PyQt4/pyqtconfig.py
python3-pyqt4: /usr/lib/python3/dist-packages/PyQt4/pyqtconfig_nd4.py
python3-pyqt4-dbg: /usr/lib/python3/dist-packages/PyQt4/pyqtconfig_d4.py

I only have the names of the MinGW-w64/MSYS2 packages available to
me and not their file contents, but if you currently do not have
mingw-w64-x86_64-python3-pyqt4 installed, I suggest you install
that package to see if that solves the missing pyqtconfig issue.

By the way, this missing files problem will likely occur many more
times for you with MinGW-w64/MSYS2. And you obviously should not have
to guess (which is what we are doing here) at MinGW-w64/MSYS2 packages
you need to install based on what files Debian provides in their own
packages.  Therefore, I strongly suggest you should ask on the MSYS2
list what is the recommended equivalent of the Debian apt-file command
to search all (both installed and uninstalled) MinGW-w64/MSYS2
packages for any given filename pattern.

This issue of finding filename patterns in system packages has come up
before between us, and at that time I did a google search and found
somone else had asked that question on the MSYS2 list several years
ago, but the answer provided then did not work for you. So it is now
time for you to ask that question directly and iterate with the list
until you do have an answer that actually works for modern
MinGW-w64/MSYS2.

I will now take a look at your latest report tarball and comment
separately on what I find there.

Alan

__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________

Re: [Plplot-devel] Planning for the release of 5.13.0

[Arjen M. <Arj...@de...>]

Hi Alan,



My apologises - I thought I had selected the right one. Here is the correct tarball. More comments below.



Regards,



Arjen

> -----Original Message-----
> From: Alan W. Irwin [mailto:ir...@be...]
> Sent: Wednesday, August 09, 2017 10:51 AM
>
> That is a report for Cygwin which you apparently ran on 2017-08-07, but which you
> have not described yet.  Is that the one where you addressed all the remaining
> Cygwin platform regressions?
>
No, I thought I would run the tests again without the "-DNON_TRANSITIVE" option to make sure nothing ontowards would occur.

> What I requested was the report for the MinGW-w64/MSYS2 test you did recently
> with Qt4 and diffutils installed where there were unexpected PostScript differences.
>
Note:

I noticed there is a difference between the results with the "MSYS Makefiles" generator and the "Unix Makefiles" generator using MinGW-w64/MSYS2. It is subtle (see the extra files with only the output from CMake) but it causes wxWidgets not to be found in the latter case. The recommendation should probably be (at this moment) to use the former, though in principle there should not matter, I think.

No luck sofar with PyQt4 - it appears I need both SIP (got that), pyqt4 (got that) and pyqtconfig (presenting a puzzle). The latter component has me puzzled, as I do not know anything about Python package management. First I downloaded a tarball for pyqtconfig, from PyPI, but it has no instructions on the actual installation. It seems I need PIP, so I got that, but that does not show up as a runnable command (either in the command shell or in the Python shell). So I am stuck, unless it is a matter of unpacking the tarball in the right location.

Please advise me on how to proceed.



DISCLAIMER: This message is intended exclusively for the addressee(s) and may contain confidential and privileged information. If you are not the intended recipient please notify the sender immediately and destroy this message. Unauthorized use, disclosure or copying of this message is strictly prohibited. The foundation 'Stichting Deltares', which has its seat at Delft, The Netherlands, Commercial Registration Number 41146461, is not liable in any way whatsoever for consequences and/or damages resulting from the improper, incomplete and untimely dispatch, receipt and/or content of this e-mail.

Re: [Plplot-devel] Planning for the release of 5.13.0

[Alan W. I. <ir...@be...>]

On 2017-08-09 06:54-0000 Arjen Markus wrote:

> Hi Alan,
>
>
>
>> -----Original Message-----
>> From: Alan W. Irwin [mailto:ir...@be...]
>> Sent: Tuesday, August 08, 2017 8:16 PM
>>
>> Hi Arjen:
>>
>> IMPORTANT.  Could I please see the full report tarball of this test which you forgot
>> to attach to your last post?  Assuming this test finished without any showstopping
>> errors, I need that report tarball (and the exact PLplot commit you tested) to
>> complete the wiki report on your fundamentally successful noninteractive testing of
>> the
>> MinGW-w64/MSYS2 platform.
>>
>
> Oops, here is the tar-ball.

That is a report for Cygwin which you apparently ran on 2017-08-07,
but which you have not described yet.  Is that the one where you
addressed all the remaining Cygwin platform regressions?

What I requested was the report for the MinGW-w64/MSYS2 test you did
recently with Qt4 and diffutils installed where there were unexpected
PostScript differences.

>
>> I think PostScript differences like above are not release critical (i.e., they are
>> something we will only want to pursue after 5.13.0 is out).  But before I declare your
>> 5.13.0 noninteractive testing completed for this particular platform, I would like you
>> to do the following:
>>
>> After sending me the above important report tarball, run this noninteractive
>> comprehensive test again, but with these additional
>> options:
>>
>> --do_clean_as_you_go no
>> --do_nondynamic no
>> --do_static no
>> --do_ctest no
>> --do_test_install_tree no
>> --do_test_traditional_install_tree no
>>
>> These options keep all plplot results for a limited test case (only the shared library
>> build, only the build-tree, only the test_noninteractive target) which has 12 times
>> less plot results than normal.  Please collect these "limited" results as follows (the
>> trailing "." is significant)
>>
>> tar zcf noninteractive_results.tar.gz -
>> C ../comprehensive_test_disposeable/shared/noninteractive/build_tree/examples/te
>> st_examples_output_dir .
>>
>
> What should be the directory in which to issue this command? I guess you want some files from that directory, given the trailing dot.

Top directory of the source tree.

The -C option internally changes to that given directory within the
comprehensive testing tree.  The "."
grabs everything in that directory, i.e., all the various kinds
of plot files generated by the test_noninteractive target built
by the comprehensive test script.

Alan


>
>> When I did the equivalent command here, the resulting tarball was 62MB so that is
>> too large to send to me by e-mail.  But can you place that tarball temporarily on
>> some website where I can download it?  And, of course, I would like you to send
>> me the report tarball for this more limited test by e-mail as well just in case there is
>> something I need to check when looking at these results.
>>
> That ought to be no problem.
>
> Regards,
>
> Arjen
>
> DISCLAIMER: This message is intended exclusively for the addressee(s) and may contain confidential and privileged information. If you are not the intended recipient please notify the sender immediately and destroy this message. Unauthorized use, disclosure or copying of this message is strictly prohibited. The foundation 'Stichting Deltares', which has its seat at Delft, The Netherlands, Commercial Registration Number 41146461, is not liable in any way whatsoever for consequences and/or damages resulting from the improper, incomplete and untimely dispatch, receipt and/or content of this e-mail.
>

__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________

Re: [Plplot-devel] Planning for the release of 5.13.0

[Arjen M. <Arj...@de...>]

Hi Alan,



> -----Original Message-----
> From: Alan W. Irwin [mailto:ir...@be...]
> Sent: Tuesday, August 08, 2017 8:16 PM
>
> Hi Arjen:
>
> IMPORTANT.  Could I please see the full report tarball of this test which you forgot
> to attach to your last post?  Assuming this test finished without any showstopping
> errors, I need that report tarball (and the exact PLplot commit you tested) to
> complete the wiki report on your fundamentally successful noninteractive testing of
> the
> MinGW-w64/MSYS2 platform.
>

Oops, here is the tar-ball.

> I think PostScript differences like above are not release critical (i.e., they are
> something we will only want to pursue after 5.13.0 is out).  But before I declare your
> 5.13.0 noninteractive testing completed for this particular platform, I would like you
> to do the following:
>
> After sending me the above important report tarball, run this noninteractive
> comprehensive test again, but with these additional
> options:
>
> --do_clean_as_you_go no
> --do_nondynamic no
> --do_static no
> --do_ctest no
> --do_test_install_tree no
> --do_test_traditional_install_tree no
>
> These options keep all plplot results for a limited test case (only the shared library
> build, only the build-tree, only the test_noninteractive target) which has 12 times
> less plot results than normal.  Please collect these "limited" results as follows (the
> trailing "." is significant)
>
> tar zcf noninteractive_results.tar.gz -
> C ../comprehensive_test_disposeable/shared/noninteractive/build_tree/examples/te
> st_examples_output_dir .
>

What should be the directory in which to issue this command? I guess you want some files from that directory, given the trailing dot.

> When I did the equivalent command here, the resulting tarball was 62MB so that is
> too large to send to me by e-mail.  But can you place that tarball temporarily on
> some website where I can download it?  And, of course, I would like you to send
> me the report tarball for this more limited test by e-mail as well just in case there is
> something I need to check when looking at these results.
>
That ought to be no problem.

Regards,

Arjen

DISCLAIMER: This message is intended exclusively for the addressee(s) and may contain confidential and privileged information. If you are not the intended recipient please notify the sender immediately and destroy this message. Unauthorized use, disclosure or copying of this message is strictly prohibited. The foundation 'Stichting Deltares', which has its seat at Delft, The Netherlands, Commercial Registration Number 41146461, is not liable in any way whatsoever for consequences and/or damages resulting from the improper, incomplete and untimely dispatch, receipt and/or content of this e-mail.

Re: [Plplot-devel] Planning for the release of 5.13.0

[Alan W. I. <ir...@be...>]

On 2017-08-08 09:32-0000 Arjen Markus wrote:

[...] 
[A]n interesting report on Lua:
>
> Comparison test using psc device
>
> ...
>
> lua
>
>  Missing examples            :
>
>  Differing graphical output  :  14a 19 23
>
>  Missing stdout              :
>
>  Differing stdout            :  31
>

> These differences pop up in various configurations. I have not seen
them in the Cygwin results, so this is something specific to
MinGW-w64/MSYS2.

Hi Arjen:

IMPORTANT.  Could I please see the full report tarball of this test
which you forgot to attach to your last post?  Assuming this test
finished without any showstopping errors, I need that report tarball
(and the exact PLplot commit you tested) to complete the wiki report
on your fundamentally successful noninteractive testing of the
MinGW-w64/MSYS2 platform.

I think PostScript differences like above are not release critical
(i.e., they are something we will only
want to pursue after 5.13.0 is out).  But before I declare your 5.13.0
noninteractive testing completed for this particular platform, I would
like you to do the following:

After sending me the above important report tarball, run this
noninteractive comprehensive test again, but with these additional
options:

--do_clean_as_you_go no
--do_nondynamic no
--do_static no
--do_ctest no
--do_test_install_tree no
--do_test_traditional_install_tree no

These options keep all plplot results for a limited test case (only the shared
library build, only the build-tree, only the test_noninteractive target)
which has 12 times less plot results than normal.  Please collect
these "limited" results as follows (the trailing "." is significant)

tar zcf noninteractive_results.tar.gz -C ../comprehensive_test_disposeable/shared/noninteractive/build_tree/examples/test_examples_output_dir .

When I did the equivalent command here, the resulting tarball was 62MB
so that is too large to send to me by e-mail.  But can you place that
tarball temporarily on some website where I can download it?  And, of
course, I would like you to send me the report tarball for this more
limited test by e-mail as well just in case there is something I need
to check when looking at these results.

The point is I would like to view all these plot results for this
"new" platform to see what they look like, and I have the bash
command-line skills, access to the ImageMagick "display" application,
and time and patience to do that.  Of course, for future reference I
will teach you how to do all this for yourself on all your platforms,
but you have more constraints on your time than I do so I am willing
to volunteer to do this monumental plot viewing task.  (And if my
eyeballs have not fallen out and I have not died from boredom due to
this task, I would likely be willing to repeat this task twice more
when you finish off the noninteractive testing on the Cygwin and MSVC
platforms.  :-) )

Alan
__________________________

Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________

Re: [Plplot-devel] Planning for the release of 5.13.0

[Alan W. I. <ir...@be...>]

On 2017-08-08 09:57-0000 Arjen Markus wrote:

> Hi Alan,
>
>
>
>> -----Original Message-----
>> From: Alan W. Irwin [mailto:ir...@be...]
>> Sent: Tuesday, August 08, 2017 11:46 AM
>>
>> So if you use make -j8 (or so) and ctest -j8 for your comprehensive test it will really
>> go ~4 times faster because those -j options will utilize your four cores completely.
>> But you have found in the past that the -j option gives unreliable results for both
>> make and ctest for both the Cygwin and MinGW-w64/MSYS2 platforms.  So you
>> have dropped these -j options with the result that you only use one of your cores,
>> but at least you do get slow but reliable results that way.
>>
> I explicitly use the build command "make" because of that, but I still see four instances of make running, as well as six instances of bash. This is with message "make VERBOSE=1 test_noninteractive in the installed examples build tree" as the last visible text.
>
> Not at all sure what this means, the information I get from the task manager does not reveal much about what these processes are actually doing (like in which directory etc.)

The cmake application configures a recursive series of Makefiles,
i.e., one make invokes another which invokes another.  And likely the
same is true of the shell (in this case bash) that is used to help
execute eacho of those nested Makefile rules.  So the real test would
be provided by a cpu meter which would measure the activity of your 4
cpus. And I predict without the -j options that would show an average
usage of 25 per cent for each of them or in other words you are only
using 1/4th the power of your machine with no -j options.

Alan
__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________