Menu
▾
▴
plplot-devel
|
From: Alan W. I. <ir...@be...> - 2017-02-13 06:51:52
|
On 2017-02-12 23:08-0000 Phil Rosenberg wrote: > Hi Alan > > The short answer is yes use a transmit and receive function to send > the data, but no don't create a new object every time you want to > send something. The current private object is definitely the way to > go. OK. I plan to stick with my local instance approach for now (automatically constructing a PLTwoSemaphore instance and destroying it again for each call to my transmitBytes and receiveBytes methods), but move to the private object approach for efficiency later on once I understand (hopefully with some further help from you) exactly what it does at run time compared with the local instance approach. > I should have used a transmit and receive function in the current > code. If I wasn't going to trample your uncommitted changes I would > actually do that right now. The reason is partitioning of > functionality. The job of the Memory map, as it stands is to provide a > space in memory where the wxDevice can write data. You are absolutely > correct that it would be much better to have a send and receive method Yes, that was my exact motivation. [...] > However, if you pull out the semaphores and have wxPLDevice look after > those then we have messed up our nice separation of functionality > again. Our wxPLDevice now knows that we are communicating with > semaphores and memory maps. > So what I should have done (and this probably came from the way things > evolved and got more complicated than I expected) is created a > PLCommunicator class. Then I could have inherited from this to make a > PLMemoryMapCommunicator and all the code from wxDevice::Transmit, > should go in there. In fact if you can give me some time to do that > without stepping on your uncommitted edits then I will as I think it > will make your life easier. Actually, I am right in the middle of modifying wxPLDevice::TransmitBuffer and wxPlFrame::ReadTransmission to use the transmitBytes and receiveBytes methods of PLMemoryMap (which are now completely implemented and they build, but I haven't pushed that commit because I have no way of testing those methods until the modifications to wxPLDevice::TransmitBuffer and wxPlFrame::ReadTransmission are done). So I have invested considerable time in understanding those routines as they are now organized. Also, my ongoing changes for the -DPL_WXWIDGETS_IPC2=ON case clearly separate assembling the data in wxPLDevice::TransmitBuffer and acting on it in wxPlFrame::ReadTransmission from the task of transmitting the assembled array of bytes from wxPLDevice::TransmitBuffer and receiving that same array of bytes in wxPlFrame::ReadTransmission. So my separation work (already mostly completed now) should give you a good start on that desired reorganization. So please hold off until later to reorganize the code. To summarize, here are the next steps I plan to do. (1) Get the two-semaphores approach fundamentally working one way from -dev wxwidgets to wxPLViewer by ignoring interactive examples and temporarily dropping text size feedback from wxPLViewer back to -dev wxwidgets and ultimately the PLplot core. (2) Refine that approach to make it two-way so that the -DPL_WXWIDGETS_IPC2=ON result passes all tests (including text size feedback as well as including example 1 with the -locate option) that are currently passed by the -DPL_WXWIDGETS_IPC2=OFF case. (3) Measure the efficiency of that local instance method and then try the private object method (once I understand exactly what it does at run-time) to see how much that efficiency improves. (4) Decide (with you) based on the simplicity and speed of the two-semaphores approach compared with the one-semaphore approach which approach we will take after that. (5) If the decision is for two semaphores, then the further steps I have in mind would be to (i) (Alan) implement a _named_ two-semaphores variation on the unnamed two-semaphores approach, (ii) (Phil) implement a named two-semaphores approach for the Windows case, and (iii) (Alan or Phil) completely remove all one-semaphore code. But if our decision goes against the two-semaphores approach, then (i') (Alan or Phil) strip all the two-semaphores code out instead but (ii') (Phil) keep the idea of separating the transmitBytes and receiveBytes methods (which have no logic other than to transmit and receive arrays of bytes) from everything else. It is not at all clear how long these steps are going to take, but I am hoping for a time scale of something like one day each for (1) through (3). And I hope you don't mind holding off any of the changes you are thinking about (especially the reorganizing ones) since such changes would slow me down, and we, in any case, really have no idea where we are going with the wxwidgets IPC until (4) is done. More later as I continue to make progress on (1) through (3). Alan __________________________ Alan W. Irwin Astronomical research affiliation with Department of Physics and Astronomy, University of Victoria (astrowww.phys.uvic.ca). Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); the Time Ephemerides project (timeephem.sf.net); PLplot scientific plotting software package (plplot.sf.net); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __________________________ Linux-powered Science __________________________ |
|
From: Alan W. I. <ir...@be...> - 2017-02-13 19:05:52
|
On 2017-02-12 22:51-0800 Alan W. Irwin wrote: > I plan to stick with my local instance approach for now (automatically > constructing a PLTwoSemaphore instance and destroying it again for > each call to my transmitBytes and receiveBytes methods), but move to > the private object approach for efficiency later on once I understand > (hopefully with some further help from you) exactly what it does at > run time compared with the local instance approach. Hi Phil: I have thought a lot more about this efficiency question using my current mental model of the private object approach (if we applied it for the case of PLMemoryMap accessing PLTwoSemapores). Please correct that mental model if I have made a mistake with it, but if that model is correct, then I predict (see below) there will be negligible efficiency differences between the present local instance approach and the private object approach. I assume to make PLTwoSemapores accessible as a private object from PLMemoryMap, I would move the current code in the PLTwoSemapores constructor to a create method, replace the current PLTwoSemapores constructor with one without arguments that basically forces the two semaphore addresses to be invalid (NULL) values, and move the current destructor code to a delete method so the destructor becomes a no-op. Then whenever PLMemoryMap was instantiated, the private PLTwoSemapores object that is mentioned in its private: section is also automatically instantiated (with the invalid addresses). Then within transmitBytes and receiveBytes methods, the create method for the private PLTwoSemapores object is called first, and the destroy method for that same object is called last. And ultimately when the PLMemoryMap object is destroyed, the no-op destructor for its private PLTwoSemapores object is also automatically called. If that is the correct model, then the code in create and destroy is the same as in the current PLTwoSemapores instantiation and destruction that occurs in the current transmitBytes and receiveBytes methods. So is your efficiency concern not that create and destroy code, but other code that must be executed if you do a full-blown instantiation and destruction of the PLTwoSemapores object? Or are you also concerned with the cpu cycles required to call the create and destroy methods for PLTwoSemapores every time the transmitBytes and receiveBytes methods were called? For example, you _could_ call the appropriate create and destroy methods outside transmitBytes and receiveBytes (and likely less often if you used a lot of care), but I would prefer not to do that because that non-locality makes the code much harder to understand (say when we look at it again 5 years from now) and I believe the number of cpu cycles saved should be negligible compared with the number of cpu cycles actually required to execute transmitBytes and receiveBytes. Come to think of it, the number of cpu cycles needed for a full-blown instantiation and destruction of a PLTwoSemapores object within transmitBytes and receiveBytes should also be negligible compared to even just the cpu costs of the memcpy calls that do the necessary chunking up of the array of bytes to be sent via shared memory with transitBytes and the similar memcpy costs on the receiving end to reassemble the array of bytes, and the argument becomes stronger if you add in the times for sem_wait to finish on the transmission and receive ends. So unless my mental model above is incorrect, I predict we will find no discernible difference in efficiency when comparing between the current local instance approach and the private object approach at least for the use of PLTwoSemapores by the transmitBytes and receiveBytes methods of PLMemoryMap. But I am willing to implement the private object approach variant (controlled by a preprocessing macro) and measure its efficiency compared with the current local instance approach to confirm (or disprove) that prediction, and once we have those test results, we can decide which approach to take. Alan __________________________ Alan W. Irwin Astronomical research affiliation with Department of Physics and Astronomy, University of Victoria (astrowww.phys.uvic.ca). Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); the Time Ephemerides project (timeephem.sf.net); PLplot scientific plotting software package (plplot.sf.net); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __________________________ Linux-powered Science __________________________ |
|
From: <p.d...@gm...> - 2017-02-13 20:24:37
|
I have absolutely no efficiency concerns. The cpu cycles will be totally negligible compared to the time to copy the memory. To be honest I rarely stress about efficiency. Premature optimisation is usually a bad thing – don't forget that the compiler has the ability to totally optimise away your classes anyway if it sees fit. My concern is more style and maintenance related. Placing all the communication code in one class or a family of parent and child classes makes sense logically which makes maintenance easier and allows different communication methods to be swapped in should we need in the future – for example it would mean that the change from the current ipc method to a new one would be a single line of code, once that new method is written. If the idea of creating a “null” object then calling a create method to fully initialise it does not sit right with you then you could instead set up the private member as a pointer which is initialised to null and use new to create the object when needed. But this doesn't really have any advantages. The class could have a method called something like isInitialised which allows it to be checked just like checking a pointer is null. But with a pointer there could be a path where its deletion is missed and the pointer goes out of scope and resources leak. The other option is to use a smart pointer. This is a class that acts like a pointer but will automatically delete the object when it is no longer needed – the exact timing depends on the type of smart pointer, which may be shared or unique. This is the joy of C++ 😊 We can automate a lot of things that need to be dealt with manually in C and if you make use of them it can avoid bugs ever occurring. So it makes sense to use them to their max. Read the book I recommended 😉 it actually has a whole section on writing a smart pointer class – although it is a bit out of date, as since C++ 11 there are built in smart pointers called std::unique_ptr and std::shared_ptr, but reading about how they work from first principles is a really good intro to good C++ style. Phil Sent from my Windows 10 phone From: Alan W. Irwin Sent: 13 February 2017 19:05 To: Phil Rosenberg Cc: PLplot development list Subject: Re: [Plplot-devel] The status of the wxwidgets IPC development On 2017-02-12 22:51-0800 Alan W. Irwin wrote: > I plan to stick with my local instance approach for now (automatically > constructing a PLTwoSemaphore instance and destroying it again for > each call to my transmitBytes and receiveBytes methods), but move to > the private object approach for efficiency later on once I understand > (hopefully with some further help from you) exactly what it does at > run time compared with the local instance approach. Hi Phil: I have thought a lot more about this efficiency question using my current mental model of the private object approach (if we applied it for the case of PLMemoryMap accessing PLTwoSemapores). Please correct that mental model if I have made a mistake with it, but if that model is correct, then I predict (see below) there will be negligible efficiency differences between the present local instance approach and the private object approach. I assume to make PLTwoSemapores accessible as a private object from PLMemoryMap, I would move the current code in the PLTwoSemapores constructor to a create method, replace the current PLTwoSemapores constructor with one without arguments that basically forces the two semaphore addresses to be invalid (NULL) values, and move the current destructor code to a delete method so the destructor becomes a no-op. Then whenever PLMemoryMap was instantiated, the private PLTwoSemapores object that is mentioned in its private: section is also automatically instantiated (with the invalid addresses). Then within transmitBytes and receiveBytes methods, the create method for the private PLTwoSemapores object is called first, and the destroy method for that same object is called last. And ultimately when the PLMemoryMap object is destroyed, the no-op destructor for its private PLTwoSemapores object is also automatically called. If that is the correct model, then the code in create and destroy is the same as in the current PLTwoSemapores instantiation and destruction that occurs in the current transmitBytes and receiveBytes methods. So is your efficiency concern not that create and destroy code, but other code that must be executed if you do a full-blown instantiation and destruction of the PLTwoSemapores object? Or are you also concerned with the cpu cycles required to call the create and destroy methods for PLTwoSemapores every time the transmitBytes and receiveBytes methods were called? For example, you _could_ call the appropriate create and destroy methods outside transmitBytes and receiveBytes (and likely less often if you used a lot of care), but I would prefer not to do that because that non-locality makes the code much harder to understand (say when we look at it again 5 years from now) and I believe the number of cpu cycles saved should be negligible compared with the number of cpu cycles actually required to execute transmitBytes and receiveBytes. Come to think of it, the number of cpu cycles needed for a full-blown instantiation and destruction of a PLTwoSemapores object within transmitBytes and receiveBytes should also be negligible compared to even just the cpu costs of the memcpy calls that do the necessary chunking up of the array of bytes to be sent via shared memory with transitBytes and the similar memcpy costs on the receiving end to reassemble the array of bytes, and the argument becomes stronger if you add in the times for sem_wait to finish on the transmission and receive ends. So unless my mental model above is incorrect, I predict we will find no discernible difference in efficiency when comparing between the current local instance approach and the private object approach at least for the use of PLTwoSemapores by the transmitBytes and receiveBytes methods of PLMemoryMap. But I am willing to implement the private object approach variant (controlled by a preprocessing macro) and measure its efficiency compared with the current local instance approach to confirm (or disprove) that prediction, and once we have those test results, we can decide which approach to take. Alan __________________________ Alan W. Irwin Astronomical research affiliation with Department of Physics and Astronomy, University of Victoria (astrowww.phys.uvic.ca). Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); the Time Ephemerides project (timeephem.sf.net); PLplot scientific plotting software package (plplot.sf.net); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __________________________ Linux-powered Science __________________________ |
|
From: Alan W. I. <ir...@be...> - 2017-02-13 21:53:26
|
On 2017-02-13 20:24-0000 p.d...@gm... wrote: > I have absolutely no efficiency concerns. The cpu cycles will be totally negligible compared to the time to copy the memory. To be honest I rarely stress about efficiency. Premature optimisation is usually a bad thing – don't forget that the compiler has the ability to totally optimise away your classes anyway if it sees fit. Hi Phil: I completely agree with you on the perils of premature optimization, and it sounds like for both of us, if it is a choice between code clarity and some minor efficiency issue, code clarity wins every time. > My concern is more style and maintenance related. Placing all the communication code in one class or a family of parent and child classes makes sense logically which makes maintenance easier and allows different communication methods to be swapped in should we need in the future – for example it would mean that the change from the current ipc method to a new one would be a single line of code, once that new method is written. Good point which I finally understand now. And it sounds from what you have said that my mental model of what needs to be done to convert my present local instance method to the private object method is correct so I will just go ahead and make that implementation change without bothering with testing the efficiency of the local instance versus private object methods. Thanks very much for the above instructive feedback! Now that this issue has been cleared up, I hope to have some useful results to report soon. Alan __________________________ Alan W. Irwin Astronomical research affiliation with Department of Physics and Astronomy, University of Victoria (astrowww.phys.uvic.ca). Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); the Time Ephemerides project (timeephem.sf.net); PLplot scientific plotting software package (plplot.sf.net); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __________________________ Linux-powered Science __________________________ |
|
From: Alan W. I. <ir...@be...> - 2017-02-14 09:09:07
|
Hi Phil:
I have found a bug in the "catch(char *)" commands used in
drivers/wxwidgets.cpp. I have discovered this bug from my attempts to
use throw("some message") style exceptions in my (currently unpushed)
-DPL_WXWIDGETS_IPC2=ON version of wxPLDevice::TransmitBuffer. Those
catches did not work (i.e., the messages were being ignored) until I
changed them to "catch(const char *)" (because const char * and NOT
char * is the type of any fixed string such as "some message"). Note
this fix is recommended by
<http://stackoverflow.com/questions/33410536/how-to-catch-char-exceptions-in-c/33410593>.
Now that I am finally getting specific exception messages for my
-DPL_WXWIDGETS_IPC2=ON code, I am debugging a specific exception that
has been revealed by this change. When that effort is done (some time
tomorrow after I get some sleep), I plan to change "catch(char *)" to
"catch(const char *)" everywhere in drivers/wxwidgets.cpp to follow
what I have done for the -DPL_WXWIDGETS_IPC2=ON version of
wxPLDevice::TransmitBuffer. This fix should make the messages for all
exceptions caught by that drivers/wxwidgets.cpp code much more
specific.
I think you can likely tell from this particular episode and other
evidence that I am getting much more comfortable with C++ now. So
that, the above bug fix, and some of your reorganization ideas
inspired by my transmitBytes and receiveBytes methods of the
PLMemoryMap class are already good results from this effort.
Alan
__________________________
Alan W. Irwin
Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).
Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________
Linux-powered Science
__________________________
|
|
From: Alan W. I. <ir...@be...> - 2017-02-17 07:09:56
|
Hi Phil:
I am still progressing at a steady pace with this wxwidgets IPC
development effort. There are no additional commits to share with you
at this time (but coming soon!), and I appreciate that you haven't
caused me any conflict resolution issues as I work on what previously
had been largely your wxwidgets code.
I have wxPLDevice::TransmitBuffer completely implemented for
for -DPL_WXWIDGETS_IPC2=ON. There I transmit the bytes in two
batches. One batch for everything but the PLplot buffer data,
and one batch (if needed) exclusively for that buffer data. The
result builds without issues, but I haven't committed it yet
because I am not (quite) in a position to test it yet.
I have started the corresponding -DPL_WXWIDGETS_IPC2=ON implementation
of wxPlFrame::ReadTransmission, and when that is done I should be able
to test my work and commit it. That commit will ignore the issue of
transmitting interactive keystrokes back from wxPLViewer to -dev
wxwidgets, but I have made a lot of general progress in that area as
well, see below.
I have figured out what I think is a fool-proof way for wxPLViewer to
use the read semaphore to coordinate when -dev wxwidgets needs to read
what wxPLViewer has written to the PLMemoryMap. I implemented this
idea as two additional methods of the PLMemoryMap class. I tested
that new code by using it to communicate the wxPLViewer update of
header->viewerOpenFlag = 1;
back to -dev wxwidgets (to signal back all is well with the
PLMemoryMap instance on the wxPLViewer), and I got a good result with
no requirement of having to go into a wxMilliSleep loop on the -dev
wxwidgets side to wait for wxPLViewer to make contact. So I think
using this general approach is also going to work well for
transmitting back interactive keystroke information from wxPLViewer to
-dev wxwidgets.
Note, these two new methods could also be used to send back text size
information (if, for example, we wanted to do something other than a
simple scaling of text size when resizing the graph). However, I just
checked with the second page of the second example (where lots of
numbers are inscribed almost exactly in boxes) and both -dev xwin and
-dev qtwidget clearly do nothing but simple scaling of the text size
when resizing. Thus, for the -dev wxwidgets case there is no need to
send back text size with each resize event, and indeed today I
discovered (as expected now I have seen what is done for other
interactive devices) all the text size measurement was done completely
on the -dev wxwidgets side with no feedback at all from the wxPLViewer
side. But if you wanted to consolidate that measurement of text size
with the actual rendering of that same text (all on the wxPLViewer
side), it could be done with this method.
Alan
__________________________
Alan W. Irwin
Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).
Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________
Linux-powered Science
__________________________
|
|
From: Alan W. I. <ir...@be...> - 2017-02-24 08:19:15
|
Hi Phil: At long last I have some preliminary code (commit 635df69) for you to look at that produces (for -DPL_WXWIDGETS_IPC2=ON) good results for one-page plots such as x00c, x01c, and x24c, but which currently has a bug for multipage plots which I am trying to understand. Sorry this took so long, but I implemented many iterations of this code as I refined my ideas. And my debugging efforts were considerably slowed down by a major g++/gdb issue where the "next" command does not reliably indicate exactly what command it is going to be executing, i.e., it can be a couple of lines off (even for the completely unoptimized case). So for a while there I was trying to figure out why the code was jumping outside loops according to "next" until I finally realized what was going on. The new approach separates the IPC from the assembly of information on the -dev wxwidgets side and the rendering of that data on the wxPLViewer side. It also clearly separates plbuf data from header data and never uses the shared memory space for semi-permanent storage of header information. (Instead, header information is transmitted from dev -wxwidgets to wxPLViewer or vice versa via a specific header location in shared memory and stored in local private variables.) The result is each call to transmitBuffer by -dev wxwidgets sends separate header information and (if needed) plbuf information, while a call to ReadTransmission by wxPLViewer reads all headers and accumulates and plots plbuf information sent by those transmitBuffer calls. In my opinion the result of the above separations is much improved code clarity, and I hope you agree with that assessment. Also, the new code appears to be fast; all the plbuf data was assembled and transferred to wxPLViewer in less than a second for the 10 pages of example 8. I am currently trying to track down a bug (uncaught exception on the wxPLViewer side) that shows up whenever there is an attempt to plot multiple pages. I tried to copy the code stanzas from the old wxplframe.cpp code to the #ifdef PL_WXWIDGETS_IPC2 case for each transmission type, but I may have missed something that is critical to the multipage case. Also for #ifdef PL_WXWIDGETS_IPC2, one call to ReadTransmission reads all pages of plbuf and should plot those pages. That's a big chance in design from ReadTransmission which for the #ifndef PL_WXWIDGETS_IPC2 case reads and plots at most one page. Is there something more I have to do to accomodate that design change so that the multipage case is displayed properly by wxPLViewer when #ifdef PL_WXWIDGETS_IPC? Alan __________________________ Alan W. Irwin Astronomical research affiliation with Department of Physics and Astronomy, University of Victoria (astrowww.phys.uvic.ca). Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); the Time Ephemerides project (timeephem.sf.net); PLplot scientific plotting software package (plplot.sf.net); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __________________________ Linux-powered Science __________________________ |
|
From: Alan W. I. <ir...@be...> - 2017-02-25 05:32:19
|
On 2017-02-24 00:19-0800 Alan W. Irwin wrote: > I am currently trying to track down a bug (uncaught exception on the > wxPLViewer side) that shows up whenever there is an attempt to plot > multiple pages. I tried to copy the code stanzas from the old > wxplframe.cpp code to the #ifdef PL_WXWIDGETS_IPC2 case for each > transmission type, but I may have missed something that is critical to > the multipage case. Also for #ifdef PL_WXWIDGETS_IPC2, one call to > ReadTransmission reads all pages of plbuf and should plot those pages. > That's a big chance in design from ReadTransmission which for the > #ifndef PL_WXWIDGETS_IPC2 case reads and plots at most one page. > > Is there something more I have to do to accomodate that design change > so that the multipage case is displayed properly by wxPLViewer > when #ifdef PL_WXWIDGETS_IPC? Hi Phil: You should ignore the above question because it turns out there is no general multipage issue for the -DPL_WXWIDGETS_IPC2=ON case! I discovered that result by doing extensive tests (running every single C example by hand using -dev wxwidgets) today. In fact, those tests showed everything working now for -DPL_WXWIDGETS_IPC2=ON with the following exceptions: 1. Examples 9 and 16 failed the first time I executed them with the throw message "PLMemoryMap::moveBytesWriter: attempt to start transfer with semaphores not in correct blocked state." That exception is thrown when m_twoSemaphores.areBothSemaphoresBlocked() fails when starting PLMemoryMap::moveBytesWriter. And the problem occurred on the first call to moveBytesWriter just after transfer of header information from wxPLViewer to -dev wxwidgets using moveBytesReaderReversed and moveBytesWriterReversed. But that failure (one or both semaphores are not blocked with zero values) should be impossible because the fact is both semaphores are initialized in that expected (blocked) state (that check succeeded for the initial call to moveBytesReaderReversed on the wxPLViewer side), and at the end of moveBytesReaderReversed when that header transfer succeeded (as measured on the -dev wxwidgets side) the check is made again that the semaphores are left in the proper blocked state. So I am pretty sure something must be clobbering the semaphores after the call to moveBytesReaderReversed is finished on the wxPLViewer side and before moveBytesWriter is called on the -dev wxwidgets side. But what? What severely complicates debugging this issue, is examples 9 and 16 have run flawlessly today after the first attempt that generated the above message. And the exact same generic sequence (transfer header from wxPLViewer to -dev wxwidgets and start tranferring data the other way with that call to moveBytesWriter) happens for every example with no occurrences of this error (at least so far). However, if something is getting clobbered (the only hypothesis that seems to make sense to me concerning the above results), then valgrind on -dev wxwidgets and/or wxPLViewer should be able to find what the trouble is. 2. I still have not implemented interactivity so I didn't bother to try example 1 with the -locate option. And for the same reason you should not expect example 20 to work (the only one of our examples that is interactive by default). 3. Examples 2 and 14 fail with wxPLViewer issuing the following message: Caught unhandled unknown exception; terminating This was the issue I incorrectly thought was a multipage issue yesterday, but it instead it is confined to just these two examples and must be due to some different way these examples set up plots that exposes an actual reproducible bug in the present -DPL_WXWIDGETS_IPC2=ON case which I am attempting to track down now. I cannot find this throw message anywhere in our own code, and indeed it instead appears (see <http://wxwidgets.10942.n7.nabble.com/Better-exception-handling-td87900.html>) that message is a generic wxwidgets response to uncaught exceptions. So my first step for this issue is to attempt to catch all exceptions in our code rather than passing them on, uncaught, to wxwidgets. In sum, the -DPL_WXWIDGETS_IPC2=ON has been largely matured with the only issues left being 1. an extremely elusive bug that ends up as an "impossible" "PLMemoryMap::moveBytesWriter: attempt to start transfer with semaphores not in correct blocked state." error message on rare and irreproducible occasions. 2. Interactivity not implemented. 3. Some easily reproduced bug exposed by example 2 and 14. I think 2 and 3 should be straightforward to deal with, and if my "something getting clobbered" hypothesis to explain 1 is correct, then with the help of valgrind that should be straightforward to solve as well. So I am extremely pleased that today's results showed so many of the examples are actually working fine now, and I am hoping to get all of them (and also the -locate option for example 1) working soon with -DPL_WXWIDGETS_IPC2=ON code. So stay tuned.... Alan __________________________ Alan W. Irwin Astronomical research affiliation with Department of Physics and Astronomy, University of Victoria (astrowww.phys.uvic.ca). Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); the Time Ephemerides project (timeephem.sf.net); PLplot scientific plotting software package (plplot.sf.net); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __________________________ Linux-powered Science __________________________ |
|
From: Alan W. I. <ir...@be...> - 2017-02-25 07:45:38
|
On 2017-02-24 21:32-0800 Alan W. Irwin wrote: > In sum, the -DPL_WXWIDGETS_IPC2=ON has been largely matured with the > only [3 issues left]. Hi Phil: The corollary to this maturation which I forgot to mention is all remaining changes to the -DPL_WXWIDGETS_IPC2=ON code should be small. So I thank you for your consideration in not causing me a lot of conflicts up to now, but from now on please feel free to start reorganizing the wxwidgets code along the lines you mentioned when you saw the direction I was going with my first commit in this -DPL_WXWIDGETS_IPC2=ON series. It is also time to remind you of follow-up goals that I mentioned before once -DPL_WXWIDGETS_IPC2=ON has been completely matured (by solving the remaining 3 issues I mentioned and verifying there are no compromises on speed with this approach). Those goals are as follows: * (Alan) implement a small variation on the -DPL_WXWIDGETS_IPC2=ON approach so that it also works with named semaphores. * (You, assuming you like the code clarity of the -DPL_WXWIDGETS_IPC2=ON approach) implement a small variation of that named semaphores variant which allows it to work on Windows. * (Alan, once those first two goals are completed and we are both happy with the results) Massive code cleanup to completely drop the present Unix + Windows -DPL_WXWIDGETS_IPC2=OFF approach. Alan __________________________ Alan W. Irwin Astronomical research affiliation with Department of Physics and Astronomy, University of Victoria (astrowww.phys.uvic.ca). Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); the Time Ephemerides project (timeephem.sf.net); PLplot scientific plotting software package (plplot.sf.net); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __________________________ Linux-powered Science __________________________ |
|
From: <p.d...@gm...> - 2017-02-25 12:12:43
|
>But that failure (one or both semaphores are not blocked with zero >values) should be impossible because the fact is both semaphores are >initialized in that expected (blocked) state (that check succeeded for >the initial call to moveBytesReaderReversed on the wxPLViewer side), >and at the end of moveBytesReaderReversed when that header transfer >succeeded (as measured on the -dev wxwidgets side) the check is made >again that the semaphores are left in the proper blocked state. Welcome to the world of multithread bugs and race conditions 😊. I haven't looked at the code but my first guess should be some sort of race condition. You cannot rely on any operation completing in one process before the other, unless you have an explicit check for it. I seem to remember having an initialisation flag in the shared memory that gets set by wxPlViewer to indicate all initialisation is complete and the viewer is ready for communication to begin. If you don't have a similar check then you can’t rely on things being ready, including the semaphores being initialised. Note that “overtaking” can occur anywhere, including midway through a single line of code. Phil So I am pretty sure something must be clobbering the semaphores after the call to moveBytesReaderReversed is finished on the wxPLViewer side and before moveBytesWriter is called on the -dev wxwidgets side. But what? What severely complicates debugging this issue, is examples 9 and 16 have run flawlessly today after the first attempt that generated the above message. And the exact same generic sequence (transfer header from wxPLViewer to -dev wxwidgets and start tranferring data the other way with that call to moveBytesWriter) happens for every example with no occurrences of this error (at least so far). However, if something is getting clobbered (the only hypothesis that seems to make sense to me concerning the above results), then valgrind on -dev wxwidgets and/or wxPLViewer should be able to find what the trouble is. 2. I still have not implemented interactivity so I didn't bother to try example 1 with the -locate option. And for the same reason you should not expect example 20 to work (the only one of our examples that is interactive by default). 3. Examples 2 and 14 fail with wxPLViewer issuing the following message: Caught unhandled unknown exception; terminating This was the issue I incorrectly thought was a multipage issue yesterday, but it instead it is confined to just these two examples and must be due to some different way these examples set up plots that exposes an actual reproducible bug in the present -DPL_WXWIDGETS_IPC2=ON case which I am attempting to track down now. I cannot find this throw message anywhere in our own code, and indeed it instead appears (see <http://wxwidgets.10942.n7.nabble.com/Better-exception-handling-td87900.html>) that message is a generic wxwidgets response to uncaught exceptions. So my first step for this issue is to attempt to catch all exceptions in our code rather than passing them on, uncaught, to wxwidgets. In sum, the -DPL_WXWIDGETS_IPC2=ON has been largely matured with the only issues left being 1. an extremely elusive bug that ends up as an "impossible" "PLMemoryMap::moveBytesWriter: attempt to start transfer with semaphores not in correct blocked state." error message on rare and irreproducible occasions. 2. Interactivity not implemented. 3. Some easily reproduced bug exposed by example 2 and 14. I think 2 and 3 should be straightforward to deal with, and if my "something getting clobbered" hypothesis to explain 1 is correct, then with the help of valgrind that should be straightforward to solve as well. So I am extremely pleased that today's results showed so many of the examples are actually working fine now, and I am hoping to get all of them (and also the -locate option for example 1) working soon with -DPL_WXWIDGETS_IPC2=ON code. So stay tuned.... Alan __________________________ Alan W. Irwin Astronomical research affiliation with Department of Physics and Astronomy, University of Victoria (astrowww.phys.uvic.ca). Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); the Time Ephemerides project (timeephem.sf.net); PLplot scientific plotting software package (plplot.sf.net); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __________________________ Linux-powered Science __________________________ |
|
From: Alan W. I. <ir...@be...> - 2017-02-26 01:44:42
|
On 2017-02-25 12:12-0000 p.d...@gm... wrote: > > >> But that failure (one or both semaphores are not blocked with zero >> values) should be impossible because the fact is both semaphores are >> initialized in that expected (blocked) state (that check succeeded for >> the initial call to moveBytesReaderReversed on the wxPLViewer side), >> and at the end of moveBytesReaderReversed when that header transfer >> succeeded (as measured on the -dev wxwidgets side) the check is made >> again that the semaphores are left in the proper blocked state. > > Welcome to the world of multithread bugs and race conditions 😊. I haven't looked at the code but my first guess should be some sort of race condition. You cannot rely on any operation completing in one process before the other, unless you have an explicit check for it. I seem to remember having an initialisation flag in the shared memory that gets set by wxPlViewer to indicate all initialisation is complete and the viewer is ready for communication to begin. If you don't have a similar check then you can’t rely on things being ready, including the semaphores being initialised. Note that “overtaking” can occur anywhere, including midway through a single line of code. Hi Phil: Thanks for that welcome. :-) Last night my hypothesis (something clobbered) to explain why this "PLMemoryMap::moveBytesWriter: attempt to start transfer with semaphores not in correct blocked state." exception was being thrown went down in flames because I got an absolutely clean report from valgrind from both the -dev wxwidgets and wxPLViewer side. So your additional feedback concerning the likely possibility of race conditions to explain this bug was quite useful. I believe you are referring above in the -DPL_WXWIDGETS_IPC2=OFF case to wxPLViewer setting the viewerOpenFlag of the header to signal -dev wxwidgets that the wxPLViewer is properly initialized and -dev wxwidgets waiting to proceed until that flag is set. When you get a chance to look at the -DPL_WXWIDGETS_IPC2=ON code case, you will see that is exactly how moveBytesReaderReversed is initially used on the wxPLViewer side and moveBytesWriterReversed initially used on the -dev wxwidgets side. However, I certainly agree mutual use of the same resource (shared memory) is a tricky world. And now that you have encouraged me to think about races, I discovered there is indeed a race condition that could explain this bug. I have now worked around that race (commit 4e6932e) and please see that commit message for more commentary concerning this type of race. Assuming I really did understand this race, I am virtually positive my simple crude fix will deal with it without any noticeable reduction in speed. However, time will tell about that. So at this time I am pretty sure only two issues are left with the -DPL_WXWIDGETS_IPC2=ON case. One of those is to implement interactivity, and the other (which I plan to start working on now) is to deal with the issue (or issues?) where wxwidgets detects an uncaught exception being thrown every time examples 2 or 14 are executed. Thanks once again for your useful feedback above. Alan __________________________ Alan W. Irwin Astronomical research affiliation with Department of Physics and Astronomy, University of Victoria (astrowww.phys.uvic.ca). Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); the Time Ephemerides project (timeephem.sf.net); PLplot scientific plotting software package (plplot.sf.net); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __________________________ Linux-powered Science __________________________ |
|
From: Alan W. I. <ir...@be...> - 2017-02-27 06:51:10
|
On 2017-02-25 17:44-0800 Alan W. Irwin wrote: > However, I certainly agree mutual use of the same resource (shared > memory) is a tricky world. And now that you have encouraged me to > think about races, I discovered there is indeed a race condition that > could explain this bug. I have now worked around that race (commit > 4e6932e) and please see that commit message for more commentary > concerning this type of race. Assuming I really did understand this > race, I am virtually positive my simple crude fix will deal with it > without any noticeable reduction in speed. However, time will tell about > that. Hi Phil: I think 10 ms sleep used in the above crude workaround would likely always work going forward because it would be pretty unusual for the OS scheduler to not give a process access to the cpu for essentially 10 million instructions. Nevertheless, that argument does depend on process speed and assumptions about scheduler details so having thought a lot more about this, I would far prefer to avoid sleep workarounds for race conditions not only on these grounds but also as simply a matter of good IPC style. Therefore, I plan to turn the current two-semaphore approach into a three semaphore approach where m_wsem and m_rsem will continue to be used for the details of a complete transfer of an array of bytes, but an additional m_tsem semaphore (where "t" stands for transfer) will be used so that only one such transfer of bytes can be done at a given time. As far as I can tell, this change means I can completely drop the moveBytesReaderReversed variant of moveBytesWriter and the moveBytesWriterReversed variant of moveBytesReader which is a really nice simplification. Furthermore, I plan to rename moveBytesWriter to transmitBytes and moveBytesReader to receiveBytes where both transmitBytes and receiveBytes will be used by either of -dev wxwidgets or wxPLViewer as needed depending simply on the direction of data flow. The additional m_tsem semaphore will be initialized to 1; transmitBytes will start by calling sem_wait on that semaphore and will end by calling sem_post on that semaphore. That simple changes means if wxPLViewer uses transmitBytes to send data that is received by -dev wxwidgets with receiveBytes, and then -dev wxwidgets follows up by calling transmitBytes to send data back that is received by wxPLViewer with a call to receiveBytes, that second use of transmitBytes will be halted by the sem_wait until that first use of transmitBytes is entirely completed, i.e., any call by either side of the IPC connection to transmitBytes cannot possibly race with a previous call to that routine by either side. Anyhow, I like this pure semaphore way to avoid the race condition much more than the 10 ms sleep, and I hope to get it completely implemented tomorrow. Alan __________________________ Alan W. Irwin Astronomical research affiliation with Department of Physics and Astronomy, University of Victoria (astrowww.phys.uvic.ca). Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); the Time Ephemerides project (timeephem.sf.net); PLplot scientific plotting software package (plplot.sf.net); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __________________________ Linux-powered Science __________________________ |
|
From: <p.d...@gm...> - 2017-02-27 07:58:15
|
Hi Alan Sorry, I still haven't had chance to look at your code (poorly 4 year old over the weekend). But you are definitely correct to not use a sleep to avoid a race. I think particularly in situations where the system is under strain all bets are off regarding timings. And like you said, it's just not good practice. Phil Sent from my Windows 10 phone From: Alan W. Irwin Sent: 27 February 2017 06:51 To: p.d...@gm... Cc: PLplot development list Subject: Re: [Plplot-devel] The status of the wxwidgets IPC development On 2017-02-25 17:44-0800 Alan W. Irwin wrote: > However, I certainly agree mutual use of the same resource (shared > memory) is a tricky world. And now that you have encouraged me to > think about races, I discovered there is indeed a race condition that > could explain this bug. I have now worked around that race (commit > 4e6932e) and please see that commit message for more commentary > concerning this type of race. Assuming I really did understand this > race, I am virtually positive my simple crude fix will deal with it > without any noticeable reduction in speed. However, time will tell about > that. Hi Phil: I think 10 ms sleep used in the above crude workaround would likely always work going forward because it would be pretty unusual for the OS scheduler to not give a process access to the cpu for essentially 10 million instructions. Nevertheless, that argument does depend on process speed and assumptions about scheduler details so having thought a lot more about this, I would far prefer to avoid sleep workarounds for race conditions not only on these grounds but also as simply a matter of good IPC style. Therefore, I plan to turn the current two-semaphore approach into a three semaphore approach where m_wsem and m_rsem will continue to be used for the details of a complete transfer of an array of bytes, but an additional m_tsem semaphore (where "t" stands for transfer) will be used so that only one such transfer of bytes can be done at a given time. As far as I can tell, this change means I can completely drop the moveBytesReaderReversed variant of moveBytesWriter and the moveBytesWriterReversed variant of moveBytesReader which is a really nice simplification. Furthermore, I plan to rename moveBytesWriter to transmitBytes and moveBytesReader to receiveBytes where both transmitBytes and receiveBytes will be used by either of -dev wxwidgets or wxPLViewer as needed depending simply on the direction of data flow. The additional m_tsem semaphore will be initialized to 1; transmitBytes will start by calling sem_wait on that semaphore and will end by calling sem_post on that semaphore. That simple changes means if wxPLViewer uses transmitBytes to send data that is received by -dev wxwidgets with receiveBytes, and then -dev wxwidgets follows up by calling transmitBytes to send data back that is received by wxPLViewer with a call to receiveBytes, that second use of transmitBytes will be halted by the sem_wait until that first use of transmitBytes is entirely completed, i.e., any call by either side of the IPC connection to transmitBytes cannot possibly race with a previous call to that routine by either side. Anyhow, I like this pure semaphore way to avoid the race condition much more than the 10 ms sleep, and I hope to get it completely implemented tomorrow. Alan __________________________ Alan W. Irwin Astronomical research affiliation with Department of Physics and Astronomy, University of Victoria (astrowww.phys.uvic.ca). Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); the Time Ephemerides project (timeephem.sf.net); PLplot scientific plotting software package (plplot.sf.net); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __________________________ Linux-powered Science __________________________ |
|
From: Alan W. I. <ir...@be...> - 2017-02-27 23:34:40
|
On 2017-02-26 22:51-0800 Alan W. Irwin wrote: [....] > Therefore, I plan to turn the current two-semaphore approach into a > three semaphore approach where m_wsem and m_rsem will continue to be > used for the details of a complete transfer of an array of bytes, but > an additional m_tsem semaphore (where "t" stands for transfer) will be > used so that only one such transfer of bytes can be done at a given > time. As far as I can tell, this change means I can completely drop > the moveBytesReaderReversed variant of moveBytesWriter and the > moveBytesWriterReversed variant of moveBytesReader which is a really > nice simplification. Furthermore, I plan to rename moveBytesWriter to > transmitBytes and moveBytesReader to receiveBytes where both > transmitBytes and receiveBytes will be used by either of -dev > wxwidgets or wxPLViewer as needed depending simply on the direction of > data flow. > > The additional m_tsem semaphore will be initialized to 1; > transmitBytes will start by calling sem_wait on that semaphore and > will end by calling sem_post on that semaphore. That simple changes > means if wxPLViewer uses transmitBytes to send data that is received > by -dev wxwidgets with receiveBytes, and then -dev wxwidgets follows > up by calling transmitBytes to send data back that is received by > wxPLViewer with a call to receiveBytes, that second use of > transmitBytes will be halted by the sem_wait until that first use of > transmitBytes is entirely completed, i.e., any call by either side of > the IPC connection to transmitBytes cannot possibly race with a > previous call to that routine by either side. > > Anyhow, I like this pure semaphore way to avoid the race condition > much more than the 10 ms sleep, and I hope to get it completely > implemented tomorrow. Hi Phil: Done as of commit c39f93d. As a result, once again I think I am at a stage in this -DPL_WXWIDGETS_IPC2=ON development where all further changes will be small ones so I encourage you to look at this code in anticipation that you will want to reorganize it along the lines you discussed before. The current status is all examples work (i.e., there are no obvious run-time errors and there are no rendering issues other than previously known ones for the -DPL_WXWIDGETS_IPC2=OFF case) other than 02 and 14 (where an exception is consistently thrown by wxPLViewer for currently unknown reasons), and examples 01 (in locate mode) and 20 (because interactivity not implemented yet). I next plan to work on the example 2 issue. Alan __________________________ Alan W. Irwin Astronomical research affiliation with Department of Physics and Astronomy, University of Victoria (astrowww.phys.uvic.ca). Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); the Time Ephemerides project (timeephem.sf.net); PLplot scientific plotting software package (plplot.sf.net); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __________________________ Linux-powered Science __________________________ |
|
From: Alan W. I. <ir...@be...> - 2017-02-28 20:19:15
|
On 2017-02-27 15:34-0800 Alan W. Irwin wrote: > The current status is all examples work (i.e., there are no obvious > run-time errors and there are no rendering issues other than > previously known ones for the -DPL_WXWIDGETS_IPC2=OFF case) other than > 02 and 14 (where an exception is consistently thrown by wxPLViewer for > currently unknown reasons), and examples 01 (in locate mode) and 20 > (because interactivity not implemented yet). I next plan to work on > the example 2 issue. Hi Phil: The example 2 and 14 issues for -DPL_WXWIDGETS_IPC2=ON turned out to have the same cause which has now (commit 245815f) been fixed. My remaining plans for the -DPL_WXWIDGETS_IPC2=ON case are as follows: 1. Implement interactivity so that C example 1 works with -locate mode and C example 20 works. 2. Named semaphore variant for POSIX systems. 3. Named semaphore variant for Windows systems. (I should be able to implement this following what is done for -DPL_WXWIDGETS_IPC2=OFF, but I will need your help to test this variant). I don't believe any of these remaining develoments are going to require large code changes. Alan __________________________ Alan W. Irwin Astronomical research affiliation with Department of Physics and Astronomy, University of Victoria (astrowww.phys.uvic.ca). Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); the Time Ephemerides project (timeephem.sf.net); PLplot scientific plotting software package (plplot.sf.net); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __________________________ Linux-powered Science __________________________ |
|
From: Alan W. I. <ir...@be...> - 2017-03-02 08:26:59
Attachments:
wxwidgets_interactive.patch.gz
|
On 2017-02-28 12:19-0800 Alan W. Irwin wrote: > My remaining plans for the -DPL_WXWIDGETS_IPC2=ON case are as follows: > > 1. Implement interactivity so that C example 1 works with -locate mode > and C example 20 works. Hi Phil: I am having trouble with implementing interactivity and could use some help from you to figure this out although I will be trying independently to figure this out on my own using gdb. The attached patch implements the required (as far as I can tell) small changes to implement interactivity for -DPL_WXWIDGETS_IPC2=ON. As a result of these changes, the entire code should have the following logic for dealing with locate mode for -DPL_WXWIDGETS_IPC2=ON: * In example 1 after all normal PLplot calls to generate the plot have been called (which fills up plbuf with 24272 bytes of information, see below), the locate mode loop generates a series of plGetCursor calls. * Each of those plGetCursor calls ends up (for -dev wxwidgets) as a call to wxPLDevice::Locate and TransmitBuffer( pls, transmissionLocate ); * TransmitBuffer calls transmitBytes to send header information (including transmissionLocate = 4 and plbufAmountToTransmit = 24272 bytes in plbuf to be received by wxPLViewer using receiveBytes. The results are Before transmitBytes transmissionType = 4 plbufAmountToTransmit = 24272 viewerOpenFlag = 1 locateModeFlag = 1 completeFlag = 0 After receiveBytes transmissionType = 4 plbufAmountToTransmit = 24272 viewerOpenFlag = 1 locateModeFlag = 1 completeFlag = 0 Successful read of plbuf i.e., that transmission of the plbuf information and transmissionLocate to wxPLViewer was a complete success. * Because plbufAmountToTransmit is non-zero, wxPLViewer appends that plbuf information to the correct element of the m_pageBuffers array and conditionally (although the result is the condition is true because of the above 24272 says there is a lot more data to plot) calls SetPageAndUpdate as per normal which should plot the page in preparation for the locate mode. * After plbuf is processed, then transmissionLocate calls SetPageAndUpdate one more time (which has no effect because everything should be plotted now), then sets m_locateMode = true * Here my understanding gets fuzzy, but wxPlFrame::OnMouse is part of a table of events which I presume means it is called whenever there is any mouse activity. But when m_locateMode is true, then in wxPlFrame::OnMouse information about the latest mouse click is collected into the m_header.graphicsIn struct, and then the whole header is sent back to -dev wxwidgets with transmit_Bytes (which because of the three-semaphore design will wait for the end of the previous transmit_Bytes before proceeding so I don't think we are up against any race condition here) where wxPLDevice::Locate receives it with receiveBytes, and passes it back to the example 1 locate loop which prints out values collected for a particular mouse click, then (unless there is an exit from locate mode by a mouse click off of a viewport keeps repeating the whole process indefinitely. I think this design should "just work" (especially because my changes to implement it in the attached patch are so small and straightforward). What happens in practice is the whole code just hangs without even displaying the plot (something it should do regardless of anything else right after the above "Successful read of plbuf" message). I am pretty sure the problem is due to some minor bug. For example, the wxPlFrame::OnKey routine is just empty for the -DPL_WXWIDGETS_IPC2=ON case, and maybe I need to implement something there as well for that case, but I cannot see how that would stop the whole display showing (as it does for example 1 if the -locate option is not used), and I don't see how it would stop wxPlFrame::OnMouse from working properly. So if you spot that bug please let me know. Of course, the answer may be there is a fundamental flaw in the above design so it you feel that is the case please let me know. And your clarification of my fuzzy understanding of when wxPlFrame::OnMouse gets called would be helpful as well. Alan __________________________ Alan W. Irwin Astronomical research affiliation with Department of Physics and Astronomy, University of Victoria (astrowww.phys.uvic.ca). Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); the Time Ephemerides project (timeephem.sf.net); PLplot scientific plotting software package (plplot.sf.net); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __________________________ Linux-powered Science __________________________ |
|
From: Alan W. I. <ir...@be...> - 2017-03-03 21:13:32
|
On 2017-03-03 16:02-0000 Phil Rosenberg wrote: > Hi Alan > Sorry for not replying sooner. I was waiting until I had time to > actually read the code, which unfortunately I haven't had chance to do > still, but I didn't want to leave it any longer without at least > guessing what might be happening. Hi Phil: Your guesses are quite helpful so keep them coming. :-) > > So a couple of things spring to mind. Firstly you must at some point > renter the event loop to allow the viewer to start doing stuff again. > If you are sat in a loop in you communication code just waiting for > something to happen then it never will because the execution point is > in your communication code and not dealing with the mouse pointer. > calling wxYield will go check the event loop, deal with that stuff > then return, but it's often not the best solution because nested calls > to wxYield aren't permitted. If possible it is better to return from > your function and use some other event to trigger your return > communication. Perhaps this could be at the end of the method that > deals with capturing mouse activity. > > Another thing is that to cause the plot to be redrawn you must tell > wxWidgets that there is something new to draw. I think the method for > this is Refresh(). Then you must allow the event loop to be entered to > allow the redraw to actually take place. > > A final thing to think about is that locate must be dealt with > immediately. It sounded like you were holding locate requests and > continuing to send plot commands. But maybe I misunderstood this as > doing so would hang the plplot side while pllocate awaited it's > return. > > As I said I haven't looked at the code, just read your email, but > these are the first places to look as a guess. > > Also there is a debug #define which when used means that plplot does > not launch the viewer, instead it just sits and waits for you to > launch it manually. This way you can launch the viewer in a debugger > to see what is going on in there. Thanks, for those ideas which I haven't completely absorbed yet, but I will look closely at them later. The reason for putting that off is there is a new surprising result I have established which is even for -DPL_WXWIDGETS_IPC2=OFF examples/c/x01c -dev wxwidgets -locate does not work correctly, i.e., there is no display of the plot until after you have clicked the mouse away from where you guess a viewport will appear. So the net result is locate mode must be finished before the plot can appear. So it was this code I was trying to follow for the -DPL_WXWIDGETS_IPC2=ON case, but that is obviously not a good model to follow. (By the way, I would not advise working on that -DPL_WXWIDGETS_IPC2=OFF issue because there is a good possibility (assuming you like everything about the new -DPL_WXWIDGETS_IPC2=ON approach once you have had a chance to look at it) that the -DPL_WXWIDGETS_IPC2=OFF code will be completely removed once I get -DPL_WXWIDGETS_IPC2=ON working for this example.) Just now I tried the third (deprecated) version of the wxwidgets code using -DOLD_WXWIDGETS=ON, and that works fine with the above example with both mouse clicks and key presses being properly reported. Therefore, I have a lot of hope that a deeper look into that deprecated code will give me a model I can follow to get the above example to work correctly with -DPL_WXWIDGETS_IPC2=ON. So please remain ready to answer questions this weekend just in case I need some further help to try and translate how that deprecated code implements interactivity to the -DPL_WXWIDGETS_IPC2=ON case. Alan __________________________ Alan W. Irwin Astronomical research affiliation with Department of Physics and Astronomy, University of Victoria (astrowww.phys.uvic.ca). Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); the Time Ephemerides project (timeephem.sf.net); PLplot scientific plotting software package (plplot.sf.net); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __________________________ Linux-powered Science __________________________ |
|
From: Alan W. I. <ir...@be...> - 2017-03-04 03:25:25
|
Hi Phil: I have now had more of a chance to read your e-mail so I have some more detailed responses to it below. On 2017-03-03 16:02-0000 Phil Rosenberg wrote: [...] > So a couple of things spring to mind. Firstly you must at some point > renter the event loop to allow the viewer to start doing stuff again. > If you are sat in a loop in you communication code just waiting for > something to happen then it never will because the execution point is > in your communication code and not dealing with the mouse pointer. I am glad you brought up wxPLViewer events because I am still trying to understand those. For example, I presume (from the name) that wxPlFrame::OnCheckTimer is called every time the event timer fires, and that will continue at various rates (depending on what is set) until the timer is killed when m_header.completeFlag is true. And similarly, wxPlFrame::OnMouse is called every time there is a mouse event, and so on for all other events in the event table. So until the timer is stopped, I assume the net effect is wxPlFrame::OnCheckTimer (which in turn calls ReadTransmission that processes all the different transmission types sent by -dev wxwidgets) is effectively run simultaneously with wxPlFrame::OnMouse and the rest of the responses to different events. Have I got what goes on in event processing interpreted correctly? Of course, in direct answer to your concern, if ReadTransmission just blocked waiting for a semaphore, then wxPLViewer would hang, but that blocking should not happen unless -dev wxwidgets has incorrectly terminated its transmission type requests to ReadTransmission, and there is no sign of that type of bug in the code right now according to debug messages being printed out by ReadTransmission. In fact, according to those, readTransmission returns (as does OnCheckTimer that called it), and the hang occurs elsewhere (before the example 1 plotbuf that is created before transmissionType == transmissionLocate is sent from the -dev wxwidgets side) is plotted. So the first order of business is to get those plotbuf commands plotted. > calling wxYield will go check the event loop, deal with that stuff > then return, but it's often not the best solution because nested calls > to wxYield aren't permitted. If possible it is better to return from > your function and use some other event to trigger your return > communication. Perhaps this could be at the end of the method that > deals with capturing mouse activity. Just to remind you, all that transmissionLocate does is call SetPageAndUpdate() (just like what happens for most other transmission types), and then it sets m_locateMode = true; which has no effect on the rest of transmissionLocate but which has a profound effect on how wxPlFrame::OnMouse is executed, i.e., that event immediately returns with that variable is false and processes a mouse event whenever that variable is true. So all that may be working fine, and the issue is simply to get the plbuf commands plotted. > Another thing is that to cause the plot to be redrawn you must tell > wxWidgets that there is something new to draw. I think the method for > this is Refresh(). Then you must allow the event loop to be entered to > allow the redraw to actually take place. SetPageAndUpdate (referred to above) does normally call Refresh() if there is anything more to be plotted. So it appears the current ReadTransmission code does everything you suggest, i.e., handles transmissionLocate flawlessly. However, I now realize there is an issue in wxPlFrame::OnMouse which attempts (when m_locateMode = true) to collect mouse information and send it back to -dev wxwidgets and evidentially that is happening in the wrong order (before the plot is displayed). How can I absolutely assure that the plot is displayed before wxPlFrame::OnMouse is entered? > A final thing to think about is that locate must be dealt with > immediately. It sounded like you were holding locate requests and > continuing to send plot commands. Locate is dealt with as follows (which I think you can classify as "immediately"). The code in question is in the patch I sent you. We have the following sequence of calls in wxPLDevice::Locate TransmitBuffer( pls, transmissionLocate ); m_outputMemoryMap.receiveBytes( true, &m_header, sizeof ( MemoryMapHeader ) ); *graphicsIn = m_header.graphicsIn; The first sends both transmissionLocate and plbuf to wxPLViewer which processes those in ReadTransmission as I have described above (also confirmed by debug messages). Then that code returns and after that the plot should be displayed (which isn't happening for some reason) and the event loop should be running. After that when a mouse event occurs wxPlFrame::OnMouse should be entered and that method collects information about that mouse event in a header, sends that header back to -dev wxwidgets with transmitBytes which is received by the above receiveBytes command and sent back to example 1 for printout via graphicsIn. I hope that is clear. In any case, my next steps are to confirm that wxPlFrame::OnMouse is being entered properly each time a mouse button is clicked, and to try to discover why there is no display of the plot before that happens (for both -DPL_WXWIDGETS_IPC2=ON and -DPL_WXWIDGETS_IPC2=OFF). > As I said I haven't looked at the code, just read your email, but > these are the first places to look as a guess. > > Also there is a debug #define which when used means that plplot does > not launch the viewer, instead it just sits and waits for you to > launch it manually. This way you can launch the viewer in a debugger > to see what is going on in there. Thanks again, Phil, for these remarks which seem to imply I am on the right track so I am likely looking for a small bug rather than a gross misdesign. By the way, I have now looked at the third (deprecated) version of the code, and that is more sophisticated in how it treats keyboard and mouse events, e.g., special crosshairs in locate mode). So I plan to do a lot more with wxPlFrame::OnKey and wxPlFrame::OnMouse once I can get the plot displayed properly before they are entered. Alan __________________________ Alan W. Irwin Astronomical research affiliation with Department of Physics and Astronomy, University of Victoria (astrowww.phys.uvic.ca). Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); the Time Ephemerides project (timeephem.sf.net); PLplot scientific plotting software package (plplot.sf.net); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __________________________ Linux-powered Science __________________________ |
|
From: Alan W. I. <ir...@be...> - 2017-03-04 19:59:44
|
On 2017-02-28 12:19-0800 Alan W. Irwin wrote: > On 2017-02-27 15:34-0800 Alan W. Irwin wrote: [...] > My remaining plans for the -DPL_WXWIDGETS_IPC2=ON case are as follows: > > 1. Implement interactivity so that C example 1 works with -locate mode > and C example 20 works. > > 2. Named semaphore variant for POSIX systems. > > 3. Named semaphore variant for Windows systems. (I should be able to > implement this following what is done for -DPL_WXWIDGETS_IPC2=OFF, but > I will need your help to test this variant). Hi Phil: I have committed (79987df) my implementation for step 1. above which should be equivalent to the patch I circulated here. This implementation does not work, i.e., examples/c/x01c -dev wxwidgets works fine but examples/c/x01c -dev wxwidgets -locate hangs, but I think it is close because the -DPL_WXWIDGETS_IPC2=OFF case has similar problems (see the commit message for further details). Since the problem appears to be pre-existing I would appreciate you fixing this yourself once you get a chance to look at and evaluate the -DPL_WXWIDGETS_IPC2=ON code. For now, my plan is to continue with steps 2. and 3. above. Alan __________________________ Alan W. Irwin Astronomical research affiliation with Department of Physics and Astronomy, University of Victoria (astrowww.phys.uvic.ca). Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); the Time Ephemerides project (timeephem.sf.net); PLplot scientific plotting software package (plplot.sf.net); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __________________________ Linux-powered Science __________________________ |
|
From: Alan W. I. <ir...@be...> - 2017-03-10 07:08:03
|
To Phil and Pedro:
I have now (commit 096527c) implemented named semaphores variants (both
POSIX and Windows) of the 3-semaphore approach that was previously
only implemented for POSIX unnamed semaphores. This should complete
my planned wxwidgets work (except for the test of unnamed semaphores
capability mentioned below), and the rest will likely be up to you
guys.
As far as I can tell on POSIX platforms the named semaphores variant
of the three-semaphores approach gives identical plotted
results to the unnamed semaphores variant.
I obviously could not test the the Windows variant of the
three-semaphores approach that I implemented so I am leaving
that up to you. Use the cmake option -DPLPLOT_WX_DEBUG_OUTPUT=ON
to get useful debug information about the header and plbuf results of
transmitBytes and receiveBytes from both the -dev wxwidgets side of
the IPC and the wxPLViewer side of the IPC. Use the cmake option
-DPL_WXWIDGETS_IPC3=ON to exercise the 3-semaphore approach on any
platform.
Unnamed POSIX semaphores are known not to work on Mac OS X (and likely
a number of other proprietary Unices) and Windows. Thus, for the
-DPL_WXWIDGETS_IPC3=ON case I plan to implement a CMake test that will
determine whether PL_HAVE_UNNAMED_POSIX_SEMAPHORES is OFF or ON for a
given platform without need for user input. However, as an interim
measure until I get that test implemented our build system sets
PL_HAVE_UNNAMED_POSIX_SEMAPHORES to OFF by default and knowledgable
users that _know_ their platform (e.g., Linux and likely the *BSD
variants) supports unnamed semaphores can try such semaphores by
setting -DPL_HAVE_UNNAMED_POSIX_SEMAPHORE=ON.
I did a large number of timing tests for the various POSIX variants
which I refer to below as
* "IPC3 unnamed semaphores" (i.e., -DPL_WXWIDGETS_IPC3=ON
-DPL_HAVE_UNNAMED_POSIX_SEMAPHORES=ON).
* "IPC3 named semaphores" (i.e., -DPL_WXWIDGETS_IPC3=ON
-DPL_HAVE_UNNAMED_POSIX_SEMAPHORES=OFF).
* "IPC single mutex" (i.e., -DPL_WXWIDGETS_IPC3=OFF which effectively uses
the same code that Phil implemented before I made the additions
to that code for the above two cases).
One apparent result is "IPC3 unnamed semaphores" and "IPC3 named
semaphores" are the same speed within the very large replication noise
(see below) with just one example where "IPC3 unnamed semaphores"
appears to be significantly faster (by a factor of 1.7). But take
this result with a huge grain of salt, because it doesn't make much
sense that only one example is affected by some timing difference
between these two cases. In any case, I believe these two cases
should be essentially identical in speed (i.e., the time taken to
access the semaphores should be negligible compared to the time
required for memcpy to transmit header and plbuf data through the
shared memory on the transmitBytes side and extract header and plbuf
data from the shared memory on the receiveBytes side). Also, I have
proved (for example 8 which has a very large plbuf filled with plot
directives) that generation of that plbuf data and transfer of those
data from -dev wxwidgets to wxPLViewer takes less than 0.8 seconds. So
assuming transfer takes less than generation (and I assume
substantially less), variants on how that transfer is done should
hardly affect the timing of example 8, and the affect on other
examples of such variants should be much less.
Another apparent result is "IPC single mutex" is significantly (factor
of two) faster _and_ slower than the other two depending on which
example you use for the comparison with the faster results
substantially outnumbering the slower results. But there doesn't seem
to be any pattern to which examples are faster and which slower, and
in any case I think the memcpy time should be nearly identical in all cases
(e.g., for the "IPC single mutex" case where a circular buffer is used
you still have to copy bytes to that buffer on the transmitting side
and copy those bytes from that buffer on the receiving side just as in
the 3-semaphores approach with or without named semaphores).
I now provide more details on how I got these timing results in case
someone can figure out why they are so unreliable.
My timing results were done using the time command with its output
reformatting to a single line corresponding to the real time
interval using
export TIMEFORMAT=$'real\t%3R'
(After all tests were done I restored the normal 3-line time format by
export TIMEFORMAT=$'\nreal\t%3lR\nuser\t%3lU\nsys\t%3lS'
.) After building the test_c_wxwidgets target (to insure all
dependencies are built), I then collected time information for -dev
wxwidgets on most of our C examples (with exceptions 08, 17, 20, 25,
31, 32, 33, and 34) typically using something like the following:
(for N in $(seq --format='%02.0f' 0 30 |grep -vE '08|17|20|25'); do echo $N; (time examples/c/x${N}c -dev wxwidgets >&/dev/null); sleep 5; done) >| ../IPC3_timing_ON_OFF_alt5.txt 2>&1
Note these time commands only measure the time taken by the -dev
wxwidgets side of the IPC, and do not measure the time required by the
other side of the IPC (wxPLViewer). The purpose of the sleep command
is to allow time for the wxPLViewer command to finish so that it
(usually except in those cases where wxPLViewer takes longer than 5
seconds) does not interfere with the time measurement of subsequent
examples.
Unfortunately, the fundamental result I have from such tests is these
results are not reliable! One issue is if I run the above command
multiple times I get substantially different results for all examples.
For example, you can easily get time variations of 20 per cent for the
same standard example from one run to the next if using the -np option
for the examples/c/x${N}c commands, and it is even worse without that
option (as above) because it is hard to be exactly consistent with how
you run through the pages of an example by hitting the enter key by
hand. In other words, both with and without the -np option, the cpu
time (or more likely the memory or some other resource) consumed by
wxPLViewer is likely interfering with the timing of the
examples/c/x${N}c command in an inconsistent way.
It should be possible to reduce this problem by a very large degree by
making wxPLViewer a lot more efficient. A particularly egregious
result is example 8 (which is why it was skipped above). As discussed
before, it takes something like 3 seconds per page for wxPLViewer to
render the 10 pages of example 8 _once that example is completely done
with all IPC finished_. And during that 3 seconds per page wxPLViewer
consumes virtually no cpu. In comparison, xcairo and wxwidgets
generates and displays all 10 pages of that example in less than 3
seconds so we are discussing at least an order of magnitude
discrepancy (and virtually all of it idle time for the wxPLViewer
case) in time required to render all 10 pages of the example.
That huge disparity is a puzzle to me since on Linux the wxwidgets
library is essentially a wrapper for a subset of the GTK+ library
suite which is accessed more directly by xcairo. Therefore, you would
expect -dev xcairo and wxPLViewer (invoked by -dev wxwidgets) to take
roughly the same amount of time to complete for example 8. Anyhow,
example 8 has lots (probably thousands) of plfill calls so I am
wondering if wxPLViewer is translatiing plfill into some generic
wxwidgets library call that is extraordinarily inefficient, and there
might be an alternative wxwidgets library API to use to make fills
that would be much more efficient than what we are using now? But the
inefficiency of wxPLViewer after an example has completed may be
nothing to do with that because a major clue is virtually no CPU time
is consumed by (independent, i.e., after IPC is finished) wxPLViewer.
Instead it appears to be spending the vast majority of its time
waiting for some non-IPC event. So perhaps the issue is the event
setup that is used for wxPLViewer is inefficient for some reason. (For
example, it is possible one of the types of events that are handled
may currently be firing essentially continuously.) Anyhow, a careful
review of that event handling code should be done with debug printout
each time an event fires to make sure there are no continuously firing
events.
I am also concerned with the reliability of any timing results for
the -np option which simply shows blank screens in all cases. Probably
the "IPC3 unnamed semaphores", "IPC3 named semaphores", and "IPC
single mutex" are implemented correctly because they are no obvious
run-time errors such as segfaults, and for the two IPC3 cases with the
-DPLPLOT_WX_DEBUG_OUTPUT=ON cmake option you get printouts verifying
the data that is sent is received properly. Nevertheless, there is
currently no visual plot rendering evidence to support (or refute)
that conclusion for the -np case.
The solution to this issue is for wxPLViewer to display plot results
immediately as the parts of plbuf are acquired to verify the rendering
is done correctly. (This "immediate" approach is already done for most
of the other PLplot interactive devices including the old version of
wxwidgets.) That change would likely allow the -locate option to work
correctly for example 1 and should also solve the example 17 issue
where only the final result is displayed rather than properly plotting
the intermediate results that are required to get to that final
result.
In sum, I believe we need to make the two improvements above (solve
the wxPLViewer order of magnitude inefficiency issue and display
wxPLViewer plots immediately for each part of plbuf that is received)
before we can get believable timing results. Therefore it is much too
soon to judge any of the "IPC3 unnamed semaphores", "IPC3 named
semaphores", and "IPC single mutex" approaches based on any apparent
timing issues that are occurring now, and once we get more reliable
timing my guess is we will see no important difference in timing
results between the three different IPC methods that are currently
implemented.
For now (except to implement a CMake test that would determine
PL_HAVE_UNNAMED_POSIX_SEMAPHORES without user input), I believe I am
done with this project, and I request you follow up with these steps:
* Build and test wxwidgets on Windows using -DPL_WXWIDGETS_IPC3=ON
-DPL_HAVE_UNNAMED_POSIX_SEMAPHORE=OFF to
verify that this variant of the three-semaphores approach is working
correctly on Windows. I am fairly confident it will work (or at
least any issues will be small typographical ones) because I copied
the syntax for initializing, posting, waiting for, and destroying the
3 semaphores for the Windows case following how that was done for
the Windows variant of the "IPC single mutex" case.
* Evaluate the code clarity of the "IPC3 unnamed semaphores" and "IPC3
named semaphores" approaches compared to the "IPC single mutex"
approach. My opinion is the three-semaphores approach (whether
using the unnamed POSIX, named POSIX, or named Windows semaphores
variants) has much improved code clarity compared to the "IPC single
mutex" approach. But it will be interesting to see if your
independent assessment of that question also supports that
conclusion. :-)
* Get
examples/c/x01c -locate -dev wxwidgets
to work for all three approaches. The fundamental issue here is the
wxPLViewer end as currently designed refuses to render the plot for
partial plbuf results before -dev wxwidgets finishes sending the
complete plbuf. So at least part of the fix here is to redesign
wxPLViewer to allow rendering of partial plbuf results. (See
discussion above concerning two other benefits of that approach.)
Alan
__________________________
Alan W. Irwin
Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).
Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________
Linux-powered Science
__________________________
|
|
From: Alan W. I. <ir...@be...> - 2017-03-11 22:56:34
|
To Phil and Pedro: On 2017-03-09 23:07-0800 Alan W. Irwin wrote: [...] > Unnamed POSIX semaphores are known not to work on Mac OS X (and likely > a number of other proprietary Unices) and Windows. Thus, for the > -DPL_WXWIDGETS_IPC3=ON case I plan to implement a CMake test that will > determine whether PL_HAVE_UNNAMED_POSIX_SEMAPHORES is OFF or ON for a > given platform without need for user input. DONE as of commit 14e70d7. This completes my planned IPC work on wxwidgets, and I look forward to your follow ups to this work. Alan __________________________ Alan W. Irwin Astronomical research affiliation with Department of Physics and Astronomy, University of Victoria (astrowww.phys.uvic.ca). Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); the Time Ephemerides project (timeephem.sf.net); PLplot scientific plotting software package (plplot.sf.net); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __________________________ Linux-powered Science __________________________ |
|
From: Alan W. I. <ir...@be...> - 2017-03-13 06:27:44
|
On 2017-03-09 23:07-0800 Alan W. Irwin wrote:
[...]
> In sum, I believe we need to make the two improvements above (solve
> the wxPLViewer order of magnitude inefficiency issue and display
> wxPLViewer plots immediately for each part of plbuf that is received)
> before we can get believable timing results. Therefore it is much too
> soon to judge any of the "IPC3 unnamed semaphores", "IPC3 named
> semaphores", and "IPC single mutex" approaches based on any apparent
> timing issues that are occurring now, and once we get more reliable
> timing my guess is we will see no important difference in timing
> results between the three different IPC methods that are currently
> implemented.
To Phil and Pedro:
I finally figured out a way to substantially reduce how the wxPLViewer
app was interfering with timing measurements for -dev wxwidgets!
To work around the wxPLViewer rendering inefficiency problem mentioned
above, I ensured nothing is rendered by that app by making a local
modification of utils/wxplframe.cpp to #define the PLPLOT_WX_NOPLOT
macro. (See commit 7cfd6ce to see exactly the few lines of code that
are skipped when that local modification is made). To work around the
uncertainty about what the -np option is actually doing, I don't use
that option for my timing tests (see the command below). Finally,
(although I am not sure this makes much difference but I state this
for the record) I also used a minimal desktop (fvwm) rather than KDE
to vastly reduce the number of background desktop tasks that are
contending for the cpu and memory when doing timing tests. So for a
given CMake configuration, and after setting
export TIMEFORMAT=$'real\t%3R'
to use a one-line real time difference format for time results, a
timing run is done using the following command
(for N in $(seq --format='%02.0f' 0 34 |grep -vE '20'); do
echo $N; examples/c/x${N}c -dev wxwidgets >&/dev/null; killall
-9 wxPLViewer; (time examples/c/x${N}c -dev wxwidgets >&/dev/null); killall
-9 wxPLViewer ; done) >| <captured results file> 2>&1
where the name of the <captured results file> depends on the
configuration being tested to keep the results conveniently separated.
Within that command, I run each of the 34 (!) examples twice so that the
second timed run will be done with cached memory results. After each
example completes I kill the corresponding wxPLViewer instance because
there is a slight bug with the PLPLOT_WX_NOPLOT macro such that it
leaves an empty wxPLViewer GUI rather than properly terminating that.
With the above procedure I now have reasonable timing consistency.
For example, timing results differ for a given example
by a maximum of 13 ms from one run of the above command to the next
for identical configurations.
For the -DPL_WXWIDGETS_IPC3=ON case if I compare results derived from
the above command for both named and unnamed semaphores, the
differences are within the 13 ms noise of the time measurement. That
is an important result which means that eventually in the interests of
simplifying our code (and because unnamed semaphores are not
universally available on all platforms) we will likely want to remove
the unnamed semaphores variant and retain the named semaphores variant
of the 3-semaphores approach, i.e., drop all code that currently is
only compiled when the PL_HAVE_UNNAMED_POSIX_SEMAPHORES macro is
#defined.
If I compare results derived from the above command for
-DPL_WXWIDGETS_IPC3=ON (three semaphores approach) versus
-DPL_WXWIDGETS_IPC3=OFF (one mutex, circular buffer approach), I get the
following result from ndiff (where the -abs 0.100 option
ignores absolute time difference less than 100 ms, "ON_OFF" refers
to results collected with -DPL_WXWIDGETS_IPC3=ON
-DPL_HAVE_UNNAMED_POSIX_SEMAPHORES=OFF, and "OFF" refers to results
collected with -DPL_WXWIDGETS_IPC3=OFF):
ndiff -abs .100 ../IPC3_timing_ON_OFF_alt11.txt ../IPC3_timing_OFF_alt11.txt
16c16 example 07
< real 0.392
--- field 2 absolute error 1.14e-01
> real 0.506
30c30 example 14
< real 0.521
--- field 2 absolute error 1.42e-01
> real 0.379
36c36 example 17
< real 10.474
--- field 2 absolute error 1.65e-01
> real 10.309
54c54 example 27
< real 0.292
--- field 2 absolute error 1.11e-01
> real 0.403
66c66 example 33
< real 1.543
--- field 2 absolute error 1.07e+00
> real 2.610
### Maximum absolute error in matching lines = 9.00e-02 at line 8 field 2
The above results mean the three-semaphores IPC approach is more
efficient than the single mutex + circular buffer IPC approach for 3
out of these 5 examples (i.e., 60 per cent of the examples) with
differences > 100 ms. However, if I had printed out all differences
greater than the maximum timing noise of ~13 ms that fraction drops to
~20 per cent.
Of course, for multiple cpu hardware (such as my two-cpu PC), circular
buffers do have a general speed advantage in ideal circumstances since
one cpu can be writing to the buffer with -dev wxwidgets and the other
cpu can be reading from that buffer with wxPLViewer. Whereas with the
three semaphores approach writing to the buffer is blocked while it is
being read from and reading from the buffer is blocked while it is
being written to. So that is a factor of two advantage to circular
buffers in ideal circumstances for multiple cpu hardware. However,
there are some additional overheads with the circular buffer approach
(such as timing loops and checks that reading is not getting ahead of
writing) which it appears under certain circumstances (i.e., 60 per
cent of the large difference examples and 20 per cent of all examples)
are larger than that theoretical saving.
Of course, in general, IPC takes a relatively small time to complete
compared to the rendering cost (e.g., example 33 with more than 100
pages takes only a few seconds for IPC) so I believe the above
differences in IPC cost are not that important in the grand scheme of
things. So I suggest (based on the much more important factor of code
clarity) that eventually (say, once you guys have dealt with the issue
with the -locate option for example 1) we should use the three named
semaphores IPC approach exclusively (by forcing the PL_WXWIDGETS_IPC3
macro to always be #defined and the PL_HAVE_UNNAMED_POSIX_SEMAPHORES
macro to always be #undefined). And once we are completely happy with
that configuration (and presumably before the next release) we should
follow up by doing a massive code cleanup to drop the code that is
ignored when we use those forced settings.
Please do fix the -locate issue and evaluate the code clarity for
yourself for the case when PL_WXWIDGETS_IPC3 is #defined and
PL_HAVE_UNNAMED_POSIX_SEMAPHORE #undefined sooner rather than later.
N.B. The reason why I am emphasizing "sooner" is it is important to
deal with such issues relatively early in this release cycle rather
than in some last-minute rush before a release when our judgement will
likely be impaired by release stress. Also, I am extremely familiar
with the -DPL_WXWIDGETS_IPC3=ON version of the code now so I can
easily answer your questions about it while that won't be as much the
case later. So your attention to these wxwidgets requests from me as
soon as you have any spare time at all would be much appreciated!
Alan
__________________________
Alan W. Irwin
Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).
Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________
Linux-powered Science
__________________________
|
|
From: Alan W. I. <ir...@be...> - 2017-03-30 21:00:29
|
The new three named semaphores approach to wxwidgets IPC that I implemented should "just work" on all platforms. The reason why I claim that is the method works on Linux and POSIX named semaphores are widely supported on all POSIX platforms (so should work on all free Unix platforms and Mac OS X and other proprietary Unix platforms). Also, I just checked and Cygwin apparently has complete support for POSIX named semaphores. So the new IPC method should just work in that case as well. Furthermore, I copied how Phil handled (in his earlier implementation of a different IPC method) the Window mutex equivalent of named POSIX semaphores for the case when WIN32 is #defined. Therefore, the new IPC method should just work in that case (the MSVC and MinGW-w64/MSYS2 Windows platforms) as well. However, currently the only test result I have for the new IPC method is for the Linux platform so testing is requested for all the other platforms I mentioned above. I have now (commit 25d120e) disabled all versions of IPC other than the new three named semaphores approach to insure all further testing will be with that new IPC method and as the first step in the planned removal of all other IPC methods we have tried in the past. The commit message indicates the comprehensive test script options that should be used to do the requested testing. Those options insured that the comprehensive test of our wxwidgets device and bindings only took 15 minutes to complete on Linux. I am especially interested in equivalent comprehensive test wxwidgets results for Cygwin, MinGW-w64/MSYS2, and Mac OS X since previous attempts to use that bash test script have worked well on all those platforms. Theoretically, that comprehensive bash script should also work on MSVC (if you set environment variables to access Cygwin or MinGW-w64/MSYS2 unix functionality such as bash.exe). But so far, nobody has figured out exactly how to do that so you may be limited to just hand testing of -dev wxwidgets on MSVC. Alan __________________________ Alan W. Irwin Astronomical research affiliation with Department of Physics and Astronomy, University of Victoria (astrowww.phys.uvic.ca). Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); the Time Ephemerides project (timeephem.sf.net); PLplot scientific plotting software package (plplot.sf.net); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __________________________ Linux-powered Science __________________________ |
|
From: Alan W. I. <ir...@be...> - 2017-03-30 22:15:08
|
On 2017-03-30 14:00-0700 Alan W. Irwin wrote: > [....] The > commit [25d120e] message indicates the comprehensive test script options that > should be used to do the requested testing [of wxwidgets]. P.S. Those options included -DPLPLOT_WX_DEBUG_OUTPUT=ON -DPLPLOT_WX_NANOSEC=ON. The first of those adds lots of debugging messages which normally are not needed. The second of those puts nanosec time stamps on those debugging messages with an experimental method that normally should only work on Linux. So they worked for me on Linux, but I suggest for general comprehensive testing of our wxwidgets device and wxwidgets binding you drop both these options, but especially the second one on non-Linux platforms. Alan __________________________ Alan W. Irwin Astronomical research affiliation with Department of Physics and Astronomy, University of Victoria (astrowww.phys.uvic.ca). Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); the Time Ephemerides project (timeephem.sf.net); PLplot scientific plotting software package (plplot.sf.net); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __________________________ Linux-powered Science __________________________ |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X