So I've been playing about with the Win32 port of 5.4.2 (full Win32 port, not Win32-qv) with a view to having a dual targetting setup that mimics the QK behaviour as closely as possible (within the obvious real time shortcomings of Windows).
Assuming a super basic setup:
AO L - low priority AO with Win32 priority set to thread_priority_normal
AO H - high priority AO with Win32 priority set to thread_priority_highest
When I test, I'm not seeing the behaviour I would expect when posting an event from a AO L to AO H. If AO H is idle and waiting on its windows event, I would expect to see an immediate preemptive switch to AO H when AO L posts to it and sets its event object, just like the QK.
However, I seem to see more time slice like scheduling where the context switch is not imediate and occurs after AO L has fully completed its dispatch call, much like the vanilla scheduler.
Whats going on here? Is there something wrong with my code or am I expecting too much of windows context switches on wait events? To make matters worse, I'm sure it used to work like I am expecting when I last looked at it a few years back, but could be my dodgy memory. Any thoughts you can offer would be most appreciated.
Cheers
Lawrence
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I guess that the behavior you observe is due to the preemption handling in Windows. In fact, I would suggest that you make a simple experiment, in which you take QP entirely out of the picture, and create two Win32 threads (by means of the CreateThread API) with two different priorities. Next, you block the high-priority thread on a Win32 object (by means of the WaitForSingleObject API), and then you signal this object from the low-priority thread. You can then experimentally find out if the preemption happens immediately (like in QK), or later.
Also, please note that the behavior might depend on the version of Windows and even on your privileges, for example if you run as an Administrator.
If you perform any of these experiments, please post your findings for others to benefit from.
--MMS
Last edit: Quantum Leaps 2015-08-13
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thanks for the advice. So I took myself back to school and setup some basic tests (using Win 7) as you suggested and the penny has dropped. My confusion stemmed from an ambiguous understanding of the Windows event signalling mechanism, for some reason I was thinking the low priority object would block when posting to the higher object (ala QK), but obviously this is not the case, it has mearly signalled a request to the higher object to unlock and excecute. For whatever reason there is a always a non deterministic delay before Windows excecutes its scheduler and switches to the higher AO, but the key thing being the lower AO will continue to excute after the call to SetEvent until the switch occurs, which explains the behaviour I observed.
Obviously nothing can be done about Windows introducing delays to the task switch, but its a shame there isnt a better way to syncronise between AO's, something that can block and release when higher AO's become available to run which would closely mimic the QK behaviour.
So a couple of questions:
Is is correct to assume running the Win32-qv port (and eliminating the dependance on Win threads) will offer idnetical behaviour (it terms of QP event sequencing) to an embedded target running its own qv port (ie ARM qv port). I cant think of any reason why this would not be the case?
Based on this, while in theory the full Win32 port seems like a nice idea, I cant see why in practice you would want to use it over the Win32-qv port. It introduces non deterministic behaviour in terms of event sequencing and with the limitiations of Win32 priority levels and corresponding use of round robin scheduling you are effectively loosing the notion of priority based excecution for your active objects. I appreciate that most of the time this is not a big deal, but I have been caught out a few times already when moving code over to the embedded target. What does this port buy you over the qv version? Am I missing something?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
First, welcome to the world of very, very "soft real-time" desktop operating systems...
The win32-qv port has been created to remove as much as possible the preemption non-determinism in Windows. This was done upon requests from customers, who wanted to test groups of AOs. (Unit testing of individual AOs in isolation does not require the full scheduler and can be done by dispatching events directly to AOs). So, yes, I would agree that as long as Windows scheduler is not involved, the event sequencing in a win32-qv application should be the same on Windows as in an embedded target running the actual QV cooperative kernel. However, as soon as Windows scheduling is involved, the sequencing might differ (e.g., time events are generated from another Windows thread, so Windows scheduling is involved even in the win32-qv port).
Regarding the full win32 port with each AO running in its own Win32 thread, this is a "by-the-book" implementation of active objects for Windows to build applications with this paradigm. It does not guarantee the same exact sequencing of events as a true hard-real-time scheduler would, but I don't think that anybody can provide such guarantees on a desktop OS. This poses an additional challenge to structure applications such that the differences in event sequencing due to scheduling would not matter.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I have to confess I'm playing devils advocate a little here, we have been using the QT QPC port for dual target development for the last couple of years and it has been a revelation, to the point where our entire development cycle has shifted to primary development on the desktop with the occasional drop onto the hardware to test out drivers etc. It's only when I got it into my head recently to take it to the next level and try for full QK emulation that I have discovered what you already knew, its not happening on a big OS.
On the subject of cross development using QP, some thoughts I have:
I noticed the QT C port has been removed, was it not popular? I found the combined power of QT for high level environment emulation mixed with QPC let us do some pretty awesome work, was surprised to see it go. Following on from our discussion, I'm thinking a QT-qv port in C would make a really great embedded emulation development platform.
Having been working with QT for some time I revisited the Win32 port to take a proper look at scheduling QP on a big OS. I can see the attraction of Win32 in terms of porting QP, with the easy access to the Win message pump, but, I have to say Win32 is the work of the devil. Trying to do anything complex in terms of environment emulation is such a major pain. Sure the win gui wrapper helps, but when you start adding in network sockets for backplane emulation, ASIC emulation etc it really sucks, so much easier on QT. What would be awesome would be to able to use C#/.NET for developing your support package coupled in with the QP port. Just an idea.
Cheers
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thanks for letting me know that the QP/C++-Qt integration gets some mileage. It's always good to know, because these things cost a lot of time to develop and maintain.
The QP/C-Qt integration has been dropped precisely because it was not clear how widely it was used. I hope that everybody understands that maintaining each QP port and set of examples slows down every release and represents a considerable cost of lost opportunity. I mean, the same time and resources could be used to develop something more widely used. For example, the bi-directional QS/QSPY software tracing has been planned for years, but only recently it is being done. (BTW, a very nice, extendible, QS/QSPY overhaul is coming really soon...)
But going back to the benefits of "dual targeting", I really don't get why it is not the norm yet. Everywhere I go, the first question I ask is whether they leverage the shiny PCs that sit under their noses. To my surprise they rarely use PCs to develop and test the embedded code. It's like choosing to walk instead of driving a nice car that they already have. Go figure...
--MMS
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
One more log for the fire. I ported QP to Windows Embedded (CE) which is (or can be made to be ) hard real time, unlike big Windows which has a very complated non-deterministic kernel scheduler.
You do have the ability to make CE very deterministic although I admit there are several subtle factors that have to be concidered.
The suggestion to use C# is opening a can of worms if you are concerned with real time dead lines. C# (and Java) have a garbage collector that runs at high priority and you, as a user, have absolutly no control over when or how long it will run. So unless you want to take toltal control over your C# object creation (which totally defeats the purpose of using the language in the first place) there are better alternatives.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
That's a good point and another justification for the "regular" (multi-threaded) QP-Win32 port. The Win32 API can, in principle, be real-time and there are many implementations of this API in various versions of Windws. WinCE is one example. I'm not sure about WinRT and perhaps others...
Speaking of C#, I keep receiveing inquiries about "QP/C#" version. I suppose that an Active Object framework in C# could be an interesting idea (e.g., see qf4net), but this wouild go beyond core expertise of Quantum Leaps and would spread the resources too thin, I guess.
--MMS
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi Miro
So I've been playing about with the Win32 port of 5.4.2 (full Win32 port, not Win32-qv) with a view to having a dual targetting setup that mimics the QK behaviour as closely as possible (within the obvious real time shortcomings of Windows).
Assuming a super basic setup:
When I test, I'm not seeing the behaviour I would expect when posting an event from a AO L to AO H. If AO H is idle and waiting on its windows event, I would expect to see an immediate preemptive switch to AO H when AO L posts to it and sets its event object, just like the QK.
However, I seem to see more time slice like scheduling where the context switch is not imediate and occurs after AO L has fully completed its dispatch call, much like the vanilla scheduler.
Whats going on here? Is there something wrong with my code or am I expecting too much of windows context switches on wait events? To make matters worse, I'm sure it used to work like I am expecting when I last looked at it a few years back, but could be my dodgy memory. Any thoughts you can offer would be most appreciated.
Cheers
Lawrence
I guess that the behavior you observe is due to the preemption handling in Windows. In fact, I would suggest that you make a simple experiment, in which you take QP entirely out of the picture, and create two Win32 threads (by means of the
CreateThreadAPI) with two different priorities. Next, you block the high-priority thread on a Win32 object (by means of theWaitForSingleObjectAPI), and then you signal this object from the low-priority thread. You can then experimentally find out if the preemption happens immediately (like in QK), or later.Also, please note that the behavior might depend on the version of Windows and even on your privileges, for example if you run as an Administrator.
If you perform any of these experiments, please post your findings for others to benefit from.
--MMS
Last edit: Quantum Leaps 2015-08-13
Thanks for the advice. So I took myself back to school and setup some basic tests (using Win 7) as you suggested and the penny has dropped. My confusion stemmed from an ambiguous understanding of the Windows event signalling mechanism, for some reason I was thinking the low priority object would block when posting to the higher object (ala QK), but obviously this is not the case, it has mearly signalled a request to the higher object to unlock and excecute. For whatever reason there is a always a non deterministic delay before Windows excecutes its scheduler and switches to the higher AO, but the key thing being the lower AO will continue to excute after the call to SetEvent until the switch occurs, which explains the behaviour I observed.
Obviously nothing can be done about Windows introducing delays to the task switch, but its a shame there isnt a better way to syncronise between AO's, something that can block and release when higher AO's become available to run which would closely mimic the QK behaviour.
So a couple of questions:
Is is correct to assume running the Win32-qv port (and eliminating the dependance on Win threads) will offer idnetical behaviour (it terms of QP event sequencing) to an embedded target running its own qv port (ie ARM qv port). I cant think of any reason why this would not be the case?
Based on this, while in theory the full Win32 port seems like a nice idea, I cant see why in practice you would want to use it over the Win32-qv port. It introduces non deterministic behaviour in terms of event sequencing and with the limitiations of Win32 priority levels and corresponding use of round robin scheduling you are effectively loosing the notion of priority based excecution for your active objects. I appreciate that most of the time this is not a big deal, but I have been caught out a few times already when moving code over to the embedded target. What does this port buy you over the qv version? Am I missing something?
First, welcome to the world of very, very "soft real-time" desktop operating systems...
The win32-qv port has been created to remove as much as possible the preemption non-determinism in Windows. This was done upon requests from customers, who wanted to test groups of AOs. (Unit testing of individual AOs in isolation does not require the full scheduler and can be done by dispatching events directly to AOs). So, yes, I would agree that as long as Windows scheduler is not involved, the event sequencing in a win32-qv application should be the same on Windows as in an embedded target running the actual QV cooperative kernel. However, as soon as Windows scheduling is involved, the sequencing might differ (e.g., time events are generated from another Windows thread, so Windows scheduling is involved even in the win32-qv port).
Regarding the full win32 port with each AO running in its own Win32 thread, this is a "by-the-book" implementation of active objects for Windows to build applications with this paradigm. It does not guarantee the same exact sequencing of events as a true hard-real-time scheduler would, but I don't think that anybody can provide such guarantees on a desktop OS. This poses an additional challenge to structure applications such that the differences in event sequencing due to scheduling would not matter.
I have to confess I'm playing devils advocate a little here, we have been using the QT QPC port for dual target development for the last couple of years and it has been a revelation, to the point where our entire development cycle has shifted to primary development on the desktop with the occasional drop onto the hardware to test out drivers etc. It's only when I got it into my head recently to take it to the next level and try for full QK emulation that I have discovered what you already knew, its not happening on a big OS.
On the subject of cross development using QP, some thoughts I have:
I noticed the QT C port has been removed, was it not popular? I found the combined power of QT for high level environment emulation mixed with QPC let us do some pretty awesome work, was surprised to see it go. Following on from our discussion, I'm thinking a QT-qv port in C would make a really great embedded emulation development platform.
Having been working with QT for some time I revisited the Win32 port to take a proper look at scheduling QP on a big OS. I can see the attraction of Win32 in terms of porting QP, with the easy access to the Win message pump, but, I have to say Win32 is the work of the devil. Trying to do anything complex in terms of environment emulation is such a major pain. Sure the win gui wrapper helps, but when you start adding in network sockets for backplane emulation, ASIC emulation etc it really sucks, so much easier on QT. What would be awesome would be to able to use C#/.NET for developing your support package coupled in with the QP port. Just an idea.
Cheers
Thanks for letting me know that the QP/C++-Qt integration gets some mileage. It's always good to know, because these things cost a lot of time to develop and maintain.
The QP/C-Qt integration has been dropped precisely because it was not clear how widely it was used. I hope that everybody understands that maintaining each QP port and set of examples slows down every release and represents a considerable cost of lost opportunity. I mean, the same time and resources could be used to develop something more widely used. For example, the bi-directional QS/QSPY software tracing has been planned for years, but only recently it is being done. (BTW, a very nice, extendible, QS/QSPY overhaul is coming really soon...)
But going back to the benefits of "dual targeting", I really don't get why it is not the norm yet. Everywhere I go, the first question I ask is whether they leverage the shiny PCs that sit under their noses. To my surprise they rarely use PCs to develop and test the embedded code. It's like choosing to walk instead of driving a nice car that they already have. Go figure...
--MMS
One more log for the fire. I ported QP to Windows Embedded (CE) which is (or can be made to be ) hard real time, unlike big Windows which has a very complated non-deterministic kernel scheduler.
You do have the ability to make CE very deterministic although I admit there are several subtle factors that have to be concidered.
The suggestion to use C# is opening a can of worms if you are concerned with real time dead lines. C# (and Java) have a garbage collector that runs at high priority and you, as a user, have absolutly no control over when or how long it will run. So unless you want to take toltal control over your C# object creation (which totally defeats the purpose of using the language in the first place) there are better alternatives.
That's a good point and another justification for the "regular" (multi-threaded) QP-Win32 port. The Win32 API can, in principle, be real-time and there are many implementations of this API in various versions of Windws. WinCE is one example. I'm not sure about WinRT and perhaps others...
Speaking of C#, I keep receiveing inquiries about "QP/C#" version. I suppose that an Active Object framework in C# could be an interesting idea (e.g., see qf4net), but this wouild go beyond core expertise of Quantum Leaps and would spread the resources too thin, I guess.
--MMS