Investigating creating a Windows kernel-mode port

[wade dawson]

Hi All. I've used QP in the past on a number of embedded projects. I am now looking for possible solutions to help tame some of the more complicated, distributed logic design challenges in a Windows kernel mode (kmdf) driver. I'm very familiar with when, where, and what you can and cannot do in km and have already been considering the port itself. While AOs might be of use at some point, I'm initially interested in getting QHSM and QEvt (static and pool-based) working. We would be looking to commercially license QP for this purpose.

Are there examples of HSM only usage of QP without AOs?

Thanks for any insight,
Wade.

[Quantum Leaps]

I'm initially interested in getting QHSM and QEvt (static and pool-based) working.

Yes, that's possible, because the QP framework has a layered structure with the QHsm/QMsm base classes separated out into the QEP layer ("Quantum Event Processor"). This means that you can use just the QHsm/QMsm base classes without using the QActive/QMActive subclasses. I suppose that this is the approach taken by NASA JPL in the Martian missions flight software , where the "Samek's state machines" have been used, but without the active objects. (NASA is using their own event-driven architecture very similar to active objects.)

Are there examples of HSM only usage of QP without AOs?

Yes, inside the qpc|qpcpp\examples\workstation directory:

• calc, calc1, calc1_sub, calc2 - calculator example in several versions
• comp - "Orthogonal Component" state pattern, where the Alarm component is QHsm
• dpp-comp - "Dining Philosophers Problem" with "Orthogonal Component"
• game-gui - "Fly 'n' Shoot" game, where the Mines are "Orthogonal Components" of the Tunnel
• history_qhsm, history_qmsm - transition to history, two versions
• qhsmtst - the QHsm-test example from the PSiCC2 book
• qmsmtst - the QMsm-test example similar to QHsm-test

[Panopticon]

I've also used "just HSMs" for complex interrupts (ISRs) in bit-banging communications protocols where the original ISR was a mess of flags. Think of things like PPP, HDLC, or bit-banging UART or even I2C or SPI. Sometimes due to low power requirements, you do almost everything in ISRs and very little if anything in "background" code (where AOs would run).

I realize that this is literally very far from Wade's specific use case (Windows / non-interrupt) but conceptually it is almost identical; using the QEP component of the QP, a nicely-encapsulated blob of functionality, without the rest of the QP. It can be done easily.

[wade dawson]

Thanks for the info, Panopticon. I've had a think while designing the HSM and delivering events, etc., for this problem and it's really looking like full QF (w/o QSPY) will be well worth the porting effort (though truth be told, I've gotten it ported in about 2 hours other than the issue below - thanks Miro!).

While I'm not using ISRs in the embedded sense, QF will be processing events from differing IRQLs, (Interrupt Request Level), thereby suffering from some of the same (or worse) restrictions present in the embedded realm. One of these restrictions is the inability to use any blocking waits (spinlocks only) at the elevated IRQLs (like ISRs) that several of the processing events will be generated at, so, an AO on a high-priority, PASSIVE level thread will allow implementing straight-forward RTC semantics by simply blocking for things like resource mutexes, etc. in the event handlers .

The issue I'm looking for advice on is how to deal with the few QF static heap allocations like:

static WDFSPINLOCK l_win32CritSect;
static WDFWAITLOCK l_startupCritSect;
static bool  l_isRunning;      // flag indicating when QF is running

Memory accessed at elevated IRQLs needs to be allocated with specific attributes pertaining to caching, paging, and execution-prevention to name a few.

Ideally I'd allocate them somewhere then let QF know about them somehow...
Any hints on how to do this would be greatly appreciated!

Thanks again, Miro, for all of your great work!

[Quantum Leaps]

I'm not sure that I understand your question/problem. The static objects you list are not on any heap. They are in the .bss section pre-allocated at link time. If you need to place them in some other memory region, you most likely have to do it through some compiler-specific extension (e.g., #pragma or attribute(...)).

But it seems to me that you need to create a special QP port to "win-kernel" anyway. So, in that port you can do whatever you like with your variables. Also, it seems to me that the Win32 critical section won't be an appropriate mechanism for critical section in the kernel mode.

I'm not sure why you talk about "blocking on resource mutexes, etc.) Wasn't the whole reason of going with QP in the first place to avoid any blocking and sharing of resources (and thus any need for mutual exclusion)?

--MMS

[wade dawson]

On Jun 29, 2021, at 8:57 PM, Quantum Leaps quantum-leaps@users.sourceforge.net wrote:

> I'm not sure that I understand your question/problem. The static objects you list are not on any heap. They are in the .bss section pre-allocated at link time. If you need to place them in some other memory region, you most likely have to do it through some compiler-specific extension (e.g., #pragma or attribute(...)).

The segment or .section the allocations end up in are not as important here as the attributes of the actual PTE (page table entry) for the physical page that the variables end up in... ie dynamically allocated with a call to ExAllocateNonPagedPool(...)... we use overloads of new() and delete() to ensure these are correct. But yes, certain segments automatically have their pages’ PTEs adjusted by the loader so as to be non-paged, no-execute, etc, but the de-facto is that you allocate explicitly (C) or overload new / delete (c++) for the specific attributes that you require.

*> But it seems to me that you need to create a special QP port to "win-kernel" anyway. So, in that port you can do whatever you like with your variables. Also, it seems to me that the Win32 critical section won't be an appropriate mechanism for critical section in the kernel mode.

Indeed... I’m using kernel fast mutexes for the QF startup lock and high-irql safe spinlock for QF’s critical sections because events can be queued at high IRQLs. Is the aforementioned critical section really “my variable”, though? Doesn’t it belong to QF? If I (the port-er) do indeed own qfinit() can i simply add a context pointer parameter then use this in the port’s critical section accessors? (Correctly allocate all essential stuff in a struct then pass this to qfinit(), of course modifying appropriately the port code to dereference this struct pointer)
*
I’ll go back and look again at qf_port.cpp/.hpp. It looks like I may have misunderstood the ownership boundaries.

> I'm not sure why you talk about "blocking on resource mutexes, etc.) Wasn't the whole reason of going with QP in the first place to avoid any blocking and sharing of resources (and this any need for mutual exclusion)?

That was one of the motivating factors, yes, but closer examination reveals that for this problem mutexes/locks won’t solve it anyway. Consider the situation where the AO is processing an event, accessing a pointer to an object that represents a hardware device and associated data structures (pcie regs for instance) created outside the scope of the AO in question. The OS can, at any time, asynchronously notify you (via callbacks) that the device has been removed, at which time you get your ONLY chance to delete /cleanup the object and return. There are NO magic std::shared_ptr<> types here so any in-flight event using pointers to the object will be in trouble as the pointer becomes invalid. While the DEVICE_REMOVED_SIG is queued to the AO that is using the object, I need to guarantee that the object remains valid (operations to the device can fail, that’s ok, but pointers must be valid) until the DEVICE_REMOVED_SIG event is processed, driving the sm to a state where no further accesses will occur..

I guess one solution would be to create a Qevt-derivative with a waitable system event (KEVENT) as a payload, queue this new event with a DEVICE_REMOVED_SIG to the AO from the removal callback, then immediately enter a wait on the KEVENT on the OS callback thread. Once the wait is complete, the object can safely be deleted, invalidating the pointer. The AO's handler signals the passed KEVENT in its DEVICE_REMOVED_SIG handler and gets driven to an “uninitialised” state where the pointer will no longer be touched.
The OS fully expects that it may take a bit of time to ensure all operations are completed on a device before completing removal so a reasonable wait is ok.
-wade

Like many other folks, I'm sure, I'm replacing discrete chunks of an existing design rather than having the luxury of taking a wholistic, ground-up, QP-based redesign approach. In this case I'm replacing the USB audio streaming packet setup, teardown and processing of an existing driver.

-Wade

[wade dawson]

Hi Miro. The statics are apparently more of a problem and not just limited to "code under my control." :

1>QP-WDF.lib(qf_dyn.obj) : warning LNK4210: .CRT section exists; there may be unhandled static initializers or terminators
1>QP-WDF.lib(qf_time.obj) : warning LNK4210: .CRT section exists; there may be unhandled static initializers or terminators

The linker doesn't point out the culprits (that I'm aware of) but if I had to guess I'd say:
qf_time.cpp:

// Package-scope objects *****************************************************
QTimeEvt QF::timeEvtHead_[QF_MAX_TICK_RATE]; // heads of time event lists

qf_dyn.cpp:

// Package-scope objects *****************************************************
QF_EPOOL_TYPE_ QF_pool_[QF_MAX_EPOOL]; // allocate the event pools
std::uint_fast8_t QF_maxPool_; // number of initialized event pools

Might you have any suggestions as to how I might proceed (without just blatantly hacking something :/ )

Thanks for your time,
-wade

[Quantum Leaps]

Initialization of uninitialized static objects (to zero) is part of the C language standard. So, I would say that the kernel mode program seems not to be compliant (?)

If you are not using time events, then you should be able to remove qf_time.cpp from the build.

--MMS