Re: [Numpy-discussion] memory position of numpy arrays

[David C. <da...@ar...>]

Lars Friedrich wrote:
> Andrew,
>
> thanks for your fast answer.
>
> Am Montag, den 23.10.2006, 00:45 -0700 schrieb Andrew Straw: 
>   
>> It sounds like your hardware drivers may be buggy -- you should only get
>> segfaults, not (the Windows equivalent of) kernel panics, when your
>> userspace code accesses wrong memory.
>>     
>
> When I started to write this thing, I had a lot of bluescreens. I think
> they occured when my Python program and the hardware driver were
> accessing the same memory location at the same time. I think the
> hardware uses some DMA-technique.
>   
I don't know anything about your device, but a driver directly accessing 
a memory buffer from a userland program sounds like a bug to me. I am 
far from being knowledgeable about os programming, but kernel and user 
space are two different adress spaces,  so I don't see how a user-land 
program could access directly memory from  kernel-space (the buffer from 
your hardware). There has to be copy somewhere, or fancier methods of 
sharing data; otherwise, this is really unsafe (using data from userland 
in kernel land is sure to cause problems....).
>> But if you have buggy hardware drivers, I suppose it's possible that
>> locking the memory will help. This wouldn't be the domain of numpy,
>> however. In Linux, this is acheived with a system call such as mlock()
>> or mlockall(). You'll have to figure out the appropriate call in
>> Windows. Thinking about it, if your hardware drivers require the memory
>> to be locked, they should do it themselves.
>>     
>
> I am not really sure about the term "locking". Does that mean that this
> part is not paged, or that this part is not accessed by two entities at
> the same time? Or both?
>
> Would I do these mlock()-calls in my C-Dll? If Yes, what would happen,
> if I tried to access the numpy-array from python, during the time it is
> locked?
>
>   
In this context, this means preventing the memory manager from putting 
the corresponding pages to the hard-drive. This is actually part of 
POSIX (I cannot remember which one), not linux-specific. Windows has a 
similar API, but I remember having seen somewhere that it is not 
reliable (ie it can swap pages out); I cannot find the corresponding API 
right now.
>
> It is a camera with its own PCI-frame-grabber card. I am using it in
> "continuous-acquisition-mode" so I suppose, the driver is ready for
> *long* uses... Anyway, if the bluescreens continue I will have to switch
> to the "single-frame-acquisition-mode" and prepare a single buffer for
> every frame to grab.
>
> Of course, there would be a different way: I could allocate the buffer
> in C, in the dll. After data retrievel using the hardware API I could
> then copy the data to some numpy-array, still using C. But this would
> make my C-coded dll longer, thus harder to maintain.
>   
I don't understand this either: how allocating in C changes the problem ?

David