Menu
▾
▴
numpy-discussion
|
From: Michael S. <msu...@gm...> - 2006-10-11 20:30:43
|
Hi, I know that it's a perennial topic on the list, but I haven't been able to find my answer in the archives. After running the installation on a RedHat Linux machine, I'm getting the import error: "/usr/lib/libblas.so.3: undefined symbol: e_wsfe". Judging from earlier exchanges here, it seems that I need to add libg2c (which this machine does have in /usr/lib, unlike g2c) somewhere between 'f77blas' and 'cblas', but I'm not sure where I should make this change. Not being well versed in Python distributions, I tried my luck with a few candidates and the import error remains. The machine should be running gcc. Thanks for any help. Michael |
|
From: Michael S. <msu...@gm...> - 2006-10-13 19:23:29
|
Following up on my own message for archival purposes, after getting local help. If you're having a problem like this, read the file called INSTALL.txt. The current NumPy tarball doesn't have this file, but the SciPy tarball does. You may need to reinstall Atlas/Lapack libraries using a different compiler. Michael On 10/11/06, Michael Subotin <msu...@gm...> wrote: > > Hi, > > I know that it's a perennial topic on the list, but I haven't been able to > find my answer in the archives. After running the installation on a RedHat > Linux machine, I'm getting the import error: "/usr/lib/libblas.so.3: > undefined symbol: e_wsfe". Judging from earlier exchanges here, it seems > that I need to add libg2c (which this machine does have in /usr/lib, unlike > g2c) somewhere between 'f77blas' and 'cblas', but I'm not sure where I > should make this change. Not being well versed in Python distributions, I > tried my luck with a few candidates and the import error remains. The > machine should be running gcc. > > Thanks for any help. > > Michael > |
|
From: Lars F. <lfr...@im...> - 2006-10-23 06:25:29
|
Hello all, to interact with some hardware (data retrieval), I use the following scheme (Windows, Python 2.4): * in Python, I create a numpy array as a buffer * I pass this array to a self written .dll using ctypes * the C-code in the .dll passes the pointer to the buffer to the API of the hardware; then the API starts writing data to my buffer * from python I can use a helper function of the .dll to know which part of the buffer is safe to be read at the moment; so I can copy this part to a different numpy array and work with the data This works quite well, but today I got a blue screen, reporting some paging-problem. It occured after doing some other stuff like moving some windows on the screen and importing pylab, and I heard the harddisk working hard, so I assume that the part of memory my buffer is in, was paged to the harddisk. This is a problem, since the hardware driver will continuously try to write to this specific memory location I gave to it. My primary question is, how to avoid a numpy-array being paged. In the ideal case there would be a flag to set, that makes sure that this array is always at this position in physical memory. Of course I am also interested in other people's work on hardware access. Do you think the above discribed way is a good one? To me it was the best way because I could do as much as possible in Python and keep may C-coded .dll very small. Is anyone doing similar things? Thanks for every comment Lars -- Dipl.-Ing. Lars Friedrich Optical Measurement Technology Department of Microsystems Engineering -- IMTEK University of Freiburg Georges-Köhler-Allee 102 D-79110 Freiburg Germany phone: +49-761-203-7531 fax: +49-761-203-7537 room: 01 088 email: lfr...@im... |
|
From: Andrew S. <str...@as...> - 2006-10-23 07:41:06
|
It sounds like your hardware drivers may be buggy -- you should only get segfaults, not (the Windows equivalent of) kernel panics, when your userspace code accesses wrong memory. But if you have buggy hardware drivers, I suppose it's possible that locking the memory will help. This wouldn't be the domain of numpy, however. In Linux, this is acheived with a system call such as mlock() or mlockall(). You'll have to figure out the appropriate call in Windows. Thinking about it, if your hardware drivers require the memory to be locked, they should do it themselves. However, I'm not convinced this is the real issue. It seems at least equally likely that your hardware drivers were developed with particular pattern of timing when accessing the buffers, but now you may be attempting to hold a buffer longer (preventing the driver writing to it) than the developer ever tested. It shouldn't blue-screen, but it does... I think it quite likely that you have some buggy hardware drivers. What hardware is it? -Andrew Lars Friedrich wrote: > Hello all, > > to interact with some hardware (data retrieval), I use the following > scheme (Windows, Python 2.4): > > * in Python, I create a numpy array as a buffer > * I pass this array to a self written .dll using ctypes > * the C-code in the .dll passes the pointer to the buffer to the API of > the hardware; then the API starts writing data to my buffer > * from python I can use a helper function of the .dll to know which part > of the buffer is safe to be read at the moment; so I can copy this part > to a different numpy array and work with the data > > This works quite well, but today I got a blue screen, reporting some > paging-problem. It occured after doing some other stuff like moving some > windows on the screen and importing pylab, and I heard the harddisk > working hard, so I assume that the part of memory my buffer is in, was > paged to the harddisk. This is a problem, since the hardware driver will > continuously try to write to this specific memory location I gave to it. > > My primary question is, how to avoid a numpy-array being paged. In the > ideal case there would be a flag to set, that makes sure that this array > is always at this position in physical memory. > > Of course I am also interested in other people's work on hardware > access. Do you think the above discribed way is a good one? To me it was > the best way because I could do as much as possible in Python and keep > may C-coded .dll very small. Is anyone doing similar things? > > Thanks for every comment > > Lars > > > |
|
From: Lars F. <lfr...@im...> - 2006-10-24 05:40:47
|
Andrew, thanks for your fast answer. Am Montag, den 23.10.2006, 00:45 -0700 schrieb Andrew Straw: > It sounds like your hardware drivers may be buggy -- you should only get > segfaults, not (the Windows equivalent of) kernel panics, when your > userspace code accesses wrong memory. When I started to write this thing, I had a lot of bluescreens. I think they occured when my Python program and the hardware driver were accessing the same memory location at the same time. I think the hardware uses some DMA-technique. > But if you have buggy hardware drivers, I suppose it's possible that > locking the memory will help. This wouldn't be the domain of numpy, > however. In Linux, this is acheived with a system call such as mlock() > or mlockall(). You'll have to figure out the appropriate call in > Windows. Thinking about it, if your hardware drivers require the memory > to be locked, they should do it themselves. I am not really sure about the term "locking". Does that mean that this part is not paged, or that this part is not accessed by two entities at the same time? Or both? Would I do these mlock()-calls in my C-Dll? If Yes, what would happen, if I tried to access the numpy-array from python, during the time it is locked? > However, I'm not convinced > this is the real issue. It seems at least equally likely that your > hardware drivers were developed with particular pattern of timing when > accessing the buffers, but now you may be attempting to hold a buffer > longer (preventing the driver writing to it) than the developer ever > tested. It shouldn't blue-screen, but it does... > I think it quite likely that you have some buggy hardware drivers. What > hardware is it? It is a camera with its own PCI-frame-grabber card. I am using it in "continuous-acquisition-mode" so I suppose, the driver is ready for *long* uses... Anyway, if the bluescreens continue I will have to switch to the "single-frame-acquisition-mode" and prepare a single buffer for every frame to grab. Of course, there would be a different way: I could allocate the buffer in C, in the dll. After data retrievel using the hardware API I could then copy the data to some numpy-array, still using C. But this would make my C-coded dll longer, thus harder to maintain. Is my way a common way? I mean, letting python/numpy do the memory allocation by creating a numpy-array with zeros in and passing its memory location to the hardware-API? Thanks Lars -- Dipl.-Ing. Lars Friedrich Optical Measurement Technology Department of Microsystems Engineering -- IMTEK University of Freiburg Georges-Köhler-Allee 102 D-79110 Freiburg Germany phone: +49-761-203-7531 fax: +49-761-203-7537 room: 01 088 email: lfr...@im... |
|
From: A. M. A. <per...@gm...> - 2006-10-24 06:22:25
|
On 24/10/06, Lars Friedrich <lfr...@im...> wrote: > I am not really sure about the term "locking". Does that mean that this > part is not paged, or that this part is not accessed by two entities at > the same time? Or both? There are two kinds of locking, and really, you probably want both. But mlock() just ensures that the virtual memory stays in actual RAM. > Is my way a common way? I mean, letting python/numpy do the memory > allocation by creating a numpy-array with zeros in and passing its > memory location to the hardware-API? It's not necessary to do it this way. I think a more usual approach would be to create the buffer however is convenient in your C code, then provide its address to numpy. You can then use the ndarray function from python to tell it how to interpret that buffer as an array. Since the C code is creating the buffer, you can make sure it is in a special locked area of memory, ensure that the garbage collector never comes calling for it, or whatever you like. If you're having problems with driver stability, though, you may be safest having your C code copy the buffer into a numpy array in one shot - then you have complete control over when and how the DMA memory is accessed. (In C, I'm afraid, but for this sort of thing C is well-suited.) A. M. Archibald |
|
From: David C. <da...@ar...> - 2006-10-24 06:29:58
|
Lars Friedrich wrote: > Andrew, > > thanks for your fast answer. > > Am Montag, den 23.10.2006, 00:45 -0700 schrieb Andrew Straw: > >> It sounds like your hardware drivers may be buggy -- you should only get >> segfaults, not (the Windows equivalent of) kernel panics, when your >> userspace code accesses wrong memory. >> > > When I started to write this thing, I had a lot of bluescreens. I think > they occured when my Python program and the hardware driver were > accessing the same memory location at the same time. I think the > hardware uses some DMA-technique. > I don't know anything about your device, but a driver directly accessing a memory buffer from a userland program sounds like a bug to me. I am far from being knowledgeable about os programming, but kernel and user space are two different adress spaces, so I don't see how a user-land program could access directly memory from kernel-space (the buffer from your hardware). There has to be copy somewhere, or fancier methods of sharing data; otherwise, this is really unsafe (using data from userland in kernel land is sure to cause problems....). >> But if you have buggy hardware drivers, I suppose it's possible that >> locking the memory will help. This wouldn't be the domain of numpy, >> however. In Linux, this is acheived with a system call such as mlock() >> or mlockall(). You'll have to figure out the appropriate call in >> Windows. Thinking about it, if your hardware drivers require the memory >> to be locked, they should do it themselves. >> > > I am not really sure about the term "locking". Does that mean that this > part is not paged, or that this part is not accessed by two entities at > the same time? Or both? > > Would I do these mlock()-calls in my C-Dll? If Yes, what would happen, > if I tried to access the numpy-array from python, during the time it is > locked? > > In this context, this means preventing the memory manager from putting the corresponding pages to the hard-drive. This is actually part of POSIX (I cannot remember which one), not linux-specific. Windows has a similar API, but I remember having seen somewhere that it is not reliable (ie it can swap pages out); I cannot find the corresponding API right now. > > It is a camera with its own PCI-frame-grabber card. I am using it in > "continuous-acquisition-mode" so I suppose, the driver is ready for > *long* uses... Anyway, if the bluescreens continue I will have to switch > to the "single-frame-acquisition-mode" and prepare a single buffer for > every frame to grab. > > Of course, there would be a different way: I could allocate the buffer > in C, in the dll. After data retrievel using the hardware API I could > then copy the data to some numpy-array, still using C. But this would > make my C-coded dll longer, thus harder to maintain. > I don't understand this either: how allocating in C changes the problem ? David |
|
From: Andrew S. <str...@as...> - 2006-10-24 17:06:15
|
David Cournapeau wrote: > I don't know anything about your device, but a driver directly accessing > a memory buffer from a userland program sounds like a bug to me. David, DMA memory (yes, I know thats an example of RAS Syndrome, apologies) allows hardware to fill a chunk of RAM and then hand it over to a userspace program. In my experience, RAM used for this purpose must be pre-allocated, usually in a ring-buffer type arrangement. So this is normal operating procedure for something like a frame grabber and not a bug at all. However, the fact that Lars is able to elicit blue screens from a user-mode program indicates driver bugs to me. It's likely, however, that once he gets his program operating within the bounds of what the developers tested, it'll work fine. Lars, for now I suggest doing what AM Archibald suggests and doing a memcpy to copy your framebuffers immediately into non-DMA memory and then hand that DMA memory back to the hardware driver. (Typically these drivers have function calls that indicate whether you "own" that part of memory -- this is, confusingly also called "locking" in the thread sense, as opposed to "locking" in the memory page sense.) Finally, whether you allocate memory in C or in numpy makes little difference. But if you want to use numpy, empty() will be presumably faster than zeros(). And it will have the advantage of doing memory management via Python's standard ref-counting. Cheers! Andrew |
|
From: Lars F. <lfr...@im...> - 2006-10-26 08:52:24
|
Hello, thanks for your comments. Am Mittwoch, den 25.10.2006, 11:28 +0900 schrieb David Cournapeau: > Andrew Straw wrote: > > David Cournapeau wrote: > > > >> I don't know anything about your device, but a driver directly accessing > >> a memory buffer from a userland program sounds like a bug to me. > >> > > David, DMA memory (yes, I know thats an example of RAS Syndrome, > > apologies) allows hardware to fill a chunk of RAM and then hand it over > > to a userspace program. In my experience, RAM used for this purpose must > > be pre-allocated, usually in a ring-buffer type arrangement. So this is > > normal operating procedure for something like a frame grabber and not a > > bug at all. > > > What I understood from former emails was that the user is allocating a > memory buffer, and that it gives this memory buffer to the hardware. In > this sense, I don't see how it is possible to avoid kernel panic or > equivalents. If on the contrary, the driver gives you the memory buffer, > then, ok, by eg a mmap-like call, you can access directly the device > memory, but within a range fixed by the driver, which is valid if the > driver is not buggy. The API of the camera provides a function pl_exp_start_cont(hCam, buffer, size) The usage is to pass a Camera-Identifier "hCam", a pointer "buffer" and a size "size" in bytes. The camera will then start to acquire frames and put them to the buffer, which has to be size-bytes long. When the function reaches the end of the buffer, it will start at the beginning again (circular buffer mode). There is also a function available to ask for the last buffer position that is written to. The buffer has to be allocated by the caller of this API-function, so I guess it will be in *userland*. To have enough time to read a frame, the buffer has to be sufficiently big. I don't know what is sufficient, but I made it 50 frames big. With an acquisition time of lets say 10ms I have about 0.5seconds to get a frame, that was recently written to the buffer. I know, these things are not numpy and even not Python-questions, but C-questions, and I think I should go and read something about it. Does anyone have a recommendation where to find information on this memory-locking things? How is this kind of programming called? Driver-Programming? Hardware-Programming? System-Programming? Thanks Lars -- Dipl.-Ing. Lars Friedrich Optical Measurement Technology Department of Microsystems Engineering -- IMTEK University of Freiburg Georges-Köhler-Allee 102 D-79110 Freiburg Germany phone: +49-761-203-7531 fax: +49-761-203-7537 room: 01 088 email: lfr...@im... |
|
From: Lars F. <lfr...@im...> - 2006-10-27 05:55:09
|
Am Donnerstag, den 26.10.2006, 19:08 +0900 schrieb David Cournapeau: > By the way, I found the information about locking pages into memory for > windows: > > http://msdn.microsoft.com/library/default.asp?url=/library/en-us/memory/base/virtuallock.asp > Thanks for the link, I now use a virtuallock in my Dll. I don't get the "paging-error"-bluescreens anymore, but now I get other ones ("DRIVER_IRQL_NOT_LESS_OR_EQUAL"), but I think this is another issue, and I am working on it... If anyone is using python / numpy / ctypes for hardware control (say, Cameras with grabber-cards or fire-wire / DCAM; National Instruments acquisition cards using NIDAQmx, ...) I am interested in discussion! Lars -- Dipl.-Ing. Lars Friedrich Optical Measurement Technology Department of Microsystems Engineering -- IMTEK University of Freiburg Georges-Köhler-Allee 102 D-79110 Freiburg Germany phone: +49-761-203-7531 fax: +49-761-203-7537 room: 01 088 email: lfr...@im... |
|
From: Gael V. <gae...@no...> - 2006-10-27 06:46:40
|
On Fri, Oct 27, 2006 at 07:55:06AM +0200, Lars Friedrich wrote: > If anyone is using python / numpy / ctypes for hardware control (say, > Cameras with grabber-cards or fire-wire / DCAM; National Instruments > acquisition cards using NIDAQmx, ...) I am interested in discussion! Worked great for me ! My approach was to write a small wrapper C (actually C++, with "extern C" linking) library that exposed only what I needed of the camera interface, in a "python-friendly" way, and to wrap it with ctypes. I controlled a "Pixis" princeton instruments camera this way. As I said, it worked surprisingly well. I can send the code as an example if you wish. Ga=EBl |
|
From: Lars F. <lfr...@im...> - 2006-10-27 11:04:10
Attachments:
pvcamDLL.cpp
|
Hello Gael (sorry, I just don't get the dots...), Am Freitag, den 27.10.2006, 08:46 +0200 schrieb Gael Varoquaux: > Worked great for me ! My approach was to write a small wrapper C > (actually C++, with "extern C" linking) library that exposed only what I > needed of the camera interface, in a "python-friendly" way, and to wrap > it with ctypes. I controlled a "Pixis" princeton instruments camera this > way. As I said, it worked surprisingly well. I can send the code as an > example if you wish. Yes, that would be really nice! I think, I am doing the same thing. In the attachment, you find my dll-code, which I am still working on --so it is not ready to use!--, if you have any comments I will be happy. Lars -- Dipl.-Ing. Lars Friedrich Optical Measurement Technology Department of Microsystems Engineering -- IMTEK University of Freiburg Georges-Köhler-Allee 102 D-79110 Freiburg Germany phone: +49-761-203-7531 fax: +49-761-203-7537 room: 01 088 email: lfr...@im... |
|
From: David C. <da...@ar...> - 2006-10-25 02:29:19
|
Andrew Straw wrote: > David Cournapeau wrote: > >> I don't know anything about your device, but a driver directly accessing >> a memory buffer from a userland program sounds like a bug to me. >> > David, DMA memory (yes, I know thats an example of RAS Syndrome, > apologies) allows hardware to fill a chunk of RAM and then hand it over > to a userspace program. In my experience, RAM used for this purpose must > be pre-allocated, usually in a ring-buffer type arrangement. So this is > normal operating procedure for something like a frame grabber and not a > bug at all. > What I understood from former emails was that the user is allocating a memory buffer, and that it gives this memory buffer to the hardware. In this sense, I don't see how it is possible to avoid kernel panic or equivalents. If on the contrary, the driver gives you the memory buffer, then, ok, by eg a mmap-like call, you can access directly the device memory, but within a range fixed by the driver, which is valid if the driver is not buggy. That's why I don't understand the paging problem and why allocating anything from C or python would change anything (I think windows can page out kernel, contrary to linux, but I doubt it can page out DMA areas), because the user does not allocate anything in this scenario. But again, this is just what I would think from "common sense", and I have never done any system programming, so I may just miss something, David |
|
From: David C. <da...@ar...> - 2006-10-26 10:08:54
|
Lars Friedrich wrote: > Hello, > > thanks for your comments. > > Am Mittwoch, den 25.10.2006, 11:28 +0900 schrieb David Cournapeau: > >> Andrew Straw wrote: >> >>> David Cournapeau wrote: >>> >>> >>>> I don't know anything about your device, but a driver directly accessing >>>> a memory buffer from a userland program sounds like a bug to me. >>>> >>>> >>> David, DMA memory (yes, I know thats an example of RAS Syndrome, >>> apologies) allows hardware to fill a chunk of RAM and then hand it over >>> to a userspace program. In my experience, RAM used for this purpose must >>> be pre-allocated, usually in a ring-buffer type arrangement. So this is >>> normal operating procedure for something like a frame grabber and not a >>> bug at all. >>> >>> >> What I understood from former emails was that the user is allocating a >> memory buffer, and that it gives this memory buffer to the hardware. In >> this sense, I don't see how it is possible to avoid kernel panic or >> equivalents. If on the contrary, the driver gives you the memory buffer, >> then, ok, by eg a mmap-like call, you can access directly the device >> memory, but within a range fixed by the driver, which is valid if the >> driver is not buggy. >> > > The API of the camera provides a function > pl_exp_start_cont(hCam, buffer, size) > > The usage is to pass a Camera-Identifier "hCam", a pointer "buffer" and > a size "size" in bytes. The camera will then start to acquire frames and > put them to the buffer, which has to be size-bytes long. When the > function reaches the end of the buffer, it will start at the beginning > again (circular buffer mode). There is also a function available to ask > for the last buffer position that is written to. The buffer has to be > allocated by the caller of this API-function, so I guess it will be in > *userland*. > In this case, I think the data are copied from the device to your buffer. > To have enough time to read a frame, the buffer has to be sufficiently > big. I don't know what is sufficient, but I made it 50 frames big. With > an acquisition time of lets say 10ms I have about 0.5seconds to get a > frame, that was recently written to the buffer. > You have no way to check if the ringbuffer is full or not ? In audio programming under linux, what happens normally is that you have a ring buffer locked in memory (that is cannot be paged out), and you need to check if your read/write pointers overlap to detect a buffer overrun. > I know, these things are not numpy and even not Python-questions, but > C-questions, and I think I should go and read something about it. Does > anyone have a recommendation where to find information on this > memory-locking things? How is this kind of programming called? > Driver-Programming? Hardware-Programming? System-Programming? > > Real time programming, maybe ? Without a knowledge of the API of your device, it is hard to know what the problem is for me. By the way, I found the information about locking pages into memory for windows: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/memory/base/virtuallock.asp It may not work as expected, though, as it happens quite often with windows :) David |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X