From: Antonino I. <tri...@gm...> - 2007-03-27 17:35:11
Attachments:
microplotlib.py
microlib.gladep
|
Hi to the list, I'm searching to display in realtime some data read serial port. The data is a 2D matrix and is read element wise from the serial, one pixel each one (or more) seconds. I'm running the script from "ipython -pylab" using the command "run scriptname". After "loading" the script I interactively launch the function that start the data acquisition and I would like to see the acquired data so far. In the loop that perform the data acquisition I'm currently calling this function to plot data: def plot_data_in_itinere(data, **kwargs): global first_plot d = data.ravel()[find(data.ravel() > 0)] m = d.mean() vr = round(5*d.std() / m, 3) #clf() title('Scanning Data') xlabel('Mean Value: '+str(int(round(m)))+' -- Relative Variation: '+str(vr)) b,t = ylim() ylim(t,b) imshow(data.astype(float), interpolation='nearest', vmin=0, origin='lower', **kwargs) if first_plot: colorbar() first_plot = False However the call to the function is "blocking". So the acquisition time is longer (not only I have to wait the data but I have to wait the plot too). Furthermore, this function plot the new data above the old one, so the plot becomes slower and slower while the acquisition goes on. If I uncommented the cla() line, I get a plot that is blank most of the time and that shows the data only for a fraction of second while the new plot is performed. Is there a way to update (or substitute) the current showed matrix data, deleting the old plots (in the axis). I have to confess that I've have not understand well how the pylab interactive mode works. Ideally the plot should be performed in background, while the script goes on and wait the next data arrival. I've tried to do this with a simple gtk app that embeds a matplotlib plot. Hoever, I don't know how to send the application in background. I've tried to do something like this: class PlotScanApp: ... class GuiThread(threading.Thread): def run(self): gtk.main() if __name__ == '__main__': stdout = sys.stdout a = array(randn(100)).reshape(10,10) p = PlotScanApp(a) p.start() print 'Start image update:' for i in range(10): sleep(1) stdout.write('.') stdout.flush() p.plot(array(randn(100)).reshape(10,10)) the PlotScanApp implement the gui (full files attached). Running the script the image should be updated with random data each 1 sec. but only the first image is showed. I suppose this is not the way to put a gui drawing app in background... Any hints in how to do in the right way this gtk app that when required update the plot? Thank a lot. Ciao, ~ Antonio |
From: Ken M. <mc...@ii...> - 2007-03-28 15:10:34
|
On Mar 27, 2007, at 12:35 PM, Antonino Ingargiola wrote: > > I'm searching to display in realtime some data read serial port. The > data is a 2D matrix and is read element wise from the serial, one > pixel each one (or more) seconds. You shouldn't have any problems making this happen, although it requires a lot more legwork than creating normal interactive plots. > I'm running the script from "ipython -pylab" using the command "run > scriptname". After "loading" the script I interactively launch the > function that start the data acquisition and I would like to see the > acquired data so far. I've never tried to do something like this using pylab before, so I probably can't be of much help there. However, I can point you toward an example of how to do it from within wxPython using a timer. > In the loop that perform the data acquisition I'm currently calling > this function to plot data: > > def plot_data_in_itinere(data, **kwargs): > global first_plot > d = data.ravel()[find(data.ravel() > 0)] > m = d.mean() > vr = round(5*d.std() / m, 3) > #clf() > title('Scanning Data') > xlabel('Mean Value: '+str(int(round(m)))+' -- Relative > Variation: '+str(vr)) > b,t = ylim() > ylim(t,b) > imshow(data.astype(float), > interpolation='nearest', > vmin=0, > origin='lower', **kwargs) > if first_plot: > colorbar() > first_plot = False > > However the call to the function is "blocking". So the acquisition > time is longer (not only I have to wait the data but I have to wait > the plot too). You should probably do the acquisition asynchronously by running it in a separate thread. That thread would read in the data one point at a time, perform any pre-processing, and post the results to a place that's shared between it and the main plotting thread. The main thread would periodically check and see if the shared data has changed and redraw the plot if needed. I'm not sure how hard this is to do in a reasonable way in pylab, but I've used this approach before in wxPython GUIs. > Furthermore, this function plot the new data above the > old one, so the plot becomes slower and slower while the acquisition > goes on. If I uncommented the cla() line, I get a plot that is blank > most of the time and that shows the data only for a fraction of second > while the new plot is performed. ` You might want to consider create a mock data source that generates a stream of values from some pre-collected data or Python's "random" module. That would let you work on debugging the plotting end of things first. It would also make it easier for you to share your code with the list. > Is there a way to update (or substitute) the current showed matrix > data, deleting the old plots (in the axis). I have to confess that > I've have not understand well how the pylab interactive mode works. pylab.imshow() returns a matplotlib.image.AxesImage object. It looks like you can update the data array using its set_data() method. The class documentation is available on the matplotlib website: http://matplotlib.sourceforge.net/matplotlib.image.html#AxesImage > Ideally the plot should be performed in background, while the script > goes on and wait the next data arrival. You'd almost certainly be happier doing things the other way around. Most GUI toolkits are extremely fussy about what thread the GUI event loop runs in. For example, wxPython requires App.MainLoop() be called from the thread that first imported the wxPython module. That being said, it's possible to run the GUI thread in the background -- the iPython wizards might be able to help you figure it out. > I've tried to do this with a simple gtk app that embeds a > matplotlib plot. Hoever, I don't know how to send the application > in background. I've tried to do something like this: > <snip> > > the PlotScanApp implement the gui (full files attached). Running the > script the image should be updated with random data each 1 sec. but > only the first image is showed. I suppose this is not the way to put a > gui drawing app in background... Well, using pylab from within a GUI application is a bit dodgy to begin with. I can see a few potential problems. The first is that you're calling plotting commands from within the main thread, although the GUI is running in a background thread. You might try doing the plotting from within the GUI thread by having a timer fire once a second to redraw the plot. The second problem is that you're not calling pylab.draw(), which forces the current figure to redraw itself. Whether or not pylab is running in interactive mode might be a factor here. There's some documentation about interactive mode and the ion()/ioff() commands here: http://matplotlib.sourceforge.net/interactive.html Ken |
From: Fernando P. <fpe...@gm...> - 2007-03-28 15:14:49
|
On 3/28/07, Ken McIvor <mc...@ii...> wrote: > You'd almost certainly be happier doing things the other way around. > Most GUI toolkits are extremely fussy about what thread the GUI event > loop runs in. For example, wxPython requires App.MainLoop() be > called from the thread that first imported the wxPython module. That > being said, it's possible to run the GUI thread in the background -- > the iPython wizards might be able to help you figure it out. No, the limitation you describe is there just as much. What we do in ipython is push the *user code* execution into the secondary thread, to make sure we keep the GUI toolkits happy for the very reasons you outline. Cheers, f |
From: Antonino I. <tri...@gm...> - 2007-03-28 23:03:47
|
On 3/28/07, Ken McIvor <mc...@ii...> wrote: > On Mar 27, 2007, at 12:35 PM, Antonino Ingargiola wrote: [cut] > You should probably do the acquisition asynchronously by running it > in a separate thread. That thread would read in the data one point > at a time, perform any pre-processing, and post the results to a > place that's shared between it and the main plotting thread. The > main thread would periodically check and see if the shared data has > changed and redraw the plot if needed. I'm not sure how hard this is > to do in a reasonable way in pylab, but I've used this approach > before in wxPython GUIs. That's exactly what I'd like to do. The problem is that if I run gtk.main() (the gtk main GUI loop) in a separate thread the program stops until I do something on the GUI (for example passing the mouse on it). My understanding so far is the following. When the function that execute gtk.main() is executed, python doesn't switch thread until either 100 bytecode instructions are executed or since an I/O (potentially) blocking operation in executed. When I'm doing nothing on the GUI application, neither 100 byte code instructions are executed in the thread neither an I/O call is performed, so the whole program (including the *other* threads) stalls. Basically, I don't know which is the right way to put a Gtk GUI in background, while another thread get the data asynchronously. BTW, the timer idea is good and eliminates the need to call asynchronously the GUI thread to update the plot, which seems (the latter case) not very simple to do. > > Furthermore, this function plot the new data above the > > old one, so the plot becomes slower and slower while the acquisition > > goes on. If I uncommented the cla() line, I get a plot that is blank > > most of the time and that shows the data only for a fraction of second > > while the new plot is performed. > ` > You might want to consider create a mock data source that generates a > stream of values from some pre-collected data or Python's "random" > module. That would let you work on debugging the plotting end of > things first. It would also make it easier for you to share your > code with the list. Thanks to your suggestion to use the image.set_data method I've created a simplified script (pasted at the end) that demonstrate how to do live-update while acquiring (jep!). The last think I'm not yet able to do is to update the colorbar to autoscale with the new incoming data. The the script that follows tries to update the colorbar too but it does not work (on matplotlib 0.87 at least). Any hints? Many thanks so far... Ciao, ~ Antono PS: Here it is the script: from time import sleep from random import randint from pylab import * N = 5 def fake_data_generator(): sleep(randint(1,3)/10.0) return rand()*100 def data_aquisition_loop(): data = zeros((N,N)) for j in range(N): for i in range(N): print ' - Acquiring pixel (',i,',',j,') ... ' data[j,i] = fake_data_generator() plot_update(data) return data def setup_plot(data): global image, color_bar title('Scanning Data') image = imshow(data, interpolation='nearest', origin='upper', extent=[0,N,N,0]) color_bar = colorbar() def plot_update(data): image.set_data(data) color_bar.set_array(data) color_bar.autoscale() draw() if __name__ == '__main__': ion() setup_plot(rand(N*N).reshape(N,N)) data_aquisition_loop() |
From: Ken M. <mc...@ii...> - 2007-03-28 23:38:52
|
On Mar 28, 2007, at 6:03 PM, Antonino Ingargiola wrote: > On 3/28/07, Ken McIvor <mc...@ii...> wrote: >> You should probably do the acquisition asynchronously by running it >> in a separate thread. <snip> > > That's exactly what I'd like to do. The problem is that if I run > gtk.main() (the gtk main GUI loop) in a separate thread the program > stops until I do something on the GUI (for example passing the mouse > on it). Does it do this if you run gtk.main() in the main thread (e.g. the interpreter prompt) while doing something else in a background thread? > My understanding so far is the following. When the function > that execute gtk.main() is executed, python doesn't switch thread > until either 100 bytecode instructions are executed or since an I/O > (potentially) blocking operation in executed. I think you're more or less correct but are unaware of one important factor. Here's my understanding of how multithreading works in Python... Thread switching is controlled by something called the Global Interpreter Lock, which is implemented in an abstract way on top of an OS lock/mutex object. Only one Python thread runs at a time because the Python interpreter requires a thread to hold the GIL before it can execute bytecode instructions. The running thread holds the GIL while is executes 100 instructions. During this time other Python threads block waiting to acquire the GIL. After it has executed its 100 instructions, the running thread releases the GIL and then attempts to reacquire it. The OS ensures that things are fair by preventing one thread from reacquiring the GIL over and over again when other threads are also waiting for it. Python doesn't actually detect you do something that will block the thread, like an I/O operation. Instead, the C code implementing an I/ O operation like file.write() releases the GIL before performing the operation. This allows a different thread to acquire it and run some more bytecode instructions while the first thread performs its I/O operation. > When I'm doing nothing on the GUI application, neither 100 byte > code instructions are executed in the thread neither an I/O call is > performed, so the whole program (including the *other* threads) > stalls. I'm not sure on the details, but the C code that implements gtk.main () almost certainly releases the GIL before running the GUI event loop. Otherwise things like iPython wouldn't be able to run the GUI in a separate thread. So, that's probably not the problem. It's possible that there's another lock that's used to protect access to the gtk module. If that's the case you could be causing deadlock by calling making gtk calls from multiple threads. In my experience it's best to stick to the one-thread rule: the thread that runs the GUI's event loop is the only thread that's allowed to do anything related to the GUI. > Basically, I don't know which is the right way to put a Gtk GUI in > background, while another thread get the data asynchronously. You probably want to put the GUI in the main thread. The main thread is the only thread in a Python program that receives the KeyboardInterrupt exception when the process receives a SIGINT (i.e. when the user hits ^C in the shell). Also, most Python GUI toolkits require their event loops run from the thread that originally imported the module. > BTW, the timer idea is good and eliminates the need to call > asynchronously the GUI thread to update the plot, which seems (the > latter case) not very simple to do. Yep. It's impossible to inject arbitrary code into a Python thread; the thread has to figure out what it's supposed to do by periodically polling something or retrieving some kind of message by blocking on a queue. Blocking on a queue isn't an option for the GUI thread. You might be able to trigger Gtk signals from a separate thread but in my experience tricks like that can be, well, tricky. > Thanks to your suggestion to use the image.set_data method I've > created a simplified script (pasted at the end) that demonstrate how > to do live-update while acquiring (jep!). Glad I could help! :-) > The last think I'm not yet able to do is to update the colorbar to > autoscale with the new incoming data. The the script that follows > tries to update the colorbar too but it does not work (on matplotlib > 0.87 at least). I have no idea if this will help, but you might need to call AxesImage.changed() after calling AxesImage.set_data(). Ken |
From: Antonino I. <tri...@gm...> - 2007-04-01 09:40:16
|
Sorry for the late answer... On 3/29/07, Ken McIvor <mc...@ii...> wrote: > On Mar 28, 2007, at 6:03 PM, Antonino Ingargiola wrote: > > On 3/28/07, Ken McIvor <mc...@ii...> wrote: > >> You should probably do the acquisition asynchronously by running it > >> in a separate thread. > <snip> > > > > That's exactly what I'd like to do. The problem is that if I run > > gtk.main() (the gtk main GUI loop) in a separate thread the program > > stops until I do something on the GUI (for example passing the mouse > > on it). > > Does it do this if you run gtk.main() in the main thread (e.g. the > interpreter prompt) while doing something else in a background thread? Good idea. I will try it next time I'll do something with GUI and thread. > > My understanding so far is the following. When the function > > that execute gtk.main() is executed, python doesn't switch thread > > until either 100 bytecode instructions are executed or since an I/O > > (potentially) blocking operation in executed. > > I think you're more or less correct but are unaware of one important > factor. Here's my understanding of how multithreading works in > Python... > > Thread switching is controlled by something called the Global > Interpreter Lock, which is implemented in an abstract way on top of > an OS lock/mutex object. Only one Python thread runs at a time > because the Python interpreter requires a thread to hold the GIL > before it can execute bytecode instructions. The running thread > holds the GIL while is executes 100 instructions. During this time > other Python threads block waiting to acquire the GIL. After it has > executed its 100 instructions, the running thread releases the GIL > and then attempts to reacquire it. The OS ensures that things are > fair by preventing one thread from reacquiring the GIL over and over > again when other threads are also waiting for it. > > Python doesn't actually detect you do something that will block the > thread, like an I/O operation. Instead, the C code implementing an I/ > O operation like file.write() releases the GIL before performing the > operation. This allows a different thread to acquire it and run some > more bytecode instructions while the first thread performs its I/O > operation. Thanks for the explanation, it's clear now :). > > BTW, the timer idea is good and eliminates the need to call > > asynchronously the GUI thread to update the plot, which seems (the > > latter case) not very simple to do. > > Yep. It's impossible to inject arbitrary code into a Python thread; > the thread has to figure out what it's supposed to do by periodically > polling something or retrieving some kind of message by blocking on a > queue. Blocking on a queue isn't an option for the GUI thread. > > You might be able to trigger Gtk signals from a separate thread but > in my experience tricks like that can be, well, tricky. Good to know that this way is too tricky (at least). > > The last think I'm not yet able to do is to update the colorbar to > > autoscale with the new incoming data. The the script that follows > > tries to update the colorbar too but it does not work (on matplotlib > > 0.87 at least). > > I have no idea if this will help, but you might need to call > AxesImage.changed() after calling AxesImage.set_data(). That doesn't help :(. I'm not jet able to update the colorbar once the image has changed. I will open a new thread to discuss this problem. > Ken ~ Antonio |
From: Antonino I. <tri...@gm...> - 2007-04-01 10:59:09
|
On 4/1/07, Antonino Ingargiola <tri...@gm...> wrote: > On 3/29/07, Ken McIvor <mc...@ii...> wrote: [cut] > > > The last think I'm not yet able to do is to update the colorbar to > > > autoscale with the new incoming data. The the script that follows > > > tries to update the colorbar too but it does not work (on matplotlib > > > 0.87 at least). > > > > I have no idea if this will help, but you might need to call > > AxesImage.changed() after calling AxesImage.set_data(). > > That doesn't help :(. I'm not jet able to update the colorbar once the > image has changed. I've found a way to update the colorbar after the image has changed: image = imshow(data) colr_bar = colorbar() ... image.set_data(new_data) image.changed() color_bar.set_clim(vmax=newdata.max()) draw() The autoscale() colorbar method does not work to update a colorbar, but with the above code I can acheive the same result. Thanks again. ~ Antonio |
From: Eric F. <ef...@ha...> - 2007-04-02 00:26:05
|
Antonino Ingargiola wrote: > On 4/1/07, Antonino Ingargiola <tri...@gm...> wrote: >> On 3/29/07, Ken McIvor <mc...@ii...> wrote: > [cut] >>>> The last think I'm not yet able to do is to update the colorbar to >>>> autoscale with the new incoming data. The the script that follows >>>> tries to update the colorbar too but it does not work (on matplotlib >>>> 0.87 at least). >>> I have no idea if this will help, but you might need to call >>> AxesImage.changed() after calling AxesImage.set_data(). >> That doesn't help :(. I'm not jet able to update the colorbar once the >> image has changed. I have made a change in svn that should solve the problem. Now the following sequence works as expected (illustrated with ipython -pylab): In [1]:IM = imshow(rand(3,4)) In [2]:CB = colorbar() In [3]:IM.set_data(10*rand(3,4)) In [4]:draw() In [5]:IM.autoscale() In [6]:draw() In [7]:IM.set_clim((0,20)) In [8]:draw() The colorbar tracks the image, and either IM.autoscale or IM.set_clim causes the color mapping range to change for both the colorbar and the image. Eric > > I've found a way to update the colorbar after the image has changed: > > image = imshow(data) > colr_bar = colorbar() > > ... > > image.set_data(new_data) > image.changed() > color_bar.set_clim(vmax=newdata.max()) > draw() > > The autoscale() colorbar method does not work to update a colorbar, > but with the above code I can acheive the same result. > > Thanks again. > > ~ Antonio |