Re: [micro-manager-general] Saving images at high speed

[JonD <jo...@as...>]

Nico Stuurman-2 wrote
>> This is great news! I'm hoping to take advantage of this once the 
>> diSPIM plugin in 2.0g is a bit further along.
> 
> Good point, and you may want to bring this up with Jon Daniels.  I 
> ported a then current version of the diSPIM plugin to 2.0 a while ago, 
> but developments on both sides (diSPIM 1.4 plugin and MM 2.0) probably 
> make that port useless, so this will need to be done again.  I'll be 
> happy to help, but kind of waiting for Jon to take the lead on that.

Yes it's on our to-do list but not at the top.  Once 2.0 is "officialized"
there will be more of a push.  The good news is that my new-ish colleague
Brandon likely will be the main person working on this, and he's a better
programmer than I am ;-)



Nico Stuurman-2 wrote
> 
> <snip>
>> Initially the buffer is kept empty, but over time performance blips 
>> (maybe background system interrupts causing hiccups?) result in the 
>> accumulation of 100 or so frames in the buffer. These blips are 
>> quickly cleared out, but after a few hundred volumes are acquired the 
>> acquisition is not able to keep the buffer emptying fast enough and 
>> frames start steadily accumulating in the sequence buffer. If we use a 
>> large buffer (say 32 GB), we can finish the acquisition but it takes 
>> tens of minutes for the buffer to then empty out. This seems to be 
>> behavior tied to FPS rather than the average data throughput, as 
>> increasing the per frame size by 4x as many pixels and decreasing the 
>> effective FPS by the same ratio resolves this behavior. In terms of 
>> system details, this is a new diSPIM running 2x Hamamatsu Fusions on 
>> the 3/9 nightly build of 1.4. The workstation itself is fairly 
>> powerful with 2x 8-core xeon's running at 3.4 GHz.
> 
> I am afraid that the behavior on 2.0 will be similar.  The blips you are 
> seeing are most likely cause by the Java garbage collector, which has a 
> lot of work to do, because lots of objects are created and destroyed.  
> This is in large part due to the image metadata.  It will be a useful 
> new project to assert that image metadata processing and saving is 
> indeed a bottleneck for the type of datasets you describe, and study how 
> to reduce that bottleneck.  My first inclination to discard per image 
> metadata after the first image will not work for your situation, since 
> you will at least need the metadata tag indicating the camera the image 
> came from. I bought the O'Reailly book on "Java performance", so that is 
> a start;)

I'm not very familiar with the inner workings of metadata, but entertain a
brainstorm of an "armchair quarterback": maybe there could be a few
different categories of metadata so that some metadata (e.g. properties of
all the devices) don't need to be attached to every single image, but other
metadata tags do get attached.  Basically any metadata that isn't changed
only needs to be written once.  If you want to go crazy there could be
metadata that lives at different levels, one set for the entire acquisition,
another set for the timepoint, another set for the position, and so on down
to a set of tags that is present (and generally unique) for each individual
image.

Jon



--
Sent from: http://micro-manager.3463995.n2.nabble.com/