Re: [micro-manager-general] Saving images at high speed

[PAVAK S. <pavak@g.ucla.edu>]

Thanks guys!

Is this metadata management handled in the low level acquisition code or at
the MDA / diSPIM plugin level?

Pavak

On Wed, Apr 8, 2020, 2:47 PM JonD <jo...@as...> wrote:

> Nico Stuurman-2 wrote
> >> This is great news! I'm hoping to take advantage of this once the
> >> diSPIM plugin in 2.0g is a bit further along.
> >
> > Good point, and you may want to bring this up with Jon Daniels.  I
> > ported a then current version of the diSPIM plugin to 2.0 a while ago,
> > but developments on both sides (diSPIM 1.4 plugin and MM 2.0) probably
> > make that port useless, so this will need to be done again.  I'll be
> > happy to help, but kind of waiting for Jon to take the lead on that.
>
> Yes it's on our to-do list but not at the top.  Once 2.0 is "officialized"
> there will be more of a push.  The good news is that my new-ish colleague
> Brandon likely will be the main person working on this, and he's a better
> programmer than I am ;-)
>
>
>
> Nico Stuurman-2 wrote
> >
> > <snip>
> >> Initially the buffer is kept empty, but over time performance blips
> >> (maybe background system interrupts causing hiccups?) result in the
> >> accumulation of 100 or so frames in the buffer. These blips are
> >> quickly cleared out, but after a few hundred volumes are acquired the
> >> acquisition is not able to keep the buffer emptying fast enough and
> >> frames start steadily accumulating in the sequence buffer. If we use a
> >> large buffer (say 32 GB), we can finish the acquisition but it takes
> >> tens of minutes for the buffer to then empty out. This seems to be
> >> behavior tied to FPS rather than the average data throughput, as
> >> increasing the per frame size by 4x as many pixels and decreasing the
> >> effective FPS by the same ratio resolves this behavior. In terms of
> >> system details, this is a new diSPIM running 2x Hamamatsu Fusions on
> >> the 3/9 nightly build of 1.4. The workstation itself is fairly
> >> powerful with 2x 8-core xeon's running at 3.4 GHz.
> >
> > I am afraid that the behavior on 2.0 will be similar.  The blips you are
> > seeing are most likely cause by the Java garbage collector, which has a
> > lot of work to do, because lots of objects are created and destroyed.
> > This is in large part due to the image metadata.  It will be a useful
> > new project to assert that image metadata processing and saving is
> > indeed a bottleneck for the type of datasets you describe, and study how
> > to reduce that bottleneck.  My first inclination to discard per image
> > metadata after the first image will not work for your situation, since
> > you will at least need the metadata tag indicating the camera the image
> > came from. I bought the O'Reailly book on "Java performance", so that is
> > a start;)
>
> I'm not very familiar with the inner workings of metadata, but entertain a
> brainstorm of an "armchair quarterback": maybe there could be a few
> different categories of metadata so that some metadata (e.g. properties of
> all the devices) don't need to be attached to every single image, but other
> metadata tags do get attached.  Basically any metadata that isn't changed
> only needs to be written once.  If you want to go crazy there could be
> metadata that lives at different levels, one set for the entire
> acquisition,
> another set for the timepoint, another set for the position, and so on down
> to a set of tags that is present (and generally unique) for each individual
> image.
>
> Jon
>
>
>
> --
> Sent from: http://micro-manager.3463995.n2.nabble.com/
>
>
> _______________________________________________
> micro-manager-general mailing list
> mic...@li...
> https://lists.sourceforge.net/lists/listinfo/micro-manager-general
>