Re: [Snap-photo-devel] faster scanning

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

Ok, if you can merge it, go ahead and commit. If you have problems,
you can either e-mail me, or just overwrite my changes and I can sort
things out.

FYI, now that I'm employed (started today), my output may drop for a
couple of weeks. It should pick back up once I acquire a laptop so I
can work on Snap (among other things) on BART.

>If necessary, we can rewrite
> PictureCatalog/LivePictureCatalog to use some kind of DB rather than a
> List<>.

Okay. I guess, ideally, we'd have an "IPictureCatalog" interface where
we could swap out those components without much fuss, but let's see
how this thing works all pulled together as is...

*crosses fingers*

-Maciek

> On 7/30/06, Maciej Sakrejda <m.s...@gm...> wrote:
> > Wow. No DB at all... Well, are you letting the IWhateverFilter
> > elements do all the legwork then? I'm reluctant to make such big
> > changes without a solid reconsideration of the whole design... I'm
> > working on the Trie tests now, and should be able to commit my stuff
> > later today. There's not *that* much changes to the stuff you changed,
> > but it still might be tricky...
> >
> > I guess my main concern about dumping the DB is what avenues of
> > expansion it closes off to us--right now it seems it'd be pretty
> > trivial to expand the DB stuff to do very complex queries ((this label
> > and this label) or (this label and not that label) or earlier than
> > this date) if we figure out how to expose this to the GUI. I guess the
> > real questions are
> >
> > 1. How useful would this really be in the first place?
> > 2. How difficult would this be to replicate without the DB, if we *do*
> > want to do it?
> >
> > A four-fold speed-up is nothing to sneeze at, but I'd hate to think it
> > comes at the cost of totally crippled functionality...
> >
> > -Maciek
> >
> > On 7/30/06, Jimmy Do <cri...@gm...> wrote:
> > > Nope, not using the DB for queries. We weren't using very complex DB
> > > queries anyway because we just kept asking the DB for all pictures.
> > > Consequently, I don't think we really need QueryLanguage. But then
> > > again, maybe you guys want to keep queries and QueryLanguage, so
> > > perhaps I shouldn't merge my changes. But if you want the changes, I
> > > guess I should merge into main first and then you merge your changes,
> > > just to make sure I don't overwrite your changes.
> > >
> > > Jimmy
> > >
> > > On 7/30/06, Maciej Sakrejda <m.s...@gm...> wrote:
> > > > But we're still using the DB for queries, right? Just not for the
> > > > cache? I guess I should take a look at your changes. I'll try to do
> > > > that today, maybe. Also, should I commit my changes, and have you
> > > > merge the ScanFaster branch with that, or do you want to merge with
> > > > the current trunk and then I can merge my changes with the new trunk?
> > > >
> > > >
> > > > On 7/29/06, Jimmy Do <cri...@gm...> wrote:
> > > > > Hey guys,
> > > > >
> > > > > So I think I've got the ScanFaster branch working pretty well. Please
> > > > > check it out and see if you can break it. I'll merge it into the main
> > > > > trunk if you guys approve.
> > > > >
> > > > > Major Changes:
> > > > >  * Faster scanning by storing all IPictureInfos directly into a
> > > > > List<>. Previously, the database inserts during scans were slowing
> > > > > things down a lot (about 4 times slower with database inserts). We now
> > > > > have a separate thread that periodically syncs the current list of
> > > > > IPictureInfos into a file by serializing it. SQLite is no longer used
> > > > > for storing the cache because syncing the pictures to an SQLite DB
> > > > > took about 20 *seconds* for only 60 pictures. With serialization, it
> > > > > took less than 1 second to sync almost 1000 pictures.
> > > > > Overall, for around 1900 pictures, scanning went from about 8 minutes
> > > > > to 3 minutes.
> > > > >
> > > > >  * New design also results in quicker discovery of missing files upon
> > > > > startup. For example, close the app, remove all pictures, and start
> > > > > the app again. Before, if you had a lot of pictures, they would slowly
> > > > > disappear from the UI. Now, all removed pictures will disappear right
> > > > > away.
> > > > >
> > > > >  * Preliminary UI for showing scan progress.
> > > > >
> > > > >
> > > > > Jimmy
> > > > >
> > > > > -------------------------------------------------------------------------
> > > > > Take Surveys. Earn Cash. Influence the Future of IT
> > > > > Join SourceForge.net's Techsay panel and you'll get the chance to share your
> > > > > opinions on IT & business topics through brief surveys -- and earn cash
> > > > > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> > > > > _______________________________________________
> > > > > Snap-photo-devel mailing list
> > > > > Sna...@li...
> > > > > https://lists.sourceforge.net/lists/listinfo/snap-photo-devel
> > > > >
> > > >
> > > > -------------------------------------------------------------------------
> > > > Take Surveys. Earn Cash. Influence the Future of IT
> > > > Join SourceForge.net's Techsay panel and you'll get the chance to share your
> > > > opinions on IT & business topics through brief surveys -- and earn cash
> > > > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> > > > _______________________________________________
> > > > Snap-photo-devel mailing list
> > > > Sna...@li...
> > > > https://lists.sourceforge.net/lists/listinfo/snap-photo-devel
> > > >
> > >
> > > -------------------------------------------------------------------------
> > > Take Surveys. Earn Cash. Influence the Future of IT
> > > Join SourceForge.net's Techsay panel and you'll get the chance to share your
> > > opinions on IT & business topics through brief surveys -- and earn cash
> > > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> > > _______________________________________________
> > > Snap-photo-devel mailing list
> > > Sna...@li...
> > > https://lists.sourceforge.net/lists/listinfo/snap-photo-devel
> > >
> >
> > -------------------------------------------------------------------------
> > Take Surveys. Earn Cash. Influence the Future of IT
> > Join SourceForge.net's Techsay panel and you'll get the chance to share your
> > opinions on IT & business topics through brief surveys -- and earn cash
> > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> > _______________________________________________
> > Snap-photo-devel mailing list
> > Sna...@li...
> > https://lists.sourceforge.net/lists/listinfo/snap-photo-devel
> >
>