Re: [Snap-photo-devel] faster scanning

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 454-5900

Yeah, the IWhateverFilters have always done all the work of properly
sorting and grouping the pictures. I think the fanciest SQL query we
made was to grab pictures in chronological order.

I agree that we should carefully consider getting rid of the DB, but
it doesn't seem too hard to eventually swap in a DB in the future.
Right now, most of the system just fetches a list of IPictureInfos
from LivePictureCatalog. If necessary, we can rewrite
PictureCatalog/LivePictureCatalog to use some kind of DB rather than a
List<>.

I'll check out your changes and see if I can merge my ScanFaster
branch. Let me know if you want me to commit.

Jimmy

On 7/30/06, Maciej Sakrejda <m.s...@gm...> wrote:
> Wow. No DB at all... Well, are you letting the IWhateverFilter
> elements do all the legwork then? I'm reluctant to make such big
> changes without a solid reconsideration of the whole design... I'm
> working on the Trie tests now, and should be able to commit my stuff
> later today. There's not *that* much changes to the stuff you changed,
> but it still might be tricky...
>
> I guess my main concern about dumping the DB is what avenues of
> expansion it closes off to us--right now it seems it'd be pretty
> trivial to expand the DB stuff to do very complex queries ((this label
> and this label) or (this label and not that label) or earlier than
> this date) if we figure out how to expose this to the GUI. I guess the
> real questions are
>
> 1. How useful would this really be in the first place?
> 2. How difficult would this be to replicate without the DB, if we *do*
> want to do it?
>
> A four-fold speed-up is nothing to sneeze at, but I'd hate to think it
> comes at the cost of totally crippled functionality...
>
> -Maciek
>
> On 7/30/06, Jimmy Do <cri...@gm...> wrote:
> > Nope, not using the DB for queries. We weren't using very complex DB
> > queries anyway because we just kept asking the DB for all pictures.
> > Consequently, I don't think we really need QueryLanguage. But then
> > again, maybe you guys want to keep queries and QueryLanguage, so
> > perhaps I shouldn't merge my changes. But if you want the changes, I
> > guess I should merge into main first and then you merge your changes,
> > just to make sure I don't overwrite your changes.
> >
> > Jimmy
> >
> > On 7/30/06, Maciej Sakrejda <m.s...@gm...> wrote:
> > > But we're still using the DB for queries, right? Just not for the
> > > cache? I guess I should take a look at your changes. I'll try to do
> > > that today, maybe. Also, should I commit my changes, and have you
> > > merge the ScanFaster branch with that, or do you want to merge with
> > > the current trunk and then I can merge my changes with the new trunk?
> > >
> > >
> > > On 7/29/06, Jimmy Do <cri...@gm...> wrote:
> > > > Hey guys,
> > > >
> > > > So I think I've got the ScanFaster branch working pretty well. Please
> > > > check it out and see if you can break it. I'll merge it into the main
> > > > trunk if you guys approve.
> > > >
> > > > Major Changes:
> > > >  * Faster scanning by storing all IPictureInfos directly into a
> > > > List<>. Previously, the database inserts during scans were slowing
> > > > things down a lot (about 4 times slower with database inserts). We now
> > > > have a separate thread that periodically syncs the current list of
> > > > IPictureInfos into a file by serializing it. SQLite is no longer used
> > > > for storing the cache because syncing the pictures to an SQLite DB
> > > > took about 20 *seconds* for only 60 pictures. With serialization, it
> > > > took less than 1 second to sync almost 1000 pictures.
> > > > Overall, for around 1900 pictures, scanning went from about 8 minutes
> > > > to 3 minutes.
> > > >
> > > >  * New design also results in quicker discovery of missing files upon
> > > > startup. For example, close the app, remove all pictures, and start
> > > > the app again. Before, if you had a lot of pictures, they would slowly
> > > > disappear from the UI. Now, all removed pictures will disappear right
> > > > away.
> > > >
> > > >  * Preliminary UI for showing scan progress.
> > > >
> > > >
> > > > Jimmy
> > > >
> > > > -------------------------------------------------------------------------
> > > > Take Surveys. Earn Cash. Influence the Future of IT
> > > > Join SourceForge.net's Techsay panel and you'll get the chance to share your
> > > > opinions on IT & business topics through brief surveys -- and earn cash
> > > > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> > > > _______________________________________________
> > > > Snap-photo-devel mailing list
> > > > Sna...@li...
> > > > https://lists.sourceforge.net/lists/listinfo/snap-photo-devel
> > > >
> > >
> > > -------------------------------------------------------------------------
> > > Take Surveys. Earn Cash. Influence the Future of IT
> > > Join SourceForge.net's Techsay panel and you'll get the chance to share your
> > > opinions on IT & business topics through brief surveys -- and earn cash
> > > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> > > _______________________________________________
> > > Snap-photo-devel mailing list
> > > Sna...@li...
> > > https://lists.sourceforge.net/lists/listinfo/snap-photo-devel
> > >
> >
> > -------------------------------------------------------------------------
> > Take Surveys. Earn Cash. Influence the Future of IT
> > Join SourceForge.net's Techsay panel and you'll get the chance to share your
> > opinions on IT & business topics through brief surveys -- and earn cash
> > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> > _______________________________________________
> > Snap-photo-devel mailing list
> > Sna...@li...
> > https://lists.sourceforge.net/lists/listinfo/snap-photo-devel
> >
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys -- and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Snap-photo-devel mailing list
> Sna...@li...
> https://lists.sourceforge.net/lists/listinfo/snap-photo-devel
>