Menu
▾
▴
Re: [Snap-photo-devel] faster scanning
|
From: Jimmy D. <cri...@gm...> - 2006-08-02 08:33:21
|
Quick note: I just discovered that a recent fix I committed for correctly sorting IPictureInfos in MainWindow leads to a slow down in scanning (although it's still faster than the old implementation). I'm working on an alternative implementation that will not slow things down as much. Jimmy On 8/1/06, Jimmy Do <cri...@gm...> wrote: > On 8/1/06, Maciej Sakrejda <m.s...@gm...> wrote: > > > 2. SearchTree needs add/remove/change events, but the re-design > > > doesn't emit any. I'll try to add these events back in somewhere. > > > > Ah. Well, are we still doing FS monitoring on watched folders at all, > > then? > > Yup, we're still using FS monitoring, except the code for handling the > add/deleted/changed notifications has been unified with the code for > scanning. > > > > This seems like a feature we want to keep. I don't know how much > > this would play into the redesign, but we had talked about abstracting > > away raw FS events into actually relevant SnapFSEvents or something > > like that--to, among other things, resolve the exiv2 file update > > (copy/delete/rename) nonsense. If this is something that you think > > would dovetail nicely with your current redesign, I can jump on it now > > (I'm still feeling massively exhausted after each day of work, but > > I'll get used to it soon). > > I don't remember the details of SnapFSEvents, so I can't really say > how easy it would be to implement with this new design. I did, > however, just add PicAdded, PicRemoved, and PicChanged events (but no > code actually uses those events right now) to PictureCatalog, so > PictureCatalog might be a good place to look for implementing > SnapFSEvents. > > In that same class, I've also added a private instance of SearchTree > so that we can update the search tree as new files are scanned, added, > removed, or updated. To take advantage of this SearchTree instance, > PictureCatalog now has a Search() method for performing searches. > Great job with the SearchTree, Maciek. I just gave it a try and it > appears to be blazingly fast. I've gotten search to work now, but you > might want to take a look at everything to make sure I didn't do > anything crazy during my merge. > > Jimmy > > > Otheriwse, if that part can be just as > > easily done independently, I'll get on it in a week or two. > > > > -Maciek > > > > > > > > Hmm...actually, I think I'm going to commit with the non-functional > > > search, and then try to fix it within the next couple days. My hard > > > drive seems to be having some issues, so I'd feel safer knowing that > > > all changes are in SVN. > > > > > > Jimmy > > > > > > > > > > > > > > > On 7/30/06, Maciej Sakrejda <m.s...@gm...> wrote: > > > > > > Wow. No DB at all... Well, are you letting the IWhateverFilter > > > > > > elements do all the legwork then? I'm reluctant to make such big > > > > > > changes without a solid reconsideration of the whole design... I'm > > > > > > working on the Trie tests now, and should be able to commit my stuff > > > > > > later today. There's not *that* much changes to the stuff you changed, > > > > > > but it still might be tricky... > > > > > > > > > > > > I guess my main concern about dumping the DB is what avenues of > > > > > > expansion it closes off to us--right now it seems it'd be pretty > > > > > > trivial to expand the DB stuff to do very complex queries ((this label > > > > > > and this label) or (this label and not that label) or earlier than > > > > > > this date) if we figure out how to expose this to the GUI. I guess the > > > > > > real questions are > > > > > > > > > > > > 1. How useful would this really be in the first place? > > > > > > 2. How difficult would this be to replicate without the DB, if we *do* > > > > > > want to do it? > > > > > > > > > > > > A four-fold speed-up is nothing to sneeze at, but I'd hate to think it > > > > > > comes at the cost of totally crippled functionality... > > > > > > > > > > > > -Maciek > > > > > > > > > > > > On 7/30/06, Jimmy Do <cri...@gm...> wrote: > > > > > > > Nope, not using the DB for queries. We weren't using very complex DB > > > > > > > queries anyway because we just kept asking the DB for all pictures. > > > > > > > Consequently, I don't think we really need QueryLanguage. But then > > > > > > > again, maybe you guys want to keep queries and QueryLanguage, so > > > > > > > perhaps I shouldn't merge my changes. But if you want the changes, I > > > > > > > guess I should merge into main first and then you merge your changes, > > > > > > > just to make sure I don't overwrite your changes. > > > > > > > > > > > > > > Jimmy > > > > > > > > > > > > > > On 7/30/06, Maciej Sakrejda <m.s...@gm...> wrote: > > > > > > > > But we're still using the DB for queries, right? Just not for the > > > > > > > > cache? I guess I should take a look at your changes. I'll try to do > > > > > > > > that today, maybe. Also, should I commit my changes, and have you > > > > > > > > merge the ScanFaster branch with that, or do you want to merge with > > > > > > > > the current trunk and then I can merge my changes with the new trunk? > > > > > > > > > > > > > > > > > > > > > > > > On 7/29/06, Jimmy Do <cri...@gm...> wrote: > > > > > > > > > Hey guys, > > > > > > > > > > > > > > > > > > So I think I've got the ScanFaster branch working pretty well. Please > > > > > > > > > check it out and see if you can break it. I'll merge it into the main > > > > > > > > > trunk if you guys approve. > > > > > > > > > > > > > > > > > > Major Changes: > > > > > > > > > * Faster scanning by storing all IPictureInfos directly into a > > > > > > > > > List<>. Previously, the database inserts during scans were slowing > > > > > > > > > things down a lot (about 4 times slower with database inserts). We now > > > > > > > > > have a separate thread that periodically syncs the current list of > > > > > > > > > IPictureInfos into a file by serializing it. SQLite is no longer used > > > > > > > > > for storing the cache because syncing the pictures to an SQLite DB > > > > > > > > > took about 20 *seconds* for only 60 pictures. With serialization, it > > > > > > > > > took less than 1 second to sync almost 1000 pictures. > > > > > > > > > Overall, for around 1900 pictures, scanning went from about 8 minutes > > > > > > > > > to 3 minutes. > > > > > > > > > > > > > > > > > > * New design also results in quicker discovery of missing files upon > > > > > > > > > startup. For example, close the app, remove all pictures, and start > > > > > > > > > the app again. Before, if you had a lot of pictures, they would slowly > > > > > > > > > disappear from the UI. Now, all removed pictures will disappear right > > > > > > > > > away. > > > > > > > > > > > > > > > > > > * Preliminary UI for showing scan progress. > > > > > > > > > > > > > > > > > > > > > > > > > > > Jimmy > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------------- > > > > > > > > > Take Surveys. Earn Cash. Influence the Future of IT > > > > > > > > > Join SourceForge.net's Techsay panel and you'll get the chance to share your > > > > > > > > > opinions on IT & business topics through brief surveys -- and earn cash > > > > > > > > > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > > > > > > > > > _______________________________________________ > > > > > > > > > Snap-photo-devel mailing list > > > > > > > > > Sna...@li... > > > > > > > > > https://lists.sourceforge.net/lists/listinfo/snap-photo-devel > > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------------- > > > > > > > > Take Surveys. Earn Cash. Influence the Future of IT > > > > > > > > Join SourceForge.net's Techsay panel and you'll get the chance to share your > > > > > > > > opinions on IT & business topics through brief surveys -- and earn cash > > > > > > > > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > > > > > > > > _______________________________________________ > > > > > > > > Snap-photo-devel mailing list > > > > > > > > Sna...@li... > > > > > > > > https://lists.sourceforge.net/lists/listinfo/snap-photo-devel > > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------------- > > > > > > > Take Surveys. Earn Cash. Influence the Future of IT > > > > > > > Join SourceForge.net's Techsay panel and you'll get the chance to share your > > > > > > > opinions on IT & business topics through brief surveys -- and earn cash > > > > > > > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > > > > > > > _______________________________________________ > > > > > > > Snap-photo-devel mailing list > > > > > > > Sna...@li... > > > > > > > https://lists.sourceforge.net/lists/listinfo/snap-photo-devel > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------------- > > > > > > Take Surveys. Earn Cash. Influence the Future of IT > > > > > > Join SourceForge.net's Techsay panel and you'll get the chance to share your > > > > > > opinions on IT & business topics through brief surveys -- and earn cash > > > > > > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > > > > > > _______________________________________________ > > > > > > Snap-photo-devel mailing list > > > > > > Sna...@li... > > > > > > https://lists.sourceforge.net/lists/listinfo/snap-photo-devel > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------------- > > > > Take Surveys. Earn Cash. Influence the Future of IT > > > > Join SourceForge.net's Techsay panel and you'll get the chance to share your > > > > opinions on IT & business topics through brief surveys -- and earn cash > > > > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > > > > _______________________________________________ > > > > Snap-photo-devel mailing list > > > > Sna...@li... > > > > https://lists.sourceforge.net/lists/listinfo/snap-photo-devel > > > > > > > > > > ------------------------------------------------------------------------- > > > Take Surveys. Earn Cash. Influence the Future of IT > > > Join SourceForge.net's Techsay panel and you'll get the chance to share your > > > opinions on IT & business topics through brief surveys -- and earn cash > > > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > > > _______________________________________________ > > > Snap-photo-devel mailing list > > > Sna...@li... > > > https://lists.sourceforge.net/lists/listinfo/snap-photo-devel > > > > > > > ------------------------------------------------------------------------- > > Take Surveys. Earn Cash. Influence the Future of IT > > Join SourceForge.net's Techsay panel and you'll get the chance to share your > > opinions on IT & business topics through brief surveys -- and earn cash > > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > > _______________________________________________ > > Snap-photo-devel mailing list > > Sna...@li... > > https://lists.sourceforge.net/lists/listinfo/snap-photo-devel > > > |