flaim-users Mailing List for flaim
Brought to you by:
dsandersorem,
jcalcote
Menu
▾
▴
flaim-users — Discussions related to the use of FLAIM libraries in other projects.
You can subscribe to this list here.
| 2006 |
Jan
|
Feb
(1) |
Mar
(8) |
Apr
(7) |
May
(1) |
Jun
|
Jul
|
Aug
(8) |
Sep
(5) |
Oct
(2) |
Nov
(6) |
Dec
|
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2007 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
(2) |
Sep
|
Oct
|
Nov
|
Dec
|
| 2008 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
(1) |
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
| 2009 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
(2) |
Oct
|
Nov
|
Dec
|
|
From: John C. <joh...@gm...> - 2009-09-30 19:09:18
|
Hi everyone, The migration of FLAIM from Novell's forge site to sourceforge.net is nearly complete. The only remaining issue is the migration of email archives from the Novell forge list server to the sf.net list server. The sf.net staff is managing this task for me via a TRAC task on the sf.net support site. The FLAIM wiki site has been moved to a MediaWiki site on the new SourceForge.net flaim project. Where the old site was located at developer.novell.com, the new site is found at flaim.sourceforge.net. This was a fairly simple move, as both sites were running MediaWiki. Please note that while these wiki pages have been migrated properly, and links fixed up so they're not broken, the site still needs a little work to bring it up to date with current FLAIM development practices. Consider it a work in progress. We may decide not stay with a wiki site. Regardless of the choices we make on format, FLAIM website access will, from now on, be hosted at the following URL. * [http://developer.novell.com/wiki/index.php/FLAIM] ==> [http://flaim.sourceforge.net] All previous flaim source and binary download archives have been moved from forgeftp.novell.com to sourceforge.net. Easy access to these files is provided on the wiki site under the Download heading in the Navigation menu on the right of each of the wiki pages. * [ftp://forgeftp.novell.com/flaim] ==> [http://sourceforge.net/projects/flaim/files] The email lists themselves have been migrated to sf.net with list names identical to the previous Novell forge list names. Existing subscribers have been removed from the forge lists, and added to the sf.net lists. Also note that if you were using the digest option on the older lists, then you're now using it with your new subscriptions, as well. You should have received a subscription notification for the lists to which you've been subscribed. Please see that message in your in-box for details on using the new sf.net FLAIM email lists. * For email lists flaim-announce, flaim-devel, and flaim-users, [flaim-*@forge.novell.com] ==> [flaim-*@lists.sourceforge.net] The entire flaim Subversion repository on forge.novell.com, with all of its commit history, has been migrated to the new sf.net flaim project. * [https://forgesvn1.novell.com/svn/flaim] ==> [https://flaim.svn.sourceforge.net/svnroot/flaim] To reset your work areas, please use the "svn switch" command with the "--relocate" option as follows: $ svn switch --relocate https://forgesvn1.novell.com/svn/flaim/... https://flaim.svn.sourceforge.net/svnroot/flaim/... [path-to-work-area] Replace the ellipses on the ends of the URLs with the remaining path (e.g., "trunk"), or remove them if you've checked out the root of the repository. You can do this in TortoiseSVN by right-clicking on the project work area root directory, selecting TortoiseSVN | Relocate... and then entering the new URL into the dialog edit box (https://flaim.svn.sourceforge.net/svnroot/flaim/... Don't forget to add the proper relative path to the end of the new URL). Note that this works even if your working copy is not up to date. You can use "svn update" from your work area immediately after relocation, as if nothing had changed. Thanks for participating. John |
|
From: John C. <joh...@gm...> - 2009-09-29 11:01:50
|
Hi everyone, In December of this year, Novell's external developer site at http://forge.novell.com and http://developer.novell.com will be terminating open-source development services. The forge site will be shut down and relevant content on the developer site will be migrated to an internally maintained site limited to current Novell interests (NDK and other related projects only). As a result, over the next few days, the FLAIM project will be migrating to sourceforge.net. The migration will include the entire flaim project Subversion repository history and the FLAIM mailing list archives. What this means to you is that your Subversion workspace URL's will have to be modified to work with sourceforge repositories instead, but you will not lose anything historically speaking from the subversion repository. Additionally, the mailing list archives will be migrated from Novell's GNU mailman list server to sourceforge.net's GNU mailman list server. Again, nothing will be lost, as sourceforge.net will import our existing mbox archives. I will migrate existing email addresses from each of the three lists copied on this message to the new respective mailing lists on the sf.net site. (If you wish to opt out of this process, please send me an email message to that effect, and I'll omit your address from this migration.) Please stand by for an announcement on these lists in a few days indicating that the migration has been completed. At that time, I'll provide instructions on how to setup a sourceforge.net account (if you haven't already done so), along with old -> new URL mappings for subversion and mailing lists. Thank you for your interest in the FLAIM project! John |
|
From: John C. <joh...@gm...> - 2008-06-27 14:59:14
|
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi everyone, I'm sorry for the multiple posting here - I realize I should post only to one of the flaim lists, but I wanted to be sure to catch everyone's attention, since we haven't done much with this project lately. The FLAIM project: http://developer.novell.com/wiki/index.php/FLAIM http://forge.novell.com/modules/xfmod/project/?flaim has just been updated with a brand new Autotools build system for Unix and Linux based systems. We've not yet released a tarball containing this new build system, so if you'd like to play with it, you'll have to checkout the FLAIM projects from here: https://forgesvn1.novell.com/svn/flaim/trunk Once this is done, you can execute: $ autoreconf -i $ ./configure && make all check from the top-level (trunk) directory to build in your work area. If you wish to build in a remote build directory, just replace the second line above with: $ mkdir build $ cd build $ ../configure && make all check (or from any other relative path you wish). To build the entire set of features, you'll need to have the following packages installed: - g++ (and company) - mono (for the csharp bindings) - java (for the java bindings) - doxygen (for doxygen docs) The last three are optional, and the system will build without them, minus those features. This new build system will NOT build Windows or NetWare versions of FLAIM. I'm currently working on a Microsoft Visual Studio 2008 build system for Windows builds. If you check out the latest repository, you'll see the beginnings of this build system in various "win32" directories scattered about. Enjoy! John -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iEYEARECAAYFAkhlVUwACgkQdcgqmRY/OH8HxACgkQaIFQmrqFr2FrWmSvYmmtx3 ZvEAmQG1pgrJi0/j6TrhflUOLPJCpPns =1Sz0 -----END PGP SIGNATURE----- |
|
From: Daniel S. <dsa...@no...> - 2007-08-29 17:03:32
|
Groupwise databases run on an older version of flaim. That older version is not open source. >>> "Jim Stahlin" <ji...@st...> 8/29/2007 3:21 PM >>> I am trying to view some groupwise data files using the dbshell.exe program. I get the error "Error opening database: 0xC03F". Any ideas on why this may happen? Is the groupwise database version different from the released FLAIM? |
|
From: Jim S. <ji...@st...> - 2007-08-29 15:21:22
|
I am trying to view some groupwise data files using the dbshell.exe program. I get the error "Error opening database: 0xC03F". Any ideas on why this may happen? Is the groupwise database version different from the released FLAIM? |
|
From: Vince D. K. <vi...@vk...> - 2006-11-13 13:38:30
|
Looks like it was a corrupt or incomplete download. I deleted the file and downloaded again. It extracts properly now. > Vince, > > I was able to extract the files using WinZip version 9.0 SR-1 (6224) as > well as 7z version 4.23. > > Daniel Sanders > > >>>> "Vince D. Kimball" <vi...@vk...> 11/13/2006 8:22 AM >>> > Hi, > > I seem to be unable to extract the files from the tar.gz files for Windows > builds of Flaim. Both Winzip and 7-zip produce errors. Has anyone done > this successfully? Is there a zip archive for Windows somewhere? > > Thanks in advance! > -- Vince > > > _______________________________________________ > Flaim-users mailing list > Fla...@fo... > http://forge.novell.com/mailman/listinfo/flaim-users > _______________________________________________ > Flaim-users mailing list > Fla...@fo... > http://forge.novell.com/mailman/listinfo/flaim-users |
|
From: Vince D. K. <vi...@vk...> - 2006-11-13 13:29:51
|
Hmm, that's weird. I tried WinZip 10 and 7-zip 4.42. I'll try using a different machine tonight at home. > Vince, > > I was able to extract the files using WinZip version 9.0 SR-1 (6224) as > well as 7z version 4.23. > > Daniel Sanders > > >>>> "Vince D. Kimball" <vi...@vk...> 11/13/2006 8:22 AM >>> > Hi, > > I seem to be unable to extract the files from the tar.gz files for Windows > builds of Flaim. Both Winzip and 7-zip produce errors. Has anyone done > this successfully? Is there a zip archive for Windows somewhere? > > Thanks in advance! > -- Vince > > > _______________________________________________ > Flaim-users mailing list > Fla...@fo... > http://forge.novell.com/mailman/listinfo/flaim-users > _______________________________________________ > Flaim-users mailing list > Fla...@fo... > http://forge.novell.com/mailman/listinfo/flaim-users |
|
From: Vince D. K. <vi...@vk...> - 2006-11-13 13:28:13
|
Well, I downloaded the file from the web page using http rather than ftp, so I don't think that's the problem. > Vince, > > Please don't take offense for none is intended, but did you set your ftp > transfer mode to binary (with the bin) command before you downloaded the > file? This will generally cause such errors in archive files on Windows. > Note that Linux ftp clients automatically determine the ascii/binary > status of a download, but the Windows ftp client is not as smart. > > --john > > ----- > John Calcote (jca...@no...) > Sr. Software Engineeer > Novell, Inc. > > >>>> "Vince D. Kimball" <vi...@vk...> 11/13/06 8:22 AM >>> > Hi, > > I seem to be unable to extract the files from the tar.gz files for Windows > builds of Flaim. Both Winzip and 7-zip produce errors. Has anyone done > this successfully? Is there a zip archive for Windows somewhere? > > Thanks in advance! > -- Vince > > > _______________________________________________ > Flaim-users mailing list > Fla...@fo... > http://forge.novell.com/mailman/listinfo/flaim-users > |
|
From: John C. <jca...@no...> - 2006-11-13 12:25:36
|
Vince, Please don't take offense for none is intended, but did you set your ftp transfer mode to binary (with the bin) command before you downloaded the file? This will generally cause such errors in archive files on Windows. Note that Linux ftp clients automatically determine the ascii/binary status of a download, but the Windows ftp client is not as smart. --john ----- John Calcote (jca...@no...) Sr. Software Engineeer Novell, Inc. >>> "Vince D. Kimball" <vi...@vk...> 11/13/06 8:22 AM >>> Hi, I seem to be unable to extract the files from the tar.gz files for Windows builds of Flaim. Both Winzip and 7-zip produce errors. Has anyone done this successfully? Is there a zip archive for Windows somewhere? Thanks in advance! -- Vince _______________________________________________ Flaim-users mailing list Fla...@fo... http://forge.novell.com/mailman/listinfo/flaim-users |
|
From: Daniel S. <dsa...@no...> - 2006-11-13 12:23:12
|
Vince, I was able to extract the files using WinZip version 9.0 SR-1 (6224) as well as 7z version 4.23. Daniel Sanders >>> "Vince D. Kimball" <vi...@vk...> 11/13/2006 8:22 AM >>> Hi, I seem to be unable to extract the files from the tar.gz files for Windows builds of Flaim. Both Winzip and 7-zip produce errors. Has anyone done this successfully? Is there a zip archive for Windows somewhere? Thanks in advance! -- Vince _______________________________________________ Flaim-users mailing list Fla...@fo... http://forge.novell.com/mailman/listinfo/flaim-users |
|
From: Vince D. K. <vi...@vk...> - 2006-11-13 08:22:32
|
Hi, I seem to be unable to extract the files from the tar.gz files for Windows builds of Flaim. Both Winzip and 7-zip produce errors. Has anyone done this successfully? Is there a zip archive for Windows somewhere? Thanks in advance! -- Vince |
|
From: Daniel S. <dsa...@no...> - 2006-10-18 13:30:36
|
1. There is no SQL abstraction layer for FLAIM. JDBC is not supported. -- All data definition and data manipulation is done via the FLAIM APIs only. The FLAIM data model is designed to support arbitrarily structured records. -- It is conceivable that someone could build an SQL abstraction layer on top of this, including an ODBC or JDBC implementation. 2. We are actually in the process of evaluating whether to move FLAIM to an LGPL model. A decision will probably be forthcoming in the next few weeks. Daniel Sanders >>> "Michael Bell" <mb...@so...> 10/18/2006 1:08 PM >>> 1. Is FLAIM supported for JDBC? Is there a SQL abstraction layer? Does such a layer apply to DDL as well, or is all of that via DC and the standard FLAIM creation stuff. 2. Who would be appropriate to talk to regarding licensing terms (LGPL works for us, but GPL does not)? Thanks, mike _______________________________________________ Flaim-users mailing list Fla...@fo... http://forge.novell.com/mailman/listinfo/flaim-users |
|
From: Michael B. <mb...@so...> - 2006-10-18 13:08:53
|
1. Is FLAIM supported for JDBC? Is there a SQL abstraction layer? Does such a layer apply to DDL as well, or is all of that via DC and the standard FLAIM creation stuff. 2. Who would be appropriate to talk to regarding licensing terms (LGPL works for us, but GPL does not)? Thanks, mike |
|
From: Andrew H. <AHo...@no...> - 2006-09-20 11:24:28
|
All, FLAIM and XFLAIM i386 packages for Debian-based distros are now available on our development downloads page: http://www.bandit-project.org/index.php/FLAIM_Development_Download We have also added a "Performance and Reliability" sections to the FLAIM FAQ page: http://www.bandit-project.org/index.php/FLAIM_FAQ#Performance_and_Reliability Thanks. |
|
From: Andrew H. <AHo...@no...> - 2006-09-18 17:08:32
|
All, We are starting to build Debian binary packages (.deb) which should be compatible with Ubuntu and other Debian-based distros. The FTK x86 package can be downloaded at: ftp://forgeftp.novell.com//flaim/development/ftk/downloads/binaries/linux-x86-32/libftk_1.1.855-i386.deb FLAIM and XFLAIM x86 packages will be available in the next couple of days. We also plan on providing "Packages" files in the various download directories so that it will be possible to add our URLs to your /etc/apt/sources.list to allow simple updates via apt-get. If you want to build the packages yourself, we have added the "debbin" and "debsrc" targets to the makefile. Any feedback is appreciated. Thanks. |
|
From: Dale O. <do...@no...> - 2006-09-15 01:07:49
|
Andy, All I can say is: Woo Hoo! That is so cool. Not only the test, and the solution, and the performance and scalability improvement, but that you are a true craftsman that cares about his work. Woo hoo! --Dale On Fri, 2006-09-08 at 11:27 -0600, Andrew Hodgkinson wrote: > All, > > I was recently able to acquire some used Fibre Channel equipment, > including a cool Brocade switch with a front-panel display that can be > configured to report I/O throughput. Being excited to test the > switch, Fibre card and disk array, I decided to run a FLAIM bulk-load > via the gigatest utility. I expected to see the I/O channel maxed out > at 100 MBytes/s when forcing a checkpoint and flushing hundreds of > megabytes of dirty cache. Unfortunately, the I/O channel was hardly > being used ... in fact the highest throughput number reported by the > switch during a checkpoint was 10 MBytes/s! Given that this was my > first experience setting up a Fibre SAN, I figured that I had done > something wrong. So, to validate the SAN configuration, I conducted a > simple experiment by copying an ISO disk image from the server's > internal U320 SCSI drive to the SAN (configured with eight 18GB U160 > drives). Wow, what a difference! The switch reported a s ustained > throughput rate of about 91 MBytes/s for the duration of the copy. > This, in turn, told me two things: 1) The SAN was configured correctly > and 2) FLAIM wasn't keeping the I/O channel busy. > > This was perplexing because FLAIM tries to be ultra efficient in the > way it interacts with the disk. This includes using async and direct > I/O when available, ordering writes to minimize seeking, using > sector-aligned buffers, and various other techniques. As it turns > out, although FLAIM (and XFLAIM) use async I/O and multiple write > buffers, the code wasn't taking advantage of the fact that > out-of-order I/O completion can result in later writes completing > before earlier writes (seems somewhat obvious, right?). > Basically, there are a limited number of I/O buffers managed by FLAIM. > As FLAIM flushes dirty cache to disk, it acquires a buffer from the > buffer manager, holds onto it until the write completes, and then > releases the buffer back to the manager. When all of the write > buffers are in use, FLAIM must wait for a pending I/O to complete > before queuing an additional I/O operation. The problem is that at > any one time during a checkpoint, we may have thousands of pending > writes and it is unknown which one will complete first. FLAIM was > taking the simplistic approach of waiting for the earliest queued > write to complete; this, however, was rarely the first I/O to > complete. The result was that there were many usable I/O buffers > available, but FLAIM was unaware of this fact because it was blocked > waiting for the earliest I/O to complete and had not acknowledged the > completion of the other writes. > > Platforms that support async I/O all provide a simple way of > determining if an async I/O has completed via a call to a routine like > GetOverlappedResult (on Windows). Windows and NetWare (and possibly > Linux, Solaris, AIX, OS X, etc.) provide an additional callback-based > mechanism to notify an application that I/O has completed. Its > possible to use this callback-based notification to leverage > out-of-order I/O completion to FLAIM's advantage. Instead of waiting > for a specific I/O operation to complete so that its buffer can be > re-used, the callback can do all of the work needed to release the > buffer back to the buffer manager and also alert (via a semaphore) any > thread waiting for a buffer to become available. This results in > efficient and timely buffer re-use and has a huge impact on I/O > throughput (more details below). > > I made some code changes in the Windows-specific FTK code to use an > I/O completion callback and also added a "buffer waiter" queue, fired > up a new bulk load, and waited for the first checkpoint. Being used > to checkpoints that run for a long time (anywhere from 30 - 90 > seconds) when flushing a large amount of dirty cache, I was somewhat > disappointed to swing around to look at the Fibre switch and find that > it was reporting throughput of only 10 MBytes/s. Argggh! When I > looked at the stats on the bulk load screen, I soon realized that the > I/O stats I was seeing on the switch were attributed to RFL writes, > NOT the checkpoint ... it had already completed and the foreground > bulk load was continuing! At the start of the next checkpoint, I made > sure that I was watching the switch from the start and saw that the > throughput was 93 MBytes/s. This is an improvement of 9.3 times. > > To further validate the changes, I started an unreasonably large bulk > load and let it run overnight. This morning, to my surprise, the bulk > load (via gigatest) had completed, resulting in a database with > 2,000,000,000 (yes, TWO BILLION) objects and a total load time of just > under 6 hours (that's 5.5 million objects a minute). In terms of the > number of objects, this is the largest FLAIM database ever created > (although I would be thrilled if someone contradicted this claim). Of > course, I proudly announced these results to my wife at breakfast in a > feeble attempt to justify my recent eBay expenditures made in > acquiring all of this equipment. I'm not sure if she was impressed. > > In any case, these code changes have been checked in to the > open-source SVN repository and will be included in the > soon-to-be-released 4.9 version of FLAIM and the 5.1 version of > XFLAIM. Currently, Windows will be the primary beneficiary of these > improvements. I am still investigating similar improvements on the > other supported platforms and will send out any new information as it > becomes available. > > Thanks. > > > > _______________________________________________ > Flaim-devel mailing list > Fla...@fo... > http://forge.novell.com/mailman/listinfo/flaim-devel |
|
From: Andrew H. <AHo...@no...> - 2006-09-12 11:35:11
|
New stable versions of FLAIM, XFLAIM and FTK have been released. Builds have been posted to our download site at http://www.bandit-project.org/index.php/FLAIM_Download. FLAIM 4.9 (changes since version 4.8): Highlights: * Added support for storing 64 bit numbers * Re-architected the FTK I/O layer and cleaned up the async interfaces. * Added support for I/O completion callbacks. * Added code to pre-extend the database file(s) when forcing a checkpoint. * Implemented use of F_BlockAlloc for block cache allocations to significantly reduce the overhead associated with reading blocks from disk. Other improvements: * Fixes to take advantage of slab manager optimizations. * Changed to use new defines for colors in sprintf format string. * Fixed issues when using GNU make 3.78. * Added code to round up to the next 512 byte boundary when writing the log header. * Added option in FLAIM to disable direct I/O on Linux and Unix platforms. * Changed how super file handle is set up - for direct io mode. * Fixed issue where we weren't reading enough from the base 64 stream when encoding a key for storage in the database. * RFL was truncating the log smaller than the minimum file extend size, resulting in a lot of unneeded overhead. * Fixed memory leak and crash in checkdb utility. * Fixed misaligned I/O operations. * Updated libtool versioning information and added documentation describing how and when libtool versioning should be changed. * Enhancements and improvements to the field ID table inside of records. * Added support for running a database sweep within an update transaction. * Enhancements to reduce the overhead of growing and shrinking roll-forward and roll-back log files. * Added gigatest utility. * Updated support for AIX. * Initial support for direct and async I/O on AIX. * Added support for HP-UX built with the native compiler (aCC). * Added ha_flaim files needed to use FLAIM as a MySQL storage engine. * Updated RFL checksum code to call f_calcPacketChecksum. * Update to use more of the toolkit collation routines. * Fixed the makefile to enable optimizations when building non-debug binaries. * Added support for the "sparcgeneric" target. This will build a library without any v8plus or newer instructions. XFLAIM 5.1 (changes since version 5.0): Highlights: * Finished implementing the Java interfaces * Added javadoc documentation of the Java interfaces * Updates to take advantage of slab manager optimizations. * Added support for I/O completion callbacks. Other improvements: * Modified to use toolkit f_logXXX functions for doing formatted logging. * Changed to use java2-devel-packages instead of java2. * Background threads are now disabled in the destination database when rebuilding. * Changes to use FTK notify lists. * Updated libtool versioning information and added documentation describing how and when libtool versioning should be changed. * Rebuild code changed to handle block checksum errors correctly. * Updated support for AIX. * Updated RFL checksum code to call f_calcPacketChecksum. * Fixed the makefile to enable optimizations when building non-debug binaries. * Added support for building xedit java utility FTK 1.1 (changes since version 1.0): Highlights: * Added support for I/O completion callbacks. * Re-architected the FTK I/O layer and cleaned up the async interfaces. * Added support for f_yieldCPU() on Unix platforms. * Added support for auto-extending files when writing to minimize filesystem metadata logging. * Added support for notify lists and reader/writer locks. * Added a sector-aligned block allocator * Optimized file extend operations on Windows, Linux, Solaris, and HP-UX * Added native atomic routines for Linux on SPARC processors. * Added ftkFastCheckSumMMX for 32 and 64-bit Intel Linux. * Added F_Printf class. Other improvements: * Changed strings for embedding colors in printf() format strings. * Moved some typedefs out of the private area of the F_FixedAlloc class. Some compilers can't deal with that. * Modified ftk.h to include stdarg.h to get va_xxx functionality - on all platforms except for Netware ring 0. * Added default new operators to the F_Object class - ones without file and line number. * Updated FTK makefile to detect the Sun Studio compiler. * Added support for allocating aligned buffers on Unix platforms that don't provide the memalign() function. * Option added to disable file system caching on OS X. * Slab manager improvements to reduce mutex contention. * File I/O optimizations for Unix and Windows. * Added flag to prevent misaligned I/O operations to occur when a file has been opened in directio mode. * Changed error code mapping of EINVAL from NE_FLM_IO_PATH_TOO_LONG to NE_FLM_INVALID_PARM * Updated libtool versioning information and added documentation describing how and when libtool versioning should be changed. * Allocation of slabs will be done using valloc on various Unix platforms. * According to the documentation posix_* routines do not set errno. Needed to make a couple of code changes to account for this. * Updated support for AIX. * Initial support for direct and async I/O on AIX. * Added support for HP-UX built with the native compiler (aCC). * Added atomic increment, decrement and exchange unit tests. * Needed to use atomic "barrier" functions on OS X to ensure correct behavior. * Added native Linux atomic ops on PPC. * Added f_getRandomByte() function. * Improvements to collation routines, including an optimization that results in 10x the performance when converting US English characters for collation. * Added more collation unit tests. * Fixed bug in f_wpBreakChar that was resulting in an illegal memory access when processing character 0x01F2. * Fixed bug in f_combineWPChar that was resulting in an illegal memory access when processing characters beyond 0x0F00. * Added unit test to compare native vs mutex-based atomic ops. * Improved performance of PPC atomic ops by removing the leading sync instruction. * Added support for the "genericsparc" target. This will build a library without any v8plus or newer instructions. * Added GCC optimization flags when doing a non-debug OS X build. |
|
From: Andrew H. <AHo...@no...> - 2006-09-08 11:27:26
|
All, I was recently able to acquire some used Fibre Channel equipment, including a cool Brocade switch with a front-panel display that can be configured to report I/O throughput. Being excited to test the switch, Fibre card and disk array, I decided to run a FLAIM bulk-load via the gigatest utility. I expected to see the I/O channel maxed out at 100 MBytes/s when forcing a checkpoint and flushing hundreds of megabytes of dirty cache. Unfortunately, the I/O channel was hardly being used ... in fact the highest throughput number reported by the switch during a checkpoint was 10 MBytes/s! Given that this was my first experience setting up a Fibre SAN, I figured that I had done something wrong. So, to validate the SAN configuration, I conducted a simple experiment by copying an ISO disk image from the server's internal U320 SCSI drive to the SAN (configured with eight 18GB U160 drives). Wow, what a difference! The switch reported a sustained throughput rate of about 91 MBytes/s for the duration of the copy. This, in turn, told me two things: 1) The SAN was configured correctly and 2) FLAIM wasn't keeping the I/O channel busy. This was perplexing because FLAIM tries to be ultra efficient in the way it interacts with the disk. This includes using async and direct I/O when available, ordering writes to minimize seeking, using sector-aligned buffers, and various other techniques. As it turns out, although FLAIM (and XFLAIM) use async I/O and multiple write buffers, the code wasn't taking advantage of the fact that out-of-order I/O completion can result in later writes completing before earlier writes (seems somewhat obvious, right?). Basically, there are a limited number of I/O buffers managed by FLAIM. As FLAIM flushes dirty cache to disk, it acquires a buffer from the buffer manager, holds onto it until the write completes, and then releases the buffer back to the manager. When all of the write buffers are in use, FLAIM must wait for a pending I/O to complete before queuing an additional I/O operation. The problem is that at any one time during a checkpoint, we may have thousands of pending writes and it is unknown which one will complete first. FLAIM was taking the simplistic approach of waiting for the earliest queued write to complete; this, however, was rarely the first I/O to complete. The result was that there were many usable I/O buffers available, but FLAIM was unaware of this fact because it was blocked waiting for the earliest I/O to complete and had not acknowledged the completion of the other writes. Platforms that support async I/O all provide a simple way of determining if an async I/O has completed via a call to a routine like GetOverlappedResult (on Windows). Windows and NetWare (and possibly Linux, Solaris, AIX, OS X, etc.) provide an additional callback-based mechanism to notify an application that I/O has completed. Its possible to use this callback-based notification to leverage out-of-order I/O completion to FLAIM's advantage. Instead of waiting for a specific I/O operation to complete so that its buffer can be re-used, the callback can do all of the work needed to release the buffer back to the buffer manager and also alert (via a semaphore) any thread waiting for a buffer to become available. This results in efficient and timely buffer re-use and has a huge impact on I/O throughput (more details below). I made some code changes in the Windows-specific FTK code to use an I/O completion callback and also added a "buffer waiter" queue, fired up a new bulk load, and waited for the first checkpoint. Being used to checkpoints that run for a long time (anywhere from 30 - 90 seconds) when flushing a large amount of dirty cache, I was somewhat disappointed to swing around to look at the Fibre switch and find that it was reporting throughput of only 10 MBytes/s. Argggh! When I looked at the stats on the bulk load screen, I soon realized that the I/O stats I was seeing on the switch were attributed to RFL writes, NOT the checkpoint ... it had already completed and the foreground bulk load was continuing! At the start of the next checkpoint, I made sure that I was watching the switch from the start and saw that the throughput was 93 MBytes/s. This is an improvement of 9.3 times. To further validate the changes, I started an unreasonably large bulk load and let it run overnight. This morning, to my surprise, the bulk load (via gigatest) had completed, resulting in a database with 2,000,000,000 (yes, TWO BILLION) objects and a total load time of just under 6 hours (that's 5.5 million objects a minute). In terms of the number of objects, this is the largest FLAIM database ever created (although I would be thrilled if someone contradicted this claim). Of course, I proudly announced these results to my wife at breakfast in a feeble attempt to justify my recent eBay expenditures made in acquiring all of this equipment. I'm not sure if she was impressed. In any case, these code changes have been checked in to the open-source SVN repository and will be included in the soon-to-be-released 4.9 version of FLAIM and the 5.1 version of XFLAIM. Currently, Windows will be the primary beneficiary of these improvements. I am still investigating similar improvements on the other supported platforms and will send out any new information as it becomes available. Thanks. |
|
From: Andrew H. <AHo...@no...> - 2006-08-30 10:38:22
|
All, I've added information about the FLAIM project to Wikipedia. If anyone would like to add to the entry or make improvements, please feel free to contribute. The entry can be found here: http://en.wikipedia.org/wiki/FLAIM_Database_Engine. Thanks. |
|
From: Daniel S. <dsa...@no...> - 2006-08-17 12:14:15
|
Jeff, I don't know why the Perl bindings cannot be managed via CPAN - sounds like a good idea to me. Only things that get checked back into our source tree would need copyright assignment back to Novell. Regarding Java - should not be required to build xflaim. The Makefile looks for JAVA_HOME in the environment. If it is missing, it should skip all of the Java build stuff. Perhaps we have a bug in the Makefile. The documentation links from the FLAIM wiki page (http://developer.novell.com/wiki/index.php/Flaim) are now correct, but the links from the main project page are still incorrect. Unfortunately, I cannot change those directly myself - we are working with the IT people to get them changed or eliminated. I think that IT meant to phase them out in favor of the wiki site, but they aren't completely done with it yet. You are reading correctly that flaim concurrency is entirely by way of threads within a single process. Only one process can work on the database at a time. This tends to be very good for applications that are servers (like Novell's eDirectory). A server is a good solution for providing "multi-process" concurrent access - but nobody has ever written just a plain old "flaim server". By embedding FLAIM in a server process, you get all of the features that will make a multi-threaded server process scale well - particularly caching. But caching is also what makes it difficult for multiple processes to simultaneously have direct access to the database. If each process has its own cache, those caches would have to be kept in sync. If there were a shared cache between processes, there are other complications to deal with. Suffice it to say, our current position is that if you need multi-process concurrent access, you will need to implement some kind of server that other processes talk to via protocol. - Daniel Sanders >>> "Jeffrey W. Baker" <jw...@ac...> 8/17/2006 1:16 AM >>> On Wed, 2006-08-16 at 13:23 -0600, Daniel Sanders wrote: > Jeffrey, > > Thank you for your interest in the FLAIM project. Sorry about the > difficulty in finding it. We agree that its relative obscurity is a > problem. We are going to look at some things to get it more > visibility in the search engines. > > We would welcome your participation in the project. Nobody is > currently working on Perl bindings, so that is a great place to > contribute. C++ makes this project a bit larger than it would be if flaim were implemented in C, but it can probably be done within our lifetimes :) > We are planning on creating Debian packages - just haven't got to it > yet. Any pointers or help you would like to give us on that would > also be welcome. xflaim builds and tests on all of ubuntu 6.06 x86_64, ubuntu 6.06 ppc32, debian 3.1 x86_64, and debian 3.1 x86, so I don't forsee a big problem here. Note that I'm not one of those Debian developer in-crowd type of people. But I think I can get the packages built. Ubuntu requires fewer secret handshakes than Debian, so that will probably be the first people to accept the packages. For Ubuntu Universe repository there are hardly any requirements at all. On the build side, there are a few hitches. I didn't get any shlibs out of a 'make test' and 'make libs' fails by way of not finding Java. Java is not exactly Free in that tricky DFSG sense (and there's no real Java solution for ppc32 Linux), so I will likely try to strip out the java build target temporarily. > As with many open source projects, Novell's policy for projects it > sponsors is to have contributors assign copyright back to Novell. I > hope that isn't a problem. Of course, all contributions will be made > available under the GPL (FLAIM's current licence). Can the Perl bindings be managed through CPAN? That would be the canonical Perl way, and it will spread adoption. Every platform that has Perl also has CPAN, whereas many platforms do not have organized package libraries like the one Debian provides. As for the Debian packaging, you would probably want to keep that separate also. Because there is a certain impedance mismatch between the flaim build process and the usual gnu autotools build, I think you'll find it's best to keep that upstream (that's you) blissfully unaware of the Debian sausage making. If I make any actual changes on the flaim tree, I'll be pleased to assign those copyrights back to Novell. > If you need help using the FLAIM APIs, please let us know. We would > be glad to assist you as our time allows. The links to the documentation from Novell Forge lead to non-existent pages, but the links from the Bandit project work. Maybe fix the links on Forge, or maybe consolidate under the Bandit wiki? [helpful descriptions of flaim and xflaim elided] Just to confirm that I'm reading this correctly, is flaim concurrency entirely by way of threads within a single process? The way I read it, only one process can work on the database at a time. -Jeff Baker |
|
From: Jeffrey W. B. <jw...@ac...> - 2006-08-17 01:16:11
|
On Wed, 2006-08-16 at 13:23 -0600, Daniel Sanders wrote: > Jeffrey, > > Thank you for your interest in the FLAIM project. Sorry about the > difficulty in finding it. We agree that its relative obscurity is a > problem. We are going to look at some things to get it more > visibility in the search engines. > > We would welcome your participation in the project. Nobody is > currently working on Perl bindings, so that is a great place to > contribute. C++ makes this project a bit larger than it would be if flaim were implemented in C, but it can probably be done within our lifetimes :) > We are planning on creating Debian packages - just haven't got to it > yet. Any pointers or help you would like to give us on that would > also be welcome. xflaim builds and tests on all of ubuntu 6.06 x86_64, ubuntu 6.06 ppc32, debian 3.1 x86_64, and debian 3.1 x86, so I don't forsee a big problem here. Note that I'm not one of those Debian developer in-crowd type of people. But I think I can get the packages built. Ubuntu requires fewer secret handshakes than Debian, so that will probably be the first people to accept the packages. For Ubuntu Universe repository there are hardly any requirements at all. On the build side, there are a few hitches. I didn't get any shlibs out of a 'make test' and 'make libs' fails by way of not finding Java. Java is not exactly Free in that tricky DFSG sense (and there's no real Java solution for ppc32 Linux), so I will likely try to strip out the java build target temporarily. > As with many open source projects, Novell's policy for projects it > sponsors is to have contributors assign copyright back to Novell. I > hope that isn't a problem. Of course, all contributions will be made > available under the GPL (FLAIM's current licence). Can the Perl bindings be managed through CPAN? That would be the canonical Perl way, and it will spread adoption. Every platform that has Perl also has CPAN, whereas many platforms do not have organized package libraries like the one Debian provides. As for the Debian packaging, you would probably want to keep that separate also. Because there is a certain impedance mismatch between the flaim build process and the usual gnu autotools build, I think you'll find it's best to keep that upstream (that's you) blissfully unaware of the Debian sausage making. If I make any actual changes on the flaim tree, I'll be pleased to assign those copyrights back to Novell. > If you need help using the FLAIM APIs, please let us know. We would > be glad to assist you as our time allows. The links to the documentation from Novell Forge lead to non-existent pages, but the links from the Bandit project work. Maybe fix the links on Forge, or maybe consolidate under the Bandit wiki? [helpful descriptions of flaim and xflaim elided] Just to confirm that I'm reading this correctly, is flaim concurrency entirely by way of threads within a single process? The way I read it, only one process can work on the database at a time. -Jeff Baker |
|
From: Daniel S. <dsa...@no...> - 2006-08-16 13:23:36
|
Jeffrey, Thank you for your interest in the FLAIM project. Sorry about the difficulty in finding it. We agree that its relative obscurity is a problem. We are going to look at some things to get it more visibility in the search engines. We would welcome your participation in the project. Nobody is currently working on Perl bindings, so that is a great place to contribute. We are planning on creating Debian packages - just haven't got to it yet. Any pointers or help you would like to give us on that would also be welcome. As with many open source projects, Novell's policy for projects it sponsors is to have contributors assign copyright back to Novell. I hope that isn't a problem. Of course, all contributions will be made available under the GPL (FLAIM's current licence). There are two maintainers to the FLAIM project - Andrew Hodgkinson (aho...@no...) and myself (dsa...@no...). You can e-mail contributions to us, and we will review them and then check them into the project. If you need help using the FLAIM APIs, please let us know. We would be glad to assist you as our time allows. XFLAIM is not really a superset of FLAIM. Both FLAIM and XFLAIM rely on some common core technologies (there is a FLAIM Common Toolkit), and many other aspects are similar - transactions, caching, roll-forward logging, etc. However, the data models are different in some important ways, and the APIs are quite different. Furthermore, XFLAIM code does not know how to access a FLAIM database or vice versa. XFLAIM's data model is XML - with various extensions for data types and other issues associated with storing and retrieving XML in a database. It stores XML documents, but nodes within a document are individually addressable, updateable, and indexable. You access the database using a DOM-like interface. There is also an XPATH-like interface for doing queries. Although we support importing from external XML text documents, a programmer really only needs to use our DOM interface to create, browse, modify, delete, and access their XML documents. The way I describe it is that it is a "virtualized, persistent DOM". Once you open a database for access, it is as if all of the nodes in every document in the database is immediately accessible via DOM. Underneath the covers the nodes are brought into memory on-demand. No XML parser is required to create the DOM nodes from external text files - the DOM nodes are just there, ready to use - so you just start making DOM calls. Of course, we have a method that parses and imports external XML text documents, but it is not required to get documents into the database or to access documents that are already in the database. FLAIM's data model is records that are comprised of fields which may be hierarchically nested. Every field in a record is tagged with its field identifier, so a record can be arbitrarily complex. In fact, every record in the database may have a different structure if desired. The fact that FLAIM allows for arbitrary tagged-nested structure within a record makes it very similar to XML, but it lacks some of the other features of XML - such as element attributes. Indexing in FLAIM is based on fields or field paths within a record, but index keys only point to the record that contains the indexed fields, not to the individual fields themselves - in XFLAIM index keys point directly to the individual nodes that were used to construct the keys. Update granularity in FLAIM is individual records - only entire records may be added, modified or deleted. If you want to modify a single field within a record, you must read the entire record into memory, update the desired field, and then write the entire record back to the database. In XFLAIM, you can retrieve and update individual nodes directly - no need to retrieve the entire document. We have found that FLAIM is good for some things while XFLAIM is good for others - it just depends on what you want. FLAIM performs better on some things than XFLAIM and vice versa - again, it depends on the application's usage. XFLAIM tends to consume more disk space than FLAIM (because each node is individually addressable). I could go on about various differences and pros and cons of FLAIM versus XFLAIM. Each has strengths and weaknesses. I could probably give you more specific advice if you had some questions you wanted to ask. I hope this information helps. Good luck with what you are doing, and let us know if we can be of any assistance. Thanks, Daniel Sanders Daniel Sanders Software Engineer dsa...@no... 801-861-4193 Daniel Sanders Software Engineer dsa...@no... 801-861-4193 >>> "Jeffrey W. Baker" <jw...@ac...> 8/16/2006 11:10 AM >>> Howdy, Let me first say that I wanted to compose this email weeks ago, but found it impossible to find this project using Google. If you don't know the name FLAIM, you'll never find it with terms like novell open source database, novell berkeleydb competitor, novell xml database, or frankly any other search. And the search engine on forge.novell.com never returns. The relative obscurity of the project may be an impediment to potential users. It certainly was to me. If I hadn't read the press release on LWN, I would never have even heard of the project. Having noted that, my hope is to use FLAIM in an existing application where BerkeleyDB has proven itself to be an unending series of operational pitfalls. The application is written in Perl, so first I'll need Perl bindings. Will I be duplicating any work there? Second problem is lack of Debian packages. There do not seem to be any impediments technical nor legal to producing some. Pointers to existing work? Last question: is XFLAIM a strict superset of FLAIM 4, and therefore can a new developer safely ignore FLAIM in favor of XFLAIM? Regards, Jeffrey Baker _______________________________________________ Flaim-users mailing list Fla...@fo... http://forge.novell.com/mailman/listinfo/flaim-users |
|
From: Jeffrey W. B. <jw...@ac...> - 2006-08-16 11:10:36
|
Howdy, Let me first say that I wanted to compose this email weeks ago, but found it impossible to find this project using Google. If you don't know the name FLAIM, you'll never find it with terms like novell open source database, novell berkeleydb competitor, novell xml database, or frankly any other search. And the search engine on forge.novell.com never returns. The relative obscurity of the project may be an impediment to potential users. It certainly was to me. If I hadn't read the press release on LWN, I would never have even heard of the project. Having noted that, my hope is to use FLAIM in an existing application where BerkeleyDB has proven itself to be an unending series of operational pitfalls. The application is written in Perl, so first I'll need Perl bindings. Will I be duplicating any work there? Second problem is lack of Debian packages. There do not seem to be any impediments technical nor legal to producing some. Pointers to existing work? Last question: is XFLAIM a strict superset of FLAIM 4, and therefore can a new developer safely ignore FLAIM in favor of XFLAIM? Regards, Jeffrey Baker |
|
From: Andrew H. <AHo...@no...> - 2006-08-15 11:34:38
|
A new build of FLAIM has been posted to the Bandit site: http://www.bandit-project.org/index.php/FLAIM_Development_Download. This version of the FLAIM takes advantage of the improvements recently introduced in the cross-platform toolkit (FTK). Specifically, the following improvements were made: * Enhancement: New checksum routines were written in SPARC assembly for Linux and Solaris . This results in 5 - 8 times faster block and RFL packet checksum calculations. * Enhancement: New checksum routines for GCC-built x86 targets (Linux) were added. This results in up to 40 times faster block and RFL packet checksums by leveraging the parallel add and xor MMX instructions. * Enhancement: Native atomic operations for PowerPC Linux were added. * Enhancement: Added support for super fast file extend operations on platforms and file systems that support fast file extends (such as Win32+NTFS, HP-UX+Veritas, and Solaris+UFS) * Enhancement: Added a new block cache allocator to guarantee that block buffers are aligned on page and/or sector boundaries. This allows disk drivers to read directly into the block cache buffers when using direct I/O, thus eliminating a large memcpy on each read operation. The new allocator also reduces cache overhead from 7.5% to 1.5% (resulting in more items being cached). * Enhancement: FLAIM is now built as a v8plus binary on SPARC platforms. This allows fast atomic operations to be used. * Enhancement: Reduced the overhead of growing and shrinking roll-forward and roll-back log files. * Enhancement: Changed most I/O operations to use aligned buffers. * Enhancement: Added support for running a database sweep within an update transaction. * Bug Fix: The Ubuntu/SPARC version of FLAIM now builds correctly * Bug Fix: AIX platform support has been updated, including limited support for direct and async I/O * Bug Fix: HP-UX (PA-RISC) platform support has been updated * Bug Fix: The RFL was being truncated smaller than the minimum file extend size. This was resulting in a lot of overhead because we would subsequently extend the RFL file when writing. * Utilities: Fixed a memory leak and crash in checkdb utility. * Utilities: Added the gigatest utility for performance testing. * Build: libtool versioning information was updated and documentation describing how and when libtool versioning should be changed was added. Also changed libtool library version to be 3.0.0. * Documentation: Query retrieval functions were grouped incorrectly for doxygen. This has been fixed. * Documentation: Event functions were grouped incorrectly for doxygen. This has been fixed. Thanks. |
|
From: Andrew H. <AHo...@no...> - 2006-08-14 13:26:34
|
A new build of the FLAIM cross-platform toolkit (FTK) has been posted to the Bandit site: http://www.bandit-project.org/index.php/FLAIM_Development_Download. This version of the toolkit contains a number of optimizations that will noticeably improve the performance and resource utilization of FLAIM and XFLAIM. Specifically, this release includes the following: * New checksum routines written in SPARC assembly for Linux and Solaris . This results in 5 - 8 times faster block and RFL packet checksum calculations. * New checksum routines for GCC-built x86 targets (Linux). This results in up to 40 times faster block and RFL packet checksums by leveraging the parallel add and xor MMX instructions. * Added native atomic operations for PowerPC Linux. * Added support for super fast file extend operations on platforms and file systems that support fast file extends (such as Win32+NTFS, HP-UX+Veritas, and Solaris+UFS) * Added a new block cache allocator to guarantee that block buffers are aligned on page and/or sector boundaries. This allows disk drivers to read directly into the block cache buffers when using direct I/O, thus eliminating a large memcpy on each read operation. The new allocator also reduces cache overhead from 7.5% to 1.5% (resulting in more items being cached). Other changes include: * Support for notify lists and reader/writer locks * FTK is now built as a v8plus binary on SPARC platforms * The Ubuntu/SPARC version of FTK now builds correctly * AIX platform support has been updated, including limited support for direct and async I/O * HP-UX (PA-RISC) platform support has been updated New FLAIM and XFLAIM builds based on this version of FTK will be available soon (hopefully today). Thanks. |
