Menu
▾
▴
cweb-cvs — CVS commit messages are posted to this list.
You can subscribe to this list here.
| 2006 |
Jan
|
Feb
|
Mar
(414) |
Apr
(123) |
May
(448) |
Jun
(180) |
Jul
(17) |
Aug
(49) |
Sep
(3) |
Oct
(92) |
Nov
(101) |
Dec
(64) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2007 |
Jan
(132) |
Feb
(230) |
Mar
(146) |
Apr
(146) |
May
|
Jun
|
Jul
(34) |
Aug
(4) |
Sep
(3) |
Oct
(10) |
Nov
(12) |
Dec
(24) |
| 2008 |
Jan
(6) |
Feb
|
Mar
|
Apr
|
May
(1) |
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(11) |
Nov
(4) |
Dec
|
| 2009 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
(1) |
Oct
|
Nov
|
Dec
|
Update of /cvsroot/cweb/bigdata/src/test/com/bigdata/btree In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv13221/src/test/com/bigdata/btree Modified Files: TestIndexSegmentBuilderWithSmallTree.java MyHardReferenceQueue.java MyEvictionListener.java TestBTree.java TestReopen.java TestAll.java Log Message: Updates to make the application of the cweb license uniform in preparation for license change to GPL. Index: TestReopen.java =================================================================== RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/btree/TestReopen.java,v retrieving revision 1.1 retrieving revision 1.2 diff -C2 -d -r1.1 -r1.2 *** TestReopen.java 13 Apr 2007 15:04:17 -0000 1.1 --- TestReopen.java 16 Apr 2007 10:02:49 -0000 1.2 *************** *** 1,49 **** /** ! The Notice below must appear in each file of the Source Code of any ! copy you distribute of the Licensed Product. Contributors to any ! Modifications may add their own copyright notices to identify their ! own contributions. ! ! License: ! The contents of this file are subject to the CognitiveWeb Open Source ! License Version 1.1 (the License). You may not copy or use this file, ! in either source code or executable form, except in compliance with ! the License. You may obtain a copy of the License from ! http://www.CognitiveWeb.org/legal/license/ ! Software distributed under the License is distributed on an AS IS ! basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See ! the License for the specific language governing rights and limitations ! under the License. ! Copyrights: ! Portions created by or assigned to CognitiveWeb are Copyright ! (c) 2003-2003 CognitiveWeb. All Rights Reserved. Contact ! information for CognitiveWeb is available at ! http://www.CognitiveWeb.org ! Portions Copyright (c) 2002-2003 Bryan Thompson. ! Acknowledgements: ! Special thanks to the developers of the Jabber Open Source License 1.0 ! (JOSL), from which this License was derived. This License contains ! terms that differ from JOSL. ! Special thanks to the CognitiveWeb Open Source Contributors for their ! suggestions and support of the Cognitive Web. ! Modifications: ! */ ! /* ! * Created on Nov 17, 2006 ! */ package com.bigdata.btree; --- 1,45 ---- /** ! The Notice below must appear in each file of the Source Code of any ! copy you distribute of the Licensed Product. Contributors to any ! Modifications may add their own copyright notices to identify their ! own contributions. ! License: ! The contents of this file are subject to the CognitiveWeb Open Source ! License Version 1.1 (the License). You may not copy or use this file, ! in either source code or executable form, except in compliance with ! the License. You may obtain a copy of the License from ! http://www.CognitiveWeb.org/legal/license/ ! Software distributed under the License is distributed on an AS IS ! basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See ! the License for the specific language governing rights and limitations ! under the License. ! Copyrights: ! Portions created by or assigned to CognitiveWeb are Copyright ! (c) 2003-2003 CognitiveWeb. All Rights Reserved. Contact ! information for CognitiveWeb is available at ! http://www.CognitiveWeb.org ! Portions Copyright (c) 2002-2003 Bryan Thompson. ! Acknowledgements: ! Special thanks to the developers of the Jabber Open Source License 1.0 ! (JOSL), from which this License was derived. This License contains ! terms that differ from JOSL. ! Special thanks to the CognitiveWeb Open Source Contributors for their ! suggestions and support of the Cognitive Web. ! Modifications: + */ package com.bigdata.btree; Index: TestBTree.java =================================================================== RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/btree/TestBTree.java,v retrieving revision 1.3 retrieving revision 1.4 diff -C2 -d -r1.3 -r1.4 *** TestBTree.java 13 Apr 2007 15:04:19 -0000 1.3 --- TestBTree.java 16 Apr 2007 10:02:49 -0000 1.4 *************** *** 1,49 **** /** ! The Notice below must appear in each file of the Source Code of any ! copy you distribute of the Licensed Product. Contributors to any ! Modifications may add their own copyright notices to identify their ! own contributions. ! ! License: ! The contents of this file are subject to the CognitiveWeb Open Source ! License Version 1.1 (the License). You may not copy or use this file, ! in either source code or executable form, except in compliance with ! the License. You may obtain a copy of the License from ! http://www.CognitiveWeb.org/legal/license/ ! Software distributed under the License is distributed on an AS IS ! basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See ! the License for the specific language governing rights and limitations ! under the License. ! Copyrights: ! Portions created by or assigned to CognitiveWeb are Copyright ! (c) 2003-2003 CognitiveWeb. All Rights Reserved. Contact ! information for CognitiveWeb is available at ! http://www.CognitiveWeb.org ! Portions Copyright (c) 2002-2003 Bryan Thompson. ! Acknowledgements: ! Special thanks to the developers of the Jabber Open Source License 1.0 ! (JOSL), from which this License was derived. This License contains ! terms that differ from JOSL. ! Special thanks to the CognitiveWeb Open Source Contributors for their ! suggestions and support of the Cognitive Web. ! Modifications: ! */ ! /* ! * Created on Nov 8, 2006 ! */ package com.bigdata.btree; --- 1,45 ---- /** ! The Notice below must appear in each file of the Source Code of any ! copy you distribute of the Licensed Product. Contributors to any ! Modifications may add their own copyright notices to identify their ! own contributions. ! License: ! The contents of this file are subject to the CognitiveWeb Open Source ! License Version 1.1 (the License). You may not copy or use this file, ! in either source code or executable form, except in compliance with ! the License. You may obtain a copy of the License from ! http://www.CognitiveWeb.org/legal/license/ ! Software distributed under the License is distributed on an AS IS ! basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See ! the License for the specific language governing rights and limitations ! under the License. ! Copyrights: ! Portions created by or assigned to CognitiveWeb are Copyright ! (c) 2003-2003 CognitiveWeb. All Rights Reserved. Contact ! information for CognitiveWeb is available at ! http://www.CognitiveWeb.org ! Portions Copyright (c) 2002-2003 Bryan Thompson. ! Acknowledgements: ! Special thanks to the developers of the Jabber Open Source License 1.0 ! (JOSL), from which this License was derived. This License contains ! terms that differ from JOSL. ! Special thanks to the CognitiveWeb Open Source Contributors for their ! suggestions and support of the Cognitive Web. ! Modifications: + */ package com.bigdata.btree; Index: TestIndexSegmentBuilderWithSmallTree.java =================================================================== RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/btree/TestIndexSegmentBuilderWithSmallTree.java,v retrieving revision 1.1 retrieving revision 1.2 diff -C2 -d -r1.1 -r1.2 *** TestIndexSegmentBuilderWithSmallTree.java 13 Apr 2007 15:04:17 -0000 1.1 --- TestIndexSegmentBuilderWithSmallTree.java 16 Apr 2007 10:02:49 -0000 1.2 *************** *** 1,2 **** --- 1,45 ---- + /** + + The Notice below must appear in each file of the Source Code of any + copy you distribute of the Licensed Product. Contributors to any + Modifications may add their own copyright notices to identify their + own contributions. + + License: + + The contents of this file are subject to the CognitiveWeb Open Source + License Version 1.1 (the License). You may not copy or use this file, + in either source code or executable form, except in compliance with + the License. You may obtain a copy of the License from + + http://www.CognitiveWeb.org/legal/license/ + + Software distributed under the License is distributed on an AS IS + basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See + the License for the specific language governing rights and limitations + under the License. + + Copyrights: + + Portions created by or assigned to CognitiveWeb are Copyright + (c) 2003-2003 CognitiveWeb. All Rights Reserved. Contact + information for CognitiveWeb is available at + + http://www.CognitiveWeb.org + + Portions Copyright (c) 2002-2003 Bryan Thompson. + + Acknowledgements: + + Special thanks to the developers of the Jabber Open Source License 1.0 + (JOSL), from which this License was derived. This License contains + terms that differ from JOSL. + + Special thanks to the CognitiveWeb Open Source Contributors for their + suggestions and support of the Cognitive Web. + + Modifications: + + */ package com.bigdata.btree; Index: MyHardReferenceQueue.java =================================================================== RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/btree/MyHardReferenceQueue.java,v retrieving revision 1.1 retrieving revision 1.2 diff -C2 -d -r1.1 -r1.2 *** MyHardReferenceQueue.java 13 Apr 2007 15:04:18 -0000 1.1 --- MyHardReferenceQueue.java 16 Apr 2007 10:02:49 -0000 1.2 *************** *** 1,2 **** --- 1,45 ---- + /** + + The Notice below must appear in each file of the Source Code of any + copy you distribute of the Licensed Product. Contributors to any + Modifications may add their own copyright notices to identify their + own contributions. + + License: + + The contents of this file are subject to the CognitiveWeb Open Source + License Version 1.1 (the License). You may not copy or use this file, + in either source code or executable form, except in compliance with + the License. You may obtain a copy of the License from + + http://www.CognitiveWeb.org/legal/license/ + + Software distributed under the License is distributed on an AS IS + basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See + the License for the specific language governing rights and limitations + under the License. + + Copyrights: + + Portions created by or assigned to CognitiveWeb are Copyright + (c) 2003-2003 CognitiveWeb. All Rights Reserved. Contact + information for CognitiveWeb is available at + + http://www.CognitiveWeb.org + + Portions Copyright (c) 2002-2003 Bryan Thompson. + + Acknowledgements: + + Special thanks to the developers of the Jabber Open Source License 1.0 + (JOSL), from which this License was derived. This License contains + terms that differ from JOSL. + + Special thanks to the CognitiveWeb Open Source Contributors for their + suggestions and support of the Cognitive Web. + + Modifications: + + */ package com.bigdata.btree; Index: TestAll.java =================================================================== RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/btree/TestAll.java,v retrieving revision 1.3 retrieving revision 1.4 diff -C2 -d -r1.3 -r1.4 *** TestAll.java 13 Apr 2007 15:04:18 -0000 1.3 --- TestAll.java 16 Apr 2007 10:02:49 -0000 1.4 *************** *** 1,2 **** --- 1,45 ---- + /** + + The Notice below must appear in each file of the Source Code of any + copy you distribute of the Licensed Product. Contributors to any + Modifications may add their own copyright notices to identify their + own contributions. + + License: + + The contents of this file are subject to the CognitiveWeb Open Source + License Version 1.1 (the License). You may not copy or use this file, + in either source code or executable form, except in compliance with + the License. You may obtain a copy of the License from + + http://www.CognitiveWeb.org/legal/license/ + + Software distributed under the License is distributed on an AS IS + basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See + the License for the specific language governing rights and limitations + under the License. + + Copyrights: + + Portions created by or assigned to CognitiveWeb are Copyright + (c) 2003-2003 CognitiveWeb. All Rights Reserved. Contact + information for CognitiveWeb is available at + + http://www.CognitiveWeb.org + + Portions Copyright (c) 2002-2003 Bryan Thompson. + + Acknowledgements: + + Special thanks to the developers of the Jabber Open Source License 1.0 + (JOSL), from which this License was derived. This License contains + terms that differ from JOSL. + + Special thanks to the CognitiveWeb Open Source Contributors for their + suggestions and support of the Cognitive Web. + + Modifications: + + */ package com.bigdata.btree; Index: MyEvictionListener.java =================================================================== RCS file: /cvsroot/cweb/bigdata/src/test/com/bigdata/btree/MyEvictionListener.java,v retrieving revision 1.1 retrieving revision 1.2 diff -C2 -d -r1.1 -r1.2 *** MyEvictionListener.java 13 Apr 2007 15:04:17 -0000 1.1 --- MyEvictionListener.java 16 Apr 2007 10:02:49 -0000 1.2 *************** *** 1,2 **** --- 1,45 ---- + /** + + The Notice below must appear in each file of the Source Code of any + copy you distribute of the Licensed Product. Contributors to any + Modifications may add their own copyright notices to identify their + own contributions. + + License: + + The contents of this file are subject to the CognitiveWeb Open Source + License Version 1.1 (the License). You may not copy or use this file, + in either source code or executable form, except in compliance with + the License. You may obtain a copy of the License from + + http://www.CognitiveWeb.org/legal/license/ + + Software distributed under the License is distributed on an AS IS + basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See + the License for the specific language governing rights and limitations + under the License. + + Copyrights: + + Portions created by or assigned to CognitiveWeb are Copyright + (c) 2003-2003 CognitiveWeb. All Rights Reserved. Contact + information for CognitiveWeb is available at + + http://www.CognitiveWeb.org + + Portions Copyright (c) 2002-2003 Bryan Thompson. + + Acknowledgements: + + Special thanks to the developers of the Jabber Open Source License 1.0 + (JOSL), from which this License was derived. This License contains + terms that differ from JOSL. + + Special thanks to the CognitiveWeb Open Source Contributors for their + suggestions and support of the Cognitive Web. + + Modifications: + + */ package com.bigdata.btree; |
|
From: Bryan T. <tho...@us...> - 2007-04-16 10:02:54
|
Update of /cvsroot/cweb/bigdata/src/java/com/bigdata/util/concurrent In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv13221/src/java/com/bigdata/util/concurrent Modified Files: DaemonThreadFactory.java Log Message: Updates to make the application of the cweb license uniform in preparation for license change to GPL. Index: DaemonThreadFactory.java =================================================================== RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/util/concurrent/DaemonThreadFactory.java,v retrieving revision 1.1 retrieving revision 1.2 diff -C2 -d -r1.1 -r1.2 *** DaemonThreadFactory.java 21 Feb 2007 20:17:21 -0000 1.1 --- DaemonThreadFactory.java 16 Apr 2007 10:02:50 -0000 1.2 *************** *** 1,2 **** --- 1,45 ---- + /** + + The Notice below must appear in each file of the Source Code of any + copy you distribute of the Licensed Product. Contributors to any + Modifications may add their own copyright notices to identify their + own contributions. + + License: + + The contents of this file are subject to the CognitiveWeb Open Source + License Version 1.1 (the License). You may not copy or use this file, + in either source code or executable form, except in compliance with + the License. You may obtain a copy of the License from + + http://www.CognitiveWeb.org/legal/license/ + + Software distributed under the License is distributed on an AS IS + basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See + the License for the specific language governing rights and limitations + under the License. + + Copyrights: + + Portions created by or assigned to CognitiveWeb are Copyright + (c) 2003-2003 CognitiveWeb. All Rights Reserved. Contact + information for CognitiveWeb is available at + + http://www.CognitiveWeb.org + + Portions Copyright (c) 2002-2003 Bryan Thompson. + + Acknowledgements: + + Special thanks to the developers of the Jabber Open Source License 1.0 + (JOSL), from which this License was derived. This License contains + terms that differ from JOSL. + + Special thanks to the CognitiveWeb Open Source Contributors for their + suggestions and support of the Cognitive Web. + + Modifications: + + */ package com.bigdata.util.concurrent; |
|
From: Bryan T. <tho...@us...> - 2007-04-16 10:02:54
|
Update of /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv13221/src/java/com/bigdata/scaleup Modified Files: MasterJournal.java NameAndExtensionFilter.java SlaveJournal.java PartitionMetadata.java MetadataIndex.java Log Message: Updates to make the application of the cweb license uniform in preparation for license change to GPL. Index: PartitionMetadata.java =================================================================== RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup/PartitionMetadata.java,v retrieving revision 1.7 retrieving revision 1.8 diff -C2 -d -r1.7 -r1.8 *** PartitionMetadata.java 13 Apr 2007 15:04:24 -0000 1.7 --- PartitionMetadata.java 16 Apr 2007 10:02:49 -0000 1.8 *************** *** 1,49 **** /** ! The Notice below must appear in each file of the Source Code of any ! copy you distribute of the Licensed Product. Contributors to any ! Modifications may add their own copyright notices to identify their ! own contributions. ! License: ! The contents of this file are subject to the CognitiveWeb Open Source ! License Version 1.1 (the License). You may not copy or use this file, ! in either source code or executable form, except in compliance with ! the License. You may obtain a copy of the License from ! http://www.CognitiveWeb.org/legal/license/ ! Software distributed under the License is distributed on an AS IS ! basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See ! the License for the specific language governing rights and limitations ! under the License. ! Copyrights: ! Portions created by or assigned to CognitiveWeb are Copyright ! (c) 2003-2003 CognitiveWeb. All Rights Reserved. Contact ! information for CognitiveWeb is available at ! http://www.CognitiveWeb.org ! Portions Copyright (c) 2002-2003 Bryan Thompson. ! Acknowledgements: ! Special thanks to the developers of the Jabber Open Source License 1.0 ! (JOSL), from which this License was derived. This License contains ! terms that differ from JOSL. ! Special thanks to the CognitiveWeb Open Source Contributors for their ! suggestions and support of the Cognitive Web. ! Modifications: ! */ package com.bigdata.scaleup; import java.io.DataInput; - import java.io.DataOutput; import java.io.Externalizable; import java.io.IOException; --- 1,48 ---- /** ! The Notice below must appear in each file of the Source Code of any ! copy you distribute of the Licensed Product. Contributors to any ! Modifications may add their own copyright notices to identify their ! own contributions. ! License: ! The contents of this file are subject to the CognitiveWeb Open Source ! License Version 1.1 (the License). You may not copy or use this file, ! in either source code or executable form, except in compliance with ! the License. You may obtain a copy of the License from ! http://www.CognitiveWeb.org/legal/license/ ! Software distributed under the License is distributed on an AS IS ! basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See ! the License for the specific language governing rights and limitations ! under the License. ! Copyrights: ! Portions created by or assigned to CognitiveWeb are Copyright ! (c) 2003-2003 CognitiveWeb. All Rights Reserved. Contact ! information for CognitiveWeb is available at ! http://www.CognitiveWeb.org ! Portions Copyright (c) 2002-2003 Bryan Thompson. ! Acknowledgements: ! Special thanks to the developers of the Jabber Open Source License 1.0 ! (JOSL), from which this License was derived. This License contains ! terms that differ from JOSL. ! Special thanks to the CognitiveWeb Open Source Contributors for their ! suggestions and support of the Cognitive Web. ! Modifications: ! */ package com.bigdata.scaleup; import java.io.DataInput; import java.io.Externalizable; import java.io.IOException; Index: MasterJournal.java =================================================================== RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup/MasterJournal.java,v retrieving revision 1.7 retrieving revision 1.8 diff -C2 -d -r1.7 -r1.8 *** MasterJournal.java 13 Apr 2007 15:04:24 -0000 1.7 --- MasterJournal.java 16 Apr 2007 10:02:49 -0000 1.8 *************** *** 1,49 **** /** ! The Notice below must appear in each file of the Source Code of any ! copy you distribute of the Licensed Product. Contributors to any ! Modifications may add their own copyright notices to identify their ! own contributions. ! License: ! The contents of this file are subject to the CognitiveWeb Open Source ! License Version 1.1 (the License). You may not copy or use this file, ! in either source code or executable form, except in compliance with ! the License. You may obtain a copy of the License from ! http://www.CognitiveWeb.org/legal/license/ ! Software distributed under the License is distributed on an AS IS ! basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See ! the License for the specific language governing rights and limitations ! under the License. ! Copyrights: ! Portions created by or assigned to CognitiveWeb are Copyright ! (c) 2003-2003 CognitiveWeb. All Rights Reserved. Contact ! information for CognitiveWeb is available at ! http://www.CognitiveWeb.org ! Portions Copyright (c) 2002-2003 Bryan Thompson. ! Acknowledgements: ! Special thanks to the developers of the Jabber Open Source License 1.0 ! (JOSL), from which this License was derived. This License contains ! terms that differ from JOSL. ! Special thanks to the CognitiveWeb Open Source Contributors for their ! suggestions and support of the Cognitive Web. ! Modifications: */ - /* - * Created on Dec 22, 2006 - */ - package com.bigdata.scaleup; --- 1,45 ---- /** ! The Notice below must appear in each file of the Source Code of any ! copy you distribute of the Licensed Product. Contributors to any ! Modifications may add their own copyright notices to identify their ! own contributions. ! License: ! The contents of this file are subject to the CognitiveWeb Open Source ! License Version 1.1 (the License). You may not copy or use this file, ! in either source code or executable form, except in compliance with ! the License. You may obtain a copy of the License from ! http://www.CognitiveWeb.org/legal/license/ ! Software distributed under the License is distributed on an AS IS ! basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See ! the License for the specific language governing rights and limitations ! under the License. ! Copyrights: ! Portions created by or assigned to CognitiveWeb are Copyright ! (c) 2003-2003 CognitiveWeb. All Rights Reserved. Contact ! information for CognitiveWeb is available at ! http://www.CognitiveWeb.org ! Portions Copyright (c) 2002-2003 Bryan Thompson. ! Acknowledgements: ! Special thanks to the developers of the Jabber Open Source License 1.0 ! (JOSL), from which this License was derived. This License contains ! terms that differ from JOSL. ! Special thanks to the CognitiveWeb Open Source Contributors for their ! suggestions and support of the Cognitive Web. ! Modifications: */ package com.bigdata.scaleup; *************** *** 67,71 **** import com.bigdata.cache.WeakValueCache; import com.bigdata.isolation.UnisolatedBTree; - import com.bigdata.isolation.Value; import com.bigdata.journal.BufferMode; import com.bigdata.journal.CommitRecordIndex; --- 63,66 ---- Index: SlaveJournal.java =================================================================== RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup/SlaveJournal.java,v retrieving revision 1.8 retrieving revision 1.9 diff -C2 -d -r1.8 -r1.9 *** SlaveJournal.java 13 Apr 2007 15:04:24 -0000 1.8 --- SlaveJournal.java 16 Apr 2007 10:02:49 -0000 1.9 *************** *** 1,43 **** /** ! The Notice below must appear in each file of the Source Code of any ! copy you distribute of the Licensed Product. Contributors to any ! Modifications may add their own copyright notices to identify their ! own contributions. ! License: ! The contents of this file are subject to the CognitiveWeb Open Source ! License Version 1.1 (the License). You may not copy or use this file, ! in either source code or executable form, except in compliance with ! the License. You may obtain a copy of the License from ! http://www.CognitiveWeb.org/legal/license/ ! Software distributed under the License is distributed on an AS IS ! basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See ! the License for the specific language governing rights and limitations ! under the License. ! Copyrights: ! Portions created by or assigned to CognitiveWeb are Copyright ! (c) 2003-2003 CognitiveWeb. All Rights Reserved. Contact ! information for CognitiveWeb is available at ! http://www.CognitiveWeb.org ! Portions Copyright (c) 2002-2003 Bryan Thompson. ! Acknowledgements: ! Special thanks to the developers of the Jabber Open Source License 1.0 ! (JOSL), from which this License was derived. This License contains ! terms that differ from JOSL. ! Special thanks to the CognitiveWeb Open Source Contributors for their ! suggestions and support of the Cognitive Web. ! Modifications: */ --- 1,43 ---- /** ! The Notice below must appear in each file of the Source Code of any ! copy you distribute of the Licensed Product. Contributors to any ! Modifications may add their own copyright notices to identify their ! own contributions. ! License: ! The contents of this file are subject to the CognitiveWeb Open Source ! License Version 1.1 (the License). You may not copy or use this file, ! in either source code or executable form, except in compliance with ! the License. You may obtain a copy of the License from ! http://www.CognitiveWeb.org/legal/license/ ! Software distributed under the License is distributed on an AS IS ! basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See ! the License for the specific language governing rights and limitations ! under the License. ! Copyrights: ! Portions created by or assigned to CognitiveWeb are Copyright ! (c) 2003-2003 CognitiveWeb. All Rights Reserved. Contact ! information for CognitiveWeb is available at ! http://www.CognitiveWeb.org ! Portions Copyright (c) 2002-2003 Bryan Thompson. ! Acknowledgements: ! Special thanks to the developers of the Jabber Open Source License 1.0 ! (JOSL), from which this License was derived. This License contains ! terms that differ from JOSL. ! Special thanks to the CognitiveWeb Open Source Contributors for their ! suggestions and support of the Cognitive Web. ! Modifications: */ Index: NameAndExtensionFilter.java =================================================================== RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup/NameAndExtensionFilter.java,v retrieving revision 1.1 retrieving revision 1.2 diff -C2 -d -r1.1 -r1.2 *** NameAndExtensionFilter.java 8 Feb 2007 21:32:14 -0000 1.1 --- NameAndExtensionFilter.java 16 Apr 2007 10:02:49 -0000 1.2 *************** *** 1,2 **** --- 1,45 ---- + /** + + The Notice below must appear in each file of the Source Code of any + copy you distribute of the Licensed Product. Contributors to any + Modifications may add their own copyright notices to identify their + own contributions. + + License: + + The contents of this file are subject to the CognitiveWeb Open Source + License Version 1.1 (the License). You may not copy or use this file, + in either source code or executable form, except in compliance with + the License. You may obtain a copy of the License from + + http://www.CognitiveWeb.org/legal/license/ + + Software distributed under the License is distributed on an AS IS + basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See + the License for the specific language governing rights and limitations + under the License. + + Copyrights: + + Portions created by or assigned to CognitiveWeb are Copyright + (c) 2003-2003 CognitiveWeb. All Rights Reserved. Contact + information for CognitiveWeb is available at + + http://www.CognitiveWeb.org + + Portions Copyright (c) 2002-2003 Bryan Thompson. + + Acknowledgements: + + Special thanks to the developers of the Jabber Open Source License 1.0 + (JOSL), from which this License was derived. This License contains + terms that differ from JOSL. + + Special thanks to the CognitiveWeb Open Source Contributors for their + suggestions and support of the Cognitive Web. + + Modifications: + + */ package com.bigdata.scaleup; Index: MetadataIndex.java =================================================================== RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup/MetadataIndex.java,v retrieving revision 1.8 retrieving revision 1.9 diff -C2 -d -r1.8 -r1.9 *** MetadataIndex.java 13 Apr 2007 15:04:24 -0000 1.8 --- MetadataIndex.java 16 Apr 2007 10:02:49 -0000 1.9 *************** *** 1,45 **** /** ! The Notice below must appear in each file of the Source Code of any ! copy you distribute of the Licensed Product. Contributors to any ! Modifications may add their own copyright notices to identify their ! own contributions. ! License: ! The contents of this file are subject to the CognitiveWeb Open Source ! License Version 1.1 (the License). You may not copy or use this file, ! in either source code or executable form, except in compliance with ! the License. You may obtain a copy of the License from ! http://www.CognitiveWeb.org/legal/license/ ! Software distributed under the License is distributed on an AS IS ! basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See ! the License for the specific language governing rights and limitations ! under the License. ! Copyrights: ! Portions created by or assigned to CognitiveWeb are Copyright ! (c) 2003-2003 CognitiveWeb. All Rights Reserved. Contact ! information for CognitiveWeb is available at ! http://www.CognitiveWeb.org ! Portions Copyright (c) 2002-2003 Bryan Thompson. ! Acknowledgements: ! Special thanks to the developers of the Jabber Open Source License 1.0 ! (JOSL), from which this License was derived. This License contains ! terms that differ from JOSL. ! Special thanks to the CognitiveWeb Open Source Contributors for their ! suggestions and support of the Cognitive Web. ! Modifications: ! */ package com.bigdata.scaleup; --- 1,45 ---- /** ! The Notice below must appear in each file of the Source Code of any ! copy you distribute of the Licensed Product. Contributors to any ! Modifications may add their own copyright notices to identify their ! own contributions. ! License: ! The contents of this file are subject to the CognitiveWeb Open Source ! License Version 1.1 (the License). You may not copy or use this file, ! in either source code or executable form, except in compliance with ! the License. You may obtain a copy of the License from ! http://www.CognitiveWeb.org/legal/license/ ! Software distributed under the License is distributed on an AS IS ! basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See ! the License for the specific language governing rights and limitations ! under the License. ! Copyrights: ! Portions created by or assigned to CognitiveWeb are Copyright ! (c) 2003-2003 CognitiveWeb. All Rights Reserved. Contact ! information for CognitiveWeb is available at ! http://www.CognitiveWeb.org ! Portions Copyright (c) 2002-2003 Bryan Thompson. ! Acknowledgements: ! Special thanks to the developers of the Jabber Open Source License 1.0 ! (JOSL), from which this License was derived. This License contains ! terms that differ from JOSL. ! Special thanks to the CognitiveWeb Open Source Contributors for their ! suggestions and support of the Cognitive Web. ! Modifications: ! */ package com.bigdata.scaleup; |
|
From: Bryan T. <tho...@us...> - 2007-04-16 10:02:53
|
Update of /cvsroot/cweb/bigdata/src/java/com/bigdata/service In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv13221/src/java/com/bigdata/service Modified Files: IReducer.java IDataService.java IMapOp.java Log Message: Updates to make the application of the cweb license uniform in preparation for license change to GPL. Index: IMapOp.java =================================================================== RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/service/IMapOp.java,v retrieving revision 1.3 retrieving revision 1.4 diff -C2 -d -r1.3 -r1.4 *** IMapOp.java 13 Apr 2007 15:04:19 -0000 1.3 --- IMapOp.java 16 Apr 2007 10:02:50 -0000 1.4 *************** *** 1,2 **** --- 1,45 ---- + /** + + The Notice below must appear in each file of the Source Code of any + copy you distribute of the Licensed Product. Contributors to any + Modifications may add their own copyright notices to identify their + own contributions. + + License: + + The contents of this file are subject to the CognitiveWeb Open Source + License Version 1.1 (the License). You may not copy or use this file, + in either source code or executable form, except in compliance with + the License. You may obtain a copy of the License from + + http://www.CognitiveWeb.org/legal/license/ + + Software distributed under the License is distributed on an AS IS + basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See + the License for the specific language governing rights and limitations + under the License. + + Copyrights: + + Portions created by or assigned to CognitiveWeb are Copyright + (c) 2003-2003 CognitiveWeb. All Rights Reserved. Contact + information for CognitiveWeb is available at + + http://www.CognitiveWeb.org + + Portions Copyright (c) 2002-2003 Bryan Thompson. + + Acknowledgements: + + Special thanks to the developers of the Jabber Open Source License 1.0 + (JOSL), from which this License was derived. This License contains + terms that differ from JOSL. + + Special thanks to the CognitiveWeb Open Source Contributors for their + suggestions and support of the Cognitive Web. + + Modifications: + + */ package com.bigdata.service; Index: IReducer.java =================================================================== RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/service/IReducer.java,v retrieving revision 1.1 retrieving revision 1.2 diff -C2 -d -r1.1 -r1.2 *** IReducer.java 15 Mar 2007 16:11:11 -0000 1.1 --- IReducer.java 16 Apr 2007 10:02:50 -0000 1.2 *************** *** 1,2 **** --- 1,45 ---- + /** + + The Notice below must appear in each file of the Source Code of any + copy you distribute of the Licensed Product. Contributors to any + Modifications may add their own copyright notices to identify their + own contributions. + + License: + + The contents of this file are subject to the CognitiveWeb Open Source + License Version 1.1 (the License). You may not copy or use this file, + in either source code or executable form, except in compliance with + the License. You may obtain a copy of the License from + + http://www.CognitiveWeb.org/legal/license/ + + Software distributed under the License is distributed on an AS IS + basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See + the License for the specific language governing rights and limitations + under the License. + + Copyrights: + + Portions created by or assigned to CognitiveWeb are Copyright + (c) 2003-2003 CognitiveWeb. All Rights Reserved. Contact + information for CognitiveWeb is available at + + http://www.CognitiveWeb.org + + Portions Copyright (c) 2002-2003 Bryan Thompson. + + Acknowledgements: + + Special thanks to the developers of the Jabber Open Source License 1.0 + (JOSL), from which this License was derived. This License contains + terms that differ from JOSL. + + Special thanks to the CognitiveWeb Open Source Contributors for their + suggestions and support of the Cognitive Web. + + Modifications: + + */ package com.bigdata.service; Index: IDataService.java =================================================================== RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/service/IDataService.java,v retrieving revision 1.6 retrieving revision 1.7 diff -C2 -d -r1.6 -r1.7 *** IDataService.java 13 Apr 2007 15:04:19 -0000 1.6 --- IDataService.java 16 Apr 2007 10:02:50 -0000 1.7 *************** *** 1,49 **** /** ! The Notice below must appear in each file of the Source Code of any ! copy you distribute of the Licensed Product. Contributors to any ! Modifications may add their own copyright notices to identify their ! own contributions. ! ! License: ! The contents of this file are subject to the CognitiveWeb Open Source ! License Version 1.1 (the License). You may not copy or use this file, ! in either source code or executable form, except in compliance with ! the License. You may obtain a copy of the License from ! http://www.CognitiveWeb.org/legal/license/ ! Software distributed under the License is distributed on an AS IS ! basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See ! the License for the specific language governing rights and limitations ! under the License. ! Copyrights: ! Portions created by or assigned to CognitiveWeb are Copyright ! (c) 2003-2003 CognitiveWeb. All Rights Reserved. Contact ! information for CognitiveWeb is available at ! http://www.CognitiveWeb.org ! Portions Copyright (c) 2002-2003 Bryan Thompson. ! Acknowledgements: ! Special thanks to the developers of the Jabber Open Source License 1.0 ! (JOSL), from which this License was derived. This License contains ! terms that differ from JOSL. ! Special thanks to the CognitiveWeb Open Source Contributors for their ! suggestions and support of the Cognitive Web. ! Modifications: ! */ ! /* ! * Created on Mar 15, 2007 ! */ package com.bigdata.service; --- 1,45 ---- /** ! The Notice below must appear in each file of the Source Code of any ! copy you distribute of the Licensed Product. Contributors to any ! Modifications may add their own copyright notices to identify their ! own contributions. ! License: ! The contents of this file are subject to the CognitiveWeb Open Source ! License Version 1.1 (the License). You may not copy or use this file, ! in either source code or executable form, except in compliance with ! the License. You may obtain a copy of the License from ! http://www.CognitiveWeb.org/legal/license/ ! Software distributed under the License is distributed on an AS IS ! basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See ! the License for the specific language governing rights and limitations ! under the License. ! Copyrights: ! Portions created by or assigned to CognitiveWeb are Copyright ! (c) 2003-2003 CognitiveWeb. All Rights Reserved. Contact ! information for CognitiveWeb is available at ! http://www.CognitiveWeb.org ! Portions Copyright (c) 2002-2003 Bryan Thompson. ! Acknowledgements: ! Special thanks to the developers of the Jabber Open Source License 1.0 ! (JOSL), from which this License was derived. This License contains ! terms that differ from JOSL. ! Special thanks to the CognitiveWeb Open Source Contributors for their ! suggestions and support of the Cognitive Web. ! Modifications: + */ package com.bigdata.service; |
|
From: Bryan T. <tho...@us...> - 2007-04-16 10:02:52
|
Update of /cvsroot/cweb/bigdata/src/java/com/bigdata/istore In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv13221/src/java/com/bigdata/istore Modified Files: JournalStore.java Log Message: Updates to make the application of the cweb license uniform in preparation for license change to GPL. Index: JournalStore.java =================================================================== RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/istore/JournalStore.java,v retrieving revision 1.6 retrieving revision 1.7 diff -C2 -d -r1.6 -r1.7 *** JournalStore.java 11 Mar 2007 11:42:43 -0000 1.6 --- JournalStore.java 16 Apr 2007 10:02:49 -0000 1.7 *************** *** 1,2 **** --- 1,45 ---- + /** + + The Notice below must appear in each file of the Source Code of any + copy you distribute of the Licensed Product. Contributors to any + Modifications may add their own copyright notices to identify their + own contributions. + + License: + + The contents of this file are subject to the CognitiveWeb Open Source + License Version 1.1 (the License). You may not copy or use this file, + in either source code or executable form, except in compliance with + the License. You may obtain a copy of the License from + + http://www.CognitiveWeb.org/legal/license/ + + Software distributed under the License is distributed on an AS IS + basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See + the License for the specific language governing rights and limitations + under the License. + + Copyrights: + + Portions created by or assigned to CognitiveWeb are Copyright + (c) 2003-2003 CognitiveWeb. All Rights Reserved. Contact + information for CognitiveWeb is available at + + http://www.CognitiveWeb.org + + Portions Copyright (c) 2002-2003 Bryan Thompson. + + Acknowledgements: + + Special thanks to the developers of the Jabber Open Source License 1.0 + (JOSL), from which this License was derived. This License contains + terms that differ from JOSL. + + Special thanks to the CognitiveWeb Open Source Contributors for their + suggestions and support of the Cognitive Web. + + Modifications: + + */ package com.bigdata.istore; |
|
From: Bryan T. <tho...@us...> - 2007-04-16 10:02:31
|
Update of /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv13212/src/java/com/bigdata/rdf Modified Files: AutoIncCounter.java Log Message: Updates to make the application of the cweb license uniform in preparation for license change to GPL. Index: AutoIncCounter.java =================================================================== RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/AutoIncCounter.java,v retrieving revision 1.5 retrieving revision 1.6 diff -C2 -d -r1.5 -r1.6 *** AutoIncCounter.java 13 Apr 2007 15:02:34 -0000 1.5 --- AutoIncCounter.java 16 Apr 2007 10:02:24 -0000 1.6 *************** *** 1,2 **** --- 1,45 ---- + /** + + The Notice below must appear in each file of the Source Code of any + copy you distribute of the Licensed Product. Contributors to any + Modifications may add their own copyright notices to identify their + own contributions. + + License: + + The contents of this file are subject to the CognitiveWeb Open Source + License Version 1.1 (the License). You may not copy or use this file, + in either source code or executable form, except in compliance with + the License. You may obtain a copy of the License from + + http://www.CognitiveWeb.org/legal/license/ + + Software distributed under the License is distributed on an AS IS + basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See + the License for the specific language governing rights and limitations + under the License. + + Copyrights: + + Portions created by or assigned to CognitiveWeb are Copyright + (c) 2003-2003 CognitiveWeb. All Rights Reserved. Contact + information for CognitiveWeb is available at + + http://www.CognitiveWeb.org + + Portions Copyright (c) 2002-2003 Bryan Thompson. + + Acknowledgements: + + Special thanks to the developers of the Jabber Open Source License 1.0 + (JOSL), from which this License was derived. This License contains + terms that differ from JOSL. + + Special thanks to the CognitiveWeb Open Source Contributors for their + suggestions and support of the Cognitive Web. + + Modifications: + + */ package com.bigdata.rdf; |
|
From: Bryan T. <tho...@us...> - 2007-04-15 23:29:29
|
Update of /cvsroot/cweb/bigdata In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv6131 Modified Files: project.xml Log Message: POM updates in preparation for the license change. Index: project.xml =================================================================== RCS file: /cvsroot/cweb/bigdata/project.xml,v retrieving revision 1.5 retrieving revision 1.6 diff -C2 -d -r1.5 -r1.6 *** project.xml 15 Apr 2007 18:17:04 -0000 1.5 --- project.xml 15 Apr 2007 23:29:26 -0000 1.6 *************** *** 4,11 **** <name>bigdata</name> <package>com.bigdata</package> ! <extends>../cweb/project.xml</extends> <groupId>bigdata</groupId> <currentVersion>0.1-b1-dev</currentVersion> <organization> <name>SYSTAP, LLC</name> --- 4,13 ---- <name>bigdata</name> <package>com.bigdata</package> ! <extends>../bigdata-master-pom/project.xml</extends> <groupId>bigdata</groupId> <currentVersion>0.1-b1-dev</currentVersion> + <!-- @todo "master" project, CVSROOT (permissions). --> + <organization> <name>SYSTAP, LLC</name> *************** *** 195,198 **** --- 197,223 ---- </p> + <h2>License</h2> + + <p> + + bigdata is currently being released under GPL (version 2). However, it is + our intention to make bigdata usable by non-GPL Free/Libre and Open Source + Software (FLOSS) projects. To this end, we are considering either a FLOSS + exclusion (such as MySQL AB has issued) or a license change to something + with the same semantics as the SleepyCat License. Please bear with us + while we work through this issue. If you have any questions concerning the + license, write lic...@bi.... + + </p> + + <p> + + SYSTAP, LLC will offer commercial licenses for customers who either want the + value add (warrenty, technical support, additional regression testing), who + want to redistribute bigdata with their own commercial products, or who are + not "comfortable" with the GPL license. + + </p> + ]]> |
|
From: Bryan T. <tho...@us...> - 2007-04-15 18:17:11
|
Update of /cvsroot/cweb/bigdata/xdocs In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv6887/xdocs Modified Files: navigation.xml Log Message: Preparing bigdata for license change. POM may now be used to do a maven compile. Jini JARs were renamed to include the version (2.1). Index: navigation.xml =================================================================== RCS file: /cvsroot/cweb/bigdata/xdocs/navigation.xml,v retrieving revision 1.1 retrieving revision 1.2 diff -C2 -d -r1.1 -r1.2 *** navigation.xml 26 Aug 2006 19:30:56 -0000 1.1 --- navigation.xml 15 Apr 2007 18:17:05 -0000 1.2 *************** *** 1,4 **** <?xml version="1.0" encoding="ISO-8859-1"?> ! <project name="BigData"> <!-- --> --- 1,4 ---- <?xml version="1.0" encoding="ISO-8859-1"?> ! <project name="bigdata"> <!-- --> *************** *** 17,21 **** <properties> <author>tho...@us...</author> ! <title>BigData</title> </properties> --- 17,21 ---- <properties> <author>tho...@us...</author> ! <title>bigdata</title> </properties> *************** *** 24,30 **** <!-- horizontal links at top of page. --> <links> - <item name="Wiki" - href="http://wiki.cognitiveweb.org/BigData" - /> <item name="Javadoc" href="/apidocs/index.html" --- 24,27 ---- *************** *** 32,58 **** </links> ! <!-- A top-level, hierarchical menu --> ! <menu name="CognitiveWeb"> ! <item name="CognitiveWeb" ! href="http://www.cognitiveweb.org" ! img="http://www.cognitiveweb.org/Images/tagline.gif" /> ! <item name="CognitiveWebWiki" ! href="http://wiki.cognitiveweb.org" ! img="http://www.cognitiveweb.org/wiki/img/moinmoin.gif" /> </menu> <!-- Architecture. --> - <!-- <menu name="Architecture"> ! <item name="PDF of UML" ! href="/architecture/DBCache.pdf" ! /> ! <item name="UML (IBM System Architect)" ! href="/architecture/DBCache.emx" /> </menu> - --> <!-- References. --> --- 29,49 ---- </links> ! <!-- A top-level, hierarchical menu --> ! <menu name="bigdata"> ! <item name="bigdata" ! href="http://www.bigdata.com" /> ! <item name="sourceforge" ! href="http://sourceforge.net/projects/bigdata/" ! img="http://sourceforge.net/sflogo.php?group_id=95673&type=1" /> </menu> <!-- Architecture. --> <menu name="Architecture"> ! <item name="Web-published UML (zipped)" ! href="/architecture/bigdata-published-model.zip" /> </menu> <!-- References. --> *************** *** 62,68 **** href="http://portal.acm.org/citation.cfm?id=1995" /> - --> - </menu> <!-- Add the google(tm) search widget. There is no way to --- 53,58 ---- href="http://portal.acm.org/citation.cfm?id=1995" /> </menu> + --> <!-- Add the google(tm) search widget. There is no way to |
|
From: Bryan T. <tho...@us...> - 2007-04-15 18:17:11
|
Update of /cvsroot/cweb/bigdata In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv6887 Modified Files: project.properties project.xml Log Message: Preparing bigdata for license change. POM may now be used to do a maven compile. Jini JARs were renamed to include the version (2.1). Index: project.properties =================================================================== RCS file: /cvsroot/cweb/bigdata/project.properties,v retrieving revision 1.1 retrieving revision 1.2 diff -C2 -d -r1.1 -r1.2 *** project.properties 26 Aug 2006 19:30:13 -0000 1.1 --- project.properties 15 Apr 2007 18:17:04 -0000 1.2 *************** *** 7,11 **** # authoritative source. ! maven.repo.remote=http://www.ibiblio.org/maven/,http://www.cognitiveweb.org/maven-repository,http://proto.cognitiveweb.org/maven-repository maven.javadoc.links=\ --- 7,11 ---- # authoritative source. ! maven.repo.remote=http://mirrors.ibiblio.org/pub/mirrors/maven,http://proto.cognitiveweb.org/maven-repository maven.javadoc.links=\ Index: project.xml =================================================================== RCS file: /cvsroot/cweb/bigdata/project.xml,v retrieving revision 1.4 retrieving revision 1.5 diff -C2 -d -r1.4 -r1.5 *** project.xml 5 Feb 2007 18:17:46 -0000 1.4 --- project.xml 15 Apr 2007 18:17:04 -0000 1.5 *************** *** 1,21 **** <?xml version="1.0" encoding="ISO-8859-1"?> <project> ! <id>cweb-bigdata</id> <name>bigdata</name> ! <package>org.CognitiveWeb</package> <extends>../cweb/project.xml</extends> ! <groupId>org.CognitiveWeb</groupId> <currentVersion>0.1-b1-dev</currentVersion> <organization> ! <name>CognitiveWeb</name> ! <url>http://www.cognitiveweb.org/</url> ! <logo>http://www.cognitiveweb.org/Images/cogweb-logo.jpg</logo> </organization> <!-- required per-POM metadata. --> ! <inceptionYear>2002</inceptionYear> ! <url>http://www.cognitiveweb.org/</url> ! <logo>http://www.cognitiveweb.org/Images/tagline.gif</logo> <!-- The short description is used in the JAR manifest. For --> --- 1,21 ---- <?xml version="1.0" encoding="ISO-8859-1"?> <project> ! <id>bigdata</id> <name>bigdata</name> ! <package>com.bigdata</package> <extends>../cweb/project.xml</extends> ! <groupId>bigdata</groupId> <currentVersion>0.1-b1-dev</currentVersion> <organization> ! <name>SYSTAP, LLC</name> ! <url>http://www.bigdata.com/</url> ! <!--logo>url goes here</logo--> </organization> <!-- required per-POM metadata. --> ! <inceptionYear>2006</inceptionYear> ! <url>http://www.bigdata.com/</url> ! <!--logo>url goes here</logo--> <!-- The short description is used in the JAR manifest. For --> *************** *** 31,37 **** <p> ! bigdata is a distributed storage platform designed for high performance ! and scale out on commodity hardware. </p> --- 31,196 ---- <p> ! bigdata is a scale-out data and computing fabric designed for commodity hardware. ! ! </p> ! <p> ! ! The bigdata architecture provides named scale-out indices that are transparently ! partitioned and distributed across a cluster or grid of commodity server platforms. ! Atomic "row" operations are supported for very high concurrency. However, full ! transactions are also available for applications needing less concurrency but ! requiring atomic operations that read or write on more than one row, index partition, ! or index. The scale-out indices are B+Trees and remain balanced under insert and ! removal operations. Keys and values for btrees are variable length byte[]s (the ! keys are interpreted as unsigned byte[]s). Writes are absorbed on mutable btree instances in append only ! "journals" of ~200M capacity. On overflow, data in a journal is evicted onto read- ! optimized, immutable "index segments". The metadata service manages the index ! definitions, the index partitions, and the assignment of index partitions to data ! services. A data service encapsulates a journal to zero or more index partitions ! have been assigned, including the logic to handle overflow. A deployment is made ! up of one logical metadata service (with failover redundency) and many distinct ! data services. bigdata can provide data redundency internally (by pipelining ! writes bound for a partition across primary, secondary, ... data service instances ! for that partition) or it can be deployed over distributed RAID. bigdata itself ! is 100% Java and requires a 1.5 or better JVM. ! ! </p> ! ! <h2>Status</h2> ! ! <p> ! ! bigdata is currently in pre-release. We are seeking beta users to ! vet the architecture and work through robust scaling. The basic index structures (mutable and ! immutable btrees and bulk index build operations) and the persistent store ! structure (the journal) have all been implemented and tested. Current work ! is focused on the distributed services architecture. Aside from the test suite, ! there is an RDF database application layer (bigdata-rdf) that is being used to ! benchmark and tune the bigdata implementation. We have plans to provide an ! transactional OODBMS integration (based on the Generic Object Model) as well ! as a media index and search service (based on Lucene). ! ! </p> ! ! <h2>Architecture</h2> ! ! <p> ! ! The bigdata SOA defines two essential services and some additional services. The ! essential services are the metadata service (provides a locator service for index ! partitions on data services) and the data service (provides read, write, and concurrency ! control for index partitions). Optional services include a transaction manager (transactions ! are NOT required, so you can use bigdata as a scale-out row store ) and a job scheduler (the ! integration point for map/reduce style distributed functional programming). ! ! </p> ! ! <p> ! ! While other service fabric architectures are contemplated, bigdata services ! today use JINI 2.x to advertise themselves and do perform service discovery. ! This means that you must be running a JINI registrar in order for services to be ! able to register themselves or discover other services. ! ! </p> ! ! <p> ! ! The main building blocks for the bigdata architecture are the journal (append-only ! persistence store), the mutable B+Tree (used to absorb writes), and the read-optimized ! immutable B+Tree (aka the index segment). Highly efficient bulk index builds are used ! to transfer data absorbed by a mutable B+Tree on a journal into index segment files. ! Each for index segment contains data for a single partition of a scale-out index. In ! order to read from an index partition, a consistent view is created by dynamically ! fusing data for that index partition, including any recent writes on the current ! journal, any historical writes that are in the process of being transferred onto ! index segments, and any historical index segments that also contain data for that ! view. Periodically, index segments are merged together. Both merging index segments ! and overflow of a journal onto index segments cause historical states of the database ! to be purged in favor of the current state. Periodically, the metadata service will ! direct data services to physically delete old journals and index segments, thereby ! releasing the local disk resources. (An immortal or temporal database can be ! realized by choosing not to delete old journals, since journals have all committed ! states for the distributed database.) ! ! </p> ! ! <h2>Sparse row store</h2> ! ! <p> ! ! People familiar with Google's bigtable architecture will recognize the similarity. ! bigtable provides very high concurrency with atomic "row" updates and a space data ! model. In fact, it is trivial to realize bigtable semantics with bigdata - you ! need to exercise a specific protocol when forming the keys for your scale-out ! indices and you simply choose to NOT use transactions. A bigtable style key-value ! is formed as: ! ! </p> ! ! <pre> ! ! {columnFamily \0 primaryKey \0 columnName \0 timestamp} : {value} ! ! </pre> ! ! <p> ! ! By placing the column family identifier up front, all data in the same column ! family will be clustered together by the index. The next component is the "row" ! identifier, what you would think of as the primary key in a relational table. ! The column name comes next - only column names for non-null columns are written ! into the index. Finally, there is a timestamp column that is used either to ! record a timestamp specified by the application or a datum write time. The ! value associated with the key is simply the datum for that column in that row. ! The use of nul byte separators makes it possible to parse the key, which is ! required for various operations including index partition splits and filtering ! key scans based on column names or timestamps. ! See the KeyBuilder class ! in com.bigdata.btree for utilities that may be used to construct keys from a variety ! of components. ! ! </p> ! ! <p> ! ! Filter row scans are realized using the filter mechanism on the key scan ! interface of the data service. The history policy is specified when the ! index is created and is applied during compacting merges of index segments, ! effectively removing entries for the index partition that do not pass the ! history filter. ! ! </p> ! ! <h2>Map/reduce</h2> ! ! <p> ! ! Google's map/reduce architecture has received a lot of attention, along with ! its bigtable architecture. Map/reduce provides a means to transparently ! decompose processing across a cluster. The "map" process examines a series ! of key-value pair, emitting a set of intermediate key-value pairs for each ! input. Those intermediate key-values are then hashed (module R) onto R reduce ! processes. The inputs for the reduce processes are pre-sorted. The reduce ! process then runs some arbitrary operation on the sorted data, such as computing ! an inverted index file or loading the data into a scale-out index. ! ! </p> + <h2>Data Replication</h2> + + <p> + + If you know much about Google's architecture, you know that they deploy on + a distributed file system named "GFS" (Google File System). bigdata is + designed to provide its own data replication using pipelining of writes + across multiple data services, such that each index partition has some + target #of replicated indices. However, an alternative deployment strategy + is to deploy on a distributed file system or using NAS with built in RAID. + The best deployment configuration undoubtedly depends on the specifics of + your data center. bigtable makes an effort throughout its architecture to + only perform sustained IOs. This means that it can be a very efficient user + of both local file systems and networked file systems. + </p> *************** *** 52,67 **** <!-- copy down inheritance applies. --> ! <siteAddress>www.cognitiveweb.org</siteAddress> <siteDirectory> ! /home/groups/c/cw/cweb/htdocs/technology/projects/${pom.artifactId} </siteDirectory> <!-- copy down inheritance applies. --> <distributionSite>www.sourceforge.net</distributionSite> ! <distributionDirectory>/projects/cweb</distributionDirectory> <!-- copy down inheritance applies. --> <issueTrackingUrl> ! http://jira.cognitiveweb.org/browse/BigData </issueTrackingUrl> --- 211,231 ---- <!-- copy down inheritance applies. --> ! <siteAddress>www.bigdata.com</siteAddress> <siteDirectory> ! /home/groups/b/bi/bigd/htdocs/technology/projects/${pom.artifactId} </siteDirectory> <!-- copy down inheritance applies. --> <distributionSite>www.sourceforge.net</distributionSite> ! <distributionDirectory>/projects/bigdata</distributionDirectory> <!-- copy down inheritance applies. --> + <!-- <issueTrackingUrl> ! http://jira.bigdata.com/browse/bigdata ! </issueTrackingUrl> ! --> ! <issueTrackingUrl> ! http://sourceforge.net/tracker/?group_id=191861 </issueTrackingUrl> *************** *** 69,78 **** <repository> <connection> ! scm:cvs:pserver:anonymous@${maven.scm.cvs.host}:/cvsroot/cweb:bigdata </connection> <developerConnection> ! scm:cvs:ext:user@${maven.scm.cvs.host}:/cvsroot/cweb:bigdata </developerConnection> ! <url>http://cweb.cvs.sourceforge.net/cweb/bigdata</url> </repository> --- 233,242 ---- <repository> <connection> ! scm:cvs:pserver:anonymous@${maven.scm.cvs.host}:/cvsroot/bigdata:bigdata </connection> <developerConnection> ! scm:cvs:ext:user@${maven.scm.cvs.host}:/cvsroot/bigdata:bigdata </developerConnection> ! <url>http://bigdata.cvs.sourceforge.net/bigdata/bigdata</url> </repository> *************** *** 80,101 **** <licenses> <license> ! <name>CognitiveWeb Open Source License</name> ! <url>http://www.cognitiveweb.org/legal/license/</url> <distribution>manual</distribution> </license> </licenses> ! <!-- copy down inheritance applies. --> <mailingLists> <mailingList> <name>Announce List</name> <subscribe> ! http://lists.sourceforge.net/lists/listinfo/cweb-announce </subscribe> <unsubscribe> ! http://lists.sourceforge.net/lists/listinfo/cweb-announce </unsubscribe> <archive> ! http://sourceforge.net/mailarchive/forum.php?forum=cweb-announce </archive> </mailingList> --- 244,265 ---- <licenses> <license> ! <name>GPL (version 2)</name> ! <url>http://www.opensource.org/licenses/gpl-license.php</url> <distribution>manual</distribution> </license> </licenses> ! <!-- copy down inheritance applies. <mailingLists> <mailingList> <name>Announce List</name> <subscribe> ! http://lists.sourceforge.net/lists/listinfo/bigdata-announce </subscribe> <unsubscribe> ! http://lists.sourceforge.net/lists/listinfo/bigdata-announce </unsubscribe> <archive> ! http://sourceforge.net/mailarchive/forum.php?forum=bigdata-announce </archive> </mailingList> *************** *** 103,116 **** <name>Developer List</name> <subscribe> ! http://lists.sourceforge.net/lists/listinfo/cweb-developer </subscribe> <unsubscribe> ! http://lists.sourceforge.net/lists/listinfo/cweb-developer </unsubscribe> <archive> ! http://sourceforge.net/mailarchive/forum.php?forum=cweb-developer </archive> </mailingList> </mailingLists> <!-- copy down inheritance applies. --> --- 267,281 ---- <name>Developer List</name> <subscribe> ! http://lists.sourceforge.net/lists/listinfo/bigdata-developer </subscribe> <unsubscribe> ! http://lists.sourceforge.net/lists/listinfo/bigdata-developer </unsubscribe> <archive> ! http://sourceforge.net/mailarchive/forum.php?forum=bigdata-developer </archive> </mailingList> </mailingLists> + --> <!-- copy down inheritance applies. --> *************** *** 126,129 **** --- 291,303 ---- </developer> + <developer> + <name>Mike Personick</name> + <id>mike</id> + <email>mrp...@us...</email> + <organization>SYSTAP, LLC.</organization> + <!-- US Mountain. --> + <timezone>-7</timezone> + </developer> + </developers> *************** *** 142,145 **** --- 316,321 ---- <dependencies> + <!-- Note: Dependencies are bundled in CVS. --> + <!-- log4j integration. --> <dependency> *************** *** 149,153 **** </dependency> ! <!-- junit extensions. --> <dependency> <id>junit</id> --- 325,329 ---- </dependency> ! <!-- junit (tests only). --> <dependency> <id>junit</id> *************** *** 156,164 **** </dependency> <!-- LGPL utility classes. --> <dependency> ! <id>lgpl-util</id> ! <!--<version>3.8.1</version>--> ! <url>http://cweb.cvs.sourceforge.net/cweb/lgpl-utils/</url> </dependency> --- 332,422 ---- </dependency> + <!-- junit extensions (tests only). --> + <dependency> + <groupId>org.CognitiveWeb</groupId> + <artifactId>cweb-junit-ext</artifactId> + <version>1.1-b2-dev</version> + <url>http://www.sourceforge.net/projects/cweb</url> + </dependency> + + <!-- cweb utility classes. --> + <dependency> + <groupId>org.CognitiveWeb</groupId> + <artifactId>cweb-commons</artifactId> + <version>1.1-b2-dev</version> + <url>http://www.sourceforge.net/projects/cweb</url> + </dependency> + + <!-- Extensible Serialization framework. --> + <dependency> + <groupId>org.CognitiveWeb</groupId> + <artifactId>cweb-extser</artifactId> + <version>0.1-b2-dev</version> + <url>http://www.sourceforge.net/projects/cweb</url> + </dependency> + <!-- LGPL utility classes. --> <dependency> ! <groupId>org.CognitiveWeb</groupId> ! <artifactId>lgpl-utils</artifactId> ! <version>1.0-b1-dev</version> ! <url>http://www.sourceforge.net/projects/cweb</url> ! </dependency> ! ! <!-- Striterator support. --> ! <!-- Note: The jar name reflects the date when I downloaded it --> ! <!-- and makes it possible for me to rollback to historical CTC --> ! <!-- versions. Any recent "ctc_utils.jar" SHOULD work. Just --> ! <!-- extract the JAR and rename it. --> ! <dependency> ! <groupId>ctc</groupId> ! <artifactId>utils</artifactId> ! <jar>ctc_utils-5-4-2005.jar</jar> ! <url>http://www.cutthecrap.biz/downloads/ctc_utils.zip</url> ! </dependency> ! ! <!-- --> ! <!-- Jini --> ! <!-- --> ! <dependency> ! <groupId>jini</groupId> ! <artifactId>jini-core</artifactId> ! <version>2.1</version> ! <url>http://www.jini.org/wiki/Category:Jini_Starter_Kit</url> ! </dependency> ! <dependency> ! <groupId>jini</groupId> ! <artifactId>jini-ext</artifactId> ! <version>2.1</version> ! <url>http://www.jini.org/wiki/Category:Jini_Starter_Kit</url> ! </dependency> ! <dependency> ! <groupId>jini</groupId> ! <artifactId>reggie</artifactId> ! <version>2.1</version> ! <url>http://www.jini.org/wiki/Category:Jini_Starter_Kit</url> ! </dependency> ! <dependency> ! <groupId>jini</groupId> ! <artifactId>sun-util</artifactId> ! <version>2.1</version> ! <url>http://www.jini.org/wiki/Category:Jini_Starter_Kit</url> ! </dependency> ! <dependency> ! <groupId>jini</groupId> ! <artifactId>start</artifactId> ! <version>2.1</version> ! <url>http://www.jini.org/wiki/Category:Jini_Starter_Kit</url> ! </dependency> ! ! <!-- --> ! <!-- ICU --> ! <!-- --> ! <!-- Unicode support (compressed sort keys). --> ! <dependency> ! <groupId>icu</groupId> ! <artifactId>icu4j</artifactId> ! <version>3_6</version> ! <url>http://sourceforge.net/projects/icu/</url> </dependency> *************** *** 182,186 **** <includes> ! <include>org/CognitiveWeb/dbcache/TestAll.java</include> </includes> --- 440,444 ---- <includes> ! <include>com/bigdata/TestAll.java</include> </includes> |
|
From: Bryan T. <tho...@us...> - 2007-04-15 18:17:09
|
Update of /cvsroot/cweb/bigdata/src/java/com/bigdata/btree In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv6887/src/java/com/bigdata/btree Modified Files: IndexSegmentPlan.java Log Message: Preparing bigdata for license change. POM may now be used to do a maven compile. Jini JARs were renamed to include the version (2.1). Index: IndexSegmentPlan.java =================================================================== RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/btree/IndexSegmentPlan.java,v retrieving revision 1.1 retrieving revision 1.2 diff -C2 -d -r1.1 -r1.2 *** IndexSegmentPlan.java 13 Apr 2007 15:04:12 -0000 1.1 --- IndexSegmentPlan.java 15 Apr 2007 18:17:05 -0000 1.2 *************** *** 1,7 **** package com.bigdata.btree; - import com.bigdata.btree.TestIndexSegmentBuilderWithSmallTree; - import com.bigdata.btree.TestIndexSegmentPlan; - /** * A plan for building a B+-Tree based on an input branching factor and #of --- 1,4 ---- |
|
From: Bryan T. <tho...@us...> - 2007-04-15 18:17:09
|
Update of /cvsroot/cweb/bigdata/src/test/com/bigdata/service In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv6887/src/test/com/bigdata/service Removed Files: TestPersistentJiniService.java Log Message: Preparing bigdata for license change. POM may now be used to do a maven compile. Jini JARs were renamed to include the version (2.1). --- TestPersistentJiniService.java DELETED --- |
|
From: Bryan T. <tho...@us...> - 2007-04-15 18:17:08
|
Update of /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv6887/src/java/com/bigdata/scaleup Modified Files: PartitionedIndexView.java Log Message: Preparing bigdata for license change. POM may now be used to do a maven compile. Jini JARs were renamed to include the version (2.1). Index: PartitionedIndexView.java =================================================================== RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/scaleup/PartitionedIndexView.java,v retrieving revision 1.4 retrieving revision 1.5 diff -C2 -d -r1.4 -r1.5 *** PartitionedIndexView.java 13 Apr 2007 15:04:24 -0000 1.4 --- PartitionedIndexView.java 15 Apr 2007 18:17:05 -0000 1.5 *************** *** 95,98 **** --- 95,103 ---- * @version $Id$ * + * @todo add a restart safe data counter for each index partition that may be + * used to assign 64-bit unique identifiers in conjunction with the 32-bit + * unique (within index) partition identifier assigned by the + * {@link MetadataIndex} to each index partition. + * * @todo add a restart safe data structure tracking each partition for which * writes have been absorbed on the {@link Journal}. this could be just |
|
From: Bryan T. <tho...@us...> - 2007-04-15 18:17:08
|
Update of /cvsroot/cweb/bigdata/lib In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv6887/lib Added Files: sun-util-2.1.jar jini-ext-2.1.jar start-2.1.jar jini-core-2.1.jar reggie-2.1.jar Removed Files: sun-util.jar start.jar jini-core.jar startnow.jar jini-ext.jar reggie.jar Log Message: Preparing bigdata for license change. POM may now be used to do a maven compile. Jini JARs were renamed to include the version (2.1). --- NEW FILE: sun-util-2.1.jar --- (This appears to be a binary file; contents omitted.) --- start.jar DELETED --- --- NEW FILE: reggie-2.1.jar --- (This appears to be a binary file; contents omitted.) --- jini-core.jar DELETED --- --- NEW FILE: jini-ext-2.1.jar --- (This appears to be a binary file; contents omitted.) --- sun-util.jar DELETED --- --- reggie.jar DELETED --- --- startnow.jar DELETED --- --- NEW FILE: jini-core-2.1.jar --- (This appears to be a binary file; contents omitted.) --- NEW FILE: start-2.1.jar --- (This appears to be a binary file; contents omitted.) --- jini-ext.jar DELETED --- |
|
From: Bryan T. <tho...@us...> - 2007-04-15 18:16:14
|
Update of /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv6871/src/java/com/bigdata/rdf Modified Files: TripleStore.java Log Message: Preparing bigdata for license change. POM may now be used to do a maven compile. Jini JARs were renamed to include the version (2.1). Index: TripleStore.java =================================================================== RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/TripleStore.java,v retrieving revision 1.29 retrieving revision 1.30 diff -C2 -d -r1.29 -r1.30 *** TripleStore.java 14 Apr 2007 13:33:52 -0000 1.29 --- TripleStore.java 15 Apr 2007 18:15:55 -0000 1.30 *************** *** 138,145 **** * whether or not we need to abort a transaction in this case. * * @todo Refactor to use a delegation mechanism so that we can run with or * without partitioned indices? (All you have to do now is change the ! * class that is being extended from Journal to MasterJournal and ! * handle some different initialization properties.) * * @todo the only added cost for a quad store is the additional statement --- 138,155 ---- * whether or not we need to abort a transaction in this case. * + * @todo bnodes do not need to be store in the terms or ids indices if we + * presume that an unknown identifier is a bnode. however, we still need + * to ensure that bnode identifiers are distinct or the same when and + * where appropriate, so we need to assign identifiers to bnodes in a + * restart-safe manner even if we "forget" the term-id mapping. + * + * @todo modify the term identifier assignment mechanism to be compatible with + * the scale-out index partitions (32-bit unique within index partition + * identified plus a restart-safe counter for each index partition). + * * @todo Refactor to use a delegation mechanism so that we can run with or * without partitioned indices? (All you have to do now is change the ! * class that is being extended from Journal to MasterJournal and handle ! * some different initialization properties.) * * @todo the only added cost for a quad store is the additional statement |
|
From: Bryan T. <tho...@us...> - 2007-04-15 17:11:56
|
Update of /cvsroot/cweb/bigdata/src/architecture In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv7557/src/architecture Removed Files: bigdata object journal.ppt Log Message: Removed original design document for the object store since it is now stale. --- bigdata object journal.ppt DELETED --- |
|
From: Bryan T. <tho...@us...> - 2007-04-14 14:39:22
|
Update of /cvsroot/cweb/bigdata-rdf In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv6515 Added Files: LICENSE.txt Log Message: added license file. --- NEW FILE: LICENSE.txt --- The Notice below must appear in each file of the Source Code of any copy you distribute of the Licensed Product. Contributors to any Modifications may add their own copyright notices to identify their own contributions. License: The contents of this file are subject to the CognitiveWeb Open Source License Version 1.1 (the License). You may not copy or use this file, in either source code or executable form, except in compliance with the License. You may obtain a copy of the License from http://www.CognitiveWeb.org/legal/license/ Software distributed under the License is distributed on an AS IS basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License. Copyrights: Portions created by or assigned to CognitiveWeb are Copyright (c) 2003-2003 CognitiveWeb. All Rights Reserved. Contact information for CognitiveWeb is available at http://www.CognitiveWeb.org Portions Copyright (c) 2002-2003 Bryan Thompson. Acknowledgements: Special thanks to the developers of the Jabber Open Source License 1.0 (JOSL), from which this License was derived. This License contains terms that differ from JOSL. Special thanks to the CognitiveWeb Open Source Contributors for their suggestions and support of the Cognitive Web. Modifications: |
|
From: Bryan T. <tho...@us...> - 2007-04-14 13:50:02
|
Update of /cvsroot/cweb/bigdata-rdf/src/test/com/bigdata/rdf/inf In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv20481/src/test/com/bigdata/rdf/inf Modified Files: TestFullForwardClosure.java Log Message: Added code (commented out) to load the word net schema + nouns data set in the full forward closure test. Index: TestFullForwardClosure.java =================================================================== RCS file: /cvsroot/cweb/bigdata-rdf/src/test/com/bigdata/rdf/inf/TestFullForwardClosure.java,v retrieving revision 1.7 retrieving revision 1.8 diff -C2 -d -r1.7 -r1.8 *** TestFullForwardClosure.java 14 Apr 2007 13:33:49 -0000 1.7 --- TestFullForwardClosure.java 14 Apr 2007 13:49:58 -0000 1.8 *************** *** 83,89 **** * hand-crafted data sets to test the rule implementations. */ store.loadData(new File("data/alibaba_v41.rdf"),"",false); ! // store.loadData(new File("data/nciOncology.owl"),"",false); store.fullForwardClosure(); --- 83,97 ---- * hand-crafted data sets to test the rule implementations. */ + store.loadData(new File("data/alibaba_v41.rdf"),"",false); ! ! store.loadData(new File("data/nciOncology.owl"),"",false); + /* + * Wordnet schema + nouns (two source files). + */ + // store.loadData(new File("data/wordnet-20000620.rdfs"), "", false); + // store.loadData(new File("data/wordnet_nouns-20010201.rdf"), "", false); + store.fullForwardClosure(); |
|
From: Bryan T. <tho...@us...> - 2007-04-14 13:50:02
|
Update of /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/sail In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv20481/src/java/com/bigdata/rdf/sail Modified Files: SimpleRdfRepository.java Log Message: Added code (commented out) to load the word net schema + nouns data set in the full forward closure test. Index: SimpleRdfRepository.java =================================================================== RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/sail/SimpleRdfRepository.java,v retrieving revision 1.2 retrieving revision 1.3 diff -C2 -d -r1.2 -r1.3 *** SimpleRdfRepository.java 13 Apr 2007 15:02:34 -0000 1.2 --- SimpleRdfRepository.java 14 Apr 2007 13:49:58 -0000 1.3 *************** *** 366,369 **** --- 366,373 ---- } + /* + * @todo make closure optional? Is closure required by the Sesame test + * suite? Mark entailments in the KB using the btree entry value? + */ tripleStore.fullForwardClosure(); |
|
From: Bryan T. <tho...@us...> - 2007-04-14 13:33:58
|
Update of /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/inf In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv14591/src/java/com/bigdata/rdf/inf Modified Files: AbstractRuleRdf.java InferenceEngine.java Rule.java AbstractRuleRdfs511.java RuleRdf01.java RuleRdfs11.java SPO.java AbstractRuleRdfs2379.java AbstractRuleRdfs68101213.java Added Files: SPOBuffer.java Log Message: Modified RuleRdfs11 to use a binary search to locate possible matches during the self-join. This gives a big speedup on the nciOncology dataset. There is more work to do here. I think that RuleRdfs11 should fix point all by itself in order to compute the transative closure of the subClassOf relationship as quickly as possible (in as few rounds). This the join is handled in the abstract rule shared by the subPropertyOf rule, this will also provide a fast fixed point for that relation. Index: InferenceEngine.java =================================================================== RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/inf/InferenceEngine.java,v retrieving revision 1.13 retrieving revision 1.14 diff -C2 -d -r1.13 -r1.14 *** InferenceEngine.java 13 Apr 2007 20:37:04 -0000 1.13 --- InferenceEngine.java 14 Apr 2007 13:33:51 -0000 1.14 *************** *** 320,323 **** --- 320,329 ---- */ final int BUFFER_SIZE = 100 * Bytes.kilobyte32; + + /* + * Note: Unlike the parser buffer, making statements distinct appears + * to slow things down significantly (2x slower!). + */ + final boolean distinct = false; final Rule[] rules = this.rules; *************** *** 337,341 **** int round = 0; ! /* * This is a buffer that is used to hold entailments so that we can --- 343,352 ---- int round = 0; ! ! /* ! * The temporary store used to accumulate the entailments. ! */ ! TempTripleStore tmpStore = new TempTripleStore(); ! /* * This is a buffer that is used to hold entailments so that we can *************** *** 347,362 **** * tmp store. */ ! final SPO[] buffer = new SPO[BUFFER_SIZE]; ! ! /* ! * The temporary store used to accumulate the entailments. ! */ ! TempTripleStore entailments = new TempTripleStore(); ! Stats totalStats = new Stats(); while (true) { ! final int numEntailmentsBefore = entailments.getStatementCount(); for (int i = 0; i < nrules; i++) { --- 358,368 ---- * tmp store. */ ! final SPOBuffer buffer = new SPOBuffer(tmpStore, BUFFER_SIZE, distinct); ! Stats totalStats = new Stats(); while (true) { ! final int numEntailmentsBefore = tmpStore.getStatementCount(); for (int i = 0; i < nrules; i++) { *************** *** 368,372 **** int nbefore = ruleStats.numComputed; ! rule.apply( ruleStats, buffer, entailments ); int nnew = ruleStats.numComputed - nbefore; --- 374,378 ---- int nbefore = ruleStats.numComputed; ! rule.apply( ruleStats, buffer ); int nnew = ruleStats.numComputed - nbefore; *************** *** 422,426 **** } ! final int numEntailmentsAfter = entailments.getStatementCount(); if ( numEntailmentsBefore == numEntailmentsAfter ) { --- 428,437 ---- } ! /* ! * Flush the statements in the buffer to the temporary store. ! */ ! buffer.flush(); ! ! final int numEntailmentsAfter = tmpStore.getStatementCount(); if ( numEntailmentsBefore == numEntailmentsAfter ) { *************** *** 437,441 **** final long insertStart = System.currentTimeMillis(); ! final int numInserted = transferBTrees(entailments); final long insertTime = System.currentTimeMillis() - insertStart; --- 448,452 ---- final long insertStart = System.currentTimeMillis(); ! final int numInserted = copyStatements(tmpStore); final long insertTime = System.currentTimeMillis() - insertStart; *************** *** 477,487 **** /** ! * Copies the entailments from the temporary store into the main store. * * @param tmpStore * ! * @return The #of entailments inserted into the main store. */ ! private int transferBTrees( TempTripleStore tmpStore ) { int numInserted = 0; --- 488,500 ---- /** ! * Copies the statements from the temporary store into the main store. * * @param tmpStore * ! * @return The #of statements inserted into the main store (the count only ! * reports those statements that were not already in the main ! * store). */ ! public int copyStatements( TempTripleStore tmpStore ) { int numInserted = 0; Index: AbstractRuleRdfs511.java =================================================================== RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/inf/AbstractRuleRdfs511.java,v retrieving revision 1.6 retrieving revision 1.7 diff -C2 -d -r1.6 -r1.7 *** AbstractRuleRdfs511.java 13 Apr 2007 20:37:04 -0000 1.6 --- AbstractRuleRdfs511.java 14 Apr 2007 13:33:51 -0000 1.7 *************** *** 47,53 **** import com.bigdata.rdf.KeyOrder; ! import com.bigdata.rdf.TempTripleStore; ! public class AbstractRuleRdfs511 extends AbstractRuleRdf { --- 47,70 ---- import com.bigdata.rdf.KeyOrder; ! import com.bigdata.rdf.TripleStore; + /** + * Abstract rule for chain triple patterns where the object position in the + * first triple pattern is the same variable as the subject position in the + * second triple pattern and where the predicate is bound to the same constant + * for both triple patterns and also appears in the predicate position in the + * entailed triple. + * + * <pre> + * triple(?u,C,?x) :- + * triple(?u,C,?v), + * triple(?v,C,?x). + * </pre> + * + * where C is a constant. + * + * @author <a href="mailto:tho...@us...">Bryan Thompson</a> + * @version $Id$ + */ public class AbstractRuleRdfs511 extends AbstractRuleRdf { *************** *** 62,66 **** } ! public Stats apply( final Stats stats, SPO[] buffer, TempTripleStore tmpStore ) { final long computeStart = System.currentTimeMillis(); --- 79,83 ---- } ! public Stats apply( final Stats stats, SPOBuffer buffer) { final long computeStart = System.currentTimeMillis(); *************** *** 89,93 **** final SPO[] stmts1 = store.getStatements(store.getPOSIndex(), KeyOrder.POS, pkey, pkey1); ! stats.stmts1 += stmts1.length; --- 106,110 ---- final SPO[] stmts1 = store.getStatements(store.getPOSIndex(), KeyOrder.POS, pkey, pkey1); ! stats.stmts1 += stmts1.length; *************** *** 97,132 **** // a clone of the answer set // SPO[] stmts2 = stmts1.clone(); ! final SPO[] stmts2 = stmts1; ! ! stats.stmts2 += stmts2.length; ! int n = 0; // the simplest n^2 algorithm for (int i = 0; i < stmts1.length; i++) { // printStatement(stmts1[i]); ! for (int j = 0; j < stmts2.length; j++) { ! ! if (stmts1[i].o == stmts2[j].s) { ! ! buffer[n++] = new SPO(stmts1[i].s, p, stmts2[j].o); ! ! if (n == buffer.length) { ! insertStatements(buffer, n, tmpStore); ! n = 0; ! } ! stats.numComputed++; ! } } } - - insertStatements( buffer, n, tmpStore ); stats.computeTime += System.currentTimeMillis() - computeStart; --- 114,160 ---- // a clone of the answer set // SPO[] stmts2 = stmts1.clone(); ! // final SPO[] stmts2 = stmts1; ! // stats.stmts2 += stmts2.length; // the simplest n^2 algorithm for (int i = 0; i < stmts1.length; i++) { + SPO left = stmts1[i]; + // printStatement(stmts1[i]); ! /* ! * Search for the index of the first statement having left.s as its ! * subject. Note that the object is NULL, so this should always ! * return a negative index which we then convert to the insert ! * position. The insert position is the first index at which a ! * matching statement would be found. We then scan statements from ! * that point. As soon as there is no match (and it may be that ! * there is no match even on the first statement tested) we break ! * out of the inner loop and continue with the outer loop. ! */ ! int j = Arrays.binarySearch(stmts1, new SPO(left.o, p, ! TripleStore.NULL), SPOComparator.INSTANCE); ! if (j < 0) { ! // Convert the position to obtain the insertion point. ! j = -j - 1; ! ! } ! ! // process only the stmts with left.s as their subject. ! for (; j < stmts1.length; j++) { ! if (left.o != stmts1[j].s) break; ! ! buffer.add(new SPO(left.s, p, stmts1[j].o)); + stats.numComputed++; + } } stats.computeTime += System.currentTimeMillis() - computeStart; Index: AbstractRuleRdfs2379.java =================================================================== RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/inf/AbstractRuleRdfs2379.java,v retrieving revision 1.5 retrieving revision 1.6 diff -C2 -d -r1.5 -r1.6 *** AbstractRuleRdfs2379.java 13 Apr 2007 20:37:04 -0000 1.5 --- AbstractRuleRdfs2379.java 14 Apr 2007 13:33:52 -0000 1.6 *************** *** 47,51 **** import com.bigdata.rdf.KeyOrder; - import com.bigdata.rdf.TempTripleStore; public abstract class AbstractRuleRdfs2379 extends AbstractRuleRdf { --- 47,50 ---- *************** *** 61,65 **** } ! public Stats apply( final Stats stats, final SPO[] buffer, TempTripleStore tmpStore ) { final long computeStart = System.currentTimeMillis(); --- 60,64 ---- } ! public Stats apply( final Stats stats, final SPOBuffer buffer) { final long computeStart = System.currentTimeMillis(); *************** *** 69,74 **** stats.stmts1 += stmts1.length; - int n = 0; - for (int i = 0; i < stmts1.length; i++) { --- 68,71 ---- *************** *** 79,91 **** for (int j = 0; j < stmts2.length; j++) { ! buffer[n++] = buildStmt3(stmts1[i], stmts2[j]); ! ! if (n == buffer.length) { ! ! insertStatements(buffer, n, tmpStore); ! ! n = 0; ! ! } stats.numComputed++; --- 76,80 ---- for (int j = 0; j < stmts2.length; j++) { ! buffer.add(buildStmt3(stmts1[i], stmts2[j])); stats.numComputed++; *************** *** 95,100 **** } - insertStatements(buffer, n, tmpStore); - stats.computeTime += System.currentTimeMillis() - computeStart; --- 84,87 ---- Index: RuleRdf01.java =================================================================== RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/inf/RuleRdf01.java,v retrieving revision 1.7 retrieving revision 1.8 diff -C2 -d -r1.7 -r1.8 *** RuleRdf01.java 13 Apr 2007 20:37:04 -0000 1.7 --- RuleRdf01.java 14 Apr 2007 13:33:51 -0000 1.8 *************** *** 46,50 **** import com.bigdata.btree.IEntryIterator; import com.bigdata.rdf.KeyOrder; - import com.bigdata.rdf.TempTripleStore; --- 46,49 ---- *************** *** 60,64 **** } ! public Stats apply( final Stats stats, final SPO[] buffer, TempTripleStore btree ) { final long computeStart = System.currentTimeMillis(); --- 59,63 ---- } ! public Stats apply( final Stats stats, final SPOBuffer buffer ) { final long computeStart = System.currentTimeMillis(); *************** *** 66,71 **** long lastP = -1; - int n = 0; - IEntryIterator it = store.getPOSIndex().rangeIterator(null,null); --- 65,68 ---- *************** *** 82,95 **** lastP = stmt.p; ! buffer[n++] = new SPO(stmt.p, store.rdfType.id, ! store.rdfProperty.id); ! ! if (n == buffer.length) { ! ! insertStatements(buffer, n, btree); ! ! n = 0; ! ! } stats.numComputed++; --- 79,84 ---- lastP = stmt.p; ! buffer.add(new SPO(stmt.p, store.rdfType.id, ! store.rdfProperty.id)); stats.numComputed++; *************** *** 99,104 **** } - insertStatements( buffer, n, btree ); - stats.computeTime += System.currentTimeMillis() - computeStart; --- 88,91 ---- Index: SPO.java =================================================================== RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/inf/SPO.java,v retrieving revision 1.2 retrieving revision 1.3 diff -C2 -d -r1.2 -r1.3 *** SPO.java 26 Jan 2007 20:51:25 -0000 1.2 --- SPO.java 14 Apr 2007 13:33:52 -0000 1.3 *************** *** 46,49 **** --- 46,50 ---- import com.bigdata.rdf.KeyOrder; import com.bigdata.rdf.RdfKeyBuilder; + import com.bigdata.rdf.TripleStore; /** *************** *** 129,131 **** --- 130,232 ---- } + /** + * The #of times this SPO is encountered in an {@link SPOBuffer}. + * + * @todo drop this field if making {@link SPO}s distinct in the + * {@link SPOBuffer} does not prove cost effective. + */ + int count = 0; + + // private int hashCode = 0; + // + // /** + // * @todo validate the manner in which we are combining the hash codes for + // * the individual components of the triple (each component uses the + // * same hash code algorithm as {@link Long#hashCode()}). + // */ + // public int hashCode() { + // + // if(hashCode==0) { + // + // // compute and cache. + // hashCode = (int) ((s ^ (s >>> 32)) | (p ^ (p >>> 32)) | (o ^ (o >>> 32))); + // + // } + // + // return hashCode; + // + // } + + /** + * Imposes s:p:o ordering based on termIds. + */ + public int compareTo(Object other) { + + if (other == this) { + + return 0; + + } + + final SPO stmt1 = this; + + final SPO stmt2 = (SPO) other; + + /* + * Note: logic avoids possible overflow of [long] by not computing the + * difference between two longs. + */ + int ret; + + ret = stmt1.code - stmt2.code; + + if (ret == 0) { + + ret = stmt1.s < stmt2.s ? -1 : stmt1.s > stmt2.s ? 1 : 0; + + if (ret == 0) { + + ret = stmt1.p < stmt2.p ? -1 : stmt1.p > stmt2.p ? 1 : 0; + + if (ret == 0) { + + ret = stmt1.o < stmt2.o ? -1 : stmt1.o > stmt2.o ? 1 : 0; + + } + + } + + } + + return ret; + + } + + /** + * True iff the statements are the same object or if they have the code and + * the same same term identifiers assigned for the subject, predicate and + * object positions. + */ + public boolean equals(SPO stmt2) { + + if (stmt2 == this) + return true; + + return this.code == stmt2.code && this.s == stmt2.s + && this.p == stmt2.p && this.o == stmt2.o; + + } + + /** + * Return a representation of the statement using the term identifiers (the + * identifers are NOT resolved to terms). + * + * @see TripleStore#toString(long, long, long) + */ + public String toString() { + + return (""+s+","+p+","+o); + + } + } --- NEW FILE: SPOBuffer.java --- /** The Notice below must appear in each file of the Source Code of any copy you distribute of the Licensed Product. Contributors to any Modifications may add their own copyright notices to identify their own contributions. License: The contents of this file are subject to the CognitiveWeb Open Source License Version 1.1 (the License). You may not copy or use this file, in either source code or executable form, except in compliance with the License. You may obtain a copy of the License from http://www.CognitiveWeb.org/legal/license/ Software distributed under the License is distributed on an AS IS basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License. Copyrights: Portions created by or assigned to CognitiveWeb are Copyright (c) 2003-2003 CognitiveWeb. All Rights Reserved. Contact information for CognitiveWeb is available at http://www.CognitiveWeb.org Portions Copyright (c) 2002-2003 Bryan Thompson. Acknowledgements: Special thanks to the developers of the Jabber Open Source License 1.0 (JOSL), from which this License was derived. This License contains terms that differ from JOSL. Special thanks to the CognitiveWeb Open Source Contributors for their suggestions and support of the Cognitive Web. Modifications: */ /* * Created on Apr 13, 2007 */ package com.bigdata.rdf.inf; import java.util.HashMap; import java.util.Map; import com.bigdata.rdf.TempTripleStore; import com.bigdata.rdf.TripleStore; import com.bigdata.rdf.rio.Buffer; /** * A buffer for distinct {@link SPO}s. * * @author <a href="mailto:tho...@us...">Bryan Thompson</a> * @version $Id$ */ public class SPOBuffer { /** * The array in which the statements are stored. */ final private SPO[] stmts; /** * The #of statements currently in the buffer. */ private int numStmts; /** * Map used to filter out duplicate statements. */ final private Map<SPO,SPO> distinctStmtMap; /** * The backing store into which the statements are added when the buffer * overflows. */ protected final TempTripleStore store; /** * The buffer capacity -or- <code>-1</code> if the {@link Buffer} object * is signaling that no more buffers will be placed onto the queue by the * producer and that the consumer should therefore terminate. */ protected final int capacity; /** * When true only distinct statements are stored in the buffer. */ protected final boolean distinct; /** * Create a buffer. * * @param store * The database into which the terms and statements will be * inserted. * @param capacity * The maximum #of Statements, URIs, Literals, or BNodes that the * buffer can hold. * @param distinct * When true only distinct terms and statements are stored in the * buffer. */ public SPOBuffer(TempTripleStore store, int capacity, boolean distinct) { this.store = store; this.capacity = capacity; this.distinct = distinct; if (capacity == -1) { stmts = null; distinctStmtMap = null; return; } else { stmts = new SPO[capacity]; if (distinct) { distinctStmtMap = new HashMap<SPO, SPO>(capacity); } else { distinctStmtMap = null; } } } /** * Returns true there are no slots remaining in the statements array. Under * those conditions adding another statement to the buffer could cause an * overflow. * * @return True if the buffer might overflow if another statement were * added. */ final private boolean nearCapacity() { if (numStmts + 1 > capacity) return true; return false; } /** * Uniquify a statement. * * @param stmt * * @return Either the statement or the pre-existing statement in the buffer * with the same data. */ protected SPO getDistinctStatement(SPO stmt) { assert distinct == true; SPO existingStmt = distinctStmtMap.get(stmt); if (existingStmt != null) { // return the pre-existing statement. return existingStmt; } else { // put the new statement in the map. if (distinctStmtMap.put(stmt, stmt) != null) { throw new AssertionError(); } // return the new statement. return stmt; } } /** * Flush any buffer statements to the backing store. */ public void flush() { if (numStmts > 0) overflow(); } private void overflow() { /* * batch insert statements into the store. */ store.addStatements(stmts, numStmts); /* * reset the buffer. */ numStmts = 0; if(distinctStmtMap!=null) { distinctStmtMap.clear(); } } /** * Adds the statement into the buffer. When the buffer is * {@link #nearCapacity()} the statements in the buffer will be flushed into * the backing store. * * @param stmt * The statement. * * @see #nearCapacity() */ public void add( SPO stmt ) { if(nearCapacity()) { overflow(); } if(distinct) { SPO tmp = getDistinctStatement(stmt); if(tmp.count++ == 0){ stmts[numStmts++] = tmp; } } else { stmts[numStmts++] = stmt; } } /** * Dumps the state of the buffer on {@link System#err}. * * @param store * The terms in the statements are resolved against this store. */ public void dump(InferenceEngine store) { System.err.println("capacity="+capacity); System.err.println("numStmts="+numStmts); if(distinct) { System.err.println("#distinct="+distinctStmtMap.size()); } for (int i = 0; i < numStmts; i++) { SPO stmt = stmts[i]; System.err.println("#" + i + "\t" + store.toString(stmt.s, stmt.p, stmt.o)); } } } Index: Rule.java =================================================================== RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/inf/Rule.java,v retrieving revision 1.5 retrieving revision 1.6 diff -C2 -d -r1.5 -r1.6 *** Rule.java 13 Apr 2007 20:37:04 -0000 1.5 --- Rule.java 14 Apr 2007 13:33:51 -0000 1.6 *************** *** 44,47 **** --- 44,48 ---- package com.bigdata.rdf.inf; + import com.bigdata.btree.BTree; import com.bigdata.rdf.TempTripleStore; *************** *** 102,105 **** --- 103,110 ---- * Apply the rule, creating entailments that are inserted into the temporary * store. + * <p> + * Note: the {@link BTree} class is NOT safe for concurrent modification + * under traversal so implementations of this method need to buffer the + * statements that they will insert. See {@link SPOBuffer} * * @param stats *************** *** 107,113 **** * @param buffer * Used to buffer entailments so that we can perform batch btree ! * operations. ! * @param tmpStore ! * The temporary store into which the entailments are placed. * * @return The statistics object. --- 112,117 ---- * @param buffer * Used to buffer entailments so that we can perform batch btree ! * operations. Entailments are batch inserted into the backing ! * store (for the buffer) when the buffer overflows. * * @return The statistics object. *************** *** 120,128 **** * operation variant for insert. * - * @todo the btree class is NOT safe for concurrent modification under - * traversal so implementations of this method need to bufferQueue the - * statements that they will insert. */ ! public abstract Stats apply( final Stats stats, final SPO[] buffer, TempTripleStore tmpStore ); /** --- 124,129 ---- * operation variant for insert. * */ ! public abstract Stats apply( final Stats stats, final SPOBuffer buffer ); /** Index: AbstractRuleRdf.java =================================================================== RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/inf/AbstractRuleRdf.java,v retrieving revision 1.7 retrieving revision 1.8 diff -C2 -d -r1.7 -r1.8 *** AbstractRuleRdf.java 13 Apr 2007 20:37:04 -0000 1.7 --- AbstractRuleRdf.java 14 Apr 2007 13:33:51 -0000 1.8 *************** *** 44,53 **** package com.bigdata.rdf.inf; - import java.util.Arrays; - import org.openrdf.model.URI; import com.bigdata.btree.IIndex; - import com.bigdata.rdf.TempTripleStore; public abstract class AbstractRuleRdf extends Rule { --- 44,50 ---- *************** *** 59,114 **** } - /** - * Copies the entailments from the array into the {@link TempTripleStore}. - * - * @param stmts - * The source statements. - * - * @param n - * The #of statements in the buffer. - * - * @param store - * The target store. - */ - protected void insertStatements(SPO[] stmts, int n, TempTripleStore store) { - - // deal with the SPO index - IIndex spo = store.getSPOIndex(); - Arrays.sort(stmts,0,n,SPOComparator.INSTANCE); - for ( int i = 0; i < n; i++ ) { - byte[] key = store.keyBuilder.statement2Key - ( stmts[i].s, stmts[i].p, stmts[i].o - ); - if ( !spo.contains(key) ) { - spo.insert(key, null); - } - } - - // deal with the POS index - IIndex pos = store.getPOSIndex(); - Arrays.sort(stmts,0,n,POSComparator.INSTANCE); - for ( int i = 0; i < n; i++ ) { - byte[] key = store.keyBuilder.statement2Key - ( stmts[i].p, stmts[i].o, stmts[i].s - ); - if ( !pos.contains(key) ) { - pos.insert(key, null); - } - } - - // deal with the OSP index - IIndex osp = store.getOSPIndex(); - Arrays.sort(stmts,0,n,OSPComparator.INSTANCE); - for ( int i = 0; i < n; i++ ) { - byte[] key = store.keyBuilder.statement2Key - ( stmts[i].o, stmts[i].s, stmts[i].p - ); - if ( !osp.contains(key) ) { - osp.insert(key, null); - } - } - - } - // /** // * Copies the statements into the primary store. --- 56,59 ---- *************** *** 174,197 **** // } - protected void printStatement( SPO stmt ) { - - IIndex ndx = store.getIdTermIndex(); - - URI s = (URI) ndx.lookup(store.keyBuilder.id2key(stmt.s)); - - URI p = (URI) ndx.lookup(store.keyBuilder.id2key(stmt.p)); - - URI o = (URI) ndx.lookup(store.keyBuilder.id2key(stmt.o)); - - System.err.println(abbrev(s)+","+abbrev(p)+","+abbrev(o)); - - } - - protected String abbrev( URI uri ) { - - return uri.getURI().substring(uri.getURI().lastIndexOf('#')); - - } - // protected TempTripleStore convert( SPO[] stmts ) { // --- 119,122 ---- Index: RuleRdfs11.java =================================================================== RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/inf/RuleRdfs11.java,v retrieving revision 1.3 retrieving revision 1.4 diff -C2 -d -r1.3 -r1.4 *** RuleRdfs11.java 31 Jan 2007 05:23:20 -0000 1.3 --- RuleRdfs11.java 14 Apr 2007 13:33:51 -0000 1.4 *************** *** 52,56 **** * triple(?u,rdfs:subClassOf,?x) :- * triple(?u,rdfs:subClassOf,?v), ! * triple(?v,rdf:subClassOf,?x). * </pre> */ --- 52,56 ---- * triple(?u,rdfs:subClassOf,?x) :- * triple(?u,rdfs:subClassOf,?v), ! * triple(?v,rdfs:subClassOf,?x). * </pre> */ Index: AbstractRuleRdfs68101213.java =================================================================== RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/inf/AbstractRuleRdfs68101213.java,v retrieving revision 1.7 retrieving revision 1.8 diff -C2 -d -r1.7 -r1.8 *** AbstractRuleRdfs68101213.java 13 Apr 2007 20:37:04 -0000 1.7 --- AbstractRuleRdfs68101213.java 14 Apr 2007 13:33:52 -0000 1.8 *************** *** 46,50 **** import com.bigdata.btree.IEntryIterator; import com.bigdata.rdf.KeyOrder; - import com.bigdata.rdf.TempTripleStore; --- 46,49 ---- *************** *** 61,65 **** } ! public Stats apply( final Stats stats, final SPO[] buffer, TempTripleStore tmpStore ) { final long computeStart = System.currentTimeMillis(); --- 60,64 ---- } ! public Stats apply( final Stats stats, final SPOBuffer buffer) { final long computeStart = System.currentTimeMillis(); *************** *** 90,102 **** long _o = head.o.isVar() ? stmt.s : head.o.id; ! buffer[n++] = new SPO(_s, _p, _o); ! ! if (n == buffer.length) { ! ! insertStatements(buffer, n, tmpStore); ! ! n = 0; ! ! } stats.numComputed++; --- 89,93 ---- long _o = head.o.isVar() ? stmt.s : head.o.id; ! buffer.add( new SPO(_s, _p, _o) ); stats.numComputed++; *************** *** 104,109 **** } - insertStatements( buffer, n, tmpStore ); - stats.computeTime += System.currentTimeMillis() - computeStart; --- 95,98 ---- |
|
From: Bryan T. <tho...@us...> - 2007-04-14 13:33:57
|
Update of /cvsroot/cweb/bigdata-rdf/src/test/com/bigdata/rdf/inf In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv14591/src/test/com/bigdata/rdf/inf Modified Files: TestFullForwardClosure.java TestMagicSets.java AbstractInferenceEngineTestCase.java Added Files: TestRuleRdfs11.java Log Message: Modified RuleRdfs11 to use a binary search to locate possible matches during the self-join. This gives a big speedup on the nciOncology dataset. There is more work to do here. I think that RuleRdfs11 should fix point all by itself in order to compute the transative closure of the subClassOf relationship as quickly as possible (in as few rounds). This the join is handled in the abstract rule shared by the subPropertyOf rule, this will also provide a fast fixed point for that relation. Index: AbstractInferenceEngineTestCase.java =================================================================== RCS file: /cvsroot/cweb/bigdata-rdf/src/test/com/bigdata/rdf/inf/AbstractInferenceEngineTestCase.java,v retrieving revision 1.8 retrieving revision 1.9 diff -C2 -d -r1.8 -r1.9 *** AbstractInferenceEngineTestCase.java 22 Mar 2007 21:11:29 -0000 1.8 --- AbstractInferenceEngineTestCase.java 14 Apr 2007 13:33:49 -0000 1.9 *************** *** 85,94 **** properties = super.getProperties(); ! // properties.setProperty(Options.BUFFER_MODE, BufferMode.Transient ! // .toString()); ! properties.setProperty(Options.BUFFER_MODE, getBufferMode().toString()); ! // properties.setProperty(Options.BUFFER_MODE, BufferMode.Disk ! // .toString()); ! // properties.setProperty(Options.SLOT_SIZE, ""+Bytes.kilobyte32); if(properties.getProperty(Options.FILE)==null) { properties.setProperty(Options.FILE, getName()+".jnl"); --- 85,94 ---- properties = super.getProperties(); ! if(properties.getProperty(Options.BUFFER_MODE)==null) { ! ! // override if not specified. ! properties.setProperty(Options.BUFFER_MODE, getBufferMode().toString()); ! ! } if(properties.getProperty(Options.FILE)==null) { properties.setProperty(Options.FILE, getName()+".jnl"); *************** *** 97,101 **** properties.setProperty(Options.BASENAME, getName()); } - // properties.setProperty(Options.INITIAL_EXTENT,""+getInitialExtent()); } --- 97,100 ---- *************** *** 118,122 **** protected BufferMode getBufferMode() { ! return BufferMode.Direct; } --- 117,121 ---- protected BufferMode getBufferMode() { ! return BufferMode.Transient; } Index: TestFullForwardClosure.java =================================================================== RCS file: /cvsroot/cweb/bigdata-rdf/src/test/com/bigdata/rdf/inf/TestFullForwardClosure.java,v retrieving revision 1.6 retrieving revision 1.7 diff -C2 -d -r1.6 -r1.7 *** TestFullForwardClosure.java 13 Apr 2007 20:37:04 -0000 1.6 --- TestFullForwardClosure.java 14 Apr 2007 13:33:49 -0000 1.7 *************** *** 71,75 **** super(name); } ! /** * Test of full forward closure. --- 71,75 ---- super(name); } ! /** * Test of full forward closure. *************** *** 83,88 **** * hand-crafted data sets to test the rule implementations. */ ! // store.loadData(new File("data/alibaba_v41.rdf"),"",false); ! store.loadData(new File("data/nciOncology.owl"),"",false); store.fullForwardClosure(); --- 83,88 ---- * hand-crafted data sets to test the rule implementations. */ ! store.loadData(new File("data/alibaba_v41.rdf"),"",false); ! // store.loadData(new File("data/nciOncology.owl"),"",false); store.fullForwardClosure(); --- NEW FILE: TestRuleRdfs11.java --- /** The Notice below must appear in each file of the Source Code of any copy you distribute of the Licensed Product. Contributors to any Modifications may add their own copyright notices to identify their own contributions. License: The contents of this file are subject to the CognitiveWeb Open Source License Version 1.1 (the License). You may not copy or use this file, in either source code or executable form, except in compliance with the License. You may obtain a copy of the License from http://www.CognitiveWeb.org/legal/license/ Software distributed under the License is distributed on an AS IS basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the specific language governing rights and limitations under the License. Copyrights: Portions created by or assigned to CognitiveWeb are Copyright (c) 2003-2003 CognitiveWeb. All Rights Reserved. Contact information for CognitiveWeb is available at http://www.CognitiveWeb.org Portions Copyright (c) 2002-2003 Bryan Thompson. Acknowledgements: Special thanks to the developers of the Jabber Open Source License 1.0 (JOSL), from which this License was derived. This License contains terms that differ from JOSL. Special thanks to the CognitiveWeb Open Source Contributors for their suggestions and support of the Cognitive Web. Modifications: */ /* * Created on Apr 13, 2007 */ package com.bigdata.rdf.inf; import org.openrdf.model.URI; import org.openrdf.vocabulary.RDFS; import com.bigdata.rdf.TempTripleStore; import com.bigdata.rdf.inf.Rule.Stats; import com.bigdata.rdf.model.OptimizedValueFactory._URI; /** * @see RuleRdfs11 * * @author <a href="mailto:tho...@us...">Bryan Thompson</a> * @version $Id$ */ public class TestRuleRdfs11 extends AbstractInferenceEngineTestCase { /** * */ public TestRuleRdfs11() { } /** * @param name */ public TestRuleRdfs11(String name) { super(name); } public void test_rdfs11() { URI A = new _URI("http://www.foo.org/A"); URI B = new _URI("http://www.foo.org/B"); URI C = new _URI("http://www.foo.org/C"); URI rdfsSubClassOf = new _URI(RDFS.SUBCLASSOF); store.addStatement(A, rdfsSubClassOf, B); store.addStatement(B, rdfsSubClassOf, C); assertTrue(store.containsStatement(A, rdfsSubClassOf, B)); assertTrue(store.containsStatement(B, rdfsSubClassOf, C)); assertFalse(store.containsStatement(A, rdfsSubClassOf, C)); // store.fullForwardClosure(); Stats stats = new Stats(); TempTripleStore tmpStore = new TempTripleStore(); final int capacity = 10; final boolean distinct = false; SPOBuffer buffer = new SPOBuffer(tmpStore,capacity,distinct); // apply the rule. store.rdfs11.apply(stats, buffer); // dump entailments on the console. buffer.dump(store); assertEquals("numComputed",stats.numComputed,1); // flush entailments into the temporary store. buffer.flush(); /* * transfer the entailments from the temporary store to the primary * store. */ assertEquals("#copied",1,store.copyStatements(tmpStore)); /* * validate the state of the primary store. */ assertTrue(store.containsStatement(A, rdfsSubClassOf, B)); assertTrue(store.containsStatement(B, rdfsSubClassOf, C)); assertTrue(store.containsStatement(A, rdfsSubClassOf, C)); } } Index: TestMagicSets.java =================================================================== RCS file: /cvsroot/cweb/bigdata-rdf/src/test/com/bigdata/rdf/inf/TestMagicSets.java,v retrieving revision 1.8 retrieving revision 1.9 diff -C2 -d -r1.8 -r1.9 *** TestMagicSets.java 13 Apr 2007 20:37:04 -0000 1.8 --- TestMagicSets.java 14 Apr 2007 13:33:49 -0000 1.9 *************** *** 161,169 **** * Applies the base rule iff the {@link Magic} is matched. */ ! public Rule.Stats apply( Stats stats, SPO[] buffer, TempTripleStore tmpStore ) { if(match()) { ! return rule.apply( stats, buffer, tmpStore ); } --- 161,169 ---- * Applies the base rule iff the {@link Magic} is matched. */ ! public Rule.Stats apply( Stats stats, SPOBuffer buffer) { if(match()) { ! return rule.apply( stats, buffer ); } |
|
From: Bryan T. <tho...@us...> - 2007-04-14 13:33:55
|
Update of /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv14591/src/java/com/bigdata/rdf Modified Files: TempTripleStore.java TripleStore.java Log Message: Modified RuleRdfs11 to use a binary search to locate possible matches during the self-join. This gives a big speedup on the nciOncology dataset. There is more work to do here. I think that RuleRdfs11 should fix point all by itself in order to compute the transative closure of the subClassOf relationship as quickly as possible (in as few rounds). This the join is handled in the abstract rule shared by the subPropertyOf rule, this will also provide a fast fixed point for that relation. Index: TempTripleStore.java =================================================================== RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/TempTripleStore.java,v retrieving revision 1.4 retrieving revision 1.5 diff -C2 -d -r1.4 -r1.5 *** TempTripleStore.java 13 Apr 2007 15:02:34 -0000 1.4 --- TempTripleStore.java 14 Apr 2007 13:33:52 -0000 1.5 *************** *** 48,51 **** --- 48,52 ---- package com.bigdata.rdf; + import java.util.Arrays; import java.util.Locale; import java.util.UUID; *************** *** 59,63 **** --- 60,67 ---- import com.bigdata.journal.TemporaryStore; import com.bigdata.rawstore.Bytes; + import com.bigdata.rdf.inf.OSPComparator; + import com.bigdata.rdf.inf.POSComparator; import com.bigdata.rdf.inf.SPO; + import com.bigdata.rdf.inf.SPOComparator; import com.bigdata.rdf.model.OptimizedValueFactory._Statement; import com.bigdata.rdf.serializers.StatementSerializer; *************** *** 232,235 **** --- 236,288 ---- /** + * Copies the entailments from the array into the {@link TempTripleStore}. + * + * @param stmts + * The source statements. + * + * @param n + * The #of statements in the buffer. + */ + public void addStatements(SPO[] stmts, int n ) { + + // deal with the SPO index + IIndex spo = getSPOIndex(); + Arrays.sort(stmts,0,n,SPOComparator.INSTANCE); + for ( int i = 0; i < n; i++ ) { + byte[] key = keyBuilder.statement2Key + ( stmts[i].s, stmts[i].p, stmts[i].o + ); + if ( !spo.contains(key) ) { + spo.insert(key, null); + } + } + + // deal with the POS index + IIndex pos = getPOSIndex(); + Arrays.sort(stmts,0,n,POSComparator.INSTANCE); + for ( int i = 0; i < n; i++ ) { + byte[] key = keyBuilder.statement2Key + ( stmts[i].p, stmts[i].o, stmts[i].s + ); + if ( !pos.contains(key) ) { + pos.insert(key, null); + } + } + + // deal with the OSP index + IIndex osp = getOSPIndex(); + Arrays.sort(stmts,0,n,OSPComparator.INSTANCE); + for ( int i = 0; i < n; i++ ) { + byte[] key = keyBuilder.statement2Key + ( stmts[i].o, stmts[i].s, stmts[i].p + ); + if ( !osp.contains(key) ) { + osp.insert(key, null); + } + } + + } + + /** * Writes out some usage details on System.err. */ Index: TripleStore.java =================================================================== RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/TripleStore.java,v retrieving revision 1.28 retrieving revision 1.29 diff -C2 -d -r1.28 -r1.29 *** TripleStore.java 13 Apr 2007 15:02:34 -0000 1.28 --- TripleStore.java 14 Apr 2007 13:33:52 -0000 1.29 *************** *** 66,74 **** import com.bigdata.btree.BTree; - import com.bigdata.btree.Errors; - import com.bigdata.btree.IBatchOp; import com.bigdata.btree.IEntryIterator; import com.bigdata.btree.IIndex; - import com.bigdata.btree.ISimpleBTree; import com.bigdata.btree.KeyBuilder; import com.bigdata.journal.ICommitRecord; --- 66,71 ---- *************** *** 92,96 **** import com.bigdata.rdf.serializers.StatementSerializer; import com.bigdata.rdf.serializers.TermIdSerializer; - import com.bigdata.scaleup.MasterJournal; import com.bigdata.scaleup.PartitionedIndexView; import com.bigdata.scaleup.SlaveJournal; --- 89,92 ---- *************** *** 554,557 **** --- 550,583 ---- /** + * Externalizes a statement using an appreviated syntax. + */ + public String toString( long s, long p, long o ) { + + IIndex ndx = getIdTermIndex(); + + URI s1 = (URI) ndx.lookup(keyBuilder.id2key(s)); + + URI p1 = (URI) ndx.lookup(keyBuilder.id2key(p)); + + URI o1 = (URI) ndx.lookup(keyBuilder.id2key(o)); + + return ("< "+abbrev(s1)+", "+abbrev(p1)+", "+abbrev(o1)+" >"); + + } + + // @todo substitute in well know namespaces (rdf, rdfs, etc). + private String abbrev( URI uri ) { + + String t = uri.getURI(); + + int index = t.lastIndexOf('#'); + + if(index==-1) return t; + + return t.substring(index); + + } + + /** * Return true if the statement exists in the store (non-batch API). * |
|
From: Bryan T. <tho...@us...> - 2007-04-13 20:37:11
|
Update of /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/inf In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv5139/src/java/com/bigdata/rdf/inf Modified Files: InferenceEngine.java AbstractRuleRdf.java Rule.java RuleRdfs09.java AbstractRuleRdfs511.java AbstractRuleRdfs2379.java AbstractRuleRdfs68101213.java RuleRdf01.java RuleRdfs07.java Log Message: Fussed with the inference engine and got a modest performance boost. Rdfs5/11 needs to be rewritten to be smart about exploiting the bufferes triples, e.g., using a binary search on the buffer to locate matches in the self-join. Index: RuleRdfs07.java =================================================================== RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/inf/RuleRdfs07.java,v retrieving revision 1.1 retrieving revision 1.2 diff -C2 -d -r1.1 -r1.2 *** RuleRdfs07.java 31 Jan 2007 15:52:54 -0000 1.1 --- RuleRdfs07.java 13 Apr 2007 20:37:04 -0000 1.2 *************** *** 44,49 **** package com.bigdata.rdf.inf; ! ! public class RuleRdfs07 extends AbstractRuleRdfs2379 { --- 44,52 ---- package com.bigdata.rdf.inf; ! /** ! * <pre> ! * <a rdfs:subPropertyOf b> AND <u a y> IMPLIES <u b y> ! * </pre> ! */ public class RuleRdfs07 extends AbstractRuleRdfs2379 { *************** *** 59,63 **** --- 62,68 ---- protected SPO buildStmt3( SPO stmt1, SPO stmt2 ) { + return new SPO( stmt2.s, stmt1.o, stmt2.o ); + } Index: AbstractRuleRdfs511.java =================================================================== RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/inf/AbstractRuleRdfs511.java,v retrieving revision 1.5 retrieving revision 1.6 diff -C2 -d -r1.5 -r1.6 *** AbstractRuleRdfs511.java 12 Apr 2007 23:59:21 -0000 1.5 --- AbstractRuleRdfs511.java 13 Apr 2007 20:37:04 -0000 1.6 *************** *** 45,53 **** import java.util.Arrays; - import java.util.Vector; import com.bigdata.rdf.KeyOrder; import com.bigdata.rdf.TempTripleStore; - import com.bigdata.rdf.inf.Rule.Stats; --- 45,51 ---- *************** *** 64,72 **** } ! public Stats apply( TempTripleStore entailments ) { ! ! Stats stats = new Stats(); ! long computeStart = System.currentTimeMillis(); // the predicate is fixed for all parts of the rule. --- 62,68 ---- } ! public Stats apply( final Stats stats, SPO[] buffer, TempTripleStore tmpStore ) { ! final long computeStart = System.currentTimeMillis(); // the predicate is fixed for all parts of the rule. *************** *** 91,121 **** // in POS order. ! SPO[] stmts1 = store.getStatements(store.getPOSIndex(), KeyOrder.POS, pkey, pkey1); // in SPO order. Arrays.sort(stmts1,SPOComparator.INSTANCE); // a clone of the answer set ! SPO[] stmts2 = stmts1.clone(); - Vector<SPO> stmts3 = new Vector<SPO>(BUFFER_SIZE); // the simplest n^2 algorithm ! for( int i = 0; i < stmts1.length; i++ ) { // printStatement(stmts1[i]); ! for ( int j = 0; j < stmts2.length; j++ ) { ! if ( stmts1[i].o == stmts2[j].s ) { ! if (stmts3.size() == BUFFER_SIZE) { ! dumpBuffer ! ( stmts3.toArray( new SPO[stmts3.size()] ), ! entailments ! ); ! stmts3.clear(); } ! stmts3.add( new SPO(stmts1[i].s, p, stmts2[j].o) ); stats.numComputed++; } } } ! if(debug)dumpBuffer( stmts3.toArray( new SPO[stmts3.size()] ), entailments ); ! stats.computeTime = System.currentTimeMillis() - computeStart; return stats; --- 87,134 ---- // in POS order. ! final SPO[] stmts1 = store.getStatements(store.getPOSIndex(), ! KeyOrder.POS, pkey, pkey1); ! ! stats.stmts1 += stmts1.length; ! // in SPO order. Arrays.sort(stmts1,SPOComparator.INSTANCE); + // a clone of the answer set ! // SPO[] stmts2 = stmts1.clone(); ! final SPO[] stmts2 = stmts1; ! ! stats.stmts2 += stmts2.length; ! ! int n = 0; // the simplest n^2 algorithm ! for (int i = 0; i < stmts1.length; i++) { ! // printStatement(stmts1[i]); ! ! for (int j = 0; j < stmts2.length; j++) { ! ! if (stmts1[i].o == stmts2[j].s) { ! ! buffer[n++] = new SPO(stmts1[i].s, p, stmts2[j].o); ! ! if (n == buffer.length) { ! ! insertStatements(buffer, n, tmpStore); ! ! n = 0; } ! stats.numComputed++; } + } + } ! ! insertStatements( buffer, n, tmpStore ); ! stats.computeTime += System.currentTimeMillis() - computeStart; return stats; Index: AbstractRuleRdfs2379.java =================================================================== RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/inf/AbstractRuleRdfs2379.java,v retrieving revision 1.4 retrieving revision 1.5 diff -C2 -d -r1.4 -r1.5 *** AbstractRuleRdfs2379.java 12 Apr 2007 23:59:21 -0000 1.4 --- AbstractRuleRdfs2379.java 13 Apr 2007 20:37:04 -0000 1.5 *************** *** 45,54 **** import java.util.Arrays; - import java.util.Vector; import com.bigdata.rdf.KeyOrder; import com.bigdata.rdf.TempTripleStore; - public abstract class AbstractRuleRdfs2379 extends AbstractRuleRdf { --- 45,52 ---- *************** *** 63,93 **** } ! public Stats apply( TempTripleStore entailments ) { ! ! Stats stats = new Stats(); ! long computeStart = System.currentTimeMillis(); - // create a place to hold the entailments - Vector<SPO> stmts3 = new Vector<SPO>(BUFFER_SIZE); - SPO[] stmts1 = getStmts1(); ! for ( int i = 0; i < stmts1.length; i++ ) { ! SPO[] stmts2 = getStmts2( stmts1[i] ); ! for ( int j = 0; j < stmts2.length; j++ ) { ! if (stmts3.size() == BUFFER_SIZE) { ! dumpBuffer ! ( stmts3.toArray( new SPO[stmts3.size()] ), ! entailments ! ); ! stmts3.clear(); } ! stmts3.add( buildStmt3( stmts1[i], stmts2[j] ) ); stats.numComputed++; } } - if(debug) dumpBuffer( stmts3.toArray( new SPO[stmts3.size()] ), entailments ); ! stats.computeTime = System.currentTimeMillis() - computeStart; return stats; --- 61,101 ---- } ! public Stats apply( final Stats stats, final SPO[] buffer, TempTripleStore tmpStore ) { ! final long computeStart = System.currentTimeMillis(); SPO[] stmts1 = getStmts1(); ! ! stats.stmts1 += stmts1.length; ! ! int n = 0; ! ! for (int i = 0; i < stmts1.length; i++) { ! ! SPO[] stmts2 = getStmts2(stmts1[i]); ! ! stats.stmts2 += stmts2.length; ! ! for (int j = 0; j < stmts2.length; j++) { ! ! buffer[n++] = buildStmt3(stmts1[i], stmts2[j]); ! ! if (n == buffer.length) { ! ! insertStatements(buffer, n, tmpStore); ! ! n = 0; ! } ! stats.numComputed++; + } + } ! insertStatements(buffer, n, tmpStore); ! ! stats.computeTime += System.currentTimeMillis() - computeStart; return stats; *************** *** 95,111 **** } ! // default behavior is to use POS index to match body[0] and then sort ! // using POSComparator since the default for body[1] is the POS index ! // again (is the sort even necessary?) protected SPO[] getStmts1() { // use the POS index to look up the matches for body[0], the more // constrained triple byte[] fromKey = store.keyBuilder.statement2Key( body[0].p.id, 0, 0 ); byte[] toKey = store.keyBuilder.statement2Key( body[0].p.id+1, 0, 0 ); ! SPO[] stmts1 = store.getStatements(store.getPOSIndex(), KeyOrder.POS, fromKey, toKey); // make sure the statements are in POS order, since we are going to be // doing lookups against the POS index in a moment Arrays.sort(stmts1,POSComparator.INSTANCE); --- 103,126 ---- } ! /** ! * default behavior is to use POS index to match body[0] and then sort using ! * POSComparator since the default for body[1] is the POS index again (is ! * the sort even necessary?) ! */ protected SPO[] getStmts1() { // use the POS index to look up the matches for body[0], the more // constrained triple + byte[] fromKey = store.keyBuilder.statement2Key( body[0].p.id, 0, 0 ); + byte[] toKey = store.keyBuilder.statement2Key( body[0].p.id+1, 0, 0 ); ! ! SPO[] stmts1 = store.getStatements(store.getPOSIndex(), KeyOrder.POS, ! fromKey, toKey); // make sure the statements are in POS order, since we are going to be // doing lookups against the POS index in a moment + Arrays.sort(stmts1,POSComparator.INSTANCE); *************** *** 114,124 **** } ! // default behavior is to join the subject of stmt1 with the predicate ! // of body[1] using the POS index protected SPO[] getStmts2( SPO stmt1 ) { byte[] fromKey = store.keyBuilder.statement2Key(stmt1.s, 0, 0); byte[] toKey = store.keyBuilder.statement2Key(stmt1.s+1, 0, 0); ! return store.getStatements(store.getPOSIndex(), KeyOrder.POS, fromKey, toKey); } --- 129,144 ---- } ! /** ! * default behavior is to join the subject of stmt1 with the predicate of ! * body[1] using the POS index. ! */ protected SPO[] getStmts2( SPO stmt1 ) { byte[] fromKey = store.keyBuilder.statement2Key(stmt1.s, 0, 0); + byte[] toKey = store.keyBuilder.statement2Key(stmt1.s+1, 0, 0); ! ! return store.getStatements(store.getPOSIndex(), KeyOrder.POS, fromKey, ! toKey); } *************** *** 126,128 **** protected abstract SPO buildStmt3( SPO stmt1, SPO stmt2 ); ! } \ No newline at end of file --- 146,148 ---- protected abstract SPO buildStmt3( SPO stmt1, SPO stmt2 ); ! } Index: RuleRdf01.java =================================================================== RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/inf/RuleRdf01.java,v retrieving revision 1.6 retrieving revision 1.7 diff -C2 -d -r1.6 -r1.7 *** RuleRdf01.java 13 Apr 2007 15:02:34 -0000 1.6 --- RuleRdf01.java 13 Apr 2007 20:37:04 -0000 1.7 *************** *** 44,49 **** package com.bigdata.rdf.inf; - import java.util.Vector; - import com.bigdata.btree.IEntryIterator; import com.bigdata.rdf.KeyOrder; --- 44,47 ---- *************** *** 62,75 **** } ! public Stats apply( TempTripleStore btree ) { ! ! Stats stats = new Stats(); ! ! long computeStart = System.currentTimeMillis(); ! Vector<SPO> entailments = new Vector<SPO>(BUFFER_SIZE); long lastP = -1; IEntryIterator it = store.getPOSIndex().rangeIterator(null,null); --- 60,71 ---- } ! public Stats apply( final Stats stats, final SPO[] buffer, TempTripleStore btree ) { ! final long computeStart = System.currentTimeMillis(); long lastP = -1; + int n = 0; + IEntryIterator it = store.getPOSIndex().rangeIterator(null,null); *************** *** 78,97 **** it.next(); ! SPO stmt = ! new SPO(KeyOrder.POS,store.keyBuilder,it.getKey()); if ( stmt.p != lastP ) { lastP = stmt.p; ! if (entailments.size() == BUFFER_SIZE) { ! dumpBuffer ! ( entailments.toArray( new SPO[entailments.size()] ), ! btree ! ); ! entailments.clear(); } ! entailments.add ! ( new SPO(stmt.p, store.rdfType.id, store.rdfProperty.id) ); stats.numComputed++; --- 74,96 ---- it.next(); ! stats.stmts1++; ! ! SPO stmt = new SPO(KeyOrder.POS, store.keyBuilder, it.getKey()); if ( stmt.p != lastP ) { lastP = stmt.p; + + buffer[n++] = new SPO(stmt.p, store.rdfType.id, + store.rdfProperty.id); ! if (n == buffer.length) { ! ! insertStatements(buffer, n, btree); ! ! n = 0; ! } ! stats.numComputed++; *************** *** 99,105 **** } - if(debug)dumpBuffer( entailments.toArray( new SPO[entailments.size()] ), btree ); ! stats.computeTime = System.currentTimeMillis() - computeStart; return stats; --- 98,105 ---- } ! insertStatements( buffer, n, btree ); ! ! stats.computeTime += System.currentTimeMillis() - computeStart; return stats; Index: RuleRdfs09.java =================================================================== RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/inf/RuleRdfs09.java,v retrieving revision 1.3 retrieving revision 1.4 diff -C2 -d -r1.3 -r1.4 *** RuleRdfs09.java 9 Feb 2007 20:18:56 -0000 1.3 --- RuleRdfs09.java 13 Apr 2007 20:37:04 -0000 1.4 *************** *** 47,50 **** --- 47,55 ---- + /** + * <pre> + * <u rdfs:subClassOf x> AND <v rdf:type u> IMPLIES <v rdf:type x> + * </pre> + */ public class RuleRdfs09 extends AbstractRuleRdfs2379 { *************** *** 61,74 **** protected SPO[] getStmts2( SPO stmt1 ) { ! byte[] fromKey = ! store.keyBuilder.statement2Key(store.rdfType.id,stmt1.s,0); ! byte[] toKey = ! store.keyBuilder.statement2Key(store.rdfType.id,stmt1.s+1,0); ! return store.getStatements(store.getPOSIndex(), KeyOrder.POS, fromKey, toKey); } protected SPO buildStmt3( SPO stmt1, SPO stmt2 ) { return new SPO( stmt2.s, store.rdfType.id, stmt1.o ); } --- 66,84 ---- protected SPO[] getStmts2( SPO stmt1 ) { ! byte[] fromKey = store.keyBuilder.statement2Key(store.rdfType.id, ! stmt1.s, 0); ! ! byte[] toKey = store.keyBuilder.statement2Key(store.rdfType.id, ! stmt1.s + 1, 0); ! ! return store.getStatements(store.getPOSIndex(), KeyOrder.POS, fromKey, ! toKey); } protected SPO buildStmt3( SPO stmt1, SPO stmt2 ) { + return new SPO( stmt2.s, store.rdfType.id, stmt1.o ); + } Index: InferenceEngine.java =================================================================== RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/inf/InferenceEngine.java,v retrieving revision 1.12 retrieving revision 1.13 diff -C2 -d -r1.12 -r1.13 *** InferenceEngine.java 13 Apr 2007 15:02:34 -0000 1.12 --- InferenceEngine.java 13 Apr 2007 20:37:04 -0000 1.13 *************** *** 54,60 **** --- 54,62 ---- import com.bigdata.journal.BufferMode; import com.bigdata.journal.Options; + import com.bigdata.rawstore.Bytes; import com.bigdata.rdf.KeyOrder; import com.bigdata.rdf.TempTripleStore; import com.bigdata.rdf.TripleStore; + import com.bigdata.rdf.inf.Rule.Stats; import com.bigdata.rdf.inf.TestMagicSets.MagicRule; import com.bigdata.rdf.model.OptimizedValueFactory._URI; *************** *** 62,66 **** /** * Adds support for RDFS inference. ! * <p> * A fact always has the form: * --- 64,68 ---- /** * Adds support for RDFS inference. ! * <p> * A fact always has the form: * *************** *** 269,286 **** /** ! * Compute the complete forward closure of the store using a ! * set-at-a-time inference strategy. * ! * @todo refactor so that we can close a document that we are loading ! * before it is inserted into the main database. * ! * @todo can the dependency array among the rules be of use to us when ! * we are computing full foward closure as opposed to using magic ! * sets to answer a specific query? */ public void fullForwardClosure() { final Rule[] rules = this.rules; final int nrules = rules.length; --- 271,330 ---- /** ! * Compute the complete forward closure of the store using a set-at-a-time ! * inference strategy. ! * <p> ! * The general approach is a series of rounds in which each rule is applied ! * to all data in turn. The rules directly embody queries that cause only ! * the statements which can trigger the rule to be visited. Since most rules ! * require two antecedents, this typically means that the rules are running ! * two range queries and performing a join operation in order to identify ! * the set of rule firings. Entailments computed in each round are fed back ! * into the source against which the rules can match their preconditions, so ! * derived entailments may be computed in a succession of rounds. The ! * process halts when no new entailments are computed in a given round. ! * <p> * ! * @todo Rules can be computed in parallel using a pool of worker threads. ! * The round ends when the queue of rules to process is empty and all ! * workers are done. * ! * @todo The entailments computed in each round are inserted back into the ! * primary triple store at the end of the round. The purpose of this ! * is to read from a fused view of the triples already in the primary ! * store and those that have been computed by the last application of ! * the rules. This is necessary in order for derived entailments to be ! * computed. However, an alternative approach would be to explicitly ! * read from a fused view of the indices in the temporary store and ! * those in the primary store (or simply combining their iterators in ! * the rules). This could have several advantages and an approach like ! * this is necessary in order to compute entailments at query time ! * that are not to be inserted back into the kb. ! * ! * @todo can the dependency array among the rules be of use to us when we ! * are computing full foward closure as opposed to using magic sets to ! * answer a specific query? ! * ! * @todo The SPO[] buffer might be better off as a more interesting ! * structure that only accepted the distinct statements. This was a ! * big win for the batch oriented parser and would probably eliminate ! * even more duplicates in the context of the inference engine. */ public void fullForwardClosure() { + /* + * @todo configuration paramater. + * + * There is a factor of 2 performance difference for a sample data set + * from a buffer size of one (unordered inserts) to a buffer size of + * 10k. + */ + final int BUFFER_SIZE = 100 * Bytes.kilobyte32; + final Rule[] rules = this.rules; + final long[] timePerRule = new long[rules.length]; + + final int[] entailmentsPerRule = new int[rules.length]; + final int nrules = rules.length; *************** *** 294,320 **** int round = 0; TempTripleStore entailments = new TempTripleStore(); while (true) { ! int numComputed = 0; ! ! long computeTime = 0; ! ! int numEntailmentsBefore = entailments.getStatementCount(); for (int i = 0; i < nrules; i++) { Rule rule = rules[i]; ! Rule.Stats stats = rule.apply( entailments ); ! numComputed += stats.numComputed; ! computeTime += stats.computeTime; } ! int numEntailmentsAfter = entailments.getStatementCount(); if ( numEntailmentsBefore == numEntailmentsAfter ) { --- 338,426 ---- int round = 0; + /* + * This is a buffer that is used to hold entailments so that we can + * insert them into the indices using ordered insert operations (much + * faster than random inserts). The buffer is reused by each rule. The + * rule assumes that the buffer is empty and just keeps a local counter + * of the #of entailments that it has inserted into the buffer. When the + * buffer overflows, those entailments are transfered enmass into the + * tmp store. + */ + final SPO[] buffer = new SPO[BUFFER_SIZE]; + + /* + * The temporary store used to accumulate the entailments. + */ TempTripleStore entailments = new TempTripleStore(); + Stats totalStats = new Stats(); + while (true) { ! final int numEntailmentsBefore = entailments.getStatementCount(); for (int i = 0; i < nrules; i++) { + Stats ruleStats = new Stats(); + Rule rule = rules[i]; ! int nbefore = ruleStats.numComputed; ! rule.apply( ruleStats, buffer, entailments ); ! ! int nnew = ruleStats.numComputed - nbefore; ! // #of statements examined by the rule. ! int nstmts = ruleStats.stmts1 + ruleStats.stmts2; ! ! long elapsed = ruleStats.computeTime; ! ! timePerRule[i] += elapsed; ! ! entailmentsPerRule[i] = ruleStats.numComputed; // Note: already a running sum. ! ! long stmtsPerSec = (nstmts == 0 || elapsed == 0L ? 0 ! : ((long) (((double) nstmts) / ((double) elapsed) * 1000d))); ! ! if (DEBUG||true) { ! log.debug("round# " + round + ", " ! + rule.getClass().getSimpleName() ! + ", entailments=" + nnew + ", #stmts1=" ! + ruleStats.stmts1 + ", #stmts2=" ! + ruleStats.stmts2 + ", #stmtsExaminedPerSec=" ! + stmtsPerSec ! ); ! } ! ! totalStats.numComputed += ruleStats.numComputed; ! ! totalStats.computeTime += ruleStats.computeTime; } + + if(true) { ! /* ! * Show times for each rule so far. ! */ ! System.err.println("rule \tms\t#entms\tentms/ms"); ! ! for(int i=0; i<timePerRule.length; i++) { ! ! System.err.println(rules[i].getClass().getSimpleName() ! + "\t" ! + timePerRule[i] ! + "\t" ! + entailmentsPerRule[i] ! + "\t" ! + (timePerRule[i] == 0 ? 0 : entailmentsPerRule[i] ! / timePerRule[i])); ! ! } ! ! } ! ! final int numEntailmentsAfter = entailments.getStatementCount(); if ( numEntailmentsBefore == numEntailmentsAfter ) { *************** *** 325,339 **** } ! long insertStart = System.currentTimeMillis(); ! ! int numInserted = transferBTrees( entailments ); ! ! long insertTime = System.currentTimeMillis() - insertStart; ! ! if(DEBUG){ StringBuilder debug = new StringBuilder(); debug.append( "round #" ).append( round++ ).append( ": " ); ! debug.append( numComputed ).append( " computed in " ); ! debug.append( computeTime ).append( " millis, " ); debug.append( numInserted ).append( " inserted in " ); debug.append( insertTime ).append( " millis " ); --- 431,449 ---- } ! /* ! * Transfer the entailments into the primary store so that derived ! * entailments may be computed. ! */ ! final long insertStart = System.currentTimeMillis(); ! ! final int numInserted = transferBTrees(entailments); ! ! final long insertTime = System.currentTimeMillis() - insertStart; ! ! if (DEBUG) { StringBuilder debug = new StringBuilder(); debug.append( "round #" ).append( round++ ).append( ": " ); ! debug.append( totalStats.numComputed ).append( " computed in " ); ! debug.append( totalStats.computeTime ).append( " millis, " ); debug.append( numInserted ).append( " inserted in " ); debug.append( insertTime ).append( " millis " ); *************** *** 347,354 **** final int lastStatementCount = getStatementCount(); ! if(INFO) { ! log.info("Closed store in " + elapsed + "ms yeilding " ! + lastStatementCount + " statements total, " + ! (lastStatementCount - firstStatementCount) + " inferences"); } --- 457,475 ---- final int lastStatementCount = getStatementCount(); ! if (INFO) { ! ! final int inferenceCount = lastStatementCount - firstStatementCount; ! ! log.info("Computed closure of store in " ! + elapsed ! + "ms yeilding " ! + lastStatementCount ! + " statements total, " ! + (inferenceCount) ! + " inferences" ! + ", entailmentsPerSec=" ! + ((long) (((double) inferenceCount) ! / ((double) elapsed) * 1000d))); ! } *************** *** 358,370 **** * Copies the entailments from the temporary store into the main store. * ! * @param entailments * * @return The #of entailments inserted into the main store. */ ! private int transferBTrees( TempTripleStore entailments ) { int numInserted = 0; ! IEntryIterator it = entailments.getSPOIndex().rangeIterator(null, null); while (it.hasNext()) { it.next(); --- 479,491 ---- * Copies the entailments from the temporary store into the main store. * ! * @param tmpStore * * @return The #of entailments inserted into the main store. */ ! private int transferBTrees( TempTripleStore tmpStore ) { int numInserted = 0; ! IEntryIterator it = tmpStore.getSPOIndex().rangeIterator(null, null); while (it.hasNext()) { it.next(); *************** *** 376,380 **** } ! it = entailments.getPOSIndex().rangeIterator(null, null); while (it.hasNext()) { it.next(); --- 497,501 ---- } ! it = tmpStore.getPOSIndex().rangeIterator(null, null); while (it.hasNext()) { it.next(); *************** *** 385,389 **** } ! it = entailments.getOSPIndex().rangeIterator(null, null); while (it.hasNext()) { it.next(); --- 506,510 ---- } ! it = tmpStore.getOSPIndex().rangeIterator(null, null); while (it.hasNext()) { it.next(); Index: Rule.java =================================================================== RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/inf/Rule.java,v retrieving revision 1.4 retrieving revision 1.5 diff -C2 -d -r1.4 -r1.5 *** Rule.java 12 Apr 2007 23:59:21 -0000 1.4 --- Rule.java 13 Apr 2007 20:37:04 -0000 1.5 *************** *** 69,74 **** abstract public class Rule { - static protected final boolean debug = false; - /** * The inference engine. --- 69,72 ---- *************** *** 102,118 **** /** ! * Apply the rule to the statement in the store. * ! * @param entailments ! * The temporary triple store used to hold entailments. * ! * @return Statistics related to what the rule did. * ! * @todo support conditional insert in the btree so that we do not have ! * to do a lookup/insert combination. * ! * @todo we could store proofs in the value for a statement or in a ! * proofs index. doing that efficiently would require a ! * concatenation operation variant for insert. * * @todo the btree class is NOT safe for concurrent modification under --- 100,122 ---- /** ! * Apply the rule, creating entailments that are inserted into the temporary ! * store. * ! * @param stats ! * Returns statistics on the rule application as a side effect. ! * @param buffer ! * Used to buffer entailments so that we can perform batch btree ! * operations. ! * @param tmpStore ! * The temporary store into which the entailments are placed. * ! * @return The statistics object. * ! * @todo support conditional insert in the btree so that we do not have to ! * do a lookup/insert combination. * ! * @todo we could store proofs in the value for a statement or in a proofs ! * index. doing that efficiently would require a concatenation ! * operation variant for insert. * * @todo the btree class is NOT safe for concurrent modification under *************** *** 120,125 **** * statements that they will insert. */ ! abstract public Stats apply( TempTripleStore entailments ); ! /** --- 124,128 ---- * statements that they will insert. */ ! public abstract Stats apply( final Stats stats, final SPO[] buffer, TempTripleStore tmpStore ); /** *************** *** 130,138 **** public static class Stats { public int numComputed; long computeTime; } ! } \ No newline at end of file --- 133,164 ---- public static class Stats { + /** + * #of matches for the triple pattern for the first antecedent of the rule. + */ + public int stmts1; + + /** + * #of matches for the triple pattern for the second antecedent of the + * rule (if there are two). + */ + public int stmts2; + + /** + * #of statements considered. + */ + public int numConsidered; + + /** + * #of entailments computed. + */ public int numComputed; + /** + * Time to compute the entailments and store them within the + * {@link TempTripleStore} in milliseconds. + */ long computeTime; } ! } Index: AbstractRuleRdf.java =================================================================== RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/inf/AbstractRuleRdf.java,v retrieving revision 1.6 retrieving revision 1.7 diff -C2 -d -r1.6 -r1.7 *** AbstractRuleRdf.java 13 Apr 2007 15:02:34 -0000 1.6 --- AbstractRuleRdf.java 13 Apr 2007 20:37:04 -0000 1.7 *************** *** 48,59 **** import org.openrdf.model.URI; - import com.bigdata.btree.IEntryIterator; import com.bigdata.btree.IIndex; - import com.bigdata.rdf.KeyOrder; import com.bigdata.rdf.TempTripleStore; public abstract class AbstractRuleRdf extends Rule { - - protected final int BUFFER_SIZE = 10*1024*1024; public AbstractRuleRdf(InferenceEngine store, Triple head, Pred[] body) { --- 48,55 ---- *************** *** 63,122 **** } ! public abstract Stats apply( TempTripleStore entailments ); ! ! protected void dumpBuffer( SPO[] stmts, TempTripleStore btree ) { ! ! // deal with the SPO index ! IIndex spo = btree.getSPOIndex(); ! Arrays.sort(stmts,SPOComparator.INSTANCE); ! for ( int i = 0; i < stmts.length; i++ ) { ! byte[] key = btree.keyBuilder.statement2Key ! ( stmts[i].s, stmts[i].p, stmts[i].o ! ); ! if ( !spo.contains(key) ) { ! spo.insert(key, null); ! } ! } ! ! // deal with the POS index ! IIndex pos = btree.getPOSIndex(); ! Arrays.sort(stmts,POSComparator.INSTANCE); ! for ( int i = 0; i < stmts.length; i++ ) { ! byte[] key = btree.keyBuilder.statement2Key ! ( stmts[i].p, stmts[i].o, stmts[i].s ! ); ! if ( !pos.contains(key) ) { ! pos.insert(key, null); ! } ! } ! ! // deal with the OSP index ! IIndex osp = btree.getOSPIndex(); ! Arrays.sort(stmts,OSPComparator.INSTANCE); ! for ( int i = 0; i < stmts.length; i++ ) { ! byte[] key = btree.keyBuilder.statement2Key ! ( stmts[i].o, stmts[i].s, stmts[i].p ! ); ! if ( !osp.contains(key) ) { ! osp.insert(key, null); ! } ! } ! ! } ! ! protected int insertEntailments( SPO[] entailments ) { ! ! int numAdded = 0; // deal with the SPO index IIndex spo = store.getSPOIndex(); ! Arrays.sort(entailments,SPOComparator.INSTANCE); ! for ( int i = 0; i < entailments.length; i++ ) { byte[] key = store.keyBuilder.statement2Key ! ( entailments[i].s, entailments[i].p, entailments[i].o ); if ( !spo.contains(key) ) { spo.insert(key, null); - numAdded++; } } --- 59,85 ---- } ! /** ! * Copies the entailments from the array into the {@link TempTripleStore}. ! * ! * @param stmts ! * The source statements. ! * ! * @param n ! * The #of statements in the buffer. ! * ! * @param store ! * The target store. ! */ ! protected void insertStatements(SPO[] stmts, int n, TempTripleStore store) { // deal with the SPO index IIndex spo = store.getSPOIndex(); ! Arrays.sort(stmts,0,n,SPOComparator.INSTANCE); ! for ( int i = 0; i < n; i++ ) { byte[] key = store.keyBuilder.statement2Key ! ( stmts[i].s, stmts[i].p, stmts[i].o ); if ( !spo.contains(key) ) { spo.insert(key, null); } } *************** *** 124,131 **** // deal with the POS index IIndex pos = store.getPOSIndex(); ! Arrays.sort(entailments,POSComparator.INSTANCE); ! for ( int i = 0; i < entailments.length; i++ ) { byte[] key = store.keyBuilder.statement2Key ! ( entailments[i].p, entailments[i].o, entailments[i].s ); if ( !pos.contains(key) ) { --- 87,94 ---- // deal with the POS index IIndex pos = store.getPOSIndex(); ! Arrays.sort(stmts,0,n,POSComparator.INSTANCE); ! for ( int i = 0; i < n; i++ ) { byte[] key = store.keyBuilder.statement2Key ! ( stmts[i].p, stmts[i].o, stmts[i].s ); if ( !pos.contains(key) ) { *************** *** 136,143 **** // deal with the OSP index IIndex osp = store.getOSPIndex(); ! Arrays.sort(entailments,OSPComparator.INSTANCE); ! for ( int i = 0; i < entailments.length; i++ ) { byte[] key = store.keyBuilder.statement2Key ! ( entailments[i].o, entailments[i].s, entailments[i].p ); if ( !osp.contains(key) ) { --- 99,106 ---- // deal with the OSP index IIndex osp = store.getOSPIndex(); ! Arrays.sort(stmts,0,n,OSPComparator.INSTANCE); ! for ( int i = 0; i < n; i++ ) { byte[] key = store.keyBuilder.statement2Key ! ( stmts[i].o, stmts[i].s, stmts[i].p ); if ( !osp.contains(key) ) { *************** *** 145,159 **** } } - - return numAdded; } ! protected int insertEntailments2( TempTripleStore entailments ) { ! ! return insertEntailments( convert( entailments ) ); ! ! } ! protected void printStatement( SPO stmt ) { --- 108,177 ---- } } } ! // /** ! // * Copies the statements into the primary store. ! // * ! // * @todo refactor for common code and #of statements parameters with ! // * {@link #dumpBuffer(SPO[], int, TempTripleStore)}, which copies the ! // * statements into the temporary store. ! // * ! // * @param entailments ! // * The statements. ! // * ! // * @return The #of statements actually added to the store. ! // */ ! // protected int insertEntailments( SPO[] entailments ) { ! // ! // int numAdded = 0; ! // ! // // deal with the SPO index ! // IIndex spo = store.getSPOIndex(); ! // Arrays.sort(entailments,SPOComparator.INSTANCE); ! // for ( int i = 0; i < entailments.length; i++ ) { ! // byte[] key = store.keyBuilder.statement2Key ! // ( entailments[i].s, entailments[i].p, entailments[i].o ! // ); ! // if ( !spo.contains(key) ) { ! // spo.insert(key, null); ! // numAdded++; ! // } ! // } ! // ! // // deal with the POS index ! // IIndex pos = store.getPOSIndex(); ! // Arrays.sort(entailments,POSComparator.INSTANCE); ! // for ( int i = 0; i < entailments.length; i++ ) { ! // byte[] key = store.keyBuilder.statement2Key ! // ( entailments[i].p, entailments[i].o, entailments[i].s ! // ); ! // if ( !pos.contains(key) ) { ! // pos.insert(key, null); ! // } ! // } ! // ! // // deal with the OSP index ! // IIndex osp = store.getOSPIndex(); ! // Arrays.sort(entailments,OSPComparator.INSTANCE); ! // for ( int i = 0; i < entailments.length; i++ ) { ! // byte[] key = store.keyBuilder.statement2Key ! // ( entailments[i].o, entailments[i].s, entailments[i].p ! // ); ! // if ( !osp.contains(key) ) { ! // osp.insert(key, null); ! // } ! // } ! // ! // return numAdded; ! // ! // } ! ! // protected int insertEntailments2( TempTripleStore entailments ) { ! // ! // return insertEntailments( getStatements( entailments ) ); ! // ! // } ! protected void printStatement( SPO stmt ) { *************** *** 176,214 **** } ! protected TempTripleStore convert( SPO[] stmts ) { ! ! TempTripleStore tts = new TempTripleStore(); ! ! for ( int i = 0; i < stmts.length; i++ ) { ! ! tts.addStatement( stmts[i].s, stmts[i].p, stmts[i].o ); ! ! } ! ! return tts; ! ! } ! protected SPO[] convert( TempTripleStore tts ) { ! ! SPO[] stmts = new SPO[tts.getStatementCount()]; ! ! int i = 0; ! ! IIndex ndx_spo = tts.getSPOIndex(); ! ! IEntryIterator it = ndx_spo.rangeIterator(null, null); ! ! while ( it.hasNext() ) { ! ! it.next(); ! ! stmts[i++] = new SPO(KeyOrder.SPO, tts.keyBuilder, it.getKey()); ! ! } ! ! return stmts; ! ! } } \ No newline at end of file --- 194,240 ---- } ! // protected TempTripleStore convert( SPO[] stmts ) { ! // ! // TempTripleStore tts = new TempTripleStore(); ! // ! // for ( int i = 0; i < stmts.length; i++ ) { ! // ! // tts.addStatement( stmts[i].s, stmts[i].p, stmts[i].o ); ! // ! // } ! // ! // return tts; ! // ! // } ! // /** ! // * Extracts all statements in the store into an {@link SPO}[]. ! // * ! // * @param store ! // * The store. ! // * ! // * @return The array of statements. ! // */ ! // protected SPO[] getStatements( TempTripleStore store ) { ! // ! // SPO[] stmts = new SPO[store.getStatementCount()]; ! // ! // int i = 0; ! // ! // IIndex ndx_spo = store.getSPOIndex(); ! // ! // IEntryIterator it = ndx_spo.rangeIterator(null, null); ! // ! // while ( it.hasNext() ) { ! // ! // it.next(); ! // ! // stmts[i++] = new SPO(KeyOrder.SPO, store.keyBuilder, it.getKey()); ! // ! // } ! // ! // return stmts; ! // ! // } } \ No newline at end of file Index: AbstractRuleRdfs68101213.java =================================================================== RCS file: /cvsroot/cweb/bigdata-rdf/src/java/com/bigdata/rdf/inf/AbstractRuleRdfs68101213.java,v retrieving revision 1.6 retrieving revision 1.7 diff -C2 -d -r1.6 -r1.7 *** AbstractRuleRdfs68101213.java 13 Apr 2007 15:02:34 -0000 1.6 --- AbstractRuleRdfs68101213.java 13 Apr 2007 20:37:04 -0000 1.7 *************** *** 44,49 **** package com.bigdata.rdf.inf; - import java.util.Vector; - import com.bigdata.btree.IEntryIterator; import com.bigdata.rdf.KeyOrder; --- 44,47 ---- *************** *** 63,73 **** } ! public Stats apply( TempTripleStore btree ) { ! ! Stats stats = new Stats(); ! ! long computeStart = System.currentTimeMillis(); ! Vector<SPO> entailments = new Vector<SPO>(BUFFER_SIZE); byte[] startKey = store.keyBuilder.statement2Key --- 61,67 ---- } ! public Stats apply( final Stats stats, final SPO[] buffer, TempTripleStore tmpStore ) { ! final long computeStart = System.currentTimeMillis(); byte[] startKey = store.keyBuilder.statement2Key *************** *** 79,108 **** ); ! IEntryIterator it = store.getPOSIndex().rangeIterator(startKey,endKey); while ( it.hasNext() ) { it.next(); - SPO stmt = - new SPO(KeyOrder.POS,store.keyBuilder,it.getKey()); long _s = head.s.isVar() ? stmt.s : head.s.id; long _p = head.p.isVar() ? stmt.s : head.p.id; long _o = head.o.isVar() ? stmt.s : head.o.id; ! if (entailments.size() == BUFFER_SIZE) { ! dumpBuffer ! ( entailments.toArray( new SPO[entailments.size()] ), ! btree ! ); ! entailments.clear(); } ! entailments.add( new SPO(_s, _p, _o) ); stats.numComputed++; } - if(debug)dumpBuffer( entailments.toArray( new SPO[entailments.size()] ), btree ); ! stats.computeTime = System.currentTimeMillis() - computeStart; return stats; --- 73,110 ---- ); ! IEntryIterator it = store.getPOSIndex().rangeIterator(startKey, endKey); ! ! int n = 0; while ( it.hasNext() ) { it.next(); + stats.stmts1++; + + SPO stmt = new SPO(KeyOrder.POS, store.keyBuilder, it.getKey()); + + // @todo review -- should this be substituting stmt.s in each case? long _s = head.s.isVar() ? stmt.s : head.s.id; long _p = head.p.isVar() ? stmt.s : head.p.id; long _o = head.o.isVar() ? stmt.s : head.o.id; ! buffer[n++] = new SPO(_s, _p, _o); ! ! if (n == buffer.length) { ! ! insertStatements(buffer, n, tmpStore); ! ! n = 0; ! } ! stats.numComputed++; } ! insertStatements( buffer, n, tmpStore ); ! ! stats.computeTime += System.currentTimeMillis() - computeStart; return stats; |
|
From: Bryan T. <tho...@us...> - 2007-04-13 20:37:10
|
Update of /cvsroot/cweb/bigdata-rdf/src/test/com/bigdata/rdf/inf In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv5139/src/test/com/bigdata/rdf/inf Modified Files: TestFullForwardClosure.java TestMagicSets.java Log Message: Fussed with the inference engine and got a modest performance boost. Rdfs5/11 needs to be rewritten to be smart about exploiting the bufferes triples, e.g., using a binary search on the buffer to locate matches in the self-join. Index: TestFullForwardClosure.java =================================================================== RCS file: /cvsroot/cweb/bigdata-rdf/src/test/com/bigdata/rdf/inf/TestFullForwardClosure.java,v retrieving revision 1.5 retrieving revision 1.6 diff -C2 -d -r1.5 -r1.6 *** TestFullForwardClosure.java 12 Apr 2007 23:59:21 -0000 1.5 --- TestFullForwardClosure.java 13 Apr 2007 20:37:04 -0000 1.6 *************** *** 83,88 **** * hand-crafted data sets to test the rule implementations. */ ! store.loadData(new File("data/alibaba_v41.rdf"),"",false); ! // store.loadData(new File("data/nciOncology.owl"),""); store.fullForwardClosure(); --- 83,88 ---- * hand-crafted data sets to test the rule implementations. */ ! // store.loadData(new File("data/alibaba_v41.rdf"),"",false); ! store.loadData(new File("data/nciOncology.owl"),"",false); store.fullForwardClosure(); Index: TestMagicSets.java =================================================================== RCS file: /cvsroot/cweb/bigdata-rdf/src/test/com/bigdata/rdf/inf/TestMagicSets.java,v retrieving revision 1.7 retrieving revision 1.8 diff -C2 -d -r1.7 -r1.8 *** TestMagicSets.java 27 Mar 2007 14:35:08 -0000 1.7 --- TestMagicSets.java 13 Apr 2007 20:37:04 -0000 1.8 *************** *** 161,169 **** * Applies the base rule iff the {@link Magic} is matched. */ ! public Rule.Stats apply( TempTripleStore entailments ) { if(match()) { ! return rule.apply( entailments ); } --- 161,169 ---- * Applies the base rule iff the {@link Magic} is matched. */ ! public Rule.Stats apply( Stats stats, SPO[] buffer, TempTripleStore tmpStore ) { if(match()) { ! return rule.apply( stats, buffer, tmpStore ); } |
|
From: Bryan T. <tho...@us...> - 2007-04-13 17:52:55
|
Update of /cvsroot/cweb/bigdata/src/java/com/bigdata/btree In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv8389/src/java/com/bigdata/btree Modified Files: BytesUtil.java Log Message: Improved options for using JNI in BytesUtil, but it is still slower in every case that I have tested. Index: BytesUtil.java =================================================================== RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/btree/BytesUtil.java,v retrieving revision 1.4 retrieving revision 1.5 diff -C2 -d -r1.4 -r1.5 *** BytesUtil.java 13 Apr 2007 17:03:09 -0000 1.4 --- BytesUtil.java 13 Apr 2007 17:52:52 -0000 1.5 *************** *** 3,6 **** --- 3,8 ---- import java.util.Comparator; + import org.apache.log4j.Logger; + /** * Class supporting operations on variable length byte[] keys. *************** *** 13,17 **** * positions in that byte[] is maintained. * <p> ! * See {@link #main(String[])} which provides a test for the JNI integration. * * @author <a href="mailto:tho...@us...">Bryan Thompson</a> --- 15,33 ---- * positions in that byte[] is maintained. * <p> ! * JNI methods are provided for unsigned byte[] comparison. However, note that ! * the JNI methods do not appear to be as fast as the pure Java methods - ! * presumably because of the overhead of going from Java to C. In order to ! * execute using the JNI methods you MUST define the optional boolean system ! * property, e.g., ! * ! * <pre> ! * java -Dcom.bigdata.btree.BytesUtil.jni=true ... ! * </pre> ! * ! * See BytesUtil.c in this package for instructions on compiling the JNI ! * methods. ! * </p> ! * See {@link #main(String[])} which provides a test for the JNI integration and ! * some pointers on how to get this running on your platform. * * @author <a href="mailto:tho...@us...">Bryan Thompson</a> *************** *** 20,23 **** --- 36,41 ---- public class BytesUtil { + protected static final Logger log = Logger.getLogger(BytesUtil.class); + /** * Flag set iff JNI linking succeeds. When this flag is false we run with *************** *** 25,29 **** * the JNI versions are used. */ ! static boolean linked; /** --- 43,47 ---- * the JNI versions are used. */ ! static boolean linked = false; /** *************** *** 39,60 **** static { ! try { ! System.loadLibrary("BytesUtil"); ! System.err.println("BytesUtil JNI linked"); ! linked = true; ! } catch (UnsatisfiedLinkError ex) { ! System.err.println("BytesUtil JNI NOT linked: " + ex); ! linked = false; } - - /* - * Strangly it appears that java is faster than JNI + C for byte[] - * comparisons! - */ - linked = false; } /** * True iff the two arrays compare as equal. This is somewhat optimized in * that it tests the array lengths first, assumes that it is being used on --- 57,125 ---- static { ! final boolean jni; ! ! String val = System.getProperty("com.bigdata.btree.BytesUtil.jni"); ! ! if (val != null) { ! ! jni = Boolean.parseBoolean(val); ! ! } else { ! ! jni = false; // Note: We will not even try to use JNI by default! ! ! } ! ! if (jni) { ! ! /* ! * Attempt to load the JNI library. ! */ ! ! loadJNILibrary(); ! } } /** + * Attempt to load the JNI library. + * <p> + * Note: this is done automatically if the optional boolean system property + * <code>com.bigdata.btree.BytesUtil.jni=true</code> is specified, e.g., + * using + * + * <pre> + * java -Dcom.bigdata.btree.BytesUtil.jni=true ... + * </pre> + * + * @return True iff the JNI library was successfully linked. + */ + public static boolean loadJNILibrary() { + + if (!linked) { + + try { + + System.loadLibrary("BytesUtil"); + + log.info("BytesUtil JNI linked"); + + linked = true; + + } catch (UnsatisfiedLinkError ex) { + + log.warn("BytesUtil JNI NOT linked: " + ex); + + linked = false; + + } + } + + return linked; + + } + + /** * True iff the two arrays compare as equal. This is somewhat optimized in * that it tests the array lengths first, assumes that it is being used on *************** *** 580,589 **** /** ! * This method tries to execute the JNI methods. ! * ! * See BytesUtil.c in this package for instructions on compiling the JNI ! * methods. However, note that the JNI methods do not appear to be as fast ! * as the pure Java methods - presumably because of the overhead of going ! * from Java to C. * <p> * In order to use the JNI library under Windows, you must specify the JNI --- 645,650 ---- /** ! * This method forces the load of the JNI library and tries to execute the ! * JNI methods. * <p> * In order to use the JNI library under Windows, you must specify the JNI *************** *** 591,597 **** * * <pre> ! * cd bigdata ! * set PATH=%PATH%;lib ! * java -cp bin com.bigdata.btree.BytesUtil * </pre> * --- 652,658 ---- * * <pre> ! * cd bigdata ! * set PATH=%PATH%;lib ! * java -cp bin com.bigdata.btree.BytesUtil * </pre> * *************** *** 601,605 **** * * <pre> ! * java -Djava.library.path=lib com.bigdata.btree.BytesUtil * </pre> * --- 662,666 ---- * * <pre> ! * java -Djava.library.path=lib com.bigdata.btree.BytesUtil * </pre> * *************** *** 613,616 **** --- 674,680 ---- public static void main(String[] args) { + // Force load of the JNI library. + loadJNILibrary(); + if( 0 != BytesUtil._compareBytes(3, new byte[]{1,2,3}, 3, new byte[]{1,2,3}) ) { |
|
From: Bryan T. <tho...@us...> - 2007-04-13 17:03:13
|
Update of /cvsroot/cweb/bigdata/src/java/com/bigdata/btree In directory sc8-pr-cvs4.sourceforge.net:/tmp/cvs-serv22157/src/java/com/bigdata/btree Modified Files: BytesUtil.java Log Message: Update to javadoc and enabling JNI loading in BytesUtil.java Index: BytesUtil.java =================================================================== RCS file: /cvsroot/cweb/bigdata/src/java/com/bigdata/btree/BytesUtil.java,v retrieving revision 1.3 retrieving revision 1.4 diff -C2 -d -r1.3 -r1.4 *** BytesUtil.java 13 Apr 2007 16:48:05 -0000 1.3 --- BytesUtil.java 13 Apr 2007 17:03:09 -0000 1.4 *************** *** 39,50 **** static { ! // try { ! // System.loadLibrary("BytesUtil"); ! // System.err.println("BytesUtil JNI linked"); ! // linked = true; ! // } catch (UnsatisfiedLinkError ex) { ! // System.err.println("BytesUtil JNI NOT linked: " + ex); ! // linked = false; ! // } /* --- 39,50 ---- static { ! try { ! System.loadLibrary("BytesUtil"); ! System.err.println("BytesUtil JNI linked"); ! linked = true; ! } catch (UnsatisfiedLinkError ex) { ! System.err.println("BytesUtil JNI NOT linked: " + ex); ! linked = false; ! } /* *************** *** 625,628 **** --- 625,630 ---- } + System.err.println("JNI library routines Ok."); + } |
![http://www.cognitiveweb.org/Images/tagline.gif"](http://www.cognitiveweb.org/Images/tagline.gif"){kind=link}
![http://www.cognitiveweb.org/wiki/img/moinmoin.gif"](http://www.cognitiveweb.org/wiki/img/moinmoin.gif"){kind=link}
