agilewiki-wiki Mailing List for AgileWiki (Page 26)
Status: Beta
Brought to you by:
blaforge
Menu
▾
▴
agilewiki-wiki — General discussion.
You can subscribe to this list here.
| 2005 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
(11) |
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2006 |
Jan
(10) |
Feb
(1) |
Mar
(29) |
Apr
(13) |
May
(119) |
Jun
(104) |
Jul
(142) |
Aug
(92) |
Sep
(86) |
Oct
(31) |
Nov
(16) |
Dec
(6) |
| 2007 |
Jan
(3) |
Feb
(10) |
Mar
(2) |
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
| 2011 |
Jan
(1) |
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
| 2012 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
(1) |
Sep
|
Oct
|
Nov
|
Dec
|
|
From: BiLaf <Bi...@Ro...> - 2006-03-03 08:26:57
|
Sorry 'bout this. --b |
|
From: BiLaf <Bi...@Ro...> - 2006-03-03 07:58:45
|
Having mail problems too, Norm. SourceForge just started blocking the Yahoo email server--a known source of spam. :-( Anyway, I just moved createArk from Init to Env and made it a static method, as well as checking for duplicate calls--the ark is now being treated as a singleton. Bill |
|
From: Bill la F. <laf...@ya...> - 2006-03-03 05:04:40
|
ReentrantReadWriteLock looks made to order. Had to write this myself in Python. Bill __________________________________________________________ Yahoo! India Matrimony: Find your partner now. Go to http://yahoo.shaadi.com |
|
From: Bill la F. <laf...@ya...> - 2006-03-03 04:34:11
|
Ther cmds.txt and displays.txt are framework resource files, with entries in the form "name class". I'm thinking of moving these up to the top-level of the class tree and combining them into an ini-like file with entries in the form "cmd.name=class" and "display.name=class", as well as adding the tuning parameters for invoking DArk. I'm also thinking of moving Init.createArk into Env and making it a static method. This method would then read the .ini file and create a HashMap from its contents. Bill __________________________________________________________ Yahoo! India Matrimony: Find your partner now. Go to http://yahoo.shaadi.com |
|
From: Norm K. <nr...@cm...> - 2006-03-03 01:48:15
|
My ISP claims to have solved the email delivery problem. Here goes. ;-n |
|
From: Norm K. <nka...@co...> - 2006-03-03 01:01:05
|
Here is yet another attempt to contact this list. ;-n |
|
From: Bill la F. <laf...@ya...> - 2006-03-03 00:44:32
|
Andy,
There is a fine line between explaining and defending. But I think the explaining is important enough to risk such confusion. I could equally argue that having so many source trees adds to the confusion to a newbie, so it should only be done if it is also well documented--which I don't think it is.
Your point about adding builds and tests is a good one. I'd like this project to be easy for someone to adopt. Regression tests are a personal weakness of mine. I do like to develop things in a way that they can be tested, but I don't leave a trail of test code behind me--the bread crumbs are too sparse for others to follow.
As for NetBeans builds, I'm a bit unhappy with what I've got. Every time I do a run in the NB environment, it recompiles over a dozen modules. But its not something I have put any effort into. And obviously it should be worked on. I do like your idea of having a common build under svn. Please say more.
As for RMI, it looks like a quick way to go. I'm rusty at it, and it is really inappropriate as it does too much that isn't needed while not doing some important things well. We don't need object serialization--everything's a string. And distributed garbage collection is a lot of baggage for what we need. But it should be easy to do and I think there may even be a community of folk who know how to write RMI clients. But I'd like to keep the RMI code and clients in separate source trees and limit how much code it contaminates. Ultimately a select-based server will give us much better speed and performance, but right now getting something working and usable is more the point. Lets talk more about this.
As for not being familiar with rolonics, I must say it took me years. But the joke is just how easy and appropriate it is. Reminds me of the shift over to oo. I had a bit of a hard time with that, too. But there was a lot more literature to help. Between the lack of reading material, no good programming examples, lack of community and overall the intimidation factors of terminology and its utmost simplicity, its a big speed bump. And there's still so many simple and easy things that need to be worked out. All I can say is relax and let it flow. It is a really very effective way to approach things, but its so natural that you can't help but try too hard. There is really hardly anything to it. And the words just get in the way.
One thing that might help is Ark vs DArk. The Ark is largely without behavior, just gets and puts. Its all dressed up in a nice API, but there's no validation. DArk addresses partness/descriptors/behavior. Any paramaterizable behavior goes in DArk, leaving Ark with fixed (if sometimes complex) logic. And once you go beyond Ark and DArk, the rest is just architecture, and perhaps not always the best of architecture.
Bill
Andy Glick <and...@ac...> wrote:
Bill,
I haven't done much RMI work, but I have done some distributed server programming. I'm willing to have you vet anything that I can put together, but I imagine that it may take me some time to come up to speed with the code that you are currently writing. I am not a rolonic initiate yet, I'm still attempting to sort out the principles.
I wanted to start a discussion about the SVN repository directory structure and the build/release infrastructure, but I am hesitant to do that lest you go off and rearrange everything again before we can conclude a conversation. In one of your recent posts to me you seemed to be defending your original directory structure which contained 2 top level source trees, ark and framework. I didn't have a problem with that structure, but you have since reordered the directory structure and pushed them down a level, and now have a set of top level trees under trunk. Again, that is OK, and I'm not trying to criticize, it is simply that you never explained your actual reasons for creating or demoting the top level artifacts, and I still am not clear on your thinking.
What I believe that I have understood is that you are:
1) using the Netbeans system's build infrastructure, which you have not committed to SVN
2) you partitioned the current SVN repository into a set of source trees, {ark, directportal, framework, textclient} because each subtree produces a single jar artifact.
Given that your stated goal is to interest more open source developers to take an interest in and to lend a hand with AgileWiki, I have some concerns about the project's current lack of build and testing infrastructure, and some suggestions for improving each. I'd be happy to discuss them, either now or at a later date.
Thanks,
Andy
---------------------------------
Jiyo cricket on Yahoo! India cricket
Yahoo! Messenger Mobile Stay in touch with your buddies all the time. |
|
From: Andy G. <and...@ac...> - 2006-03-02 19:21:45
|
At 08:19 AM 3/1/2006, Bill la Forge wrote:
>Have you done any RMI work? I'm thinking we should s et up two more
>projects--one for an RMI server, which would include an RMI text portal
>and another for the RMI client.
>
>bill
Bill,
I haven't done much RMI work, but I have done some distributed server
programming. I'm willing to have you vet anything that I can put together,
but I imagine that it may take me some time to come up to speed with the
code that you are currently writing. I am not a rolonic initiate yet, I'm
still attempting to sort out the principles.
I wanted to start a discussion about the SVN repository directory structure
and the build/release infrastructure, but I am hesitant to do that lest you
go off and rearrange everything again before we can conclude a
conversation. In one of your recent posts to me you seemed to be defending
your original directory structure which contained 2 top level source trees,
ark and framework. I didn't have a problem with that structure, but you
have since reordered the directory structure and pushed them down a level,
and now have a set of top level trees under trunk. Again, that is OK, and
I'm not trying to criticize, it is simply that you never explained your
actual reasons for creating or demoting the top level artifacts, and I
still am not clear on your thinking.
What I believe that I have understood is that you are:
1) using the Netbeans system's build infrastructure, which you have not
committed to SVN
2) you partitioned the current SVN repository into a set of source trees,
{ark, directportal, framework, textclient} because each subtree produces a
single jar artifact.
Given that your stated goal is to interest more open source developers to
take an interest in and to lend a hand with AgileWiki, I have some concerns
about the project's current lack of build and testing infrastructure, and
some suggestions for improving each. I'd be happy to discuss them, either
now or at a later date.
Thanks,
Andy
|
|
From: Bill la F. <laf...@ya...> - 2006-03-02 11:53:28
|
I am thinking now that a wrapper of Ark for descriptors is over kill. All we really need is a subclass of Ark. Descriptor logic can then reside in the subclass. So we introduce the DArk class, and move the implementation of CmdArk from Ark to DArk. (And remove "final" as needed from the Ark methods.) Then at least there is a proper place to put descriptor-driven logic, especially for validation. Bill Blog: http://laforge49.blogspot.com Web: http://AgileWiki.org --------------------------------- Jiyo cricket on Yahoo! India cricket Yahoo! Messenger Mobile Stay in touch with your buddies all the time. |
|
From: Bill la F. <laf...@ya...> - 2006-03-02 03:26:30
|
The control flows are done now for user registration. You can actually use it. It just doesn't create a user account yet. Also, I've added an Init class to the framework which creates the Ark object as well as creating the Etc and Home cabinets as needed. This class then is used in Driver.main. Bill ____________________________________________________ Jiyo cricket on Yahoo! India cricket http://in.sports.yahoo.com/cricket/ |
|
From: Bill la F. <laf...@ya...> - 2006-03-01 13:19:23
|
The andyglick user now has svn access: svn co https://svn.sourceforge.net/svnroot/compstrm compstrm I've found svn much friendlier than cvs. The mess I made only took a few minutes to recover from (I used a hammer.) And it really is nice to get your fingers burned early in the game--instead of messing up something important. Welcome aboard! Norm definately vouches for you! And it sure is nice to have someone else around who will post an email on the list. I've got about 600 posts on the FORUM and almost all from me. Have you done any RMI work? I'm thinking we should set up two more projects--one for an RMI server, which would include an RMI text portal and another for the RMI client. bill Andy Glick <and...@ac...> wrote: Bill, As you can see I've joined the list. I'm sorry that you had difficulty restructuring the SVN repository. I've suffered through some pain when restructuring SVN directory structures, which is one of the reasons that I offered to do it, I'm sorry that I didn't add a caution about potential risks. My Source Forge user name is andyglick. ------------------------------------------------------- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 _______________________________________________ Compstrm-wiki mailing list Com...@li... https://lists.sourceforge.net/lists/listinfo/compstrm-wiki --------------------------------- Jiyo cricket on Yahoo! India cricket Yahoo! Messenger Mobile Stay in touch with your buddies all the time. |
|
From: Andy G. <and...@ac...> - 2006-03-01 12:43:11
|
Bill, As you can see I've joined the list. I'm sorry that you had difficulty restructuring the SVN repository. I've suffered through some pain when restructuring SVN directory structures, which is one of the reasons that I offered to do it, I'm sorry that I didn't add a caution about potential risks. My Source Forge user name is andyglick. |
|
From: Bill la F. <laf...@ya...> - 2006-03-01 12:19:27
|
I'm thinking about implementing an RMI server around the framework--yet another jar file, of course. Looks like an easy way to make some forward progress, as well as adding utility. But still a few too many things to get done first--user registration and login take priority! Bill Blog: http://laforge49.blogspot.com Web: http://AgileWiki.org --------------------------------- Jiyo cricket on Yahoo! India cricket Yahoo! Messenger Mobile Stay in touch with your buddies all the time. |
|
From: Bill la F. <laf...@ya...> - 2006-03-01 08:39:00
|
I should mention that it was Andy who suggested changing the svn structure. My thanks! Bill __________________________________________________________ Yahoo! India Matrimony: Find your partner now. Go to http://yahoo.shaadi.com |
|
From: Bill la F. <laf...@ya...> - 2006-03-01 08:37:10
|
Well, I reorganized things on svn over lunch break. Did some bad things, some quick fixes and finally learned, I think, how to do it right. The history on Ark is kabash. But I think current state is good. I've now got trunk, branches and tags, with 4 projects under trunk. I also managed to salvage my NetBeans configuration and got everything built. I figure its better to learn this now than later, when the history is more meaningful. And its also important to me to conform to standards as much as possible. I want to attract developers to this project, not repel them! So if there are any more suggestions, please feel free! Bill __________________________________________________________ Yahoo! India Matrimony: Find your partner now. Go to http://yahoo.shaadi.com |
|
From: Bill la F. <laf...@ya...> - 2006-03-01 00:40:00
|
I'll admidt it--I didn't read svn chapter 4. So the svn layout no doubt needs updating. Perhaps the structure I chose should be pushed down under trunk? Anyway, the intent of the structure I did go with was to have ark and framework as virtually separate projects. In NetBeans, I have them as separate projects and the ark has its own jar file. Three reasons for this, really.First, the ark code does not depend on the framework code, while the framework code only depends on the ark interface(s). Second, its easier just to try learning the ark first, without worring too much about the framework. Finally, I see the ark as a handy little database. I might want to use it in places where having a back-end server is overkill. Wel, OK, one more. I see the need to add additional little projects. Like a servlet which interfaces to the backend service provided by the ark and its framework. Bill Blog: http://laforge49.blogspot.com Web: http://AgileWiki.org --------------------------------- Jiyo cricket on Yahoo! India cricket Yahoo! Messenger Mobile Stay in touch with your buddies all the time. |
|
From: Bill la F. <laf...@ya...> - 2006-02-22 12:29:18
|
Norm, Epifiny day for me, big time! FINALLY, I can see how to write all kinds of applications using AgileWiki3. And its so very easy! Five years I have waited for this vision. Things will be so much easier now. Bill Blog: http://laforge49.blogspot.com Web: http://AgileWiki.org --------------------------------- Jiyo cricket on Yahoo! India cricket Yahoo! Messenger Mobile Stay in touch with your buddies all the time. |
|
From: Bill la F. <laf...@ya...> - 2006-01-19 01:57:18
|
from time import time
from weakref import WeakValueDictionary
from heapq import heappush,heappop
from randompot import UUID
from os.path import exists
from threading import Lock
from zlib import compress,decompress
from bisect import bisect_left,insort_left
class block:
"a cachable block buffer"
def __init__(self,recnbr,blksize,bimage):
self.blksize=blksize
self.recnbr=recnbr
self.bimage=bimage
self.s=None
self.l=None
self.inrheap=False
self.inwheap=False
def gets(self):
if self.s==None:
if self.l==None:
raise Exception('empty buffer')
else:
self.s=chr(255).join(self.l)
if len(self.s)>self.blksize:
raise Exception('block puts: string greater than blksize')
return self.s
def puts(self,s):
self.bimage(self)
i=s.rfind(chr(255))
if i<0:
raise Exception('record has no terminator')
s=s[:i+1]
if len(s)>self.blksize:
raise Exception('block puts: string greater than blksize')
self.s=s
self.l=None
def getl(self):
if self.l==None:
if self.s==None:
raise Exception('empty buffer')
else:
self.l=self.s.split(chr(255))
return self.l
def putl(self,l):
self.bimage(self)
self.l=l
self.s=None
def empty(self):
return self.s==None and self.l==None
def len(self):
if self.s!=None:
return len(self.s)
if self.l!=None:
ln=0
for kv in self.l:
ln+=len(kv)+1
return l
raise Exception('empty buffer')
class cachefile:
"Keeps the last accessed blocks in RAM"
def __init__(self,file,bimage,maxrcache=1000,maxwcache=1000,blksize=8*1024):
"""
maxrcache is the number of blocks kept in RAM
maxwcache is the number of dirty blocks kept in RAM
"""
self.file=file
self.bimage=bimage
self.maxr=maxrcache
self.maxw=maxwcache
self.blksize=blksize
self.wd=WeakValueDictionary()
self.rheap=[]
self.wheap=[]
def flush(self):
"flush all pending writes"
wheap=self.wheap
dl=[]
for ts,b in wheap:
if not b.inwheap:
continue
dl.append((b.recnbr,b.gets()))
dl.sort()
for recnbr,s in dl:
s=s.ljust(self.blksize)
self.file.seek(self.blksize*recnbr)
self.file.write(s)
self.file.flush()
def buffer(self,recnbr):
"locate a buffer or create a new one"
try:
b=self.wd[recnbr]
except:
b=block(recnbr,self.blksize,self.bimage)
return b
def put(self,buffer,force=False):
"write a block"
recnbr=buffer.recnbr
self.add(recnbr,buffer)
ts=time()
buffer.timeoflastaccess=ts
if force:
buffer.inwheap=False
s=buffer.gets()
s=s.ljust(self.blksize)
self.file.seek(self.blksize*recnbr)
self.file.write(s)
return
if buffer.inwheap:
return
buffer.inwheap=True
heappush(self.wheap,(ts,buffer))
while len(self.wheap):
t,b=heappop(self.wheap)
ts=b.timeoflastaccess
if not b.inwheap:
continue
if t==ts:
b.inwheap=False
s=b.gets()
s=s.ljust(self.blksize)
self.file.seek(self.blksize*recnbr)
self.file.write(s)
continue
heappush(self.rheap,(ts,b))
def get(self,recnbr):
"locate a block or read it."
try:
buffer=self.wd[recnbr]
except:
self.file.seek(self.blksize*recnbr)
buffer=self.buffer(recnbr)
buffer.puts(self.file.read(self.blksize))
self.add(recnbr,buffer)
buffer.timeoflastaccess=time()
return buffer
def add(self,recnbr,buffer):
"add a buffer to the cache"
self.wd[recnbr]=buffer
ts=time()
buffer.timeoflastaccess=ts
if buffer.inrheap:
return
heappush(self.rheap,(ts,buffer))
buffer.inrheap=True
while len(self.rheap):
t,b=heappop(self.rheap)
ts=b.timeoflastaccess
if t==ts:
b.inrheap=False
continue
heappush(self.rheap,(ts,b))
class tranfile:
"""
Cached and transactional block I/O
self.root is the block number of the btree root block
"""
def __init__(self,filename,blksize=8*1024,maxrcache=0,maxwcache=0):
"""
filename names the database.
"before-"+filename is the name of the before-image file.
blksize is the size of the blocks
maxrcache is the number of blocks cached in RAM
maxwcache is the number of dirty blocks cached in RAM
"""
if not maxrcache:
maxrcache=(1024*1024)/blksize
if not maxwcache:
maxwcache=(1024*1024)/blksize
ms=len(UUID())
self.ms=ms
blksize=int(blksize)
self.bblksize=self.ms+16+blksize
if not exists(filename+'.wdb'):
f=file(filename+'.wdb','wb',blksize)
f.write(''.join((str(blksize).zfill(16),'1'.zfill(16),'0'.zfill(16),chr(255))))
f.close()
f=file(filename+'.wdbb','wb',self.bblksize)
f.write(''.ljust(self.bblksize))
f.close()
self.blksize=blksize
self.file=file(filename+'.wdb','r+b',blksize)
self.cache=cachefile(self.file,self.bimage,maxrcache=maxrcache,maxwcache=maxwcache,blksize=blksize)
r0=self.file.read(blksize)
bsz=long(r0[:16])
if bsz!=blksize:
raise Exception('cachefile has wrong blksize')
self.ms=len(UUID())
self.bblksize=ms+16+blksize
self.bfile=file(filename+'.wdbb','r+b',self.bblksize)
self.before={}
self.tran=None
self.bfile.seek(0,2)
sz=self.bfile.tell()/self.bblksize
self.bfile.seek(0)
bs=self.bfile.read(self.bblksize)
sz-=1
t=bs[:ms]
if t!=''.ljust(self.ms):
while True:
rn=long(bs[ms:ms+16])
s=bs[ms+16:]
b=self.cache.buffer(rn)
b.puts(s)
self.putb(b)
if not sz:
break
bs=self.bfile.read(self.bblksize)
sz-=1
nt=bs[:ms]
if nt!=t:
break
self.bfile.seek(0)
r0=self.getb(0).gets()
self.avail=long(r0[16:32])
self.root=long(r0[32:48])
def close(self):
"Update close the files"
self.file.close()
self.bfile.close()
def putnewl(self,l):
"Allocate 1 block"
a=self.avail
self.avail+=1
b=self.cache.buffer(a)
b.putl(l)
self.cache.put(b,force=True)
return a
def putb(self,buffer,force=False):
"write a block"
self.cache.put(buffer,force=force)
def getb(self,recnbr):
"read a block"
return self.cache.get(recnbr)
def bimage(self,b):
if b.empty():
return
recnbr=b.recnbr
if self.tran and (recnbr not in self.before):
s=b.gets()
self.bwrite(recnbr,s)
def bwrite(self,recnbr,s):
self.before[recnbr]=None
s=s.ljust(self.blksize)
bfile=self.bfile
bfile.write(''.join((self.tran,str(recnbr).zfill(16),s)))
bfile.flush()
def save(self,s):
"""
save a large string (len > blksize/2)
returns the starting block nbr
"""
l=len(s)
if (l+16)>self.blksize:
s=chr(252)+compress(s)
l=len(s)
s=str(l).zfill(16)+s
l+=16
bs=(l+self.blksize-1)/self.blksize
s=s.ljust(bs*self.blksize)
a=self.avail
self.avail+=bs
self.file.seek(a*self.blksize)
self.file.write(s)
return a
def load(self,recnbr):
"fetch a large string previously saved"
self.file.seek(recnbr*self.blksize)
sl=self.file.read(16)
l=long(sl)
c=self.file.read(1)
s=self.file.read(l-1)
if c!=chr(252):
return c+s
return decompress(s)
def transaction(self):
"start a transaction"
if self.tran:
raise Exception('tran in tran')
self.tran=UUID()
b=self.getb(0)
self.bwrite(0,b.gets())
def commit(self):
"commit a transaction"
if not self.tran:
raise Exception('commit before tran')
r0=''.join((str(self.blksize).zfill(16),str(self.avail).zfill(16),str(self.root).zfill(16),chr(255)))
b=self.cache.buffer(0)
b.puts(r0)
self.cache.put(b,force=True)
self.cache.flush()
self.bfile.seek(0)
self.bfile.write(''.ljust(self.bblksize))
self.bfile.flush()
self.bfile.seek(0)
self.before={}
self.tran=None
class btree:
"""
A write once, read many transactional btree file.
All keys and values must be 7-bit ascii.
Duplicate keys stored in chronological order.
Reserved characters:
chr(255) - record separator
chr(0) - key/value separator
chr(253) - indirect value indicator
chr(252) - compression indicator
"""
def __init__(self,filename,blksize=1024,maxrcache=0,maxwcache=0):
"""
filename names the database.
"before-"+filename is the name of the before-image file.
blksize is the size of the blocks
maxrcache is the number of blocks cached in RAM
maxwcache is the number of dirty blocks cached in RAM
"""
if blksize<64:
blksize=64
tf=tranfile(filename,blksize=blksize,maxrcache=maxrcache,maxwcache=maxwcache)
self.tf=tf
self.blksize=tf.blksize
self.lock=Lock()
self.putcount=0
if not tf.root:
tf.transaction()
tf.root=tf.putnewl(['L'+chr(255)])
tf.commit()
def close(self):
"Update block 0 and close the files"
self.tf.close()
def tran(self):
"start a transaction"
self.tf.transaction()
def commit(self):
"commit a transaction"
self.tf.commit()
def items(self,key,limit):
"yields k,v where limit <= k <= key, in descending order by k"
putcount=-1
while True:
if putcount!=self.putcount:
c,k,v=self._get(key)
if not k:
return
if k<limit:
return
if putcount==-1 or k!=key:
putcount=self.putcount
yield k,v
key=k
else:
self.lock.acquire()
try:
c,k,v=self._prior(c)
finally:
self.lock.release()
if not k:
return
if k<limit:
return
putcount=self.putcount
key=k
yield k,v
def haskey(self,key):
"does wdb have key"
c,k,v=self._get(key)
return k==key
def _get(self,key):
"""
Get a value.
Returns cursor,key,value.
(Cursor is None when called in a transaction.)
The returned key may be less than the argument key.
When there is no matching value, returns None,None,None.
"""
self.lock.acquire()
try:
nl,bu=self._getleaf(key)
kl=bu.getl()
l=len(kl)-1
i=bisect_left(kl,key,1,l)
if i==l:
i=l-1
if not i:
return None,None,None
k,v=kl[i].split(chr(0))
if k>key:
nl.append((bu,i))
return self._prior(nl)
if v.startswith(chr(253)):
vn=long(v[1:])
v=self.tf.load(vn)
nl.append((bu,i))
return nl,k,v
finally:
self.lock.release()
def _prior(self,nl):
"""
Get the prior value.
Returns cursor,key,value.
When there is no matching value, returns None,None,None.
"""
tf=self.tf
i=0
while len(nl):
bu,i=nl.pop()
if i>1:
break
i=0
if not i:
return None,None,None
i-=1
nl.append((bu,i))
kl=bu.getl()
k,v=kl[i].split(chr(0))
while kl[0]=='N':
bn=long(v)
bu=tf.getb(bn)
kl=bu.getl()
i=len(kl)-2
nl.append((bu,i))
k,v=kl[i].split(chr(0))
if v.startswith(chr(253)):
vn=long(v[1:])
v=self.tf.load(vn)
return nl,k,v
def _getleaf(self,key):
nl=[]
tf=self.tf
bn=tf.root
while True:
bu=tf.getb(bn)
kl=bu.getl()
if kl[0]=='L':
return nl,bu
l=len(kl)-1
i=bisect_left(kl,key,1,l)
if i==l:
i=l-1
k,sbn=kl[i].split(chr(0))
nl.append((bu,i))
bn=long(sbn)
def put(self,key,value):
"""
Add a key/value.
"""
tf=self.tf
if len(key)>self.blksize/4:
raise Exception('key length is more than blksize/4')
if not tf.tran:
raise Exception('put not in tran')
self.putcount+=1
if len(value)>self.blksize/2:
vn=tf.save(value)
value=chr(253)+str(vn)
nl,bu=self._getleaf(key)
kl=bu.getl()
f=len(kl)-2
nkv=chr(0).join((key,value))
insort_left(kl,nkv,1,len(kl)-1)
while True:
ln=0
for kv in kl:
ln+=len(kv)+1
if ln<=self.blksize:
bu.putl(kl)
tf.putb(bu)
return
ln=0
kl1=kl
kl0=[]
t=kl[0]
while ln<(self.blksize/2) and len(kl1)>2:
kv=kl1.pop(0)
ln+=len(kv)+1
kl0.append(kv)
kl0.append('')
kl1.insert(0,t)
kv0=kl0[-2]
kv1=kl1[-2]
i0=kv0.find(chr(0))
i1=kv1.find(chr(0))
k0=kv0[0:i0]
k1=kv1[0:i1]
tf=self.tf
bn0=bu.recnbr
bu.putl(kl0)
tf.putb(bu)
bn1=tf.putnewl(kl1)
v0=str(bn0)
v1=str(bn1)
if not len(nl):
print 'btree root split'
kl=['N',k0+chr(0)+v0,k1+chr(0)+v1,'']
bn=tf.putnewl(kl)
tf.root=bn
return
bu,i=nl.pop()
kl=bu.getl()
kl[i]=chr(0).join((k0,v0))
kl.insert(i+1,chr(0).join((k1,v1)))
|
|
From: Bill la F. <laf...@ya...> - 2006-01-14 05:29:22
|
from time import time
from weakref import WeakValueDictionary
from heapq import heappush,heappop
from tks.randompot import UUID
from os.path import exists
from threading import Lock
from zlib import compress,decompress
from bisect import bisect_left,insort_left
class block:
"a cachable block buffer"
def __init__(self,recnbr,blksize,bimage):
self.blksize=blksize
self.recnbr=recnbr
self.bimage=bimage
self.s=None
self.l=None
self.inrheap=False
self.inwheap=False
def gets(self):
if self.s==None:
if self.l==None:
raise Exception('empty buffer')
else:
self.s=chr(255).join(self.l)
if len(self.s)>self.blksize:
raise Exception('block puts: string greater than blksize')
return self.s
def puts(self,s):
self.bimage(self)
i=s.rfind(chr(255))
if i<0:
raise Exception('record has no terminator')
s=s[:i+1]
if len(s)>self.blksize:
raise Exception('block puts: string greater than blksize')
self.s=s
self.l=None
def getl(self):
if self.l==None:
if self.s==None:
raise Exception('empty buffer')
else:
self.l=self.s.split(chr(255))
return self.l
def putl(self,l):
self.bimage(self)
self.l=l
self.s=None
def empty(self):
return self.s==None and self.l==None
def len(self):
if self.s!=None:
return len(self.s)
if self.l!=None:
ln=0
for kv in self.l:
ln+=len(kv)+1
return l
raise Exception('empty buffer')
class cachefile:
"Keeps the last accessed blocks in RAM"
def __init__(self,file,bimage,maxrcache=1000,maxwcache=1000,blksize=8*1024):
"""
maxrcache is the number of blocks kept in RAM
maxwcache is the number of dirty blocks kept in RAM
"""
self.file=file
self.bimage=bimage
self.maxr=maxrcache
self.maxw=maxwcache
self.blksize=blksize
self.wd=WeakValueDictionary()
self.rheap=[]
self.wheap=[]
def flush(self):
"flush all pending writes"
wheap=self.wheap
dl=[]
for ts,b in wheap:
if not b.inwheap:
continue
dl.append((b.recnbr,b.gets()))
dl.sort()
for recnbr,s in dl:
s=s.ljust(self.blksize)
self.file.seek(self.blksize*recnbr)
self.file.write(s)
self.file.flush()
def buffer(self,recnbr):
"locate a buffer or create a new one"
try:
b=self.wd[recnbr]
except:
b=block(recnbr,self.blksize,self.bimage)
return b
def put(self,buffer,force=False):
"write a block"
recnbr=buffer.recnbr
self.add(recnbr,buffer)
ts=time()
buffer.timeoflastaccess=ts
if force:
buffer.inwheap=False
s=buffer.gets()
s=s.ljust(self.blksize)
self.file.seek(self.blksize*recnbr)
self.file.write(s)
return
if buffer.inwheap:
return
buffer.inwheap=True
heappush(self.wheap,(ts,buffer))
while len(self.wheap):
t,b=heappop(self.wheap)
ts=b.timeoflastaccess
if not b.inwheap:
continue
if t==ts:
b.inwheap=False
s=b.gets()
s=s.ljust(self.blksize)
self.file.seek(self.blksize*recnbr)
self.file.write(s)
continue
heappush(self.rheap,(ts,b))
def get(self,recnbr):
"locate a block or read it."
try:
buffer=self.wd[recnbr]
except:
self.file.seek(self.blksize*recnbr)
buffer=self.buffer(recnbr)
buffer.puts(self.file.read(self.blksize))
self.add(recnbr,buffer)
buffer.timeoflastaccess=time()
return buffer
def add(self,recnbr,buffer):
"add a buffer to the cache"
self.wd[recnbr]=buffer
ts=time()
buffer.timeoflastaccess=ts
if buffer.inrheap:
return
heappush(self.rheap,(ts,buffer))
buffer.inrheap=True
while len(self.rheap):
t,b=heappop(self.rheap)
ts=b.timeoflastaccess
if t==ts:
b.inrheap=False
continue
heappush(self.rheap,(ts,b))
class tranfile:
"""
Cached and transactional block I/O
self.root is the block number of the btree root block
"""
def __init__(self,filename,blksize=8*1024,maxrcache=0,maxwcache=0):
"""
filename names the database.
"before-"+filename is the name of the before-image file.
blksize is the size of the blocks
maxrcache is the number of blocks cached in RAM
maxwcache is the number of dirty blocks cached in RAM
"""
if not maxrcache:
maxrcache=(1024*1024)/blksize
if not maxwcache:
maxwcache=(1024*1024)/blksize
ms=len(UUID())
self.ms=ms
blksize=int(blksize)
self.bblksize=self.ms+16+blksize
if not exists(filename+'.wdb'):
f=file(filename+'.wdb','wb',blksize)
f.write(''.join((str(blksize).zfill(16),'1'.zfill(16),'0'.zfill(16),chr(255))))
f.close()
f=file(filename+'.wdbb','wb',self.bblksize)
f.write(''.ljust(self.bblksize))
f.close()
self.blksize=blksize
self.file=file(filename+'.wdb','r+b',blksize)
self.cache=cachefile(self.file,self.bimage,maxrcache=maxrcache,maxwcache=maxwcache,blksize=blksize)
r0=self.file.read(blksize)
bsz=long(r0[:16])
if bsz!=blksize:
raise Exception('cachefile has wrong blksize')
self.ms=len(UUID())
self.bblksize=ms+16+blksize
self.bfile=file(filename+'.wdbb','r+b',self.bblksize)
self.before={}
self.tran=None
self.bfile.seek(0,2)
sz=self.bfile.tell()/self.bblksize
self.bfile.seek(0)
bs=self.bfile.read(self.bblksize)
sz-=1
t=bs[:ms]
if t!=''.ljust(self.ms):
while True:
rn=long(bs[ms:ms+16])
s=bs[ms+16:]
b=self.cache.buffer(rn)
b.puts(s)
self.putb(b)
if not sz:
break
bs=self.bfile.read(self.bblksize)
sz-=1
nt=bs[:ms]
if nt!=t:
break
self.bfile.seek(0)
r0=self.getb(0).gets()
self.avail=long(r0[16:32])
self.root=long(r0[32:48])
def close(self):
"Update close the files"
self.file.close()
self.bfile.close()
def putnewl(self,l):
"Allocate 1 block"
a=self.avail
self.avail+=1
b=self.cache.buffer(a)
b.putl(l)
self.cache.put(b,force=True)
return a
def putb(self,buffer,force=False):
"write a block"
self.cache.put(buffer,force=force)
def getb(self,recnbr):
"read a block"
return self.cache.get(recnbr)
def bimage(self,b):
if b.empty():
return
recnbr=b.recnbr
if self.tran and (recnbr not in self.before):
s=b.gets()
self.bwrite(recnbr,s)
def bwrite(self,recnbr,s):
self.before[recnbr]=None
s=s.ljust(self.blksize)
bfile=self.bfile
bfile.write(''.join((self.tran,str(recnbr).zfill(16),s)))
bfile.flush()
def save(self,s):
"""
save a large string (len > blksize/2)
returns the starting block nbr
"""
l=len(s)
if (l+16)>self.blksize:
s=chr(252)+compress(s)
l=len(s)
s=str(l).zfill(16)+s
l+=16
bs=(l+self.blksize-1)/self.blksize
s=s.ljust(bs*self.blksize)
a=self.avail
self.avail+=bs
self.file.seek(a*self.blksize)
self.file.write(s)
return a
def load(self,recnbr):
"fetch a large string previously saved"
self.file.seek(recnbr*self.blksize)
sl=self.file.read(16)
l=long(sl)
c=self.file.read(1)
s=self.file.read(l-1)
if c!=chr(252):
return c+s
return decompress(s)
def transaction(self):
"start a transaction"
if self.tran:
raise Exception('tran in tran')
self.tran=UUID()
b=self.getb(0)
self.bwrite(0,b.gets())
def commit(self):
"commit a transaction"
if not self.tran:
raise Exception('commit before tran')
r0=''.join((str(self.blksize).zfill(16),str(self.avail).zfill(16),str(self.root).zfill(16),chr(255)))
b=self.cache.buffer(0)
b.puts(r0)
self.cache.put(b,force=True)
self.cache.flush()
self.bfile.seek(0)
self.bfile.write(''.ljust(self.bblksize))
self.bfile.flush()
self.bfile.seek(0)
self.before={}
self.tran=None
class btree:
"""
A write once, read many transactional btree file.
All keys and values must be 7-bit ascii.
Duplicate keys stored in chronological order.
Reserved characters:
chr(255) - record separator
chr(0) - key/value separator
chr(253) - indirect value indicator
chr(252) - compression indicator
"""
def __init__(self,filename,blksize=1024,maxrcache=0,maxwcache=0):
"""
filename names the database.
"before-"+filename is the name of the before-image file.
blksize is the size of the blocks
maxrcache is the number of blocks cached in RAM
maxwcache is the number of dirty blocks cached in RAM
"""
if blksize<64:
blksize=64
tf=tranfile(filename,blksize=blksize,maxrcache=maxrcache,maxwcache=maxwcache)
self.tf=tf
self.blksize=tf.blksize
self.lock=Lock()
self.putcount=0
if not tf.root:
tf.transaction()
tf.root=tf.putnewl(['L'+chr(255)])
tf.commit()
def close(self):
"Update block 0 and close the files"
self.tf.close()
def tran(self):
"start a transaction"
self.tf.transaction()
def commit(self):
"commit a transaction"
self.tf.commit()
def items(self,key,limit):
"yields k,v where limit <= k <= key, in descending order by k"
putcount=self.putcount
c,k,v=self._get(key)
if not k:
return
if k<limit:
return
yield k,v
while True:
if putcount!=self.putcount:
raise Exception("intervening put invalidates items")
self.lock.acquire()
try:
c,k,v=self._prior(c)
finally:
self.lock.release()
if not k:
return
if k<limit:
return
yield k,v
def haskey(self,key):
"does wdb have key"
c,k,v=self._get(key)
return k==key
def _get(self,key):
"""
Get a value.
Returns cursor,key,value.
(Cursor is None when called in a transaction.)
The returned key may be less than the argument key.
When there is no matching value, returns None,None,None.
"""
self.lock.acquire()
try:
nl,bu=self._getleaf(key)
kl=bu.getl()
l=len(kl)-1
i=bisect_left(kl,key,1,l)
if i==l:
i=l-1
if not i:
return None,None,None
k,v=kl[i].split(chr(0))
if k>key:
nl.append((bu,i))
return self._prior(nl)
if v.startswith(chr(253)):
vn=long(v[1:])
v=self.tf.load(vn)
nl.append((bu,i))
return nl,k,v
finally:
self.lock.release()
def _prior(self,nl):
"""
Get the prior value.
Returns cursor,key,value.
When there is no matching value, returns None,None,None.
"""
tf=self.tf
i=0
while len(nl):
bu,i=nl.pop()
if i>1:
break
i=0
if not i:
return None,None,None
i-=1
nl.append((bu,i))
kl=bu.getl()
k,v=kl[i].split(chr(0))
while kl[0]=='N':
bn=long(v)
bu=tf.getb(bn)
kl=bu.getl()
i=len(kl)-2
nl.append((bu,i))
k,v=kl[i].split(chr(0))
if v.startswith(chr(253)):
vn=long(v[1:])
v=self.tf.load(vn)
return nl,k,v
def _getleaf(self,key):
nl=[]
tf=self.tf
bn=tf.root
while True:
bu=tf.getb(bn)
kl=bu.getl()
if kl[0]=='L':
return nl,bu
l=len(kl)-1
i=bisect_left(kl,key,1,l)
if i==l:
i=l-1
k,sbn=kl[i].split(chr(0))
nl.append((bu,i))
bn=long(sbn)
def put(self,key,value):
"""
Add a key/value.
"""
tf=self.tf
if len(key)>self.blksize/4:
raise Exception('key length is more than blksize/4')
if not tf.tran:
raise Exception('put not in tran')
self.putcount+=1
if len(value)>self.blksize/2:
vn=tf.save(value)
value=chr(253)+str(vn)
nl,bu=self._getleaf(key)
kl=bu.getl()
f=len(kl)-2
nkv=chr(0).join((key,value))
insort_left(kl,nkv,1,len(kl)-1)
while True:
ln=0
for kv in kl:
ln+=len(kv)+1
if ln<=self.blksize:
bu.putl(kl)
tf.putb(bu)
return
ln=0
kl1=kl
kl0=[]
t=kl[0]
while ln<(self.blksize/2) and len(kl1)>2:
kv=kl1.pop(0)
ln+=len(kv)+1
kl0.append(kv)
kl0.append('')
kl1.insert(0,t)
kv0=kl0[-2]
kv1=kl1[-2]
i0=kv0.find(chr(0))
i1=kv1.find(chr(0))
k0=kv0[0:i0]
k1=kv1[0:i1]
tf=self.tf
bn0=bu.recnbr
bu.putl(kl0)
tf.putb(bu)
bn1=tf.putnewl(kl1)
v0=str(bn0)
v1=str(bn1)
if not len(nl):
print 'btree root split'
kl=['N',k0+chr(0)+v0,k1+chr(0)+v1,'']
bn=tf.putnewl(kl)
tf.root=bn
return
bu,i=nl.pop()
kl=bu.getl()
kl[i]=chr(0).join((k0,v0))
kl.insert(i+1,chr(0).join((k1,v1)))
|
|
From: Bill la F. <laf...@ya...> - 2006-01-13 14:04:06
|
import wdb
from tks.randompot import UUID
from tks.ts import ts,tsMax
from tkcs import tc
from os.path import exists
RECORD='$record%'
TIMESTAMP='!timestamp%'
#key prefixes
RPTC='v'
RVTC='p'
PVTC='r'
TC='t'
SEP= chr(1) #string seperator for keys and values
DELIM='HDwxR0cFv7P3abBzHBxDI3P4\n'
class TKS:
def __init__(self):
self.btree=wdb.btree('AgileWiki',blksize=1024)
self.lfile=file('AgileWiki.dmp','a')
lfile=self.lfile
lfile.write(DELIM)
lfile.write('user=admin\n')
lfile.write('time='+tc.display(tc.timestamp('.'))+'\n')
lfile.write(DELIM)
lfile.flush()
def close(self):
self.btree.close()
self.lfile.close()
def Transaction(self,R,T=None,U=None):
'Creates/returns a new transaction'
return Transaction(self,R,T=T,U=U)
def getDocument(self,R,P,time=None):
'Returns (document,type) or None'
ts=tsMax(time)
key=RPTC+SEP.join((R,P,ts,'999999'))
limit=''.join((RPTC,R,SEP,P,SEP))
k,v=self.btree.items(key,limit).next()
if not k:
return None
V1,V2,D=v.split(SEP)
if len(V1)==0:
return None
return D,V1
class Transaction:
def __init__(self,tks,R,T=None,U=None):
self.lfile=tks.lfile
self.btree=tks.btree
self.R=R
if U:
self.U=U
else:
self.U=UUID()
self.C=0
if T:
self.T=T
else:
self.T=ts()
self.log=[]
log=self.log
log.append(self.U+'\n')
log.append(self.T+'\n')
log.append(self.R+'\n')
self.btree.tran()
def commit(self):
log=self.log
log.append(DELIM)
log=''.join(self.log)
self.lfile.write(log)
self.lfile.flush()
self.btree.commit()
def put(self,pre,kl,dl):
key=pre+SEP.join(kl)
data=SEP.join(dl)
self.btree.put(key,data)
def putRPTC(self,P,V1,V2,D):
c=str(self.C).zfill(6)
self.put(RPTC,(self.R,P,self.T,c),(V1,V2,D))
self.put(TC,(self.T,c),(self.R,P))
self.C+=1
log=self.log
log.append('r '+self.R+'\n')
log.append('p '+P+'\n')
log.append('v1 '+V1+'\n')
log.append('v2 '+V2+'\n')
log.append('d '+D+'\n')
log.append(self.R+'\n')
def assignDocument(self,P,D,ext):
self.putRPTC(P,ext,'',D)
envSvc=TKS()
|
|
From: Bill la F. <laf...@ya...> - 2006-01-13 00:08:12
|
from time import time
from weakref import WeakValueDictionary
from heapq import heappush,heappop
from tks.randompot import UUID
from os.path import exists
from threading import Lock
class block:
"a cachable block buffer"
def __init__(self,recnbr,blksize,bimage):
self.blksize=blksize
self.recnbr=recnbr
self.bimage=bimage
self.s=None
self.l=None
self.inrheap=False
self.inwheap=False
def gets(self):
if self.s==None:
if self.l==None:
raise Exception('empty buffer')
else:
self.s=chr(255).join(self.l)
if len(self.s)>self.blksize:
raise Exception('block puts: string greater than blksize')
return self.s
def puts(self,s):
self.bimage(self)
i=s.rfind(chr(255))
if i<0:
raise Exception('record has no terminator')
s=s[:i+1]
if len(s)>self.blksize:
raise Exception('block puts: string greater than blksize')
self.s=s
self.l=None
def getl(self):
if self.l==None:
if self.s==None:
raise Exception('empty buffer')
else:
self.l=self.s.split(chr(255))
return self.l
def putl(self,l):
self.bimage(self)
self.l=l
self.s=None
def empty(self):
return self.s==None and self.l==None
def len(self):
if self.s!=None:
return len(self.s)
if self.l!=None:
ln=0
for kv in self.l:
ln+=len(kv)+1
return l
raise Exception('empty buffer')
class cachefile:
"Keeps the last accessed blocks in RAM"
def __init__(self,file,bimage,maxrcache=1000,maxwcache=1000,blksize=8*1024):
"""
maxrcache is the number of blocks kept in RAM
maxwcache is the number of dirty blocks kept in RAM
"""
self.file=file
self.bimage=bimage
self.maxr=maxrcache
self.maxw=maxwcache
self.blksize=blksize
self.wd=WeakValueDictionary()
self.rheap=[]
self.wheap=[]
def flush(self):
"flush all pending writes"
wheap=self.wheap
dl=[]
for ts,b in wheap:
if not b.inwheap:
continue
dl.append((b.recnbr,b.gets()))
dl.sort()
for recnbr,s in dl:
s=s.ljust(self.blksize)
self.file.seek(self.blksize*recnbr)
self.file.write(s)
self.file.flush()
def buffer(self,recnbr):
"locate a buffer or create a new one"
try:
b=self.wd[recnbr]
except:
b=block(recnbr,self.blksize,self.bimage)
return b
def put(self,buffer,force=False):
"write a block"
recnbr=buffer.recnbr
self.add(recnbr,buffer)
ts=time()
buffer.timeoflastaccess=ts
if force:
buffer.inwheap=False
s=buffer.gets()
s=s.ljust(self.blksize)
self.file.seek(self.blksize*recnbr)
self.file.write(s)
return
if buffer.inwheap:
return
buffer.inwheap=True
heappush(self.wheap,(ts,buffer))
while len(self.wheap):
t,b=heappop(self.wheap)
ts=b.timeoflastaccess
if not b.inwheap:
continue
if t==ts:
b.inwheap=False
s=b.gets()
s=s.ljust(self.blksize)
self.file.seek(self.blksize*recnbr)
self.file.write(s)
continue
heappush(self.rheap,(ts,b))
def get(self,recnbr):
"locate a block or read it."
try:
buffer=self.wd[recnbr]
except:
self.file.seek(self.blksize*recnbr)
buffer=self.buffer(recnbr)
buffer.puts(self.file.read(self.blksize))
self.add(recnbr,buffer)
buffer.timeoflastaccess=time()
return buffer
def add(self,recnbr,buffer):
"add a buffer to the cache"
self.wd[recnbr]=buffer
ts=time()
buffer.timeoflastaccess=ts
if buffer.inrheap:
return
heappush(self.rheap,(ts,buffer))
buffer.inrheap=True
while len(self.rheap):
t,b=heappop(self.rheap)
ts=b.timeoflastaccess
if t==ts:
b.inrheap=False
continue
heappush(self.rheap,(ts,b))
class tranfile:
"""
Cached and transactional block I/O
self.root is the block number of the btree root block
"""
def __init__(self,filename,blksize=8*1024,maxrcache=0,maxwcache=0):
"""
filename names the database.
"before-"+filename is the name of the before-image file.
blksize is the size of the blocks
maxrcache is the number of blocks cached in RAM
maxwcache is the number of dirty blocks cached in RAM
"""
if not maxrcache:
maxrcache=(1024*1024)/blksize
if not maxwcache:
maxwcache=(1024*1024)/blksize
ms=len(UUID())
self.ms=ms
blksize=int(blksize)
self.bblksize=self.ms+16+blksize
if not exists(filename+'.wdb'):
f=file(filename+'.wdb','wb',blksize)
f.write(''.join((str(blksize).zfill(16),'1'.zfill(16),'0'.zfill(16),chr(255))))
f.close()
f=file(filename+'.wdbb','wb',self.bblksize)
f.write(''.ljust(self.bblksize))
f.close()
self.blksize=blksize
self.file=file(filename+'.wdb','r+b',blksize)
self.cache=cachefile(self.file,self.bimage,maxrcache=maxrcache,maxwcache=maxwcache,blksize=blksize)
r0=self.file.read(blksize)
bsz=long(r0[:16])
if bsz!=blksize:
raise Exception('cachefile has wrong blksize')
self.ms=len(UUID())
self.bblksize=ms+16+blksize
self.bfile=file(filename+'.wdbb','r+b',self.bblksize)
self.before={}
self.tran=None
self.bfile.seek(0,2)
sz=self.bfile.tell()/self.bblksize
self.bfile.seek(0)
bs=self.bfile.read(self.bblksize)
sz-=1
t=bs[:ms]
if t!=''.ljust(self.ms):
while True:
rn=bs[ms:ms+16]
s=bs[ms+16:]
b=self.cache.buffer(rn)
b.puts(s)
self.putb(b)
if not sz:
break
bs=self.bfile.read(self.bblksize)
sz-=1
nt=bs[:ms]
if nt!=t:
break
self.bfile.seek(0)
r0=self.getb(0).gets()
self.avail=long(r0[16:32])
self.root=long(r0[32:48])
def close(self):
"Update close the files"
self.file.close()
self.bfile.close()
def putnewl(self,l):
"Allocate 1 block"
a=self.avail
self.avail+=1
b=self.cache.buffer(a)
b.putl(l)
self.cache.put(b,force=True)
return a
def putb(self,buffer,force=False):
"write a block"
self.cache.put(buffer,force=force)
def getb(self,recnbr):
"read a block"
return self.cache.get(recnbr)
def bimage(self,b):
if b.empty():
return
recnbr=b.recnbr
if self.tran and (recnbr not in self.before):
s=b.gets()
self.bwrite(recnbr,s)
def bwrite(self,recnbr,s):
self.before[recnbr]=None
s=s.ljust(self.blksize)
bfile=self.bfile
bfile.write(''.join((self.tran,str(recnbr).zfill(16),s)))
bfile.flush()
def save(self,s):
"""
save a large string (len > blksize/2)
returns the starting block nbr
"""
l=len(s)
s=str(l).zfill(16)+s
l+=16
bs=(l+self.blksize-1)/self.blksize
s=s.ljust(bs*self.blksize)
a=self.avail
self.avail+=bs
self.file.seek(a*blksize)
self.file.write(s)
def load(self,recnbr):
"fetch a large string previously saved"
self.file.seek(recnbr*self.blksize)
sl=self.file.read(16)
l=long(sl)
return self.file.read(l)
def transaction(self):
"start a transaction"
if self.tran:
raise Exception('tran in tran')
self.tran=UUID()
b=self.getb(0)
self.bwrite(0,b.gets())
def commit(self):
"commit a transaction"
if not self.tran:
raise Exception('commit before tran')
r0=''.join((str(self.blksize).zfill(16),str(self.avail).zfill(16),str(self.root).zfill(16),chr(255)))
b=self.cache.buffer(0)
b.puts(r0)
self.cache.put(b,force=True)
self.cache.flush()
self.bfile.seek(0)
self.bfile.write(''.ljust(self.bblksize))
self.bfile.flush()
self.bfile.seek(0)
self.before={}
self.tran=None
class btree:
"""
A write once, read many transactional btree file.
All keys and values must be 7-bit ascii.
Duplicate keys stored in chronological order.
Reserved characters:
chr(255) - record separator
chr(254) - key/value separator
chr(253) - indirect value indicator
"""
def __init__(self,filename,blksize=1024,maxrcache=0,maxwcache=0):
"""
filename names the database.
"before-"+filename is the name of the before-image file.
blksize is the size of the blocks
maxrcache is the number of blocks cached in RAM
maxwcache is the number of dirty blocks cached in RAM
"""
if blksize<64:
blksize=64
tf=tranfile(filename,blksize=blksize,maxrcache=maxrcache,maxwcache=maxwcache)
self.tf=tf
self.blksize=tf.blksize
self.lock=Lock()
self.putcount=0
if not tf.root:
tf.transaction()
tf.root=tf.putnewl(['L'+chr(255)])
tf.commit()
def close(self):
"Update block 0 and close the files"
self.tf.close()
def tran(self):
"start a transaction"
self.tf.transaction()
def commit(self):
"commit a transaction"
self.tf.commit()
def items(self,key,limit):
"yields k,v where limit <= k <= key, in descending order by k"
putcount=self.putcount
c,k,v=self._get(key)
if not k:
return
if k<limit:
return
yield k,v
while True:
if putcount!=self.putcount:
raise Exception("intervening put invalidates items")
c,k,v=self._prior(c)
if not k:
return
if k<limit:
return
yield k,v
def _get(self,key):
"""
Get a value.
Returns cursor,key,value.
(Cursor is None when called in a transaction.)
The returned key may be less than the argument key.
When there is no matching value, returns None,None,None.
"""
self.lock.acquire()
try:
nl,bu=self._getleaf(key)
kl=bu.getl()
f=len(kl)-2
for i in range(f,0,-1):
k,v=kl[i].split(chr(254))
if k<=key or i==1:
break
if k>key:
return None,None,None
if v.startswith(chr(253)):
vn=long(v[1:])
v=self.tf.load(vn)
nl.append((bu,i))
if self.tf.tran:
return None,k,v
return nl,k,v
finally:
self.lock.release()
def _prior(self,nl):
"""
Get the prior value.
Returns cursor,key,value.
When there is no matching value, returns None,None,None.
"""
self.lock.acquire()
try:
tf=self.tf
i=0
while len(nl):
bu,i=nl.pop()
if i>1:
break
i=0
if not i:
return None,None,None
i-=1
nl.append((bu,i))
kl=bu.getl()
k,v=kl[i].split(chr(254))
while kl[0]=='N':
bn=long(v)
bu=tf.getb(bn)
kl=bu.getl()
i=len(kl)-2
nl.append((bu,i))
k,v=kl[i].split(chr(254))
if v.startswith(chr(253)):
vn=long(v[1:])
v=self.tf.load(vn)
return nl,k,v
finally:
self.lock.release()
def _getleaf(self,key):
nl=[]
tf=self.tf
bn=tf.root
while True:
bu=tf.getb(bn)
kl=bu.getl()
if kl[0]=='L':
return nl,bu
f=len(kl)-2
for i in range(f,0,-1):
k,sbn=kl[i].split(chr(254))
if k<=key or i==1:
break
nl.append((bu,i))
bn=long(sbn)
def put(self,key,value):
"""
Add a key/value.
"""
tf=self.tf
if len(key)>self.blksize/4:
raise Exception('key length is more than blksize/4')
if not tf.tran:
raise Exception('put not in tran')
self.putcount+=1
if len(value)>self.blksize/2:
vn=tf.save(value)
value=chr(253)+str(vn)
nl,bu=self._getleaf(key)
kl=bu.getl()
f=len(kl)-2
if f:
for i in range(f,0,-1):
k,v=kl[i].split(chr(254))
if k<=key or i==1:
break
if k<=key:
i+=1
else:
i=1
nkv=chr(254).join((key,value))
kl.insert(i,nkv)
while True:
ln=0
for kv in kl:
ln+=len(kv)+1
if ln<=self.blksize:
bu.putl(kl)
tf.putb(bu)
return
ln=0
kl1=kl
kl0=[]
t=kl[0]
while ln<(self.blksize/2) and len(kl1)>2:
kv=kl1.pop(0)
ln+=len(kv)+1
kl0.append(kv)
kl0.append('')
kl1.insert(0,t)
kv0=kl0[1]
kv1=kl1[1]
i0=kv0.find(chr(254))
i1=kv1.find(chr(254))
k0=kv0[0:i0]
k1=kv1[0:i1]
tf=self.tf
bn0=bu.recnbr
bu.putl(kl0)
tf.putb(bu)
bn1=tf.putnewl(kl1)
v0=str(bn0)
v1=str(bn1)
if not len(nl):
print 'btree root split'
kl=['N',k0+chr(254)+v0,k1+chr(254)+v1,'']
bn=tf.putnewl(kl)
tf.root=bn
return
bu,i=nl.pop()
kl=bu.getl()
kl[i]=chr(254).join((k0,v0))
kl.insert(i+1,chr(254).join((k1,v1)))
|
|
From: Bill la F. <laf...@ya...> - 2006-01-08 12:54:00
|
Well, I've completed a brief description of each of the utilities. This may be under AwiDocs/scripts, but I'm thinking of putting it in AwiDocs/RuntimeEnvironment/scripts--I'm expanding the scope! Nothing especially useful, I just want to cover file types and directories in the RuntimeEnvironment, as well as scripts. This could make it all a lot easier for someone new to get oriented. entended entirely as a suppliment to GettingStarted, of course. Bill Send instant messages to your online friends http://in.messenger.yahoo.com |
|
From: Bill la F. <laf...@ya...> - 2006-01-08 10:45:14
|
I should have said
http://agilewiki.org/wiki/cc/AgileWiki/06/w01
(I forgot the year.)
Bill
Blog: http://laforge49.blogspot.com
Web: http://AgileWiki.org
Send instant messages to your online friends http://in.messenger.yahoo.com |
|
From: Bill la F. <laf...@ya...> - 2006-01-08 10:39:47
|
Norm,
As I know you like using email, I would suggest that we use this list--I believe you are still a member--for Ark related issues, like using a new feature or installing a new release.
As it is journaled in the AgileWiki cabinet, you can view its archive simply by visiting the appropriate weekly page, like:
http://agilewiki.org/wiki/cc/AgileWiki/w01
(or w02, w03, etc.).
I'll continue to use the forum for releases and the blog for commentary. But this list would also be a good place for technical postings too, especially as I begin work on rewriting tks.
Bill
Blog: http://laforge49.blogspot.com
Web: http://AgileWiki.org
Send instant messages to your online friends http://in.messenger.yahoo.com |
|
From: Bill la F. <laf...@ya...> - 2006-01-08 10:16:22
|
Email set to this list is now being archived in the AgileWiki Cabinet's journal, together with email from the discussion forum. Bill Blog: http://laforge49.blogspot.com Web: http://AgileWiki.org Send instant messages to your online friends http://in.messenger.yahoo.com |
14 messages has been excluded from this view by a project administrator.
