assorted-commits

[Assorted-commits] SF.net SVN: assorted: [228] python-commons/trunk/src/commons/files.py

[<yan...@us...>]

Revision: 228
          http://assorted.svn.sourceforge.net/assorted/?rev=228&view=rev
Author:   yangzhang
Date:     2008-01-12 18:30:11 -0800 (Sat, 12 Jan 2008)

Log Message:
-----------
added disk double buffer

Modified Paths:
--------------
    python-commons/trunk/src/commons/files.py

Modified: python-commons/trunk/src/commons/files.py
===================================================================
--- python-commons/trunk/src/commons/files.py	2008-01-13 02:29:58 UTC (rev 227)
+++ python-commons/trunk/src/commons/files.py	2008-01-13 02:30:11 UTC (rev 228)
@@ -11,6 +11,8 @@
 from L{invalid_filename_chars}.
 """
 
+from __future__ import with_statement
+
 import os, re, tempfile
 
 def soft_makedirs( path ):
@@ -76,3 +78,28 @@
     pattern = invalid_filename_chars_regex
     return re.sub( pattern, '_', filename )
 
+class disk_double_buffer( object ):
+    """
+    A simple disk double-buffer. One file is for reading, the other is for
+    writing, and a facility for swapping the two roles is provided.
+    """
+    def __init__( self, path ):
+        self.paths = [ path + '.0', path + '.1' ]
+        self.w, self.r = 0, 1
+        # initialize empty reader file
+        with file( self.paths[ self.r ], 'w' ): pass
+        self.reload_files()
+    def reload_files( self ):
+        self.writer = file( self.paths[ self.w ], 'w' )
+        self.reader = file( self.paths[ self.r ] )
+    def switch( self ):
+        self.r, self.w = self.w, self.r
+        self.close()
+        self.reload_files()
+    def write( self, x ):
+        self.writer.write( x )
+    def read( self, len = 8192 ):
+        return self.reader.read( len )
+    def close( self ):
+        self.reader.close()
+        self.writer.close()


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

[Assorted-commits] SF.net SVN: assorted: [232] python-commons/trunk/src/commons/files.py

[<yan...@us...>]

Revision: 232
          http://assorted.svn.sourceforge.net/assorted/?rev=232&view=rev
Author:   yangzhang
Date:     2008-01-17 15:13:45 -0800 (Thu, 17 Jan 2008)

Log Message:
-----------
added persistence to disk double buffers

Modified Paths:
--------------
    python-commons/trunk/src/commons/files.py

Modified: python-commons/trunk/src/commons/files.py
===================================================================
--- python-commons/trunk/src/commons/files.py	2008-01-17 23:11:45 UTC (rev 231)
+++ python-commons/trunk/src/commons/files.py	2008-01-17 23:13:45 UTC (rev 232)
@@ -15,6 +15,8 @@
 
 import os, re, tempfile
 
+from path import path
+
 def soft_makedirs( path ):
     """
     Emulate C{mkdir -p} (doesn't complain if it already exists).
@@ -83,9 +85,14 @@
     A simple disk double-buffer. One file is for reading, the other is for
     writing, and a facility for swapping the two roles is provided.
     """
-    def __init__( self, path ):
-        self.paths = [ path + '.0', path + '.1' ]
-        self.w, self.r = 0, 1
+    def __init__( self, path_base, do_persist = True ):
+        self.paths = [ path_base + '.0', path_base + '.1' ]
+        self.do_persist = do_persist
+        self.switch_status = path( path_base + '.switched' )
+        if not do_persist or not self.switch_status.exists():
+            self.w, self.r = 0, 1 # default
+        else:
+            self.w, self.r = 1, 0
         # initialize empty reader file
         with file( self.paths[ self.r ], 'w' ): pass
         self.reload_files()
@@ -93,8 +100,11 @@
         self.writer = file( self.paths[ self.w ], 'w' )
         self.reader = file( self.paths[ self.r ] )
     def switch( self ):
+        self.close()
+        if self.do_persist:
+            if self.w == 0: self.switch_status.touch()
+            else:           self.switch_status.remove()
         self.r, self.w = self.w, self.r
-        self.close()
         self.reload_files()
     def write( self, x ):
         self.writer.write( x )


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

[Assorted-commits] SF.net SVN: assorted: [235] python-commons/trunk/src/commons/files.py

[<yan...@us...>]

Revision: 235
          http://assorted.svn.sourceforge.net/assorted/?rev=235&view=rev
Author:   yangzhang
Date:     2008-01-17 15:57:23 -0800 (Thu, 17 Jan 2008)

Log Message:
-----------
fixed accidental wiping out of read-file in persistent disk double buffer

Modified Paths:
--------------
    python-commons/trunk/src/commons/files.py

Modified: python-commons/trunk/src/commons/files.py
===================================================================
--- python-commons/trunk/src/commons/files.py	2008-01-17 23:41:52 UTC (rev 234)
+++ python-commons/trunk/src/commons/files.py	2008-01-17 23:57:23 UTC (rev 235)
@@ -86,18 +86,18 @@
     writing, and a facility for swapping the two roles is provided.
     """
     def __init__( self, path_base, do_persist = True ):
-        self.paths = [ path_base + '.0', path_base + '.1' ]
+        self.paths = map( path, [ path_base + '.0', path_base + '.1' ] )
         self.do_persist = do_persist
         self.switch_status = path( path_base + '.switched' )
         if not do_persist or not self.switch_status.exists():
             self.w, self.r = 0, 1 # default
         else:
             self.w, self.r = 1, 0
-        # initialize empty reader file
-        with file( self.paths[ self.r ], 'w' ): pass
         self.reload_files()
     def reload_files( self ):
         self.writer = file( self.paths[ self.w ], 'w' )
+        if not self.paths[ self.r ].exists():
+            self.paths[ self.r ].touch()
         self.reader = file( self.paths[ self.r ] )
     def switch( self ):
         self.close()


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

[Assorted-commits] SF.net SVN: assorted: [688] python-commons/trunk/src/commons/files.py

[<yan...@us...>]

Revision: 688
          http://assorted.svn.sourceforge.net/assorted/?rev=688&view=rev
Author:   yangzhang
Date:     2008-04-28 22:18:57 -0700 (Mon, 28 Apr 2008)

Log Message:
-----------
added versioned_guard, versioned_cache

Modified Paths:
--------------
    python-commons/trunk/src/commons/files.py

Modified: python-commons/trunk/src/commons/files.py
===================================================================
--- python-commons/trunk/src/commons/files.py	2008-04-29 00:32:42 UTC (rev 687)
+++ python-commons/trunk/src/commons/files.py	2008-04-29 05:18:57 UTC (rev 688)
@@ -14,7 +14,7 @@
 from __future__ import with_statement
 
 import os, re, tempfile
-
+from cPickle import *
 from path import path
 
 def soft_makedirs( path ):
@@ -113,3 +113,59 @@
     def close( self ):
         self.reader.close()
         self.writer.close()
+
+def versioned_guard(path, fresh_version):
+    """
+    Maintain a version object.  This is useful for working with versioned
+    caches.
+
+    @param path: The path to the file containing the cached version object.
+    @type path: str
+
+    @param fresh_version: The actual latest version that the cached version
+    should be compared against.
+    @type fresh_version: object (any type that can be compared)
+
+    @return: True iff the cached version is obsolete (less than the fresh
+    version or doesn't exist).
+    @rtype: bool
+    """
+    try:
+        with file( path ) as f: cache_version = load(f)
+    except IOError, (errno, errstr):
+        if errno != 2: raise
+        return True
+    else:
+        if fresh_version > cache_version:
+            with file( path, 'w' ) as f: dump(fresh_version, f)
+            return True
+        else:
+            return False
+
+def versioned_cache(version_path, fresh_version, cache_path, cache_func):
+    """
+    If fresh_version is newer than the version in version_path, then invoke
+    cache_func and cache the result in cache_path (using pickle).
+
+    @param version_path: The path to the file version.
+    @type version_path: str
+
+    @param fresh_version: The actual, up-to-date version value.
+    @type fresh_version: object (any type that can be compared)
+
+    @param cache_path: The path to the cached data.
+    @type cache_path: str
+
+    @param cache_func: The function that produces the fresh data to be cached.
+    @type cache_func: function (no arguments)
+    """
+    if versioned_guard( version_path, fresh_version ):
+        # cache obsolete, force-fetch new data
+        print 'here'
+        result = cache_func()
+        with file(cache_path, 'w') as f: dump(result, f)
+        return result
+    else:
+        # cache up-to-date (should be available since dlcs-timestamp exists!)
+        print 'there'
+        with file(cache_path) as f: return load(f)


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

[Assorted-commits] SF.net SVN: assorted: [690] python-commons/trunk/src/commons/files.py

[<yan...@us...>]

Revision: 690
          http://assorted.svn.sourceforge.net/assorted/?rev=690&view=rev
Author:   yangzhang
Date:     2008-04-29 16:03:05 -0700 (Tue, 29 Apr 2008)

Log Message:
-----------
fixed bug in versioned_guard; clarified docstring on versioned_cache

Modified Paths:
--------------
    python-commons/trunk/src/commons/files.py

Modified: python-commons/trunk/src/commons/files.py
===================================================================
--- python-commons/trunk/src/commons/files.py	2008-04-29 23:00:50 UTC (rev 689)
+++ python-commons/trunk/src/commons/files.py	2008-04-29 23:03:05 UTC (rev 690)
@@ -130,23 +130,26 @@
     version or doesn't exist).
     @rtype: bool
     """
+    cache_version = None
     try:
         with file( path ) as f: cache_version = load(f)
     except IOError, (errno, errstr):
         if errno != 2: raise
+    print cache_version, fresh_version
+    if cache_version is None or fresh_version > cache_version:
+        with file( path, 'w' ) as f: dump(fresh_version, f)
         return True
     else:
-        if fresh_version > cache_version:
-            with file( path, 'w' ) as f: dump(fresh_version, f)
-            return True
-        else:
-            return False
+        return False
 
 def versioned_cache(version_path, fresh_version, cache_path, cache_func):
     """
     If fresh_version is newer than the version in version_path, then invoke
     cache_func and cache the result in cache_path (using pickle).
 
+    Note the design flaw with L{versioned_guard}: the updated version value is
+    stored immediately, rather than after updating the cache.
+
     @param version_path: The path to the file version.
     @type version_path: str
 


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.