etch-devel — Discussion regarding the development of etch

[etch-devel] SF.net SVN: etch:[144] trunk/test/alltests.rb

[<jh...@us...>]

Revision: 144
          http://etch.svn.sourceforge.net/etch/?rev=144&view=rev
Author:   jheiss
Date:     2009-11-11 00:53:09 +0000 (Wed, 11 Nov 2009)

Log Message:
-----------
Set xmllib environment variable

Modified Paths:
--------------
    trunk/test/alltests.rb

Modified: trunk/test/alltests.rb
===================================================================
--- trunk/test/alltests.rb	2009-11-11 00:48:56 UTC (rev 143)
+++ trunk/test/alltests.rb	2009-11-11 00:53:09 UTC (rev 144)
@@ -1,5 +1,8 @@
 #!/usr/bin/ruby -w
 
+# Some of the tests involve DTD validation.  For that we need LibXML.
+ENV['xmllib'] = 'libxml'
+
 #
 # Run all of the etch test cases
 #


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

[etch-devel] SF.net SVN: etch:[143] trunk/test/alltests.rb

[<jh...@us...>]

Revision: 143
          http://etch.svn.sourceforge.net/etch/?rev=143&view=rev
Author:   jheiss
Date:     2009-11-11 00:48:56 +0000 (Wed, 11 Nov 2009)

Log Message:
-----------
Add chdir to the test directory so this script can be run from anywhere.

Modified Paths:
--------------
    trunk/test/alltests.rb

Modified: trunk/test/alltests.rb
===================================================================
--- trunk/test/alltests.rb	2009-11-11 00:48:13 UTC (rev 142)
+++ trunk/test/alltests.rb	2009-11-11 00:48:56 UTC (rev 143)
@@ -5,6 +5,7 @@
 #
 
 require 'test/unit'
+Dir.chdir(File.dirname(__FILE__))
 Dir.foreach('.') do |entry|
   next unless entry =~ /\.rb$/
   # Skip this file


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

[etch-devel] SF.net SVN: etch:[142] trunk/test/actions.rb

[<jh...@us...>]

Revision: 142
          http://etch.svn.sourceforge.net/etch/?rev=142&view=rev
Author:   jheiss
Date:     2009-11-11 00:48:13 +0000 (Wed, 11 Nov 2009)

Log Message:
-----------
Add a test of an action that involves passing an environment variable.
I.e. syntax like <exec>PATH=/bin someapp</exec>

Modified Paths:
--------------
    trunk/test/actions.rb

Modified: trunk/test/actions.rb
===================================================================
--- trunk/test/actions.rb	2009-11-06 19:54:39 UTC (rev 141)
+++ trunk/test/actions.rb	2009-11-11 00:48:13 UTC (rev 142)
@@ -423,6 +423,49 @@
     assert_equal("post\n", get_file_contents("#{@repodir}/post_with_escape"), 'post with escape')
   end
 
+  def test_action_with_env
+    #
+    # Test an action involving passing an environment variable
+    #
+    
+    FileUtils.mkdir_p("#{@repodir}/source/#{@targetfile}")
+    File.open("#{@repodir}/source/#{@targetfile}/config.xml", 'w') do |file|
+      file.puts <<-EOF
+        <config>
+          <file>
+            <warning_file/>
+            <source>
+              <plain>source</plain>
+            </source>
+          </file>
+          <post>
+            <exec>TESTVAR=testvalue #{@repodir}/post_with_env</exec>
+          </post>
+        </config>
+      EOF
+    end
+    
+    sourcecontents = "This is a test of a post with an environment variable\n"
+    File.open("#{@repodir}/source/#{@targetfile}/source", 'w') do |file|
+      file.write(sourcecontents)
+    end
+    
+    File.open("#{@repodir}/post_with_env", 'w') do |file|
+      file.write <<EOF
+#!/bin/sh
+echo $TESTVAR >> #{@repodir}/post_with_env_output
+EOF
+    end
+    File.chmod(0755, "#{@repodir}/post_with_env")
+    
+    # Run etch
+    #puts "Running environment variable test"
+    run_etch(@port, @testbase)
+    
+    # Verify that the action was executed
+    assert_equal("testvalue\n", get_file_contents("#{@repodir}/post_with_env_output"), 'post with environment variable')
+  end
+
   def teardown
     stop_server(@pid)
     remove_repository(@repodir)


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

[etch-devel] SF.net SVN: etch:[141] trunk/client/etchclient.rb

[<jh...@us...>]

Revision: 141
          http://etch.svn.sourceforge.net/etch/?rev=141&view=rev
Author:   jheiss
Date:     2009-11-06 19:54:39 +0000 (Fri, 06 Nov 2009)

Log Message:
-----------
Catch version number up.

Modified Paths:
--------------
    trunk/client/etchclient.rb

Modified: trunk/client/etchclient.rb
===================================================================
--- trunk/client/etchclient.rb	2009-11-06 19:54:22 UTC (rev 140)
+++ trunk/client/etchclient.rb	2009-11-06 19:54:39 UTC (rev 141)
@@ -35,7 +35,7 @@
 require 'etch'
 
 class Etch::Client
-  VERSION = '1.18'
+  VERSION = '3.10'
   
   CONFIRM_PROCEED = 1
   CONFIRM_SKIP = 2


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

[etch-devel] SF.net SVN: etch:[140] trunk/server/lib/etch.rb

[<jh...@us...>]

Revision: 140
          http://etch.svn.sourceforge.net/etch/?rev=140&view=rev
Author:   jheiss
Date:     2009-11-06 19:54:22 +0000 (Fri, 06 Nov 2009)

Log Message:
-----------
Wrap exceptions from templates with a message indicating the template
and associated file names to help the user figure out where the error
occurred.  Already doing this for scripts, guess I forgot to do the same
for templates.

Modified Paths:
--------------
    trunk/server/lib/etch.rb

Modified: trunk/server/lib/etch.rb
===================================================================
--- trunk/server/lib/etch.rb	2009-11-06 01:04:05 UTC (rev 139)
+++ trunk/server/lib/etch.rb	2009-11-06 19:54:22 UTC (rev 140)
@@ -1339,7 +1339,13 @@
     # The binding arg ties the template's namespace to this point in the
     # code, thus ensuring that all of the variables above (@file, etc.)
     # are visible to the template code.
-    erb.result(binding)
+    begin
+      erb.result(binding)
+    rescue Exception => e
+      # Help the user figure out where the exception occurred, otherwise they
+      # just get told it happened here, which isn't very helpful.
+      raise e.exception("Exception while processing template #{template} for file #{@file}:\n" + e.message)
+    end
   end
 
   # This method runs a etch script (as specified via a <script> entry


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

[etch-devel] SF.net SVN: etch:[139] tags

[<jh...@us...>]

Revision: 139
          http://etch.svn.sourceforge.net/etch/?rev=139&view=rev
Author:   jheiss
Date:     2009-11-06 01:04:05 +0000 (Fri, 06 Nov 2009)

Log Message:
-----------
Tag 3.10 release

Modified Paths:
--------------
    Makefile

Added Paths:
-----------
    tags/release-3.10/

Modified: Makefile
===================================================================
--- Makefile	2009-11-06 00:56:57 UTC (rev 138)
+++ Makefile	2009-11-06 01:04:05 UTC (rev 139)
@@ -1,4 +1,4 @@
-VER=3.9
+VER=3.10
 TAGNAME=release-$(VER)
 
 all: dist


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

[etch-devel] SF.net SVN: etch:[138] trunk/client/etchclient.rb

[<jh...@us...>]

Revision: 138
          http://etch.svn.sourceforge.net/etch/?rev=138&view=rev
Author:   jheiss
Date:     2009-11-06 00:56:57 +0000 (Fri, 06 Nov 2009)

Log Message:
-----------
Update the temporary original placeholder files only if they aren't
currently correct, rather than updating them every time.  Some machines
may never get a given file, in which case we spit out a message to the
user and update the placeholder every time etch runs, which quickly gets
annoying.

Reverse the return value of compare_file_contents, compare_permissions 
and compare_ownership so that it makes more sense, and clean up the
code.  compare_link_destination was flipped a while back for the same 
reason.

Modified Paths:
--------------
    trunk/client/etchclient.rb

Modified: trunk/client/etchclient.rb
===================================================================
--- trunk/client/etchclient.rb	2009-11-06 00:47:38 UTC (rev 137)
+++ trunk/client/etchclient.rb	2009-11-06 00:56:57 UTC (rev 138)
@@ -606,7 +606,7 @@
 
           set_file_contents = false
           if newcontents
-            set_file_contents = compare_file_contents(file, newcontents)
+            set_file_contents = !compare_file_contents(file, newcontents)
           end
           set_permissions = nil
           set_ownership = nil
@@ -618,8 +618,8 @@
             set_permissions = true
             set_ownership = true
           else
-            set_permissions = compare_permissions(file, perms)
-            set_ownership = compare_ownership(file, uid, gid)
+            set_permissions = !compare_permissions(file, perms)
+            set_ownership = !compare_ownership(file, uid, gid)
           end
 
           # Proceed if:
@@ -783,8 +783,8 @@
           
                 # Check the permissions and ownership now to ensure they
                 # end up set properly
-                set_permissions = compare_permissions(file, perms)
-                set_ownership = compare_ownership(file, uid, gid)
+                set_permissions = !compare_permissions(file, perms)
+                set_ownership = !compare_ownership(file, uid, gid)
               end
             end
 
@@ -905,7 +905,7 @@
             if set_link_destination && !File.symlink?(file)
               set_permissions = true
             else
-              set_permissions = compare_permissions(file, perms)
+              set_permissions = !compare_permissions(file, perms)
             end
           end
           set_ownership = false
@@ -913,7 +913,7 @@
             if set_link_destination && !File.symlink?(file)
               set_ownership = true
             else
-              set_ownership = compare_ownership(file, uid, gid)
+              set_ownership = !compare_ownership(file, uid, gid)
             end
           end
 
@@ -1021,10 +1021,10 @@
               # Check the permissions and ownership now to ensure they
               # end up set properly
               if @lchmod_supported
-                set_permissions = compare_permissions(file, perms)
+                set_permissions = !compare_permissions(file, perms)
               end
               if @lchown_supported
-                set_ownership = compare_ownership(file, uid, gid)
+                set_ownership = !compare_ownership(file, uid, gid)
               end
             end
 
@@ -1107,8 +1107,8 @@
             set_permissions = true
             set_ownership = true
           else
-            set_permissions = compare_permissions(file, perms)
-            set_ownership = compare_ownership(file, uid, gid)
+            set_permissions = !compare_permissions(file, perms)
+            set_ownership = !compare_ownership(file, uid, gid)
           end
 
           # Proceed if:
@@ -1187,8 +1187,8 @@
 
               # Check the permissions and ownership now to ensure they
               # end up set properly
-              set_permissions = compare_permissions(file, perms)
-              set_ownership = compare_ownership(file, uid, gid)
+              set_permissions = !compare_permissions(file, perms)
+              set_ownership = !compare_ownership(file, uid, gid)
             end
 
             # Ensure the permissions are set properly
@@ -1512,24 +1512,18 @@
     continue_processing
   end
   
-  # Returns true if the new contents are different from the current file,
-  # or if the file does not currently exist.
+  # Returns true if the new contents are the same as the current file,
+  # false if the contents differ or if the file does not currently exist.
   def compare_file_contents(file, newcontents)
-    r = false
-
-    # If the file currently exists and is a regular file, check to see
-    # if the new contents are different.
+    # If the file currently exists and is a regular file then check to see
+    # if the new contents are the same.
     if File.file?(file) && !File.symlink?(file)
       contents = IO.read(file)
-      if newcontents != contents
-        r = true
+      if newcontents == contents
+        return true
       end
-    else
-      # The file doesn't currently exist or isn't a regular file
-      r = true
     end
-
-    r
+    false
   end
 
   # Returns true if the given file is a symlink which points to the given
@@ -1602,14 +1596,7 @@
         puts "Making directory tree #{origdir}"
         FileUtils.mkpath(origdir) if (!@dryrun)
       end
-
-      # If we're going to be using a temporary file clean up any
-      # existing one so we don't have to worry about overwriting it.
-      if save_directory_contents.nil? &&
-         (File.exist?(tmporigpath) || File.symlink?(tmporigpath))
-        remove_file(tmporigpath)
-      end
-
+      
       if File.directory?(file) && !File.symlink?(file)
         # The original "file" is a directory
         if save_directory_contents
@@ -1635,8 +1622,11 @@
           # (i.e. save_directory_contents is nil) then just save a
           # placeholder until we do get a definitive directive.
           origpath = tmporigpath
-          puts "Creating temporary original placeholder #{origpath} for directory #{file}"
-          File.open(origpath, 'w') { |file| } if (!@dryrun)
+          if !File.directory?(tmporigpath) || File.symlink?(tmporigpath)
+            puts "Creating temporary original placeholder #{tmporigpath} for directory #{file}"
+            remove_file(tmporigpath) if (!@dryrun)
+            Dir.mkdir(tmporigpath) if (!@dryrun)
+          end
           first_update = false
         else
           # Just create a directory in the originals repository with
@@ -1653,28 +1643,42 @@
         end
       elsif File.exist?(file) || File.symlink?(file)
         # The original file exists, and is not a directory
+        proceed = true
         if save_directory_contents.nil?
           origpath = tmporigpath
-          puts "Saving temporary copy of original file:  #{file} -> #{origpath}"
+          if File.exist?(tmporigpath) && !File.symlink?(tmporigpath) && compare_file_contents(tmporigpath, File.read(file))
+            proceed = false
+          else
+            puts "Saving temporary copy of original file:  #{file} -> #{origpath}"
+          end
         else
           origpath = "#{origpathbase}.ORIG"
           puts "Saving original file:  #{file} -> #{origpath}"
         end
-        filedir = File.dirname(file)
-        filebase = File.basename(file)
-        recursive_copy_and_rename(filedir, filebase, origpath) if (!@dryrun)
+        if proceed
+          filedir = File.dirname(file)
+          filebase = File.basename(file)
+          recursive_copy_and_rename(filedir, filebase, origpath) if (!@dryrun)
+        end
       else
         # If the original doesn't exist, we need to flag that so
         # that we don't try to save our generated file as an
         # original on future runs
+        proceed = true
         if save_directory_contents.nil?
           origpath = tmporigpath
-          puts "Original file #{file} doesn't exist, saving that state temporarily as #{origpath}"
+          if File.exist?(tmporigpath) && !File.symlink?(tmporigpath) && File.stat(tmporigpath).zero?
+            proceed = false
+          else
+            puts "Original file #{file} doesn't exist, saving that state temporarily as #{tmporigpath}"
+          end
         else
           origpath = "#{origpathbase}.NOORIG"
           puts "Original file #{file} doesn't exist, saving that state permanently as #{origpath}"
         end
-        File.open(origpath, 'w') { |file| } if (!@dryrun)
+        if proceed
+          File.open(origpath, 'w') { |file| } if (!@dryrun)
+        end
       end
 
       @first_update[file] = first_update
@@ -2133,36 +2137,30 @@
     gid.to_i
   end
 
-  # Returns false if the permissions of the given file match the given
-  # permissions, true otherwise.
+  # Returns true if the permissions of the given file match the given
+  # permissions, false otherwise.
   def compare_permissions(file, perms)
-    if ! File.exist?(file)
-      return true
+    if File.exist?(file)
+      st = File.lstat(file)
+      # Mask off the file type
+      fileperms = st.mode & 07777
+      if perms == fileperms
+        return true
+      end
     end
-
-    st = File.lstat(file)
-    # Mask off the file type
-    fileperms = st.mode & 07777
-    if perms == fileperms
-      return false
-    else
-      return true
-    end
+    false
   end
 
-  # Returns false if the ownership of the given file match the given UID
-  # and GID, true otherwise.
+  # Returns true if the ownership of the given file match the given UID
+  # and GID, false otherwise.
   def compare_ownership(file, uid, gid)
-    if ! File.exist?(file)
-      return true
+    if File.exist?(file)
+      st = File.lstat(file)
+      if st.uid == uid && st.gid == gid
+        return true
+      end
     end
-
-    st = File.lstat(file)
-    if st.uid == uid && st.gid == gid
-      return false
-    else
-      return true
-    end
+    false
   end
 
   def get_user_confirmation


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

[etch-devel] SF.net SVN: etch:[137] trunk

[<jh...@us...>]

Revision: 137
          http://etch.svn.sourceforge.net/etch/?rev=137&view=rev
Author:   jheiss
Date:     2009-11-06 00:47:38 +0000 (Fri, 06 Nov 2009)

Log Message:
-----------
Fix bug where specific dependency structures requested in specific ways 
on the command line might never complete due to the client and server 
never getting on the same page such that a complete dependency tree can 
be calculated.  Add test cases for same.  See code comments for details.

Modified Paths:
--------------
    trunk/server/lib/etch.rb
    trunk/test/options.rb

Modified: trunk/server/lib/etch.rb
===================================================================
--- trunk/server/lib/etch.rb	2009-11-06 00:43:46 UTC (rev 136)
+++ trunk/server/lib/etch.rb	2009-11-06 00:47:38 UTC (rev 137)
@@ -305,12 +305,43 @@
         r = generate_commands(Etch.xmltext(dependcommand), request)
         proceed = proceed && r
       end
+      
       if !proceed
         @dlogger.debug "One or more dependencies of #{file} need data from client"
+      end
+      
+      # Make sure we have the original contents for this file
+      original_file = nil
+      if request[:files] && request[:files][file] && request[:files][file][:orig]
+        original_file = request[:files][file][:orig]
+      else
+        @dlogger.debug "Need original contents of #{file} from client"
+        proceed = false
+      end
+      
+      if !proceed
+        # If any file dependency failed to generate (due to a need for orig
+        # contents from the client) then we need to tell the client to request
+        # all of the files in the dependency tree again.
+        # 
+        # For example, we have afile which depends on bfile and cfile.  The
+        # user requests afile and bfile on the command line.  The client sends
+        # sums for afile and bfile.  The server sees the need for cfile's sum, so
+        # it sends back contents for bfile and a sum request for cfile and afile
+        # (since afile depends on bfile).  The client sends sums for afile and
+        # cfile.  The server sends back contents for cfile, and a sum request for
+        # bfile and afile.  This repeats forever as the server isn't smart enough
+        # to ask for everything it needs and the client isn't smart enough to send
+        # everything.
+        depends.each { |depend| @need_orig[depend] = true }
+        
         # Tell the client to request this file again
         @need_orig[file] = true
+        
         # Strip this file's config down to the bare necessities
         filter_xml_completely!(config_xml, ['depend', 'setup'])
+        
+        # And hit the eject button
         generation_status = false
         throw :generate_done
       end
@@ -341,24 +372,6 @@
           end
         end
       end
-  
-      # Make sure we have the original contents for this file
-      original_file = nil
-      if request[:files] && request[:files][file] && request[:files][file][:orig]
-        original_file = request[:files][file][:orig]
-      else
-        @dlogger.debug "Need original contents of #{file} from client"
-        @need_orig[file] = true
-        # If there are setup commands defined for this file we need to
-        # pass those back along with our request for the original file,
-        # as the setup commands may be needed to create the original
-        # file on the node.
-        filter_xml_completely!(config_xml, ['depend', 'setup'])
-        # Nothing more can be done until we have the original file from
-        # the client
-        generation_status = false
-        throw :generate_done
-      end
     
       # Pull out any local requests
       local_requests = nil
@@ -852,6 +865,7 @@
     # If we encounter either failure or success we set it to false or :success.
     catch :generate_done do
       # Generate any other commands that this command depends on
+      dependfiles = []
       proceed = true
       Etch.xmleach(commands_xml, '/commands/depend') do |depend|
         @dlogger.debug "Generating command dependency #{Etch.xmltext(depend)}"
@@ -861,10 +875,17 @@
       # Also generate any files that this command depends on
       Etch.xmleach(commands_xml, '/commands/dependfile') do |dependfile|
         @dlogger.debug "Generating file dependency #{Etch.xmltext(dependfile)}"
+        dependfiles << Etch.xmltext(dependfile)
         r = generate_file(Etch.xmltext(dependfile), request)
         proceed = proceed && r
       end
       if !proceed
+        @dlogger.debug "One or more dependencies of #{command} need data from client"
+        # If any file dependency failed to generate (due to a need for orig
+        # contents from the client) then we need to tell the client to request
+        # all of the files in the dependency tree again.  See the big comment
+        # in generate_file for further explanation.
+        dependfiles.each { |dependfile| @need_orig[dependfile] = true }
         # Try again next time
         @retrycommands[command] = true
         generation_status = false

Modified: trunk/test/options.rb
===================================================================
--- trunk/test/options.rb	2009-11-06 00:43:46 UTC (rev 136)
+++ trunk/test/options.rb	2009-11-06 00:47:38 UTC (rev 137)
@@ -247,6 +247,183 @@
     assert_equal(origcontents, get_file_contents(cmdtargetfile3), testname + ' cmd 3')
   end
   
+  def test_file_requests_with_depends
+    #
+    # Test that the user can request specific files and commands on the
+    # command line with a dependency structure such that, in the right
+    # circumstances, a poor implementation never completes because it
+    # alternately sends/requests the orig sum for the two dependencies, never
+    # sending both at the same time.
+    #
+    # For example, we have afile which depends on bfile and cfile.  The user
+    # requests afile and bfile on the command line.  The client sends sums for
+    # afile and bfile.  The server sees the need for cfile's sum, so it sends
+    # back contents for bfile and a sum request for cfile and afile (since
+    # afile depends on bfile).  The client sends sums for afile and cfile. 
+    # The server sends back contents for cfile, and a sum request for bfile
+    # and afile.  This repeats forever as the server isn't smart enough to ask
+    # for everything it needs and the client isn't smart enough to send
+    # everything.    
+    #
+    # Yup, had this bug at one point.
+    #
+    testname = 'command line file requests with depends'
+    
+    targetfile2 = Tempfile.new('etchtest').path
+    targetfile3 = Tempfile.new('etchtest').path
+    FileUtils.mkdir_p("#{@repodir}/source/#{@targetfile}")
+    File.open("#{@repodir}/source/#{@targetfile}/config.xml", 'w') do |file|
+      file.puts <<-EOF
+        <config>
+          <depend>#{targetfile2}</depend>
+          <depend>#{targetfile3}</depend>
+          <file>
+            <warning_file></warning_file>
+            <source>
+              <plain>source</plain>
+            </source>
+          </file>
+        </config>
+      EOF
+    end
+    FileUtils.mkdir_p("#{@repodir}/source/#{targetfile2}")
+    File.open("#{@repodir}/source/#{targetfile2}/config.xml", 'w') do |file|
+      file.puts <<-EOF
+        <config>
+          <file>
+            <warning_file></warning_file>
+            <source>
+              <plain>source</plain>
+            </source>
+          </file>
+        </config>
+      EOF
+    end
+    FileUtils.mkdir_p("#{@repodir}/source/#{targetfile3}")
+    File.open("#{@repodir}/source/#{targetfile3}/config.xml", 'w') do |file|
+      file.puts <<-EOF
+        <config>
+          <file>
+            <warning_file></warning_file>
+            <source>
+              <plain>source</plain>
+            </source>
+          </file>
+        </config>
+      EOF
+    end
+    
+    sourcecontents = "Test #{testname}\n"
+    [@targetfile, targetfile2, targetfile3].each do |target|
+      File.open("#{@repodir}/source/#{target}/source", 'w') do |file|
+        file.write(sourcecontents)
+      end
+    end
+    
+    # Run etch
+    #puts "Running '#{testname}' test"
+    run_etch(@port, @testbase, false, "#{@targetfile} #{targetfile2}")
+    
+    # Verify that all were created
+    assert_equal(sourcecontents, get_file_contents(@targetfile), testname + ' filesonly file 1')
+    assert_equal(sourcecontents, get_file_contents(targetfile2), testname + ' filesonly file 2')
+    assert_equal(sourcecontents, get_file_contents(targetfile3), testname + ' filesonly file 3')
+  end
+  
+  def test_mixed_requests_with_depends
+    #
+    # Similar to previous test, but mixing file and command requests
+    #
+    testname = 'mixed command line requests with depends'
+    
+    targetfile2 = Tempfile.new('etchtest').path
+    targetfile3 = Tempfile.new('etchtest').path
+    FileUtils.mkdir_p("#{@repodir}/source/#{@targetfile}")
+    File.open("#{@repodir}/source/#{@targetfile}/config.xml", 'w') do |file|
+      file.puts <<-EOF
+        <config>
+          <depend>#{targetfile2}</depend>
+          <depend>#{targetfile3}</depend>
+          <file>
+            <warning_file></warning_file>
+            <source>
+              <plain>source</plain>
+            </source>
+          </file>
+        </config>
+      EOF
+    end
+    FileUtils.mkdir_p("#{@repodir}/source/#{targetfile2}")
+    File.open("#{@repodir}/source/#{targetfile2}/config.xml", 'w') do |file|
+      file.puts <<-EOF
+        <config>
+          <file>
+            <warning_file></warning_file>
+            <source>
+              <plain>source</plain>
+            </source>
+          </file>
+        </config>
+      EOF
+    end
+    FileUtils.mkdir_p("#{@repodir}/source/#{targetfile3}")
+    File.open("#{@repodir}/source/#{targetfile3}/config.xml", 'w') do |file|
+      file.puts <<-EOF
+        <config>
+          <file>
+            <warning_file></warning_file>
+            <source>
+              <plain>source</plain>
+            </source>
+          </file>
+        </config>
+      EOF
+    end
+    
+    sourcecontents = "Test #{testname}\n"
+    [@targetfile, targetfile2, targetfile3].each do |target|
+      File.open("#{@repodir}/source/#{target}/source", 'w') do |file|
+        file.write(sourcecontents)
+      end
+    end
+    
+    cmdtargetfile1 = Tempfile.new('etchtest').path
+    FileUtils.mkdir_p("#{@repodir}/commands/etchtest1")
+    File.open("#{@repodir}/commands/etchtest1/commands.xml", 'w') do |file|
+      file.puts <<-EOF
+        <commands>
+          <dependfile>#{targetfile2}</dependfile>
+          <dependfile>#{targetfile3}</dependfile>
+          <step>
+            <guard>
+              <exec>grep '#{testname}' #{cmdtargetfile1}</exec>
+            </guard>
+            <command>
+              <exec>printf '#{testname}' >> #{cmdtargetfile1}</exec>
+            </command>
+          </step>
+        </commands>
+      EOF
+    end
+    
+    # Put some text into the original files.
+    origcontents = "This is the original text\n"
+    [@targetfile, targetfile2, targetfile3, cmdtargetfile1].each do |target|
+      File.open(target, 'w') do |file|
+        file.write(origcontents)
+      end
+    end
+    
+    # Run etch
+    #puts "Running '#{testname}' test"
+    run_etch(@port, @testbase, false, "etchtest1 #{targetfile2}")
+    
+    # Verify that all were created
+    assert_equal(origcontents + testname, get_file_contents(cmdtargetfile1), testname + ' cmdandfile cmd 1')
+    assert_equal(sourcecontents, get_file_contents(targetfile2), testname + ' cmdandfile file 2')
+    assert_equal(sourcecontents, get_file_contents(targetfile3), testname + ' cmdandfile file 3')
+  end
+  
   def teardown
     stop_server(@pid)
     remove_repository(@repodir)


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

[etch-devel] SF.net SVN: etch:[136] trunk/test/etchtest.rb

[<jh...@us...>]

Revision: 136
          http://etch.svn.sourceforge.net/etch/?rev=136&view=rev
Author:   jheiss
Date:     2009-11-06 00:43:46 +0000 (Fri, 06 Nov 2009)

Log Message:
-----------
Comment out the sleep after the "errors expected" warning.  It slows
down a test run for little benefit, as folks probably aren't watching
the tests run most of the time, just starting a run in a window and
checking later to make sure nothing failed.

Modified Paths:
--------------
    trunk/test/etchtest.rb

Modified: trunk/test/etchtest.rb
===================================================================
--- trunk/test/etchtest.rb	2009-11-03 02:04:24 UTC (rev 135)
+++ trunk/test/etchtest.rb	2009-11-06 00:43:46 UTC (rev 136)
@@ -80,7 +80,7 @@
       puts "#"
       puts "# Errors expected here"
       puts "#"
-      sleep 3
+      #sleep 3
     end
     result = system("ruby ../client/etch --generate-all --server=http://localhost:#{port} --test-base=#{testbase} --key=keys/testkey #{extra_args}")
     if errors_expected


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

[etch-devel] SF.net SVN: etch:[135] trunk

[<jh...@us...>]

Revision: 135
          http://etch.svn.sourceforge.net/etch/?rev=135&view=rev
Author:   jheiss
Date:     2009-11-03 02:04:24 +0000 (Tue, 03 Nov 2009)

Log Message:
-----------
Loosen up the DTD a bit so that folks can filter with the same freedom
as with configs.

Modified Paths:
--------------
    trunk/etchserver-demo/commands.dtd
    trunk/etchserver-samples/commands.dtd
    trunk/test/testrepo/commands.dtd

Modified: trunk/etchserver-demo/commands.dtd
===================================================================
--- trunk/etchserver-demo/commands.dtd	2009-10-29 21:09:09 UTC (rev 134)
+++ trunk/etchserver-demo/commands.dtd	2009-11-03 02:04:24 UTC (rev 135)
@@ -3,8 +3,8 @@
 <!ELEMENT depend (#PCDATA)>
 
 <!ELEMENT step (guard, command)>
-<!ELEMENT guard (exec+)>
-<!ELEMENT command (exec+)>
+<!ELEMENT guard (exec*)>
+<!ELEMENT command (exec*)>
 
 <!ELEMENT exec (#PCDATA)>
 

Modified: trunk/etchserver-samples/commands.dtd
===================================================================
--- trunk/etchserver-samples/commands.dtd	2009-10-29 21:09:09 UTC (rev 134)
+++ trunk/etchserver-samples/commands.dtd	2009-11-03 02:04:24 UTC (rev 135)
@@ -3,8 +3,8 @@
 <!ELEMENT depend (#PCDATA)>
 
 <!ELEMENT step (guard, command)>
-<!ELEMENT guard (exec+)>
-<!ELEMENT command (exec+)>
+<!ELEMENT guard (exec*)>
+<!ELEMENT command (exec*)>
 
 <!ELEMENT exec (#PCDATA)>
 

Modified: trunk/test/testrepo/commands.dtd
===================================================================
--- trunk/test/testrepo/commands.dtd	2009-10-29 21:09:09 UTC (rev 134)
+++ trunk/test/testrepo/commands.dtd	2009-11-03 02:04:24 UTC (rev 135)
@@ -4,8 +4,8 @@
 <!ELEMENT dependfile (#PCDATA)>
 
 <!ELEMENT step (guard, command)>
-<!ELEMENT guard (exec+)>
-<!ELEMENT command (exec+)>
+<!ELEMENT guard (exec*)>
+<!ELEMENT command (exec*)>
 
 <!ELEMENT exec (#PCDATA)>
 


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

[etch-devel] SF.net SVN: etch:[134] tags

[<jh...@us...>]

Revision: 134
          http://etch.svn.sourceforge.net/etch/?rev=134&view=rev
Author:   jheiss
Date:     2009-10-29 21:09:09 +0000 (Thu, 29 Oct 2009)

Log Message:
-----------
Tag 3.9 release

Modified Paths:
--------------
    Makefile

Added Paths:
-----------
    tags/release-3.9/

Modified: Makefile
===================================================================
--- Makefile	2009-10-29 20:45:18 UTC (rev 133)
+++ Makefile	2009-10-29 21:09:09 UTC (rev 134)
@@ -1,4 +1,4 @@
-VER=3.8
+VER=3.9
 TAGNAME=release-$(VER)
 
 all: dist


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

[etch-devel] SF.net SVN: etch:[133] trunk/client/etch.8

[<jh...@us...>]

Revision: 133
          http://etch.svn.sourceforge.net/etch/?rev=133&view=rev
Author:   jheiss
Date:     2009-10-29 20:45:18 +0000 (Thu, 29 Oct 2009)

Log Message:
-----------
Improve wording

Modified Paths:
--------------
    trunk/client/etch.8

Modified: trunk/client/etch.8
===================================================================
--- trunk/client/etch.8	2009-10-29 20:44:32 UTC (rev 132)
+++ trunk/client/etch.8	2009-10-29 20:45:18 UTC (rev 133)
@@ -153,8 +153,9 @@
 .B /etc/etch/dhparams
 The Diffie-Hellman parameters used as part of the SSL connection process. Etch
 comes with a set and there's no need to generate your own, but a new set can
-be generated via "openssl dhparam" if desired. If not present the Ruby SSL
-library will warn that it is using its internal default set of parameters.
+be generated via "openssl dhparam" if desired. If this file is not present the
+Ruby SSL library will warn that it is using its internal default set of
+parameters.
 .TP
 .B /var/etch/disable_etch
 If this file is present


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

[etch-devel] SF.net SVN: etch:[132] trunk/client

[<jh...@us...>]

Revision: 132
          http://etch.svn.sourceforge.net/etch/?rev=132&view=rev
Author:   jheiss
Date:     2009-10-29 20:44:32 +0000 (Thu, 29 Oct 2009)

Log Message:
-----------
Make it easier to run etch from within the distribution directory
structure.  Remove the run_etch script I added as a poorly thought-out
hack to accomplish the same.

Add support for requesting specific commands.

Add featuring allowing the server to tell the client to retry certain
commands on the next request. The "need sum" mechanism is used to do the
same thing for files, but as commands have no original content saving
mechanism we have to have an explicit feature for retries. The server
needs this if it is unable to generate all of the prerequisites for the
command, usually because the command depends on a file for which the
server needs original content info.

Add support for files that depend on commands and vice-versa.

Put all data that extracted from the server's response into a
responsedata hash rather than a bunch of separate variables. Previous
process_file was passed only file configuration and process_commands
passed only command configuration, but with the new
files-that-depend-on-commands and vice-versa features both methods need
all configuration data.

Fix bug in get_orig_contents so that it return the contents for
temporarily saved originals in addition to finalized originals.

Modified Paths:
--------------
    trunk/client/etchclient.rb

Removed Paths:
-------------
    trunk/client/run_etch

Modified: trunk/client/etchclient.rb
===================================================================
--- trunk/client/etchclient.rb	2009-10-29 20:41:44 UTC (rev 131)
+++ trunk/client/etchclient.rb	2009-10-29 20:44:32 UTC (rev 132)
@@ -2,6 +2,13 @@
 # Etch configuration file management tool library
 ##############################################################################
 
+# Ensure we can find etch.rb if run within the development directory structure
+#   This is roughly equivalent to "../server/lib"
+serverlibdir = File.join(File.dirname(File.dirname(File.expand_path(__FILE__))), 'server', 'lib')
+if File.exist?(serverlibdir)
+  $:.unshift(serverlibdir)
+end
+
 begin
   # Try loading facter w/o gems first so that we don't introduce a
   # dependency on gems if it is not needed.
@@ -28,7 +35,7 @@
 require 'etch'
 
 class Etch::Client
-  VERSION = '1.16'
+  VERSION = '1.18'
   
   CONFIRM_PROCEED = 1
   CONFIRM_SKIP = 2
@@ -120,13 +127,14 @@
     @lchmod_supported = nil
   end
 
-  def process_until_done(files_to_generate)
+  def process_until_done(files, commands)
     # Our overall status.  Will be reported to the server and used as the
     # return value for this method.  Command-line clients should use it as
     # their exit value.  Zero indicates no errors.
     status = 0
     message = ''
     
+    # Prep http instance
     http = nil
     if !@local
       http = Net::HTTP.new(@filesuri.host, @filesuri.port)
@@ -166,9 +174,9 @@
         request = nil
         if @local
           request = {}
-          if files_to_generate && !files_to_generate.empty?
-            files_to_generate.each do |file|
-              request[:files] = {}
+          if files && !files.empty?
+            request[:files] = {}
+            files.each do |file|
               request[:files][file] = {:orig => save_orig(file)}
               local_requests = get_local_requests(file)
               if local_requests
@@ -176,18 +184,31 @@
               end
             end
           end
+          if commands && !commands.empty?
+            request[:commands] = {}
+            commands.each do |command|
+              request[:commands][command] = {}
+            end
+          end
         else
           request = get_blank_request
-          if files_to_generate && !files_to_generate.empty?
-            files_to_generate.each do |file|
-              request["files[#{CGI.escape(file)}][sha1sum]"] =
-                get_orig_sum(file)
-              local_requests = get_local_requests(file)
-              if local_requests
-                request["files[#{CGI.escape(file)}][local_requests]"] =
-                  local_requests
+          if (files && !files.empty?) || (commands && !commands.empty?)
+            if files
+              files.each do |file|
+                request["files[#{CGI.escape(file)}][sha1sum]"] =
+                  get_orig_sum(file)
+                local_requests = get_local_requests(file)
+                if local_requests
+                  request["files[#{CGI.escape(file)}][local_requests]"] =
+                    local_requests
+                end
               end
             end
+            if commands
+              commands.each do |command|
+                request["commands[#{CGI.escape(command)}]"] = '1'
+              end
+            end
           else
             request['files[GENERATEALL]'] = '1'
           end
@@ -210,23 +231,23 @@
           # Send request to server
           #
           
-          configs = nil
-          need_sums = nil
-          need_origs = nil
-          allcommands = nil
+          responsedata = {}
           if @local
             results = @etch.generate(@local, @facts, request)
-	    # FIXME: Etch#generate returns parsed XML using whatever XML
-	    # library it happens to use.  In order to avoid re-parsing
-	    # the XML we'd have to use the XML abstraction code from Etch
-	    # everwhere here.
-	    # Until then re-parse the XML using REXML.
-            #configs = results[:configs]
-            configs = {}
-            results[:configs].each {|f,c| configs[f] = REXML::Document.new(c.to_s) }
-            need_sums = {}
-            need_origs = results[:need_orig]
-            allcommands = results[:allcommands]
+            # FIXME: Etch#generate returns parsed XML using whatever XML
+            # library it happens to use.  In order to avoid re-parsing
+            # the XML we'd have to use the XML abstraction code from Etch
+            # everwhere here.
+            # Until then re-parse the XML using REXML.
+            #responsedata[:configs] = results[:configs]
+            responsedata[:configs] = {}
+            results[:configs].each {|f,c| responsedata[:configs][f] = REXML::Document.new(c.to_s) }
+            responsedata[:need_sums] = {}
+            responsedata[:need_origs] = results[:need_orig]
+            #responsedata[:allcommands] = results[:allcommands]
+            responsedata[:allcommands] = {}
+            results[:allcommands].each {|cn,c| responsedata[:allcommands][cn] = REXML::Document.new(c.to_s) }
+            responsedata[:retrycommands] = results[:retrycommands]
           else
             puts "Sending request to server #{@filesuri}: #{request.inspect}" if (@debug)
             post = Net::HTTP::Post.new(@filesuri.path)
@@ -241,32 +262,36 @@
             puts "Response from server:\n'#{response.body}'" if (@debug)
             if !response.body.nil? && !response.body.empty?
               response_xml = REXML::Document.new(response.body)
-              configs = {}
+              responsedata[:configs] = {}
               response_xml.elements.each('/files/configs/config') do |config|
                 file = config.attributes['filename']
                 # We have to make a new document so that XPath paths are
                 # referenced relative to the configuration for this
                 # specific file.
-                #configs[file] = REXML::Document.new(response_xml.elements["/files/configs/config[@filename='#{file}']"].to_s)
-                configs[file] = REXML::Document.new(config.to_s)
+                #responsedata[:configs][file] = REXML::Document.new(response_xml.elements["/files/configs/config[@filename='#{file}']"].to_s)
+                responsedata[:configs][file] = REXML::Document.new(config.to_s)
               end
-              need_sums = {}
+              responsedata[:need_sums] = {}
               response_xml.elements.each('/files/need_sums/need_sum') do |ns|
-                need_sums[ns.text] = true
+                responsedata[:need_sums][ns.text] = true
               end
-              need_origs = {}
+              responsedata[:need_origs] = {}
               response_xml.elements.each('/files/need_origs/need_orig') do |no|
-                need_origs[no.text] = true
+                responsedata[:need_origs][no.text] = true
               end
-              allcommands = {}
+              responsedata[:allcommands] = {}
               response_xml.elements.each('/files/allcommands/commands') do |command|
                 commandname = command.attributes['commandname']
                 # We have to make a new document so that XPath paths are
                 # referenced relative to the configuration for this
                 # specific file.
-                #allcommands[commandname] = REXML::Document.new(response_xml.root.elements["/files/allcommands/commands[@commandname='#{commandname}']"].to_s)
-                allcommands[commandname] = REXML::Document.new(command.to_s)
+                #responsedata[:allcommands][commandname] = REXML::Document.new(response_xml.root.elements["/files/allcommands/commands[@commandname='#{commandname}']"].to_s)
+                responsedata[:allcommands][commandname] = REXML::Document.new(command.to_s)
               end
+              responsedata[:retrycommands] = {}
+              response_xml.elements.each('/files/retrycommands/retrycommand') do |rc|
+                responsedata[:retrycommands][rc.text] = true
+              end
             else
               puts "  Response is empty" if (@debug)
               break
@@ -280,7 +305,7 @@
           # Prep a clean request hash
           if @local
             request = {}
-            if !need_origs.empty?
+            if !responsedata[:need_origs].empty?
               request[:files] = {}
             end
           else
@@ -296,14 +321,14 @@
           reset_already_processed
           # Process configs first, as they may contain setup entries that are
           # needed to create the original files.
-          configs.each_key do |file|
+          responsedata[:configs].each_key do |file|
             puts "Processing config for #{file}" if (@debug)
-            continue_processing = process(file, configs)
+            continue_processing = process_file(file, responsedata)
             if !continue_processing
               throw :stop_processing
             end
           end
-          need_sums.each_key do |need_sum|
+          responsedata[:need_sums].each_key do |need_sum|
             puts "Processing request for sum of #{need_sum}" if (@debug)
             if @local
               # If this happens we screwed something up, the local mode
@@ -324,7 +349,7 @@
             end
             need_to_loop = true
           end
-          need_origs.each_key do |need_orig|
+          responsedata[:need_origs].each_key do |need_orig|
             puts "Processing request for contents of #{need_orig}" if (@debug)
             if @local
               request[:files][need_orig] = {:orig => save_orig(need_orig)}
@@ -345,14 +370,23 @@
             end
             need_to_loop = true
           end
-         allcommands.each_key do |commandname|
+         responsedata[:allcommands].each_key do |commandname|
             puts "Processing commands #{commandname}" if (@debug)
-            continue_processing = process_commands(commandname, allcommands)
+            continue_processing = process_commands(commandname, responsedata)
             if !continue_processing
               throw :stop_processing
             end
           end
-
+          responsedata[:retrycommands].each_key do |commandname|
+            puts "Processing request to retry command #{commandname}" if (@debug)
+            if @local
+              request[:commands][commandname] = true
+            else
+              request["commands[#{CGI.escape(commandname)}]"] = '1'
+            end
+            need_to_loop = true
+          end
+          
           if !need_to_loop
             break
           end
@@ -431,7 +465,7 @@
   # Raises an exception if any fatal error is encountered
   # Returns a boolean, true unless the user indicated in interactive mode
   # that further processing should be halted
-  def process(file, configs)
+  def process_file(file, responsedata)
     continue_processing = true
     save_results = true
     exception = nil
@@ -439,7 +473,7 @@
     # We may not have configuration for this file, if it does not apply
     # to this host.  The server takes care of detecting any errors that
     # might involve, so here we can just silently return.
-    config = configs[file]
+    config = responsedata[:configs][file]
     if !config
       puts "No configuration for #{file}, skipping" if (@debug)
       return continue_processing
@@ -486,10 +520,16 @@
         
         # Process any other files that this file depends on
         config.elements.each('/config/depend') do |depend|
-          puts "Generating dependency #{depend.text}" if (@debug)
-          process(depend.text, configs)
+          puts "Processing dependency #{depend.text}" if (@debug)
+          process_file(depend.text, responsedata)
         end
-
+        
+        # Process any commands that this file depends on
+        config.elements.each('/config/dependcommand') do |dependcommand|
+          puts "Processing command dependency #{dependcommand.text}" if (@debug)
+          process_commands(dependcommand.text, responsedata)
+        end
+        
         # See what type of action the user has requested
 
         # Check to see if the user has requested that we revert back to the
@@ -1342,11 +1382,20 @@
   # Raises an exception if any fatal error is encountered
   # Returns a boolean, true unless the user indicated in interactive mode
   # that further processing should be halted
-  def process_commands(commandname, allcommands)
+  def process_commands(commandname, responsedata)
     continue_processing = true
     save_results = true
     exception = nil
     
+    # We may not have configuration for this file, if it does not apply
+    # to this host.  The server takes care of detecting any errors that
+    # might involve, so here we can just silently return.
+    command = responsedata[:allcommands][commandname]
+    if !command
+      puts "No configuration for command #{commandname}, skipping" if (@debug)
+      return continue_processing
+    end
+        
     # Skip commands we've already processed in response to <depend>
     # statements.
     if @already_processed.has_key?(commandname)
@@ -1386,14 +1435,18 @@
         # This needs to be after the circular dependency check
         lock_file(commandname)
         
-        command = allcommands[commandname]
-        
         # Process any other commands that this command depends on
         command.elements.each('/commands/depend') do |depend|
-          puts "Generating command dependency #{depend.text}" if (@debug)
-          process_commands(depend.text, allcommands)
+          puts "Processing command dependency #{depend.text}" if (@debug)
+          process_commands(depend.text, responsedata)
         end
         
+        # Process any files that this command depends on
+        command.elements.each('/commands/dependfile') do |dependfile|
+          puts "Processing file dependency #{dependfile.text}" if (@debug)
+          process_file(dependfile.text, responsedata)
+        end
+        
         # Perform each step
         command.elements.each('/commands/step') do |step|
           guard = step.elements['guard/exec'].text
@@ -1502,7 +1555,8 @@
     orig_contents = nil
     # We only send back the actual original file contents if the original is
     # a regular file, otherwise we send back an empty string.
-    if origpath =~ /\.ORIG$/ && File.file?(origpath) && !File.symlink?(origpath)
+    if (origpath =~ /\.ORIG$/ || origpath =~ /\.TMP$/) &&
+       File.file?(origpath) && !File.symlink?(origpath)
       orig_contents = IO.read(origpath)
     else
       orig_contents = ''

Deleted: trunk/client/run_etch
===================================================================
--- trunk/client/run_etch	2009-10-29 20:41:44 UTC (rev 131)
+++ trunk/client/run_etch	2009-10-29 20:44:32 UTC (rev 132)
@@ -1,11 +0,0 @@
-#!/bin/sh
-
-# This script makes it easier to run the etch client from the
-# distribution directory structure.  It is not needed when the etch
-# client is packaged and installed.
-
-RUBYLIB=../server/lib
-export RUBYLIB
-
-exec ./etch "$@"
-


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

[etch-devel] SF.net SVN: etch:[131] trunk/client/etch

[<jh...@us...>]

Revision: 131
          http://etch.svn.sourceforge.net/etch/?rev=131&view=rev
Author:   jheiss
Date:     2009-10-29 20:41:44 +0000 (Thu, 29 Oct 2009)

Log Message:
-----------
Add support for requesting specific commands.

Modified Paths:
--------------
    trunk/client/etch

Modified: trunk/client/etch
===================================================================
--- trunk/client/etch	2009-10-29 20:41:24 UTC (rev 130)
+++ trunk/client/etch	2009-10-29 20:41:44 UTC (rev 131)
@@ -4,7 +4,7 @@
 ##############################################################################
 
 # Ensure we can find etchclient.rb
-$:.unshift File.dirname(__FILE__)
+$:.unshift(File.dirname(__FILE__))
 
 require 'optparse'
 require 'etchclient'
@@ -17,7 +17,7 @@
 @generateall = nil
 
 opts = OptionParser.new(nil, 24, '  ')
-opts.banner = 'Usage: etch [options] [/path/to/config/file]'
+opts.banner = 'Usage: etch [options] [/path/to/config/file | command] [otherfile ...]'
 opts.on('--generate-all', 'Request all configuration.') do |opt|
   @generateall = opt
 end
@@ -74,11 +74,19 @@
   exit
 end
 
-# Put any leftover arguments into files_to_generate
-files_to_generate = opts.parse(ARGV)
+leftovers = opts.parse(ARGV)
+files = []
+commands = []
+leftovers.each do |leftover|
+  if leftover[0,1] == File::SEPARATOR
+    files << leftover
+  else
+    commands << leftover
+  end
+end
 
 # Display a usage message if the user did not specify a valid action to perform.
-unless (!files_to_generate.nil? && !files_to_generate.empty?) || @generateall
+if files.empty? && commands.empty? && !@generateall
   puts opts
   exit
 end
@@ -88,6 +96,6 @@
 #
 
 etchclient = Etch::Client.new(options)
-status = etchclient.process_until_done(files_to_generate)
+status = etchclient.process_until_done(files, commands)
 exit status
 


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

[etch-devel] SF.net SVN: etch:[130] trunk/server/lib/etchserver.rb

[<jh...@us...>]

Revision: 130
          http://etch.svn.sourceforge.net/etch/?rev=130&view=rev
Author:   jheiss
Date:     2009-10-29 20:41:24 +0000 (Thu, 29 Oct 2009)

Log Message:
-----------
Add support for requesting specific commands.

Add featuring allowing the server to tell the client to retry certain
commands on the next request. The "need sum" mechanism is used to do the
same thing for files, but as commands have no original content saving
mechanism we have to have an explicit feature for retries. The server
needs this if it is unable to generate all of the prerequisites for the
command, usually because the command depends on a file for which the
server needs original content info.

Modified Paths:
--------------
    trunk/server/lib/etchserver.rb

Modified: trunk/server/lib/etchserver.rb
===================================================================
--- trunk/server/lib/etchserver.rb	2009-10-29 20:41:05 UTC (rev 129)
+++ trunk/server/lib/etchserver.rb	2009-10-29 20:41:24 UTC (rev 130)
@@ -292,7 +292,7 @@
     @origbase          = "#{@configbase}/orig"
   end
 
-  def generate(files)
+  def generate(files, commands)
     #
     # Build up a list of files to generate, either from the request or from
     # the source repository if the request is for all files
@@ -354,6 +354,11 @@
       end
     end
     
+    commands.each_key do |commandname|
+      request[:commands] = {} if !request[:commands]
+      request[:commands][commandname] = {}
+    end
+    
     #
     # Process the user's request
     #
@@ -438,6 +443,15 @@
       end
       responseroot << commands_xml
     end
+    if response[:retrycommands]
+      retrycommands_xml = Etch.xmlnewelem('retrycommands')
+      response[:retrycommands].each_key do |commandname|
+        retry_xml = Etch.xmlnewelem('retrycommand')
+        Etch.xmlsettext(retry_xml, commandname)
+        retrycommands_xml << retry_xml
+      end
+      responseroot << retrycommands_xml
+    end
     
     # FIXME: clean up XML formatting
     # But only if we're in debug mode, in regular mode nobody but the


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

[etch-devel] SF.net SVN: etch:[129] trunk/server/lib/etch.rb

[<jh...@us...>]

Revision: 129
          http://etch.svn.sourceforge.net/etch/?rev=129&view=rev
Author:   jheiss
Date:     2009-10-29 20:41:05 +0000 (Thu, 29 Oct 2009)

Log Message:
-----------
Add support for files that depend on commands and vice-versa.

Add featuring allowing the server to tell the client to retry certain
commands on the next request. The "need sum" mechanism is used to do the
same thing for files, but as commands have no original content saving
mechanism we have to have an explicit feature for retries. The server
needs this if it is unable to generate all of the prerequisites for the
command, usually because the command depends on a file for which the
server needs original content info.

Modify generate_file and generate_commands methods to return a value
indicating the status of the generation. Thus if either is called due to
a dependency the caller knows whether it should proceed with its own
generation, or tell the client to retry next time if one or more
dependencies fail to generate due to a need for original content info.
generate_file previously pulled this info out of the @generation_status
hash, but it seems preferable to use a return value.

In generate_file, if a dependency fails to generate don't explicitly go
through the dependency tree and filter/need_orig each dependency. Each
file where that is necessary has already handled itself, and it's fine
if the client goes ahead and writes out other parts of the tree.

Fix bug in generate_file, we need to return the (filtered) config for
files which failed to generate due to a dependency failure, as the
config may contain setup entries.

Fix bug in xmlremove where it was calling the wrong REXML method if
asked to remove an element from a document (as opposed to another
element). The method call that was being used didn't generate an error,
but also didn't do anything. This in turn meant that
filter_xml_completely! was a no-op, which meant we were sending back
unfiltered configs when we meant to send back just depend and setup
entries.

Modified Paths:
--------------
    trunk/server/lib/etch.rb

Modified: trunk/server/lib/etch.rb
===================================================================
--- trunk/server/lib/etch.rb	2009-10-29 20:40:41 UTC (rev 128)
+++ trunk/server/lib/etch.rb	2009-10-29 20:41:05 UTC (rev 129)
@@ -43,7 +43,7 @@
   # if Logger didn't immediately blow up we'd probably end up with scrambled
   # logs as simultaneous connections tried to write at the same time.  Or
   # maybe that would work, depending on how Ruby and the OS buffer writes to
-  # the file.
+  # the file?
   def initialize(logger, debug_logger)
     @logger = logger
     @dlogger = debug_logger
@@ -178,7 +178,9 @@
     @generation_status = {}
     @configs = {}
     @need_orig = {}
-
+    @allcommands = {}
+    @retrycommands = {}
+    
     filelist.each do |file|
       @dlogger.debug "Generating #{file}"
       generate_file(file, request)
@@ -188,8 +190,6 @@
     # Generate configuration commands
     #
     
-    @allcommands = {}
-    
     commandnames = []
     if request.empty?
       @dlogger.debug "Building complete configuration commands for request from #{@fqdn}"
@@ -214,7 +214,8 @@
     
     {:configs => @configs,
      :need_orig => @need_orig,
-     :allcommands => @allcommands}
+     :allcommands => @allcommands,
+     :retrycommands => @retrycommands}
   end
 
   #
@@ -234,13 +235,16 @@
     end
     parentshash.keys.sort
   end
-
+  
+  # Returns the value of the generation_status variable, see comments in
+  # method for possible values.
   def generate_file(file, request)
     # Skip files we've already generated in response to <depend>
     # statements.
     if @already_generated[file]
       @dlogger.debug "Skipping already generated #{file}"
-      return
+      # Return the status of that previous generation
+      return @generation_status[file]
     end
 
     # Check for circular dependencies, otherwise we're vulnerable
@@ -274,46 +278,43 @@
     # As we go through the process of generating the file we'll end up with
     # four possible outcomes:
     # fatal error: raise an exception
-    # failure: we're missing needed data, generally the original file
+    # failure: we're missing needed data for this file or a dependency,
+    #          generally the original file
     # success: we successfully processed a valid configuration
     # unknown: no valid configuration nor errors encountered, probably because
-    #          filtering removed everything from the config.xml file
+    #          filtering removed everything from the config.xml file.  This
+    #          should be considered a successful outcome, it indicates the
+    #          caller/client provided us with all required data and our result
+    #          is that no action needs to be taken.
     # We keep track of which of the failure, success or unknown states we end
     # up in via the generation_status variable.  We initialize it to :unknown.
     # If we encounter either failure or success we set it to false or :success.
     catch :generate_done do
       # Generate any other files that this file depends on
       depends = []
+      proceed = true
       Etch.xmleach(config_xml, '/config/depend') do |depend|
         @dlogger.debug "Generating dependency #{Etch.xmltext(depend)}"
         depends << Etch.xmltext(depend)
-        generate_file(Etch.xmltext(depend), request)
+        r = generate_file(Etch.xmltext(depend), request)
+        proceed = proceed && r
       end
-      # If any dependency failed to generate (due to a need for orig contents
-      # from the client) then we need to unroll the whole dependency tree and
-      # punt it back to the client
-      dependency_status = depends.all? { |depend| @generation_status[depend] }
-      if !dependency_status
-        depends.each do |depend|
-          # Make sure any configuration we're returning is just the basics
-          # needed to supply orig data
-          if @configs[depend]
-            filter_xml_completely!(@configs[depend], ['depend', 'setup'])
-          end
-          # And if we weren't already planning to request orig contents for this
-          # file then stick it into the orig request list so that the client
-          # knows it needs to ask for this file again next time.
-          if !@need_orig.has_key?(depend)
-            @need_orig[depend] = true
-          end
-        end
-        # Lastly make sure that this file gets sent back appropriately
+      # Also generate any commands that this file depends on
+      Etch.xmleach(config_xml, '/config/dependcommand') do |dependcommand|
+        @dlogger.debug "Generating command dependency #{Etch.xmltext(dependcommand)}"
+        r = generate_commands(Etch.xmltext(dependcommand), request)
+        proceed = proceed && r
+      end
+      if !proceed
+        @dlogger.debug "One or more dependencies of #{file} need data from client"
+        # Tell the client to request this file again
         @need_orig[file] = true
+        # Strip this file's config down to the bare necessities
         filter_xml_completely!(config_xml, ['depend', 'setup'])
         generation_status = false
         throw :generate_done
       end
-    
+      
       # Change into the corresponding directory so that the user can
       # refer to source files and scripts by their relative pathnames.
       Dir::chdir "#{@sourcebase}/#{file}"
@@ -346,6 +347,7 @@
       if request[:files] && request[:files][file] && request[:files][file][:orig]
         original_file = request[:files][file][:orig]
       else
+        @dlogger.debug "Need original contents of #{file} from client"
         @need_orig[file] = true
         # If there are setup commands defined for this file we need to
         # pass those back along with our request for the original file,
@@ -777,8 +779,11 @@
         end
       end
     end
-  
-    if generation_status && generation_status != :unknown &&
+    
+    # In addition to successful configs return configs for files that need
+    # orig data (generation_status==false) because any setup commands might be
+    # needed to create the original file.
+    if generation_status != :unknown &&
        Etch.xmlfindfirst(config_xml, '/config/*')
       # The client needs this attribute to know to which file
       # this chunk of XML refers
@@ -789,8 +794,12 @@
     @already_generated[file] = true
     @filestack.delete(file)
     @generation_status[file] = generation_status
+    
+    generation_status
   end
   
+  # Returns the value of the generation_status variable, see comments in
+  # method for possible values.
   def generate_commands(command, request)
     # Skip commands we've already generated in response to <depend>
     # statements.
@@ -826,31 +835,67 @@
       raise "Filtered commands.xml for #{command} fails validation"
     end
     
-    # Generate any other files that this file depends on
-    Etch.xmleach(commands_xml, '/commands/depend') do |depend|
-      @dlogger.debug "Generating command dependency #{Etch.xmltext(depend)}"
-      generate_commands(Etch.xmltext(depend), request)
-    end
-    
-    # Change into the corresponding directory so that the user can
-    # refer to source files and scripts by their relative pathnames.
-    Dir::chdir "#{@commandsbase}/#{command}"
-    
-    # Check that the resulting document is consistent after filtering
-    Etch.xmleach(commands_xml, '/commands/step') do |step|
-      guard_exec_elements = Etch.xmlarray(step, 'guard/exec')
-      if check_for_inconsistency(guard_exec_elements)
-        raise "Inconsistent guard 'exec' entries for #{command}"
+    generation_status = :unknown
+    # As we go through the process of generating the command we'll end up with
+    # four possible outcomes:
+    # fatal error: raise an exception
+    # failure: we're missing needed data for this command or a dependency,
+    #          generally the original file for a file this command depends on
+    # success: we successfully processed a valid configuration
+    # unknown: no valid configuration nor errors encountered, probably because
+    #          filtering removed everything from the commands.xml file.  This
+    #          should be considered a successful outcome, it indicates the
+    #          caller/client provided us with all required data and our result
+    #          is that no action needs to be taken.
+    # We keep track of which of the failure, success or unknown states we end
+    # up in via the generation_status variable.  We initialize it to :unknown.
+    # If we encounter either failure or success we set it to false or :success.
+    catch :generate_done do
+      # Generate any other commands that this command depends on
+      proceed = true
+      Etch.xmleach(commands_xml, '/commands/depend') do |depend|
+        @dlogger.debug "Generating command dependency #{Etch.xmltext(depend)}"
+        r = generate_commands(Etch.xmltext(depend), request)
+        proceed = proceed && r
       end
-      command_exec_elements = Etch.xmlarray(step, 'command/exec')
-      if check_for_inconsistency(command_exec_elements)
-        raise "Inconsistent command 'exec' entries for #{command}"
+      # Also generate any files that this command depends on
+      Etch.xmleach(commands_xml, '/commands/dependfile') do |dependfile|
+        @dlogger.debug "Generating file dependency #{Etch.xmltext(dependfile)}"
+        r = generate_file(Etch.xmltext(dependfile), request)
+        proceed = proceed && r
       end
+      if !proceed
+        # Try again next time
+        @retrycommands[command] = true
+        generation_status = false
+        throw :generate_done
+      end
+      
+      # Change into the corresponding directory so that the user can
+      # refer to source files and scripts by their relative pathnames.
+      Dir::chdir "#{@commandsbase}/#{command}"
+      
+      # Check that the resulting document is consistent after filtering
+      Etch.xmleach(commands_xml, '/commands/step') do |step|
+        guard_exec_elements = Etch.xmlarray(step, 'guard/exec')
+        if check_for_inconsistency(guard_exec_elements)
+          raise "Inconsistent guard 'exec' entries for #{command}"
+        end
+        command_exec_elements = Etch.xmlarray(step, 'command/exec')
+        if check_for_inconsistency(command_exec_elements)
+          raise "Inconsistent command 'exec' entries for #{command}"
+        end
+      end
+      
+      # I'm not sure if we'd benefit from further checking the XML for
+      # validity.  For now we declare success if we got this far.
+      generation_status = :success
     end
     
     # If filtering didn't remove all the content then add this to the list of
     # commands to be returned to the client.
-    if Etch.xmlfindfirst(commands_xml, '/commands/*')
+    if generation_status && generation_status != :unknown &&
+       Etch.xmlfindfirst(commands_xml, '/commands/*')
       # Include the commands directory name to aid troubleshooting on the
       # client side.
       Etch.xmlattradd(Etch.xmlroot(commands_xml), 'commandname', command)
@@ -859,6 +904,9 @@
     
     @already_generated[command] = true
     @filestack.delete(command)
+    @generation_status[command] = generation_status
+    
+    generation_status
   end
   
   ALWAYS_KEEP = ['depend', 'setup', 'pre', 'test_before_post', 'post', 'test']
@@ -1198,7 +1246,11 @@
     when :libxml
       element.remove!
     when :rexml
-      xmldoc.elements.delete(element)
+      if xmldoc.node_type == :document
+        xmldoc.root.elements.delete(element)
+      else
+        xmldoc.elements.delete(element)
+      end
     else
       raise "Unknown @xmllib #{@xmllib}"
     end


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

[etch-devel] SF.net SVN: etch:[128] trunk/server/app/controllers/files_controller.rb

[<jh...@us...>]

Revision: 128
          http://etch.svn.sourceforge.net/etch/?rev=128&view=rev
Author:   jheiss
Date:     2009-10-29 20:40:41 +0000 (Thu, 29 Oct 2009)

Log Message:
-----------
Add support for requesting specific commands.

Modified Paths:
--------------
    trunk/server/app/controllers/files_controller.rb

Modified: trunk/server/app/controllers/files_controller.rb
===================================================================
--- trunk/server/app/controllers/files_controller.rb	2009-10-29 20:40:20 UTC (rev 127)
+++ trunk/server/app/controllers/files_controller.rb	2009-10-29 20:40:41 UTC (rev 128)
@@ -13,8 +13,15 @@
       # The client runs the filename through CGI.escape in case it contains
       # special characters.  Older versions of Rails automatically decoded the
       # filename, but as of Rails 2.3 we need to do it ourself.
-      files = params[:files].inject({}) { |h, (file, value)| h[CGI.unescape(file)] = value; h }
-      response = etchserver.generate(files)
+      files = {}
+      if params[:files]
+        files = params[:files].inject({}) { |h, (file, value)| h[CGI.unescape(file)] = value; h }
+      end
+      commands = {}
+      if params[:commands]
+        commands = params[:commands].inject({}) { |h, (command, value)| h[CGI.unescape(command)] = value; h }
+      end
+      response = etchserver.generate(files, commands)
       render :text => response
     rescue Exception => e
       logger.error e.message


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

[etch-devel] SF.net SVN: etch:[127] trunk/test/testrepo

[<jh...@us...>]

Revision: 127
          http://etch.svn.sourceforge.net/etch/?rev=127&view=rev
Author:   jheiss
Date:     2009-10-29 20:40:20 +0000 (Thu, 29 Oct 2009)

Log Message:
-----------
Add elements associated with files that depend on commands and
vice-versa.

Modified Paths:
--------------
    trunk/test/testrepo/commands.dtd
    trunk/test/testrepo/config.dtd

Modified: trunk/test/testrepo/commands.dtd
===================================================================
--- trunk/test/testrepo/commands.dtd	2009-10-29 20:40:08 UTC (rev 126)
+++ trunk/test/testrepo/commands.dtd	2009-10-29 20:40:20 UTC (rev 127)
@@ -1,6 +1,7 @@
-<!ELEMENT commands (depend*, step*)>
+<!ELEMENT commands (depend*, dependfile*, step*)>
 
 <!ELEMENT depend (#PCDATA)>
+<!ELEMENT dependfile (#PCDATA)>
 
 <!ELEMENT step (guard, command)>
 <!ELEMENT guard (exec+)>

Modified: trunk/test/testrepo/config.dtd
===================================================================
--- trunk/test/testrepo/config.dtd	2009-10-29 20:40:08 UTC (rev 126)
+++ trunk/test/testrepo/config.dtd	2009-10-29 20:40:20 UTC (rev 127)
@@ -1,8 +1,9 @@
-<!ELEMENT config (revert?, depend*, server_setup?, setup?, pre?, (file|link|directory|delete)+, test_before_post?, post?, test?)>
+<!ELEMENT config (revert?, depend*, dependcommand*, server_setup?, setup?, pre?, (file|link|directory|delete)+, test_before_post?, post?, test?)>
 
 <!ELEMENT revert EMPTY>
 
 <!ELEMENT depend (#PCDATA)>
+<!ELEMENT dependcommand (#PCDATA)>
 
 <!ELEMENT server_setup (exec*)>
 <!ELEMENT setup (exec*)>


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

[etch-devel] SF.net SVN: etch:[126] trunk/test/TODO

[<jh...@us...>]

Revision: 126
          http://etch.svn.sourceforge.net/etch/?rev=126&view=rev
Author:   jheiss
Date:     2009-10-29 20:40:08 +0000 (Thu, 29 Oct 2009)

Log Message:
-----------
Fix typo

Modified Paths:
--------------
    trunk/test/TODO

Modified: trunk/test/TODO
===================================================================
--- trunk/test/TODO	2009-10-29 20:39:43 UTC (rev 125)
+++ trunk/test/TODO	2009-10-29 20:40:08 UTC (rev 126)
@@ -1,7 +1,7 @@
 FIXME: Seems like we could restructure this so that the server is just started
 as needed (i.e. for the first test that runs) and then the config repo is
 cleaned out and recreated as needed. That should speed up the test runs quite
-a bit, and reduce the occurance of the occasional timing-related failures
+a bit, and reduce the occurrence of the occasional timing-related failures
 where the server doesn't finish starting up before we unleash the client.
 
 Restructure so that tests can run in local or server mode


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

[etch-devel] SF.net SVN: etch:[125] trunk/test/commands.rb

[<jh...@us...>]

Revision: 125
          http://etch.svn.sourceforge.net/etch/?rev=125&view=rev
Author:   jheiss
Date:     2009-10-29 20:39:43 +0000 (Thu, 29 Oct 2009)

Log Message:
-----------
Add test for a command with dependency on a file. (New feature)

Modified Paths:
--------------
    trunk/test/commands.rb

Modified: trunk/test/commands.rb
===================================================================
--- trunk/test/commands.rb	2009-10-29 20:39:29 UTC (rev 124)
+++ trunk/test/commands.rb	2009-10-29 20:39:43 UTC (rev 125)
@@ -11,9 +11,9 @@
 
 class EtchCommandTests < Test::Unit::TestCase
   include EtchTests
-
+  
   def setup
-    # Generate a file to use as our etch target/destination
+    # Generate a file to use as a target in commands
     @targetfile = Tempfile.new('etchtest').path
     #puts "Using #{@targetfile} as target file"
     
@@ -84,7 +84,7 @@
     #puts "Running '#{testname}' test"
     run_etch(@port, @testbase, true)
   end
-    
+  
   def test_commands_guard_succeeds
     #
     # Guard initially succeeds
@@ -116,7 +116,7 @@
     # Verify that the file was not touched
     assert_equal(testname, get_file_contents(@targetfile), testname)
   end
-    
+  
   def test_commands_multiple_steps
     #
     # Multiple steps
@@ -199,8 +199,8 @@
     # Verify that both commands ran, ordering doesn't matter
     assert_equal(['firstcmd', 'secondcmd'], get_file_contents(@targetfile).split("\n").sort, testname)
   end
-    
-  def test_commands_dependency
+  
+  def test_commands_depend
     #
     # Multiple commands with dependency
     #
@@ -246,6 +246,64 @@
     assert_equal("firstcmd\nsecondcmd\n", get_file_contents(@targetfile), testname)
   end
   
+  def test_commands_dependfile
+    #
+    # Command with dependency on a file
+    #
+    testname = 'command with file dependency'
+    
+    targetfile2 = Tempfile.new('etchtest').path
+    FileUtils.mkdir_p("#{@repodir}/source/#{targetfile2}")
+    File.open("#{@repodir}/source/#{targetfile2}/config.xml", 'w') do |file|
+      file.puts <<-EOF
+        <config>
+          <file>
+            <warning_file/>
+            <source>
+              <plain>source</plain>
+            </source>
+          </file>
+          <post>
+            <exec>sleep 3</exec>
+          </post>
+        </config>
+      EOF
+    end
+    
+    FileUtils.mkdir_p("#{@repodir}/commands/etchtest")
+    File.open("#{@repodir}/commands/etchtest/commands.xml", 'w') do |file|
+      file.puts <<-EOF
+        <commands>
+          <dependfile>#{targetfile2}</dependfile>
+          <step>
+            <guard>
+              <exec>grep '#{testname}' #{@targetfile}</exec>
+            </guard>
+            <command>
+              <exec>printf '#{testname}' >> #{@targetfile}</exec>
+            </command>
+          </step>
+        </commands>
+      EOF
+    end
+    
+    sourcecontents = "Test #{testname}\n"
+    File.open("#{@repodir}/source/#{targetfile2}/source", 'w') do |file|
+      file.write(sourcecontents)
+    end
+    
+    # Run etch
+    #puts "Running '#{testname}' test"
+    run_etch(@port, @testbase)
+    
+    # Verify that the command-generated file and the regular file were created
+    # properly
+    assert_equal(testname, get_file_contents(@targetfile), testname + ' command contents')
+    assert_equal(sourcecontents, get_file_contents(targetfile2), testname + ' file contents')
+    # And verify that they were created in the right order
+    assert(File.stat(@targetfile).mtime > File.stat(targetfile2).mtime, testname + ' ordering')
+  end
+  
   def test_commands_filtering
     #
     # Attribute filtering


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

[etch-devel] SF.net SVN: etch:[124] trunk/test/history.rb

[<jh...@us...>]

Revision: 124
          http://etch.svn.sourceforge.net/etch/?rev=124&view=rev
Author:   jheiss
Date:     2009-10-29 20:39:29 +0000 (Thu, 29 Oct 2009)

Log Message:
-----------
When testing the use of a setup command to create the original file, use
the original file contents in a script to generate the desired file, so
that we can be sure the contents generated by the setup action are sent
back to the server, in addition to the check that they are saved
properly on the client. Happened to notice that wasn't working right.

Modified Paths:
--------------
    trunk/test/history.rb

Modified: trunk/test/history.rb
===================================================================
--- trunk/test/history.rb	2009-10-29 20:38:54 UTC (rev 123)
+++ trunk/test/history.rb	2009-10-29 20:39:29 UTC (rev 124)
@@ -144,8 +144,14 @@
     # simulate a common usage of setup commands to install a package before
     # we backup the original file so that the original file has the default
     # config file contents) and ensure those contents are backed up as the
-    # original file
+    # original file.
     #
+    # Generate the file contents with a script which incorporates the original
+    # contents so that we also ensure that the client sends us the correct
+    # contents on the first try.  We once had a bug where it took a couple of
+    # tries before we achieved convergence and the client sent the correct
+    # original contents.
+    #
     
     origfile = File.join(@testbase, 'orig', "#{@targetfile}.ORIG")
     origcontents = "This is the original text"
@@ -160,7 +166,7 @@
           <file>
             <warning_file/>
             <source>
-              <plain>source</plain>
+              <script>source.script</script>
             </source>
           </file>
         </config>
@@ -168,8 +174,9 @@
     end
 
     sourcecontents = "This is a test\n"
-    File.open("#{@repodir}/source/#{@targetfile}/source", 'w') do |file|
-      file.write(sourcecontents)
+    File.open("#{@repodir}/source/#{@targetfile}/source.script", 'w') do |file|
+      file.puts("@contents << '#{sourcecontents}'")
+      file.puts("@contents << IO.read(@original_file)")
     end
 
     # Run etch
@@ -177,6 +184,7 @@
     run_etch(@port, @testbase)
 
     assert_equal(origcontents + "\n", get_file_contents(origfile), 'original backup of file via setup')
+    assert_equal(sourcecontents + origcontents + "\n", get_file_contents(@targetfile), 'contents using original backup of file via setup')
   end
   
   def test_delayed_history_setup


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

[etch-devel] SF.net SVN: etch:[123] trunk/test/depend.rb

[<jh...@us...>]

Revision: 123
          http://etch.svn.sourceforge.net/etch/?rev=123&view=rev
Author:   jheiss
Date:     2009-10-29 20:38:54 +0000 (Thu, 29 Oct 2009)

Log Message:
-----------
Break the tests up into a number of test methods rather than one giant
method. Reduces the risk of interference between the tests and makes it
easier to run specific tests for debugging purposes.

Add test for dependency on a command.

Modified Paths:
--------------
    trunk/test/depend.rb

Modified: trunk/test/depend.rb
===================================================================
--- trunk/test/depend.rb	2009-10-29 20:38:31 UTC (rev 122)
+++ trunk/test/depend.rb	2009-10-29 20:38:54 UTC (rev 123)
@@ -29,7 +29,6 @@
   end
   
   def test_depends
-
     #
     # Run a basic dependency test
     #
@@ -80,19 +79,22 @@
 
     # Run etch
     #puts "Running initial dependency test"
-    run_etch(@port, @testbase, false, '--debug')
+    run_etch(@port, @testbase, false)
 
     # Verify that the files were created properly
     assert_equal(sourcecontents, get_file_contents(@targetfile), 'dependency file 1')
     assert_equal(sourcecontents, get_file_contents(@targetfile2), 'dependency file 2')
     # And in the right order
     assert(File.stat(@targetfile).mtime > File.stat(@targetfile2).mtime, 'dependency ordering')
-
+  end
+  
+  def test_depend_request_single
     #
     # Run a dependency test where the user only requests the first
     # file on the command line
     #
-
+    testname = 'depend with single request'
+    
     FileUtils.mkdir_p("#{@repodir}/source/#{@targetfile}")
     File.open("#{@repodir}/source/#{@targetfile}/config.xml", 'w') do |file|
       file.puts <<-EOF
@@ -129,8 +131,7 @@
       EOF
     end
 
-    # Vary the source contents so we know the files were updated
-    sourcecontents = "This is a different test\n"
+    sourcecontents = "Test #{testname}\n"
     File.open("#{@repodir}/source/#{@targetfile}/source", 'w') do |file|
       file.write(sourcecontents)
     end
@@ -139,7 +140,7 @@
     end
 
     # Run etch
-    #puts "Running single request dependency test"
+    #puts "Running '#{testname}' test"
     run_etch(@port, @testbase, false, @targetfile)
 
     # Verify that the files were created properly
@@ -147,10 +148,13 @@
     assert_equal(sourcecontents, get_file_contents(@targetfile2), 'single request dependency file 2')
     # And in the right order
     assert(File.stat(@targetfile).mtime > File.stat(@targetfile2).mtime, 'single request dependency ordering')
-
+  end
+  
+  def test_circular_dependency
     #
     # Run a circular dependency test
     #
+    testname = 'circular dependency'
 
     FileUtils.mkdir_p("#{@repodir}/source/#{@targetfile}")
     File.open("#{@repodir}/source/#{@targetfile}/config.xml", 'w') do |file|
@@ -181,26 +185,90 @@
       EOF
     end
 
-    # Vary the source contents so we know the files weren't updated
-    oldsourcecontents = sourcecontents
-    sourcecontents = "This is a circular dependency test\n"
+    sourcecontents = "Test #{testname}\n"
     File.open("#{@repodir}/source/#{@targetfile}/source", 'w') do |file|
       file.write(sourcecontents)
     end
     File.open("#{@repodir}/source/#{@targetfile2}/source", 'w') do |file|
       file.write(sourcecontents)
     end
-
+    
+    # Put some text into the original files so that we can make sure they
+    # are not touched.
+    origcontents = "This is the original text\n"
+    [@targetfile, @targetfile2].each do |targetfile|
+      File.delete(targetfile)
+      File.open(targetfile, 'w') do |file|
+        file.write(origcontents)
+      end
+    end
+    
     # Run etch
-    #puts "Running circular test"
+    #puts "Running '#{testname}' test"
     run_etch(@port, @testbase, true, @targetfile)
 
     # Verify that the files weren't modified
-    assert_equal(oldsourcecontents, get_file_contents(@targetfile), 'circular dependency file 1')
-    assert_equal(oldsourcecontents, get_file_contents(@targetfile2), 'circular dependency file 2')
-
+    assert_equal(origcontents, get_file_contents(@targetfile), 'circular dependency file 1')
+    assert_equal(origcontents, get_file_contents(@targetfile2), 'circular dependency file 2')
   end
-
+  
+  def test_command_dependency
+    #
+    # Test a dependency on a command
+    #
+    testname = 'depend on command'
+    
+    FileUtils.mkdir_p("#{@repodir}/source/#{@targetfile}")
+    File.open("#{@repodir}/source/#{@targetfile}/config.xml", 'w') do |file|
+      file.puts <<-EOF
+        <config>
+          <dependcommand>etchtest</dependcommand>
+          <pre>
+            <exec>sleep 3</exec>
+          </pre>
+          <file>
+            <warning_file></warning_file>
+            <source>
+              <plain>source</plain>
+            </source>
+          </file>
+        </config>
+      EOF
+    end
+    
+    sourcecontents = "Test #{testname}\n"
+    File.open("#{@repodir}/source/#{@targetfile}/source", 'w') do |file|
+      file.write(sourcecontents)
+    end
+    
+    FileUtils.mkdir_p("#{@repodir}/commands/etchtest")
+    File.open("#{@repodir}/commands/etchtest/commands.xml", 'w') do |file|
+      file.puts <<-EOF
+        <commands>
+          <step>
+            <guard>
+              <exec>grep '#{testname}' #{@targetfile2}</exec>
+            </guard>
+            <command>
+              <exec>printf '#{testname}' >> #{@targetfile2}</exec>
+            </command>
+          </step>
+        </commands>
+      EOF
+    end
+    
+    # Run etch
+    #puts "Running '#{testname}' test"
+    run_etch(@port, @testbase)
+    
+    # Verify that the regular file and the command-generated file were created
+    # properly
+    assert_equal(sourcecontents, get_file_contents(@targetfile), testname + ' file contents')
+    assert_equal(testname, get_file_contents(@targetfile2), testname + ' command contents')
+    # And verify that they were created in the right order
+    assert(File.stat(@targetfile).mtime > File.stat(@targetfile2).mtime, testname + ' ordering')
+  end
+  
   def teardown
     stop_server(@pid)
     remove_repository(@repodir)


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

[etch-devel] SF.net SVN: etch:[122] trunk/test/options.rb

[<jh...@us...>]

Revision: 122
          http://etch.svn.sourceforge.net/etch/?rev=122&view=rev
Author:   jheiss
Date:     2009-10-29 20:38:31 +0000 (Thu, 29 Oct 2009)

Log Message:
-----------
Add a test that the user can request specific files and commands on the
command line.

Modified Paths:
--------------
    trunk/test/options.rb

Modified: trunk/test/options.rb
===================================================================
--- trunk/test/options.rb	2009-10-26 21:48:14 UTC (rev 121)
+++ trunk/test/options.rb	2009-10-29 20:38:31 UTC (rev 122)
@@ -119,6 +119,134 @@
     assert(output.size <= 23, 'help output lines')
   end
   
+  def test_specific_requests
+    #
+    # Test that the user can request specific files and commands on the
+    # command line
+    #
+    testname = 'specific command line requests'
+    
+    FileUtils.mkdir_p("#{@repodir}/source/#{@targetfile}")
+    File.open("#{@repodir}/source/#{@targetfile}/config.xml", 'w') do |file|
+      file.puts <<-EOF
+        <config>
+          <file>
+            <warning_file></warning_file>
+            <source>
+              <plain>source</plain>
+            </source>
+          </file>
+        </config>
+      EOF
+    end
+    targetfile2 = Tempfile.new('etchtest').path
+    FileUtils.mkdir_p("#{@repodir}/source/#{targetfile2}")
+    File.open("#{@repodir}/source/#{targetfile2}/config.xml", 'w') do |file|
+      file.puts <<-EOF
+        <config>
+          <file>
+            <warning_file></warning_file>
+            <source>
+              <plain>source</plain>
+            </source>
+          </file>
+        </config>
+      EOF
+    end
+    targetfile3 = Tempfile.new('etchtest').path
+    FileUtils.mkdir_p("#{@repodir}/source/#{targetfile3}")
+    File.open("#{@repodir}/source/#{targetfile3}/config.xml", 'w') do |file|
+      file.puts <<-EOF
+        <config>
+          <file>
+            <warning_file></warning_file>
+            <source>
+              <plain>source</plain>
+            </source>
+          </file>
+        </config>
+      EOF
+    end
+    
+    sourcecontents = "Test #{testname}\n"
+    [@targetfile, targetfile2, targetfile3].each do |target|
+      File.open("#{@repodir}/source/#{target}/source", 'w') do |file|
+        file.write(sourcecontents)
+      end
+    end
+    
+    cmdtargetfile1 = Tempfile.new('etchtest').path
+    FileUtils.mkdir_p("#{@repodir}/commands/etchtest1")
+    File.open("#{@repodir}/commands/etchtest1/commands.xml", 'w') do |file|
+      file.puts <<-EOF
+        <commands>
+          <step>
+            <guard>
+              <exec>grep '#{testname}' #{cmdtargetfile1}</exec>
+            </guard>
+            <command>
+              <exec>printf '#{testname}' >> #{cmdtargetfile1}</exec>
+            </command>
+          </step>
+        </commands>
+      EOF
+    end
+    cmdtargetfile2 = Tempfile.new('etchtest').path
+    FileUtils.mkdir_p("#{@repodir}/commands/etchtest2")
+    File.open("#{@repodir}/commands/etchtest2/commands.xml", 'w') do |file|
+      file.puts <<-EOF
+        <commands>
+          <step>
+            <guard>
+              <exec>grep '#{testname}' #{cmdtargetfile2}</exec>
+            </guard>
+            <command>
+              <exec>printf '#{testname}' >> #{cmdtargetfile2}</exec>
+            </command>
+          </step>
+        </commands>
+      EOF
+    end
+    cmdtargetfile3 = Tempfile.new('etchtest').path
+    FileUtils.mkdir_p("#{@repodir}/commands/etchtest3")
+    File.open("#{@repodir}/commands/etchtest3/commands.xml", 'w') do |file|
+      file.puts <<-EOF
+        <commands>
+          <step>
+            <guard>
+              <exec>grep '#{testname}' #{cmdtargetfile3}</exec>
+            </guard>
+            <command>
+              <exec>printf '#{testname}' >> #{cmdtargetfile3}</exec>
+            </command>
+          </step>
+        </commands>
+      EOF
+    end
+    
+    # Put some text into the original files so that we can make sure the ones
+    # that shouldn't get touched are not touched.
+    origcontents = "This is the original text\n"
+    [@targetfile, targetfile2, targetfile3, cmdtargetfile1, cmdtargetfile2, cmdtargetfile3].each do |target|
+      File.open(target, 'w') do |file|
+        file.write(origcontents)
+      end
+    end
+    
+    # Run etch
+    #puts "Running '#{testname}' test"
+    run_etch(@port, @testbase, false, "#{@targetfile} #{targetfile2} etchtest1 etchtest2")
+    
+    # Verify that only the requested files were created
+    assert_equal(sourcecontents, get_file_contents(@targetfile), testname + ' file 1')
+    assert_equal(sourcecontents, get_file_contents(targetfile2), testname + ' file 2')
+    assert_equal(origcontents, get_file_contents(targetfile3), testname + ' file 3')
+    # And only the requested commands were run
+    assert_equal(origcontents + testname, get_file_contents(cmdtargetfile1), testname + ' cmd 1')
+    assert_equal(origcontents + testname, get_file_contents(cmdtargetfile2), testname + ' cmd 2')
+    assert_equal(origcontents, get_file_contents(cmdtargetfile3), testname + ' cmd 3')
+  end
+  
   def teardown
     stop_server(@pid)
     remove_repository(@repodir)


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

[etch-devel] SF.net SVN: etch:[121] trunk/etchserver-samples/nventory

[<jh...@us...>]

Revision: 121
          http://etch.svn.sourceforge.net/etch/?rev=121&view=rev
Author:   jheiss
Date:     2009-10-26 21:48:14 +0000 (Mon, 26 Oct 2009)

Log Message:
-----------
Update to reflect change in nVentory get_objects API.

Modified Paths:
--------------
    trunk/etchserver-samples/nventory/nodegrouper
    trunk/etchserver-samples/nventory/nodetagger

Modified: trunk/etchserver-samples/nventory/nodegrouper
===================================================================
--- trunk/etchserver-samples/nventory/nodegrouper	2009-10-13 19:51:24 UTC (rev 120)
+++ trunk/etchserver-samples/nventory/nodegrouper	2009-10-26 21:48:14 UTC (rev 121)
@@ -13,7 +13,7 @@
 #
 
 nvclient = NVentory::Client.new
-results = nvclient.get_objects('nodes', {}, { 'name' => name }, ['node_groups'])
+results = nvclient.get_objects('nodes', {}, { 'name' => name }, {}, {}, ['node_groups'])
 
 if results.include?(name) && results[name].include?('node_groups')
   results[name]['node_groups'].each do |ng|

Modified: trunk/etchserver-samples/nventory/nodetagger
===================================================================
--- trunk/etchserver-samples/nventory/nodetagger	2009-10-13 19:51:24 UTC (rev 120)
+++ trunk/etchserver-samples/nventory/nodetagger	2009-10-26 21:48:14 UTC (rev 121)
@@ -82,7 +82,7 @@
 #
 
 nvclient = NVentory::Client.new
-results = nvclient.get_objects('nodes', {}, { 'name' => name }, ['node_groups'])
+results = nvclient.get_objects('nodes', {}, { 'name' => name }, {}, {}, ['node_groups'])
 
 tag = ''
 DEFAULT_HOURS = 4


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

[etch-devel] SF.net SVN: etch:[120] tags

[<jh...@us...>]

Revision: 120
          http://etch.svn.sourceforge.net/etch/?rev=120&view=rev
Author:   jheiss
Date:     2009-10-13 19:51:24 +0000 (Tue, 13 Oct 2009)

Log Message:
-----------
Tag 3.8 release

Modified Paths:
--------------
    Makefile

Added Paths:
-----------
    tags/release-3.8/

Modified: Makefile
===================================================================
--- Makefile	2009-10-13 18:29:59 UTC (rev 119)
+++ Makefile	2009-10-13 19:51:24 UTC (rev 120)
@@ -1,4 +1,4 @@
-VER=3.7
+VER=3.8
 TAGNAME=release-$(VER)
 
 all: dist


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.