webfetch-commits Mailing List for WebFetch (Page 2)

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

Revision: 45
          http://webfetch.svn.sourceforge.net/webfetch/?rev=45&view=rev
Author:   ikluft
Date:     2009-09-05 09:35:07 +0000 (Sat, 05 Sep 2009)

Log Message:
-----------
expand docs, expand error reporting, streamline debugging, add AUTOLOAD

Modified Paths:
--------------
    branches/v0.13/lib/WebFetch.pm

Modified: branches/v0.13/lib/WebFetch.pm
===================================================================

--- branches/v0.13/lib/WebFetch.pm	2009-08-29 00:38:38 UTC (rev 44)
+++ branches/v0.13/lib/WebFetch.pm	2009-09-05 09:35:07 UTC (rev 45)
@@ -19,15 +19,28 @@
 
 =head1 DESCRIPTION
 
-The WebFetch module is a general framework for downloading and saving
-information from the web, and for display on the web.
-It requires another module to inherit it and fill in the specifics of
-what and how to download.
-WebFetch provides a generalized interface for saving to a file
+The WebFetch module is a framework for downloading and saving
+information from the web, and for saving or re-displaying it.
+It provides a generalized interface for saving to a file
 while keeping the previous version as a backup.
-This is expected to be used for periodically-updated information
-which is run as a cron job.
+This is mainly intended for use in a cron-job to acquire
+periodically-updated information.
 
+WebFetch allows the user to specify a source and destination, and
+the input and output formats.  It is possible to write new Perl modules
+to the WebFetch API in order to add more input and output formats.
+
+The currently-provided input formats are Atom, RSS, WebFetch "SiteNews" files
+and raw Perl data structures.
+
+The currently-provided output formats are RSS, WebFetch "SiteNews" files,
+the Perl Template Toolkit, and export into a TWiki site.
+
+Some modules which were specific to pre-RSS/Atom web syndication formats
+have been deprecated.  Those modules can be found in the CPAN archive
+in WebFetch 0.10.  Those modules are no longer compatible with changes
+in the current WebFetch API.
+
 =head1 INSTALLATION
 
 After unpacking and the module sources from the tar file, run
@@ -63,41 +76,19 @@
 
 =head2 SETTING UP CRONTAB ENTRIES
 
-First of all, if you don't have crontab access or don't know what they are,
-contact your site's system administrator(s).  Only local help will do any
-good on local-configuration issues.  No one on the Internet can help.
-(If you are the administrator for your system, see the crontab(1) and
-crontab(5) manpages and nearly any book on Unix system administration.)
+If needed, see the manual pages for crontab(1), crontab(5) and any
+web sites or books on Unix system administration.
 
-Since the WebFetch command lines are usually very long, you may prefer
-to make one or more scripts as front-ends so your crontab entries aren't
-so huge.
+Since WebFetch command lines are usually very long, the user may prefer
+to make one or more scripts as front-ends so crontab entries aren't so big.
 
-Do not run the crontab entries too often - be a good net.citizen and
-do your updates no more often than necessary.
-Popular sites need their users to refrain from making automated
-requests too often because they add up on an enormous scale
-on the Internet.
-Some sites such as Freshmeat prefer no shorter than hourly intervals.
-Slashdot prefers no shorter than half-hourly intervals.
-When in doubt, ask the site maintainers what they prefer.
+Try not to run crontab entries too often - be aware if the site you're
+accessing has any resource constraints, and how often their information
+gets updated.  If they request users not to access a feed more often
+than a certain interval, respect it.  (It isn't hard to find violators
+in server logs.)  If in doubt, try every 30 minutes until more information
+becomes available.
 
-(Then again, there are a very few sites like Yahoo and CNN who don't
-mind getting the extra hits if you're going to create links to them.
-Even so, more often than every 20 minutes would still be  excessive
-to the biggest web sites.)
-
-=head2 SETTING UP SERVER-SIDE INCLUDES
-
-See the manual for your web server to make sure you have server-side include
-(SSI) enabled for the files that need it.
-(It's wasteful to enable it for all your files so be careful.)
-
-When using Apache HTTPD,
-a line like this will include a WebFetch-generated file:
-
-<!--#include file="fetch/slashdot.html"-->
-
 =head1 WebFetch FUNCTIONS
 
 The following function definitions assume B<C<$obj>> is a blessed
@@ -147,10 +138,10 @@
 		description => "unable to save: no data or nowhere to save it",
 	},
 
-	'WebFetch::Exception::NoInputHandler' => {
+	'WebFetch::Exception::NoHandler' => {
                 isa => 'WebFetch::Exception',
-		alias => 'throw_no_input_handler',
-		description => "no input handler was found",
+		alias => 'throw_no_handler',
+		description => "no handler was found",
 	},
 
 	'WebFetch::Exception::MustOverride' => {
@@ -182,10 +173,16 @@
                 description => "no module was found to run the request",
         },
 
+	'WebFetch::Exception::AutoRunFailure' => {
+                isa => 'WebFetch::TracedException',
+		alias => 'throw_autoload_fail',
+                description => "AUTORUN failed to handle function call",
+        },
+
 );
 
 # initialize class variables
-our $VERSION = '0.12';
+our $VERSION = '0.13-pre29';
 our %default_modules = (
 	"input" => {
 		"rss" => "WebFetch::Input::RSS",
@@ -203,12 +200,14 @@
 	}
 );
 our %modules;
+our $AUTOLOAD;
 my $debug;
 
-=item import( "param-name" => "value", ... )
+sub debug
+{
+	$debug and print STDERR "debug: ".join( " ", @_ )."\n";
+}
 
-=cut
-
 =item WebFetch::module_register( $module, @capabilities );
 
 This function allows a Perl module to register itself with the WebFetch API
@@ -223,9 +222,15 @@
 The @capabilities array is any number of strings as needed to list the
 capabilities which the module performs for the WebFetch API.
 The currently-recognized capabilities are "cmdline", "input" and "output".
-"config" and "storage" are reserved for future use.  The function will save
-all the capability names that the module provides.
+"config", "filter", "save" and "storage" are reserved for future use.  The
+function will save all the capability names that the module provides, without
+checking whether any code will use it.
 
+For example, the WebFetch::Output::TT module registers itself like this:
+   C<__PACKAGE__-&gt;module_register( "cmdline", "output:tt" );>
+meaning that it defines additional command-line options, and it provides an
+output format handler for the "tt" format, the Perl Template Toolkit.
+
 =cut
 
 sub module_register
@@ -258,10 +263,6 @@
 	}
 }
 
-# satisfy POD coverage test - but don't put this function in the user manual
-=pod
-=cut
-
 # module selection - choose WebFetch module based on selected file format
 # for WebFetch internal use only
 sub module_select
@@ -269,8 +270,7 @@
 	my $capability = shift;
 	my $is_optional = shift;
 
-	$debug and print STDERR "debug: "
-		."module_select($capability,$is_optional)\n";
+	debug "module_select($capability,$is_optional)";
 	# parse the capability string
 	my ( $group, $topic );
 	if ( $capability =~ /([^:]*):(.*)/ ) {
@@ -320,13 +320,12 @@
 		}
 	}
 	
-	# check if any handlers were found for this input format
+	# check if any handlers were found for this format
 	if ( ! @handlers and ! $is_optional ) {
-		throw_no_input_handler( "handler not found for $capability" );
+		throw_no_handler( "handler not found for $capability" );
 	}
 
-	$debug and print STDERR "debug: module_select: "
-		.join( " ", @handlers )."\n";
+	debug "module_select: ".join( " ", @handlers );
 	return @handlers;
 }
 
@@ -341,7 +340,7 @@
 {
 	my $group = shift;
 
-	$debug and print STDERR "debug: singular_handler($group)\n";
+	debug "singular_handler($group)";
 	my $count = 0;
 	my ( $entry, $last );
 	foreach $entry ( keys %{$modules{$group}} ) {
@@ -358,8 +357,7 @@
 	}
 
 	# if there's only one registered, that's the one to use
-	$debug and print STDERR "debug: singular_handler: "
-		."count=$count last=$last\n";
+	debug "singular_handler: count=$count last=$last";
 	return $count == 1 ? $last : undef;
 }
 
@@ -432,11 +430,13 @@
 		and ( ref $modules{cmdline} eq "ARRAY" ))
 	{
 		foreach $cli_mod ( @{$modules{cmdline}}) {
-			if ( defined @cli_mod::Options ) {
-				push @mod_options, @cli_mod::Options;
+			if ( eval "defined \@{".$cli_mod."::Options}" ) {
+				eval "push \@mod_options,"
+					."\@{".$cli_mod."::Options}";
 			}
-			if ( defined @cli_mod::Usage ) {
-				push @mod_options, @cli_mod::Usage;
+			if ( eval "defined \@{".$cli_mod."::Usage}" ) {
+				eval "push \@mod_options, \@{"
+					.$cli_mod."::Usage}";
 			}
 		}
 	}
@@ -470,7 +470,7 @@
 	if (( exists $options{debug}) and $options{debug}) {
 		$debug = 1;
 	}
-	$debug and print STDERR "debug: fetch_main\n";
+	debug "fetch_main";
 
 
 	# if either source/input or dest/output formats were not provided,
@@ -511,7 +511,7 @@
 	
 	# check if any handlers were found for this input format
 	if ( ! @handlers ) {
-		throw_no_input_handler( "input handler not found for "
+		throw_no_handler( "input handler not found for "
 			.$options{source_format});
 	}
 
@@ -519,7 +519,7 @@
 	my $pkgname;
 	my $run_count = 0;
 	foreach $pkgname ( @handlers ) {
-		$debug and print STDERR "debug: running for $pkgname\n";
+		debug "running for $pkgname";
 		eval { &WebFetch::run( $pkgname, \%options )};
 		if ( $@ ) {
 			print STDERR "WebFetch: run eval error: $@\n";
@@ -661,7 +661,7 @@
 	my $options_ref = shift;
 	my $obj;
 
-	$debug and print STDERR "debug: entered run for $run_pkg\n";
+	debug "entered run for $run_pkg";
 
 	# make sure we have the run package loaded
 	mod_load $run_pkg;
@@ -677,7 +677,7 @@
 	# create the new object
 	# this also calls the $obj->fetch() routine for the module which
 	# has inherited from WebFetch to do this
-	$debug and print STDERR "debug: run before new\n";
+	debug "run before new";
 	$obj = eval $run_pkg."->new( \%\$options_ref )";
 	if ( $@ ) {
 		throw_mod_run_failure( "module run failure: ".$@ );
@@ -686,7 +686,7 @@
 	# if the object had data for the WebFetch-embedding API,
 	# then data processing is external to the fetch routine
 	# (This externalizes the data for other software to capture it.)
-	$debug and print STDERR "run before output\n";
+	debug "run before output";
 	my $dest_format = $obj->{dest_format};
 	if ( !exists $obj->{actions}) {
 		$obj->{actions} = {};
@@ -705,22 +705,26 @@
 		throw_no_save( "save failed: no data or nowhere to save it" );
 	}
 
-	$debug and print STDERR "run before save\n";
+	debug "run before save";
 	my $result = $obj->save();
 
-	# Old WebFetch pre-0.9 API code, should not be needed any more
-	#if ( ! $result ) {
-	#	my $savable;
-	#	foreach $savable ( @{$obj->{savable}}) {
-	#		(ref $savable eq "HASH") or next;
-	#		if ( exists $savable->{error}) {
-	#			throw_save_error( "error saving in "
-	#				.$obj->{dir}
-	#				."file: ".$savable->{file}
-	#				."error: " .$savable->{error} );
-	#		}
-	#	}
-	#}
+	# check for errors, throw exception to report errors per savable item
+	if ( ! $result ) {
+		my $savable;
+		my @errors;
+		foreach $savable ( @{$obj->{savable}}) {
+			(ref $savable eq "HASH") or next;
+			if ( exists $savable->{error}) {
+				push @errors, "file: ".$savable->{file}
+					."error: " .$savable->{error};
+			}
+		}
+		if ( @errors ) {
+			throw_save_error( "error saving results in "
+				.$obj->{dir}
+				."\n".join( "\n", @errors )."\n" );
+		}
+	}
 
 	return $result ? 0 : 1;
 }
@@ -765,6 +769,10 @@
 
 URL or file path (as appropriate) to the news source
 
+=item id
+
+unique identifier string for the entry
+
 =item date
 
 a date stamp,
@@ -942,6 +950,7 @@
 sub do_actions
 {
 	my ( $self ) = @_;
+	debug "in WebFetch::do_actions";
 
 	# we *really* need the data and actions to be set!
 	# otherwise assume we're in WebFetch 0.09 compatibility mode and
@@ -962,7 +971,6 @@
 		if ( exists $modules{output}{$action_spec}) {
 			my $class;
 			foreach $class ( @{$modules{output}{$action_spec}}) {
-				print STDERR "can test on $class\n";
 				if ( $class->can( $action_handler )) {
 					$handler_ref = \&{$class."::".$action_handler};
 					last;
@@ -1258,6 +1266,24 @@
 		});
 }
 
+=item $obj->no_savables_ok
+
+This can be used by an output function which handles its own intricate output
+operation (such as WebFetch::Output::TWiki).  If the savables array is empty,
+it would cause an error.  Using this function drops a note in it which
+basically says that's OK.
+
+=cut
+
+sub no_savables_ok
+{
+	my $self = shift;
+
+	push ( @{$self->{savable}}, {
+		'ok_empty' => 1,
+		});
+}
+
 =item $obj->save
 
 This WebFetch utility function goes through all the entries in the
@@ -1330,6 +1356,11 @@
 			print STDERR "saving ".$savable->{file}."\n";
 		}
 
+		# an output module may have handled a more intricate operation
+		if ( exists $savable->{ok_empty}) {
+			last;
+		}
+
 		# verify contents of savable record
 		if ( !exists $savable->{file}) {
 			$savable->{error} = "missing file name - skipped";
@@ -1571,6 +1602,10 @@
 	return 1;
 }
 
+=item $obj->wk2fname( $wk )
+
+=cut
+
 # convert well-known name to field name
 sub wk2fname
 {
@@ -1604,6 +1639,10 @@
 	return undef;
 }
 
+=item $obj->fname2fnum( $fname )
+
+=cut
+
 # convert a field name to a field number
 sub fname2fnum
 {
@@ -1614,6 +1653,10 @@
 		? $self->{fname2fnum}{$fname} : undef;
 }
 
+=item $obj->wk2fnum( $wk )
+
+=cut
+
 # convert well-known name to field number
 sub wk2fnum
 {
@@ -1624,6 +1667,54 @@
 		? $self->{wk2fnum}{$wk} : undef;
 }
 
+=item AUTOLOAD
+
+=cut
+
+# autoloader catches calls to unknown functions
+# first try: redirect to the class which made the call, if the function exists
+# second try: act as a read-only accessor for object data
+# (want a read/write accessor? define the function explicitly)
+sub AUTOLOAD
+{
+	my $self = shift;
+	my $type = ref($self) or throw_autoload_fail "self is not an object";
+
+	my $name = $AUTOLOAD;
+	$name =~ s/.*://;   # strip fully-qualified portion
+
+	# skip all-caps special Perl functions
+	if ( $name =~ /^[A-Z]+$/ ) {
+		return;
+	}
+
+	# check for function in caller package
+	# (WebFetch may hand an input module's object to an output module)
+	my ( $package, $filename, $line ) = caller;
+	if ( $package->can( $name )) {
+		my $retval = eval $package."::".$name."( \$self, \@_ )";
+		if ( $@ ) {
+			my $e = Exception::Class->caught();
+			ref $e ? $e->rethrow
+				: throw_autoload_fail "failure in "
+					."autoloaded function: ".$e;
+		}
+		return $retval;
+	}
+
+	# act as a read-only accessor
+	# add write accessors when API can specify what's OK to write
+	if ( exists $self->{$name}) {
+		# define the sub for better efficiency next time
+		eval "sub WebFetch::$name { return \$_[0]->{$name}; }";
+		return $self->{$name};
+	}
+
+	# if we got here, we failed
+	throw_autoload_fail "function $name not found - "
+		."called by $package ($filename line $line)";
+}
+
 1;
 __END__
 # remainder of POD docs follow


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.




2001	Jan	Feb	Mar	Apr	May	Jun	Jul	Aug (1)	Sep	Oct	Nov	Dec
2003	Jan	Feb (1)	Mar	Apr	May	Jun	Jul	Aug	Sep	Oct	Nov	Dec
2009	Jan	Feb	Mar	Apr	May	Jun	Jul	Aug (12)	Sep (25)	Oct (1)	Nov	Dec

webfetch-commits Mailing List for WebFetch (Page 2)

webfetch-commits — WebFetch SVN commits