[sprog-users] Re: [ sprog-Feature Requests-1228729 ] Merge file readers/writers

[Grant M. <gr...@mc...>]

I've posted my reply to this feature request to the list because I don't
think the Sourceforge tracker system is a useful mechanism for debating
interface changes.

On Mon, 2005-06-27 at 19:25 -0700, SourceForge.net wrote:
> Feature Requests item #1228729, was opened at 2005-06-28 12:25
> Message generated for change (Tracker Item Submitted) made by Item Submitter
> Submitted By: Tim Nelson (wayland)
> Summary: Merge file readers/writers
> 
> Initial Comment:
> The number of gears is going to become unmanageable as
> soon as people really start using this.  

I'm not sure I agree with that as a starting proposition.  The palette
already supports filtering so you can limit the view by connector type
or keyword matching or both.

It's true that having one long list of all the gears wouldn't be very
useful.  That's just the default setting at the moment while the list of
core gears is small.

I am definitely open to ideas on how to make the palette more usable.  I
built it as a floating palette window and may make that an option in the
future if there's any demand.  I happen to like the way the Gimp uses
floating windows, but I know a lot of people hate it and find it
confusing.  I'm aiming for simple.

> I'd suggest having slightly more generic gears with more properties.  

That idea is definitely moving in the opposite direction from where I'm
heading.  I don't mind if people write gears that are complex and have
lots of settings, but I won't be doing that with the core gears.  The
successful Unix philosphy of using small focussed utilities and
connecting them together underlies my thinking.

> In this particular instance, I'm suggesting that the
> file reading/writing gears have a "file format" option,
> so for example, you could choose the format "Apache
> Log", and then you would no longer need the gear for
> "Parse Apache Log".  

Except then your Apache logs parsing is tied to reading local files.  
I want people to be able to source their data independently of how
they're going to process it.  We already have support for HTTP, HTTPS
and FTP via the Retrieve URL gear.  Although it would make sense to use
these protocols to retrieve Apache log files, it doesn't make sense to
build log parsing into a file transfer gear.

> You could also set an output format for the files.  eg.
> you could set the  output format to "CSV" (as a
> property), and then you could eliminate the "list to
> CSV" transform.

But what if someone needs to select an output format that we haven't
thought of (MIME encoded GZIPed XML for example)?  By building
components to do the individual steps, we can allow people to assemble
them in ways we never envisaged.

> Unfortunately that would mean that the above would need
> to be polymorphic as to the type of connector allowed,

Yes, I think that's a fatal flaw right there.

> but if you were using the "pipes" idea, you could
> colour-code the types of connector allowed, and allow
> multiple types to connect to the one thing (and use
> perl's "ref" to figure out which was which).

Colour-coding is an interesting idea.  I've given a little thought to
questions like "should somebody be able to connect an HTML data source
to a CSV parser?".  Ultimately, I think the answer to that one is yes.
Even though we're pretty sure in that example it won't be useful.

The different connector shapes that I've built so far indicate a
different API rather than a different file format.  It's kind of
analagous to you can't plug your phone into the hot water tap - it just
wouldn't make sense.  I'm not planning on having lots more connector
types but it'll be interesting to see what needs develop.

Regards
Grant