Thread: [sprog-users] Re: DBI Gear
Status: Alpha
Brought to you by:
grantm
Menu
▾
▴
sprog-users
|
From: Matthew K. <df...@ya...> - 2005-06-30 09:45:54
|
>Obviously Sprog needs the ability to source data from a database via > DBI. I'm trying to write one at the moment. >The tricky part of writing data source gears is the need for > non-blocking IO. If someone submits a query that takes several minutes > to complete, then we obviously want the "Stop" button to be functional > during that time. > Does anyone know if there are any hooks in DBI for using it > asynchronously? I'm not aware of anything built into DBI directly, but if the application needs to remain responsive while the query's executing then I would think that threads would be the best way to go. This is a bit more elegant than using fork(), particularly on platforms where Perl's support for fork is a bit of a kludge (I know it doesn't currently run on Windows, but I guess there shouldn't be anything that precludes it). The only drawback is that it makes this gear dependent on having Perl 5.8 installed. Anyway, I've got it about 50% written, when I'm close to finished I wouldn't mind getting some feedback or suggestions about it. One unrelated question: I haven't yet figured out how you're meant to modify a machine once you've constructed it (ie add or remove gears). The gears seem to connect OK when you drag them from the list on the left, but I haven't yet found a way of reconnecting gears which become disconnected in the machine (due to deleting gears), or to connect new gears in the middle of an existing machine. Is there something I'm missing ? ____________________________________________________ Do you Yahoo!? Try Yahoo! Photomail Beta: Send up to 300 photos in one email! http://au.photomail.mail.yahoo.com |
|
From: Matthew K. <df...@ya...> - 2005-06-30 10:02:08
|
One other question - I had envisaged that the gear would have null input (ie be a starting gear) and hash output (so that the values returned by the query would be passed in a hash keyed by the appropriate column names). When I set the type_in to _ and the type_out to H, sprog doesn't seem to recognise it (it doesn't show the appropriate icon on the gear). It does the same thing for type_in _ and type_out A. Is this supported at the moment, or is it just something that the GUI doesn't recognise ? Send instant messages to your online friends http://au.messenger.yahoo.com |
|
From: Grant M. <gr...@mc...> - 2005-06-30 10:19:26
|
On Thu, 2005-06-30 at 20:01 +1000, Matthew Keene wrote: > One other question - I had envisaged that the gear > would have null input (ie be a starting gear) and hash > output (so that the values returned by the query would > be passed in a hash keyed by the appropriate column > names). Sounds reasonable. > When I set the type_in to _ and the type_out > to H, sprog doesn't seem to recognise it (it doesn't > show the appropriate icon on the gear). Yes that's just a problem with the palette - I've only added mini-icons for the gear shapes I've need so far. I've just checked a new version of lib/Sprog/GtkView/Chrome.pm into CVS to fix those two. Cheers Grant |
|
From: Matthew K. <df...@ya...> - 2005-06-30 11:31:14
|
> Yes that's just a problem with the palette - I've > only added > mini-icons for the gear shapes I've need so far. > I've just > checked a new version of lib/Sprog/GtkView/Chrome.pm > into CVS > to fix those two. Thanks for that, I'll get it from CVS now and let you know how I go. ____________________________________________________ Do you Yahoo!? Try Yahoo! Photomail Beta: Send up to 300 photos in one email! http://au.photomail.mail.yahoo.com |
|
From: Matthew K. <df...@ya...> - 2005-06-30 13:07:59
|
--- Matthew Keene <df...@ya...> wrote: > > > Yes that's just a problem with the palette - I've > > only added > > mini-icons for the gear shapes I've need so far. > > I've just > > checked a new version of > lib/Sprog/GtkView/Chrome.pm > > into CVS > > to fix those two. > > Thanks for that, I'll get it from CVS now and let > you > know how I go. > > Cool, new icons work fine. Send instant messages to your online friends http://au.messenger.yahoo.com |
|
From: Grant M. <gr...@mc...> - 2005-06-30 10:11:30
|
On Thu, 2005-06-30 at 19:45 +1000, Matthew Keene wrote: > >Obviously Sprog needs the ability to source data from > >a database via DBI. > > I'm trying to write one at the moment. Great! (See I said it was an obvious need). > >The tricky part of writing data source gears is the > >need for non-blocking IO. If someone submits a query that > >takes several minutes to complete, then we obviously want > >the "Stop" button to be functional during that time. > > > Does anyone know if there are any hooks in DBI for > > using it asynchronously? > > I'm not aware of anything built into DBI directly, but > if the application needs to remain responsive while > the query's executing then I would think that threads > would be the best way to go. This is a bit more > elegant than using fork(), Except that no one would describe Perl threads as elegant. The Gtk2 libraries are also reputed to have 'issues' with threads. I haven't tried using them together but I was advised not to by someone who should know :-) > particularly on platforms > where Perl's support for fork is a bit of a kludge I intended to use IPC::Open2/3. Obviously that would use fork on *nix, but on Windows it seems to use the native spawn functions. > One unrelated question: I haven't yet figured out how > you're meant to modify a machine once you've > constructed it (ie add or remove gears). The gears > seem to connect OK when you drag them from the list on > the left, but I haven't yet found a way of > reconnecting gears which become disconnected in the > machine (due to deleting gears), or to connect new > gears in the middle of an existing machine. Is there > something I'm missing ? Yes :-) But you're not the only one. If you want to connect gear B to the output of gear A, then drag B and drop it on A. The secret is that when you drop it, the top left corner of B must be within the outline of A. I plan to redo that area and add support snapping to the closest connector as well as for moving gears with the keyboard. Cheers Grant |
|
From: Matthew K. <df...@ya...> - 2005-06-30 11:28:47
|
--- Grant McLean <gr...@mc...> wrote: > > Except that no one would describe Perl threads as > elegant. > The Gtk2 libraries are also reputed to have 'issues' > with > threads. I haven't tried using them together but I > was > advised not to by someone who should know :-) > Yes ! I just tried it and found those very 'issues'. There's also the even bigger problem of the fact that there's no 'kill' or 'destroy' method available for threads, which kind of defeats the purpose. > > > I intended to use IPC::Open2/3. Obviously that > would use > fork on *nix, but on Windows it seems to use the > native > spawn functions. > I'll try out IPC::Open* and see how I go. I'll also have a look at fork() (which I've found out does have an 'emulation' on Windows) and see if that also might help, although I don't think we want to fork the entire sprog process. I'll let you know how I go. > > Yes :-) But you're not the only one. > > If you want to connect gear B to the output of gear > A, then > drag B and drop it on A. The secret is that when > you drop it, > the top left corner of B must be within the outline > of A. I thought I'd tried all combinations of that, but I'll give it another go. > > I plan to redo that area and add support snapping to > the closest > connector as well as for moving gears with the > keyboard. > Yep, I'd have to say it's not particularly intuitive right now. ____________________________________________________ Do you Yahoo!? Try Yahoo! Photomail Beta: Send up to 300 photos in one email! http://au.photomail.mail.yahoo.com |
|
From: Matthew K. <df...@ya...> - 2005-07-01 11:42:44
Attachments:
DBIGear.pm
dbi.sprog
|
--- Grant McLean <gr...@mc...> wrote: > On Thu, 2005-06-30 at 19:45 +1000, Matthew Keene > wrote: > > >Obviously Sprog needs the ability to source data > from > > >a database via DBI. > > > > I'm trying to write one at the moment. > > Great! (See I said it was an obvious need). > I've got a synchronous DBIGear working, I'd appreciate any comments (I'll attach it here). I'm working on a design which will allow the query to run asynchronously, I'm going to start a worker process which will use sockets to communicate back to the caller. This is apparently what POE does to simulate asynchronous database calls, and I figure that it might be a bit easier to have structured (and potentially binary) communication than IPC::Open*. I'm a bit concerned that starting a worker process and using interprocess communication (of some form) is quite cumbersome and error-prone (not to mention slower), and I'll probably end up making it a user-selectable option to run it asynchronously or not. This gives the user the ability to run it in process if they're confident that the select statement will return in a reasonable amount of time or are willing to wait for it. It seems a bit like the tail wagging the dog if the whole thing has to run out of process just so someone can click the stop button, when this is probably the exceptional case. Your thoughts ? ____________________________________________________ Do you Yahoo!? Try Yahoo! Photomail Beta: Send up to 300 photos in one email! http://au.photomail.mail.yahoo.com |
|
From: Grant M. <gr...@mc...> - 2005-07-01 20:42:23
|
On Fri, 2005-07-01 at 21:42 +1000, Matthew Keene wrote:
> --- Grant McLean <gr...@mc...> wrote:
>
> > On Thu, 2005-06-30 at 19:45 +1000, Matthew Keene
> > wrote:
> > > >Obviously Sprog needs the ability to source data
> > from
> > > >a database via DBI.
> > >
> > > I'm trying to write one at the moment.
> >
> > Great! (See I said it was an obvious need).
> >
> I've got a synchronous DBIGear working,
Excellent. I just tried it and had no problems.
> I'd appreciate any comments (I'll attach it here).
And likewise, I'd appreciate comments from people who are
building gears and have questions about how things are meant
to work (or why) or comments about how things could be better.
Here are some thoughts from browsing through your code ...
Package Namespace:
You might not have got the memo, but all 'third-party' extensions
to Sprog should come under the SprogEx namespace. Also, in your
case the word 'Gear' appears redundantly. I'd suggest changing
from:
package Sprog::Gear::DBIGear;
to something like:
package SprogEx::Gear::DBIQuery;
The 'engage' Method:
Your gear does all its work in the 'engage' method. The result
is that a query which returns a million row result set will dump
a million messages in the input queue of the next gear - which
isn't how I intended it too be used.
The framework relies on cooperative multitasking, so if any one
method is working for a long time and not returning, no other
gear is getting a chance to process its incoming messages. By
sending messages in chunks, machines can work with arbitrarily
large data sets.
It's not usually a consideration for filter gears since they
just process each message as it is received. Data source gears
which generate messages are expected to be a bit 'nicer'.
The Sprog scheduler will call the 'send_data' method of a data
source gear whenever it is ready for the next chunk of data. The
gear may respond by sending more than one message, but shouldn't
send an arbitrarily large number. I'd suggest restructuring your
code so that the connect/prepare/execute happen in the engage
method, but the fetch happens in the send_data method. Something
like this:
sub send_data {
my $self = shift;
if(my $hash_ref = $self->{_sth}->fetchrow_hashref) {
$self->msg_out(record => $hash_ref) ;
return;
}
$self->{_dbh}->disconnect ;
$self->disengage ;
}
Password Input:
You can set the password text to be obscured (eg: ******), if
you want to. In Glade, select the password Entry widget and
toggle the 'Text Visible' button from 'Yes' to 'No'.
POD:
You obviously haven't gotten around to doing the POD yet, but you really
out to put your own name in the copyright section at the very least.
> I'm working on a design which will allow the query to
> run asynchronously, I'm going to start a worker
> process which will use sockets to communicate back to
> the caller. This is apparently what POE does to
> simulate asynchronous database calls, and I figure
> that it might be a bit easier to have structured (and
> potentially binary) communication than IPC::Open*.
The only reason I was thinking of IPC::Open3 was because it is
meant to handle spawning another process properly on Win32 (without
using fork emulation).
> I'm a bit concerned that starting a worker process and
> using interprocess communication (of some form) is
> quite cumbersome and error-prone (not to mention
> slower)
Ultimately, I intend to have more support in the Sprog framework
for doing this type of thing. It will be needed for other data
sources (and sinks) as well.
> and I'll probably end up making it a
> user-selectable option to run it asynchronously or
> not. This gives the user the ability to run it in
> process if they're confident that the select statement
> will return in a reasonable amount of time or are
> willing to wait for it.
Personally, I don't see the need for two different approaches,
but if you want to then the simplest approach is to just make
two gears. But whether the user chooses a different gear or a
checkbox in a dialog, it's not clear to me what they'd base
their choice on.
> It seems a bit like the tail
> wagging the dog if the whole thing has to run out of
> process just so someone can click the stop button,
> when this is probably the exceptional case. Your
> thoughts ?
It is more complex than the equivalent Perl script you'd run
from the command line. But that's the nature of working in an
event driven environment.
Cheers
Grant
|
|
From: Matthew K. <df...@ya...> - 2005-07-02 03:23:44
|
--- Grant McLean <gr...@mc...> wrote: > On Fri, 2005-07-01 at 21:42 +1000, Matthew Keene > wrote: > > --- Grant McLean <gr...@mc...> wrote: > > And likewise, I'd appreciate comments from people > who are > building gears and have questions about how things > are meant > to work (or why) or comments about how things could > be better. The obvious thing is more complete documentation. A 'How To' guide for gear developers would be excellent - the gear internals guide that you've got is a good start, but I found it a little lacking in detail (and I've just read the scheduler internals document and found some more stuff in there which now starts to make a bit more sense). I was planning on making a start on a step by step guide when I finished writing the DBI gear, sort of like a 'Gear Development for Dummies'. (We're not all rocket scientists, you know ;-)). Obviously I could only put in the little I know so far and you have to fill in the rest.... My other main complaint is not related to developing gears, but a tool usability issue. When you right click on a gear on the palette the current behaviour is that the selected item is executed when you release the right mouse button. This isn't my expectation of how context menus normally work, which is that a right mouse click simply displays the context menu, and selecting an item from the menu is a separate mouse operation (ie when the right button is released the menu stays displayed until a selection is made or the menu is dismissed). This behaviour is particularly annoying because the default selected menu item is Delete. I spent my first five or ten minutes of playing with the tool cursing at it because the gears kept disappearing every time I tried to modify their properties. Even after I figured it out it still catches me sometimes. Can you either change the context menu behaviour to be more consistent with other applications, or at the very least can you make the default selected item something other than Delete ? > > Here are some thoughts from browsing through your > code ... > > Package Namespace: > > You might not have got the memo, but all > 'third-party' extensions > to Sprog should come under the SprogEx namespace. You're right, I must have missed that memo. I must have stern words with the mail room. > > The 'engage' Method: > > Your gear does all its work in the 'engage' method. > The result > is that a query which returns a million row result > set will dump > a million messages in the input queue of the next > gear - which > isn't how I intended it too be used. > No problems - I've made the change as you suggested. As I suggested, better documentation would have made this clearer. I found that it was a little difficult to work out which methods in the various gears were related to the framework (ie overriding behaviours frm the base class) and which were for that gear (or it's parent classes) only. > > Password Input: > > You can set the password text to be obscured (eg: > ******), if > you want to. In Glade, select the password Entry > widget and > toggle the 'Text Visible' button from 'Yes' to 'No'. > Yep, although it's still in clear text in the sprog file. > > POD: > > You obviously haven't gotten around to doing the POD > yet, but you really > out to put your own name in the copyright section at > the very least. > Doesn't everybody do the documentation last, after the fun part's finished ? > One thing that I'm still a little confused by when writing the non-blocking version of this gear is how to give control back to the framework so that other events can be processed. The difficulty here is that I'm going to basically have a single event (executing the query) which will potentially consume a large amount of time before any data becomes available. I'm planning to call can_read (or some equivalent) on the IPC file handle to determine whether there the worker has produced any data to be processed, should I just return from send_data without sending any messages and let the scheduler handle the looping, or should I loop in my send_data method and make some other call in order to let the scheduler process other events ? How often will the scheduler call the input gear's send_data method if it hasn't yet started producing any data ? I'm a little conscious of consuming CPU needlessly cycling around waiting for data if it's going to take some time to arrive, and I had been planning on putting a second or two's sleep time in the loop just to make sure that it doesn't put a hard head lock on the CPU. ____________________________________________________ Do you Yahoo!? Try Yahoo! Photomail Beta: Send up to 300 photos in one email! http://au.photomail.mail.yahoo.com |
|
From: Grant M. <gr...@mc...> - 2005-07-03 12:40:43
|
On Sat, 2005-07-02 at 13:22 +1000, Matthew Keene wrote: > --- Grant McLean <gr...@mc...> wrote: > > And likewise, I'd appreciate comments from people > > who are building gears and have questions about how > > things are meant to work (or why) or comments about > > how things could be better. > > The obvious thing is more complete documentation. Ahh the perennial bugbear. As it happens though I quite enjoy writing documentation so stay tuned. > A 'How To' guide for gear developers would be excellent Yes, I'll put something up on the web site. A few pictures could make some of the concepts clearer. > - the gear internals guide that you've got is a good > start, but I found it a little lacking in detail (and > I've just read the scheduler internals document and > found some more stuff in there which now starts to > make a bit more sense). I'll probably end up turning those documents into reference pages and have the more descriptive stuff with background details on the web site. > I was planning on making a > start on a step by step guide when I finished writing > the DBI gear, sort of like a 'Gear Development for > Dummies'. (We're not all rocket scientists, you know > ;-)). Obviously I could only put in the little I know > so far and you have to fill in the rest.... I don't mind writing it and if you ask questions about things that aren't clear, then I'll have a better idea what to write about. > My other main complaint is not related to developing > gears, but a tool usability issue. When you right > click on a gear on the palette the current behaviour > is that the selected item is executed when you release > the right mouse button. I'd never noticed that, since I am in the habit of holding down the mouse button and releasing it once I've highlighted the required option. I certainly haven't done anything to make it disappear as soon as you release the button, so it looks like I'll have to do something to allow it to stay. On my systems though the delete option isn't selected unless I move the mouse after the initial click. If I just right click and release, nothing gets selected. Admittedly though the mouse only needs to move one or two pixels to make a selection. > One thing that I'm still a little confused by when > writing the non-blocking version of this gear is how > to give control back to the framework so that other > events can be processed. 'return' :-) Seriously though, it's a cooperative multitasking environment so a long as your method doesn't return, no other method (or GUI update) will get a chance. > The difficulty here is that > I'm going to basically have a single event (executing > the query) which will potentially consume a large > amount of time before any data becomes available. Yes, this is the bit that needs to be in the other process. > I'm > planning to call can_read (or some equivalent) on the > IPC file handle to determine whether there the worker > has produced any data to be processed, Glib/Gtk support a callback interface which allows you to register a handler routine which should be called when a filehandle becomes readable/writable or when a timeout expires. The ReadFile and CommandIn gears both only read from a filehandle from within IO event handlers. They both use the InputFromFH mixin class to manage that. > should I just > return from send_data without sending any messages and > let the scheduler handle the looping, Yes, that's exactly what you should do. > How often will the scheduler call the input gear's > send_data method if it hasn't yet started producing > any data ? The scheduler only calls send_data when it has no messages queued for delivery. Immediately after calling send_data, the scheduler goes to sleep, so it won't call send_data again unless something wakes it up. The most likely thing to wake it up is if your gear sends a message - which it would most likely do from inside an IO event handler. It is possible that some other gear in the machine has an IO or timeout event setup too. If it fired and sent some messages, the scheduler would wake, deliver the messages and then call send_data again on all gears that had a send_data method, before going back to sleep. So it is possible that your gear could be asked to send data more than once. It ought not to be hammered with requests though. > I'm a little conscious of consuming CPU > needlessly cycling around waiting for data if it's > going to take some time to arrive, and I had been > planning on putting a second or two's sleep time in > the loop just to make sure that it doesn't put a hard > head lock on the CPU. It shouldn't be necessary to use sleeps. Indeed if your code is sleeping, no other code is running. If your code runs in response to a filehandle becoming readable then other code can run until that happens. Regards Grant |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X