Re: [Squirrel-sql-users] Another question on data bulk loading

[Matthew D. <mda...@ja...>]

David,
 
The functionality that you are describing isn't something that SQuirreL is
designed to do.
 
You'll have more success with an ETL (Extract Transform Load) tool built
for situation that you are describing. I can recommend Talend as an open
source ETL tool that I've used (http://www.talend.com/), but you'll find
many of them with a quick search.
 
Regards,
Matt

  _____  

From: squ...@li...
[mailto:squ...@li...] On Behalf Of
David W
Sent: Thursday, 05 June, 2008 11:02
To: squ...@li...
Subject: [Squirrel-sql-users] Another question on data bulk loading



Hello, 

 

I have researched bulk loading on 

  <http://pub.eigenbase.org/wiki/LucidDbFlatFileWrapper>
http://pub.eigenbase.org/wiki/LucidDbFlatFileWrapper and

 <http://pub.eigenbase.org/wiki/LucidDbEtlTutorial#Define_Sources>
http://pub.eigenbase.org/wiki/LucidDbEtlTutorial#Define_Sources. As far as
I see, they all need to run in lucidClient. Now I have the following files

 

subreport1-2008-06-05_09-10 

subreport1-2008-06-05_09-20 

subreport1-2008-06-05_09-30 

subreport1-2008-06-05_09-40 

subreport1-2008-06-05_09-50 

subreport1-2008-06-05_10-00 

subreport1-temp 

 

subreport2-2008-06-05_09-00 

subreport2-2008-06-05_09-10 

subreport2-2008-06-05_09-20 

subreport2-2008-06-05_09-30 

subreport2-2008-06-05_09-40 

subreport2-2008-06-05_09-50 

subreport2-2008-06-05_10-00 

subreport2-temp 

 

subreport3-2008-06-05_09-00 

subreport3-2008-06-05_09-10 

subreport3-2008-06-05_09-20 

subreport3-2008-06-05_09-30 

subreport3-2008-06-05_09-40 

subreport3-2008-06-05_09-50 

subreport3-2008-06-05_10-00 

subreport3-temp 

 

The titles of these data file have this format "subreport#-timestamp". '#'
means a number from 1 to 3 that means different types of data. Data is
writing into these files on a real time basis. As the coming-in data is
really huge, it is cut and saved into seperate files every ten minutes. 

 

I need to load this data into corresponding tables on a real-time basis
continuously at the back end. It means that no one comes to run those SQL
commands anytime. All this bulk loading should be done automatically. 

 

What are the steps to do this? 

 

Thanks so much. 

 

David