[luciddb-users] Further question on bulk loading

[David W <dav...@ya...>]

Hello, 
I have researched bulk loading on 
http://pub.eigenbase.org/wiki/LucidDbFlatFileWrapper and

http://pub.eigenbase.org/wiki/LucidDbEtlTutorial#Define_Sources. As far as I see, they all need to run in lucidClient. Now I have the following files
subreport1-2008-06-05_09-10 
subreport1-2008-06-05_09-20 
subreport1-2008-06-05_09-30 
subreport1-2008-06-05_09-40 
subreport1-2008-06-05_09-50 
subreport1-2008-06-05_10-00 
subreport1-temp 
 
subreport2-2008-06-05_09-00 
subreport2-2008-06-05_09-10 
subreport2-2008-06-05_09-20 
subreport2-2008-06-05_09-30 
subreport2-2008-06-05_09-40 
subreport2-2008-06-05_09-50 
subreport2-2008-06-05_10-00 
subreport2-temp 
 
subreport3-2008-06-05_09-00 
subreport3-2008-06-05_09-10 
subreport3-2008-06-05_09-20 
subreport3-2008-06-05_09-30 
subreport3-2008-06-05_09-40 
subreport3-2008-06-05_09-50 
subreport3-2008-06-05_10-00 
subreport3-temp 
 
The titles of these data file have this format "subreport#-timestamp". '#' means a number from 1 to 3 that means different types of data. "temp" means that the system is just writing data into it. 

Data is writing into these files on a real time basis. As the coming-in data is really huge, it is cut and saved into seperate files every ten minutes. The "timestamp" represents "hour-minute". Therefore the title of these data files always changes according to the time stamp. For example, during the slot of 11:00am to 12:00pm, subreport1 will be
 
subreport1-2008-06-05_11-10 
subreport1-2008-06-05_11-20 
subreport1-2008-06-05_11-30 
subreport1-2008-06-05_11-40 
subreport1-2008-06-05_11-50 
subreport1-2008-06-05_12-00 


I need to load this data into corresponding tables on a real-time basis continuously at the back end. It means that no one comes to run SQL commands anytime. All this bulk loading should be done automatically at the back end. The challenage here is that the titles of these files change every hour. 

All data in type '1' goes to table_1. All data in type '2' goes to table_2 and data in type '3' goes to table_3. 

Can I get any suggestions? 

Thanks so much. 

David