Re: [luciddb-users] Further question on bulk loading

[Zelaine F. <zf...@lu...>]

Write a java UDR that takes a parameter that represents the timestamp in 
some way.  Have that UDR dynamically construct the name of the file 
using the parameter, and then execute the resulting dynamic SQL statement.

-- Zelaine

David W wrote:
> Hello,
> 
> I have researched bulk loading on
> 
> http://pub.eigenbase.org/wiki/LucidDbFlatFileWrapper and
> 
> _http://pub.eigenbase.org/wiki/LucidDbEtlTutorial#Define_Sources_ 
> <http://pub.eigenbase.org/wiki/LucidDbEtlTutorial/lDefine_Sources>. As 
> far as I see, they all need to run in lucidClient. Now I have the 
> following files
> 
> subreport1-2008-06-05_09-10
> 
> subreport1-2008-06-05_09-20
> 
> subreport1-2008-06-05_09-30
> 
> subreport1-2008-06-05_09-40
> 
> subreport1-2008-06-05_09-50
> 
> subreport1-2008-06-05_10-00
> 
> subreport1-temp
> 
>  
> 
> subreport2-2008-06-05_09-00
> 
> subreport2-2008-06-05_09-10
> 
> subreport2-2008-06-05_09-20
> 
> subreport2-2008-06-05_09-30
> 
> subreport2-2008-06-05_09-40
> 
> subreport2-2008-06-05_09-50
> 
> subreport2-2008-06-05_10-00
> 
> subreport2-temp
> 
>  
> 
> subreport3-2008-06-05_09-00
> 
> subreport3-2008-06-05_09-10
> 
> subreport3-2008-06-05_09-20
> 
> subreport3-2008-06-05_09-30
> 
> subreport3-2008-06-05_09-40
> 
> subreport3-2008-06-05_09-50
> 
> subreport3-2008-06-05_10-00
> 
> subreport3-temp
> 
>  
> 
> The titles of these data file have this format "subreport#-timestamp". 
> '#' means a number from 1 to 3 that means different types of data. 
> "temp" means that the system is just writing data into it.
> 
> Data is writing into these files on a real time basis. As the coming-in 
> data is really huge, it is cut and saved into seperate files every ten 
> minutes. The "timestamp" represents "hour-minute". Therefore the title 
> of these data files always changes according to the time stamp. For 
> example, during the slot of 11:00am to 12:00pm, subreport1 will be
> 
>  
> 
> subreport1-2008-06-05_11-10
> 
> subreport1-2008-06-05_11-20
> 
> subreport1-2008-06-05_11-30
> 
> subreport1-2008-06-05_11-40
> 
> subreport1-2008-06-05_11-50
> 
> subreport1-2008-06-05_12-00
> 
> 
> 
> I need to load this data into corresponding tables on a real-time basis 
> continuously at the back end. It means that no one comes to run SQL 
> commands anytime. All this bulk loading should be done automatically at 
> the back end. The challenage here is that the titles of these files 
> change every hour.
> 
> All data in type '1' goes to table_1. All data in type '2' goes to 
> table_2 and data in type '3' goes to table_3.
> 
> 
> Can I get any suggestions?
> 
> Thanks so much.
> 
> David
> 
> 
> 
> ------------------------------------------------------------------------
> 
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
> http://sourceforge.net/services/buy/index.php
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> luciddb-users mailing list
> luc...@li...
> https://lists.sourceforge.net/lists/listinfo/luciddb-users