Re: [Lmappletserver-developers] Media columns in the MetaLogViewer

[Peter S. <co...@pe...>]

Hi Roberto,

I'm not a file system expert - so I cannot tell whats the most optimal
regarding how many files to put in a folder - but your suggestion sounds
interesting. This means that we also can keep actual media file names since
each log entry has its own folder.

I'd also like a comment from Jon on this.

cheers,

Peter

On 11/1/07, Roberto Milev <rob...@gm...> wrote:
>
> Hi Peter,
>
> About the folder structure, an alternative approach would be to organize
> it to reflect the actual MetaDataLogEntry id. for example:
>
> MetaDataLogEntry id = 1001 - would map to /1/0/0/1
> MetaDataLogEntry id = 1002 - would map to /1/0/0/3
> .. etc
> MetaDataLogEntry id = 1011 - would map to /1/0/1/1
>
> in this manner every media file could be easily located within the data
> structure.
>
> I will start setting the infrastructure for the Phase One of the task. I
> will keep you posted of the progress.
>
> Regards,
> Roberto
>
>
> On 10/30/07, Peter Salomonsen <co...@pe...> wrote:
> >
> > Hi Roberto,
> >
> > I think we should split the project in two phases:
> >
> > Phase One:
> >
> > Media upload mechanisms and media file download (with correct mime types
> > etc) - but no media player. So the play button would go directly to the
> > multimediaservlet.
> >
> > Phase Two:
> >
> > Embedded Mediaplayer using e.g. FMJ.
> >
> > About the file sys:
> >
> > Splitting on date folders is good (to speed of access), but in case of
> > massive uploads, it might be better to split on MetaDataLogEntry id ranges.
> > Maybe the folder structure should reflect this - e.g. 1000 files per
> > folder (this could be customizable).
> >
> > I see the challenges regarding separating the project, and the challenge
> > regarding different data types we've also had with other data types. The
> > MetaDataLogEntry table should maybe have one field per datatype such as with
> > the numericvalue field - that we don't use today - but maybe we should.
> >
> > regards,
> >
> > Peter
> >
> > On 10/30/07, Roberto MIlev < rob...@gm...> wrote:
> > >
> > > Hi Peter,
> > >
> > > Please find attached my ideas on on implementing the new feature. My
> > > initial rough estimate for implementation is around 80 man hours.
> > >
> > > Please give me your thoughts.
> > >
> > > Regards,
> > > Roberto
> > >
> > > Peter Salomonsen wrote:
> > > > Hi Roberto -
> > > >
> > > > how is it going with the interceptor stuff - did you get to
> > > understand
> > > > how it works?
> > > >
> > > > I have a new project proposal for you. We'd like to add multimedia
> > > > capabilities to the MetaLogViewer - so that you can attach
> > > multimedia
> > > > clips to a MetaDataLogEntry. The idea will be to create new columns
> > > > that contains multimedia file names, and with a possibility to
> > > launch
> > > > a player to view /play. Embedded players would be very nice -
> > > however
> > > > supporting all the varieties of file formats may be difficult. Have
> > > to
> > > > consider whether to stick to one format ( e.g. ogg/vorbis/theora -
> > > and
> > > > use e.g. the flumotion cortado applet), or support multiple formats,
> > > > and test frameworks such as fmj (which is an open source impl of
> > > java
> > > > media framework).
> > > >
> > > > I don't think the media content itself should be stored in the
> > > > MetaDataLogEntry table, but the file name could be, and there should
> > > > be a file structure for easy retrieval.
> > > >
> > > > Get the idea?
> > > >
> > > > This makes the lmappletserver-addons.jar even more massive,  and the
> > >
> > > > question is if we could create a new Multimedia-extensions module?
> > > > Should this even be a whole new separate project?
> > > >
> > > > Is this a project for you? In that case some sketches and models
> > > would
> > > > be nice, would you have a look at it, and think of a solution
> > > proposal?
> > > >
> > > > regards,
> > > >
> > > > Peter
> > > >
> > > > On 10/23/07, *Roberto Milev* < rob...@gm...
> > > > <mailto:rob...@gm...>> wrote:
> > > >
> > > >     Hi Peter,
> > > >
> > > >     I changed and commiter orm.xml. I will be looking at the
> > > >     interceptor stuff next. I will keep you posted.
> > > >
> > > >     Regards,
> > > >     Roberto
> > > >
> > > >
> > > >     On 10/23/07, * Peter Salomonsen* < co...@pe...
> > > >     <mailto: co...@pe...>> wrote:
> > > >
> > > >         Hi Robert,
> > > >
> > > >         I've attached a metaloginterceptors property file example.
> > > >         This is a feature me and Jon has been working on, and the
> > > >         purpose is to be able to intercept calls within the
> > > >         lmappletserver-addons ejb3 session beans using default EJB3
> > > >         interceptors.
> > > >
> > > >         I wasn't aware that it was a requirement to have this file,
> > > so
> > > >         this is a bug, the intention is that this file is not
> > > >         required, but an option if you need to create an interceptor
> > > >         for some reason. And it really shouldn't be called
> > > >         metaloginterceptor, as it is really intercepting everything
> > > -
> > > >         this naming is related to our specific use case - we should
> > > >         change this however to a more generic name (maybe just
> > > >         interceptor).
> > > >
> > > >         To tell you about our use case - we use interceptors to
> > > >         perform a parallell query on searchLogEntries (historic
> > > >         searches). We have a MetaDataLogEntry table with more than a
> > >
> > > >         billion of records, and the standard query produced by
> > > >         searchLogEntries is not able to handle this. The strategy is
> > > >         then to intercept the call, and duplicate it to run in
> > > >         multiple processes / servers on smaller dataset partitions,
> > > >         and collect the resultsets when sufficient amount of threads
> > > >         has returned. Then we also gain our desired speed (which is
> > > >         instant results).
> > > >
> > > >         About table names - since the case problem is TopLink
> > > >         specific, could we rather solve this using an orm.xml for
> > > the
> > > >         unit tests? Instead of using the Table annotation?
> > > >
> > > >         regards,
> > > >
> > > >         Peter
> > > >
> > > >
> > > >         On 10/21/07, *Roberto MIlev* < rob...@gm...
> > > >         <mailto: rob...@gm...>> wrote:
> > > >
> > > >             Hi Peter,
> > > >
> > > >             You will find the changes you requested committed to
> > > SVN.
> > > >             A few notes:
> > > >
> > > >             - I was only abe to test the shownColumns changes with
> > > >             dummy hardcoded
> > > >             data, beacuse I am getting errors due to missing
> > > resource -
> > > >             lmappletserver_metaloginterceptors. I am not able to
> > > find
> > > >             this resource
> > > >             in the project. Perhaps you can shed some light on this?
> > > >             - In the mean time I also resolved the issue with the
> > > junt
> > > >             tests
> > > >             failing. The thing was that the Toplink JPA always
> > > creates
> > > >             the tables
> > > >             based on the name of the Entity in uppercase. This was
> > > not
> > > >             visible until
> > > >             i install everything on Linux. the solution is to always
> > >
> > > >             specify the
> > > >             name attribute in the @Table annotation.
> > > >
> > > >             Regards,
> > > >             Roberto
> > > >
> > > >             Peter Salomonsen wrote:
> > > >             > Hi,
> > > >             >
> > > >             > I was just going to update my metadata table by adding
> > > a
> > > >             new metadata
> > > >             > (new column). I found that the searchdto has a
> > > >             showcolumns boolean
> > > >             > array, and this causes the requirement that the
> > > metadata
> > > >             cannot change
> > > >             > order or change in size. What happened in my case was
> > > that
> > > >             > listmetadata returned more columns than the
> > > showcolumns
> > > >             array, and
> > > >             > this caused an ArrayIndexOutOfBounds exception in
> > > >             > MetaDataSearchExecutor.java on the showcolumns array.
> > > >             >
> > > >             > I think this has to be solved differently. The
> > > >             showcolumns array
> > > >             > should rather be a list of metadata_ids rather than
> > > >             booleans - so that
> > > >             > it is not dependent on me not changing the metadata
> > > order
> > > >             or number of
> > > >             > metadatas.
> > > >             >
> > > >             > Can you fix this?
> > > >             >
> > > >             > regards,
> > > >             >
> > > >             > Peter
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >     --
> > > >     Regards,
> > > >     Roberto Milev
> > > >
> > > >
> > >
> > >
> >
>
>
> --
> Regards,
> Roberto Milev